Tools we recommend for AI coding, from rg to htmlmd and callm

Our test page had 180 menu links, an advert and a short API reference buried in the middle. We ran htmlmd on it, kept only the article and went from 10,335 bytes to 292. The JavaScript example made it through, and so did the table explaining expiry and the link to the guide.

To be fair, we built that page for the test, and most of the 97% saving is its bloated menu. Your docs site will land somewhere else. We counted bytes, not tokens, and we didn’t time a real coding task, so treat the number as a demo, not a promise about your token bill.

What’s worth stealing is the habit. Trim a page before you ask a model about it. Pick out the records before you ask for an explanation. Find the callers before you ask for a review. The model gets less to read, and you can see exactly what it got.

We maintain htmlmd and callm, so both get space here. We checked which other tools were installed and exercised the recipes against small synthetic files. The starting recommendation is modest: rg, jq, git, and your project’s existing checks. Add tools as a repeated task gives you a reason.

Pick tools by the files you handle

You can install a long list and still watch your assistant write a new parser for each task. Give it the command name, the job it handles and one working example in your repository’s agent instructions. Include the version or implementation where syntax differs.

On our machine, yq is the Python wrapper around jq. A developer using the Go implementation needs different examples. We also have both sg and ast-grep; sg resolves to ast-grep here, but the longer command avoids a name collision on another machine.

Task	Tool to reach for	Useful output
Find candidate files	fd or rg --files	A list of paths within the intended scope
Find a string or regex	rg	Matches with file names and line numbers
Match a code construct	ast-grep	Syntax matches that exclude unrelated text
Select or transform JSON	jq	A checked projection, count or encoded record
Read YAML with jq filters	Python yq	Values from configuration without a custom parser
Select columns or join CSV	qsv	CSV that preserves quoting and field boundaries
Convert captured HTML	htmlmd	Markdown or text with explicit cleanup options
Select an HTML fragment	htmlq	Elements or attributes matched by CSS selectors
Request a model review	callm	A response you can save and validate
Inspect changes	git, with gh for GitHub tasks	The intended diff or structured repository data
Measure a command	hyperfine	Repeated timings and an export you can inspect

Versions matter for the flags below. We used htmlmd 0.1.2 and callm 0.10.0. Run --help before you copy anything. Don’t trust yq --version either: ours printed 0.0.0, and only its help showed which yq it was.

Put this in the repo, not in your head. A few lines in AGENTS.md are enough: this machine has Python yq, this is the command that checks the app, recipes live in that folder. Add one example and say which directory it runs from. A command copied from another project can run without errors and still read the wrong files.

Something like this:

## Tools on this machine
- Search text with `rg`; match code shapes with `ast-grep` (not `sg`).
- `yq` is the Python wrapper, so it takes jq filters.
- Convert saved HTML with `htmlmd --profile gfm --extract-selector article`.
  A missing selector still exits 0: check you didn't get the whole page.
- Before finishing, run `npm run check` from the repo root.
- Saved recipes live in `docs/recipes/`, each with a small sample input.

Ask the assistant to save a recipe after it proves useful. Save what it expects as input, the expected output and a small fixture with it. The next task can start from that command instead of another parser. Update the entry if you replace a binary or change the data format; an old command in agent instructions can be harder to notice than a missing executable.

Everything below runs against a small download. Unpack it somewhere you don’t mind making a mess and cd netcraft-ai-tools. You get both HTML pages, the data files and a mock server. The review script wants your own model key. The mock doesn’t: it hands a fake key to a server on localhost.

Find the code before preparing a review

The bundle’s repo/src/cache.js has a comment, a string and a function call:

// console.log("comment");
const note = 'console.log("string")';
console.log("actual call");

There is one executable logging call in that file. A literal search has three reasons to match:

fd -e js . repo
rg -n -F 'console.log(' repo

rg finds all three lines. The comment and string matter if you’re changing documentation or looking for embedded examples. For a task about executable calls, use a syntax pattern:

ast-grep run --lang js \
  --pattern 'console.log($$$ARGS)' \
  --json repo/src/cache.js

The AST result points to line 3 and includes "actual call" as its argument. It excludes the other two appearances. Keep single quotes around console.log($$$ARGS) in the shell; the dollar signs belong to the pattern. Read the surrounding function before editing a match in a real repository.

For a logging cleanup, we’d use those syntax matches to collect the calls for review. For an outdated example in a README, we’d use the text matches. The search choice follows the edit you intend to make. A parser for JavaScript won’t tell you where a configuration file or an HTML template mentions the same API.

Use the same care with the diff. git diff --cached covers staged changes, while git diff covers unstaged changes to tracked files. Neither includes the contents of a new untracked file. For a branch review, choose the intended base and include the related files the reviewer needs. Note the commit with the result, so after the next edit you still know which version a finding is about.

Both fd and ripgrep apply file filters. A search that misses a hidden configuration file can lead you to the wrong conclusion. Choose the directory and hidden-file options for the task, then confirm which paths you searched. The ripgrep guide explains those filters; the ast-grep introduction covers syntax patterns.

Use htmlmd to prepare documentation for an assistant

Suppose you need to change a cache adapter and check what ttl means in the API documentation. The captured page contains the answer, but also a navigation menu, styling and unrelated content. Keep the capture so you can compare the conversion with its source.

For the bundle’s reference page, this is the command that produced the 292-byte result:

htmlmd --profile gfm --extract-selector article \
  --remove-selectors .ad \
  --base-url https://docs.example.test/reference/cache \
  --remove-tracking-params true \
  captures/reference/cache.html > context.md

The .test address names our synthetic source; you don’t need to fetch it. For a captured real page, pass the URL you fetched as the base. The converter can then resolve relative links in the same context as the browser.

The result includes:

# Cache API

Read one cache record and keep its expiry.

| Field | Meaning              |
| ----- | -------------------- |
| ttl   | Seconds until expiry |

[Cache guide](https://docs.example.test/guide?version=2)

We omitted the JavaScript fence from this excerpt to keep it short. The generated file contains the original const cache = { ttl: 60 }; line and the call that follows it. Our assertions checked both, the table row and the guide URL. The converter removed utm_source and kept version=2, since that one can change which page you land on.

You can now attach context.md to a coding task with the adapter’s relevant function. The model receives the expiry definition and the example. You also have a file you can open before sending anything, and the original capture to check it against.

Check that you selected the intended content

Our missing-selector test exposed a trap. With --extract-selector '#missing', installed htmlmd 0.1.2 returned the full document and exit code 0. The resulting GFM occupied 7,135 bytes. Exit code 0 didn’t mean we’d got the article.

For a pipeline that requires a match, select a fragment first and reject an empty result:

htmlq --filename captures/reference/cache.html article > article.html
test -s article.html || exit 1
htmlmd --profile gfm --remove-selectors .ad \
  --base-url https://docs.example.test/reference/cache \
  --remove-tracking-params true article.html > context.md

With #missing, htmlq produced an empty file in our run, so test -s stopped the pipeline. With article, conversion of the fragment matched our 292-byte result. An existing element can still contain the wrong document. For this reference we checked Cache API, the expiry row and the guide link before accepting the context file.

htmlmd converts the HTML you supply. Use an HTTP client to capture pages whose response already contains the relevant content. For a page that renders it through JavaScript, capture the rendered DOM with a browser. A browser capture still needs inspection: a login screen or loading placeholder can produce valid HTML.

Choose the output your next step can read

We used GFM because the API table and code blocks mattered. For a search index, readable text may suffice. For a note archive, you may want source metadata alongside the body.

Next step	Profile to consider	Check before using it
Documentation review	gfm	Code, table rows and links survive
Text search	plain-text	The searchable terms remain; keep the source URL elsewhere
Notes with frontmatter	obsidian	Request the metadata fields you need
MDX content import	mdx-safe	Compile the target MDX and compare code examples

To extract the title, description and canonical URL into an Obsidian document:

htmlmd --profile obsidian --extract-selector article \
  --metadata-title --metadata-description --metadata-canonical-url \
  captures/reference/cache.html > cache-note.md

We found another limitation worth checking before an import. In our installed version, mdx-safe escaped braces inside the JavaScript fence as \{ and \}. That changes code a reader might copy. GFM preserved those braces in the same fixture. Choose a profile for the destination, then inspect both its rendered page and the copyable examples.

The profile documentation describes output choices. Treat a profile as a starting point and spell out the options you rely on.

Convert a whole archive without losing track of sources

The second capture in the bundle is a guide. Both files use the name cache.html, under different directories. Flattening them into one output directory would lose that distinction, so htmlmd rejects the collision.

Use recursive discovery and mirrored output:

htmlmd captures -r -m --profile gfm --extract-selector article \
  --atomic --output-dir converted --manifest manifest.json
jq '.files[] | {input, output, input_hash, output_hash}' manifest.json

-r selects descendant HTML files; -m preserves their relative paths. This run produces converted/reference/cache.md and converted/guides/cache.md. With --atomic, the converter writes each file through a temporary file and rename. That helps avoid partial individual files if a write fails; you still need to check the batch’s status.

The manifest records input and output hashes. Keep it with the captures to compare runs and see which generated documents changed. The option reference documents the batch flags and manifest.

For an existing conversion, use the same profile and cleanup settings with --check --diff. Our reproducer checked a matching output, added a line to it and confirmed that the command then failed. This gives you a check for generated-content drift without asking a model to compare documents.

The manifest identifies captured files, not their original web addresses. Keep those addresses beside it. A link such as ../guide has a different destination under /reference/cache than it has under a deeper source path. For an import into this blog, we would also need author and date fields, asset paths and a target MDX build. A code fence with altered braces would be reason to stop that import even if the file compiled.

Feeding a RAG index? Keep the source URL and the headings with every chunk. Then ask it a few questions you know the answers to and see if the right passage comes back. Clean Markdown is only the start.

Ask another model through callm

You have a selected diff, nearby definitions and a short documentation file. You can give that context to another model for a review through callm. Your coding assistant can invoke the same command, provided its environment has the credentials and permission to send the material.

callm makes the API request and handles the response. It doesn’t give the requested model access to your checkout or let it run your tests. You decide which context to attach and what to do with a finding. Ask for a file, a line and the evidence for each finding, and say plainly that no findings is an acceptable answer.

Save the prompt in review.txt and describe the output structure in review.schema.json. The bundle supplies both. Its schema requires a nonempty summary and a findings array; each finding needs a file, positive line number, evidence and suggestion.

Use an explicit provider and model. The following is the API step inside the bundle’s safe-review.sh:

callm --or -m "$NETCRAFT_REVIEW_MODEL" \
  --api-key-env NETCRAFT_REVIEW_KEY --no-stdin \
  --prompt-file review.txt -f context.md \
  --validate-schema review.schema.json --result-json \
  --timeout 60s --max-tokens 2048 > review.result.tmp.json

Configure those environment variables outside the script. Choose a model your provider offers. --or selects OpenRouter; other providers have their own presets. Model IDs, access and prices can change, so use the current catalog for that choice.

--no-stdin tells callm to use the files without waiting for an inherited pipe. The timeout bounds the request wait, and the output cap limits generation. callm also supports an input-byte cap, but that measures the serialized request; you need the model’s tokenizer or reported usage to measure tokens.

Preserve the last valid result

The script stages the response in the destination directory, checks callm’s exit code, then checks the result envelope:

jq -e '.status == "ok" and .schema_valid == true' "$staged"

Only after both checks pass does it rename that file to your destination. Run the supplied wrapper after configuring the model and key:

bash safe-review.sh context.md review.result.json
jq '.answer | fromjson' review.result.json

If the request fails, the previous destination survives. We tested that behavior with a localhost mock that returned malformed JSON, a wrong field type, an incomplete answer and a refusal. None of those responses replaced the earlier valid result. That tests the plumbing, not how good a real model’s review is.

We inspected the mock’s request bodies for the two schema flags. With --validate-schema, callm omitted a provider response-format constraint and checked the answer after receiving it. With --schema, the request included response_format.type=json_schema, and callm checked the returned answer too. Confirm that your chosen model supports those constraints before using the latter option.

The truncation case is the easiest one to miss. The mock returned answer text with finish_reason: length to simulate a token-limit failure, and callm wrapped it like this:

{
  "status": "error",
  "finish_reason": "length",
  "schema_valid": null,
  "error": {
    "code": "invalid_completion",
    "message": "incomplete response: token limit reached (length)"
  }
}

If you only grab answer, you can miss the failure. finish_reason: length means the provider stopped generation at a token limit; the text needn’t end mid-sentence. Our mock kept a complete JSON answer to check that the wrapper still rejected it. callm marks that completion as an error before checking the schema. That’s why our script checks status before using the answer. The callm docs list the other envelope fields.

A passing schema only means the JSON has the right shape. The model can say cache.js:3, and the schema can’t tell you whether line 3 is the line you sent. So we open the file there and try to reproduce the bug before changing anything. Same with any command the model suggests: read it before you run it.

Log the model you asked for, the model that answered, and the usage numbers. Ignore our mock’s numbers: 100 tokens in and 20 out are values we typed into the fixture. Real providers sometimes leave usage out, so write down that it’s missing instead of guessing. Some count cached or reasoning tokens inside the totals and some list them apart; check before you add them up. And if you’re pricing a cheaper model, count its failed attempts too, and how many of its answers you kept.

Reuse completed processing steps

callm’s example mini harness handles a bounded sequence of model calls. Each step has a prompt, local schema and named inputs. A later step can consume the earlier step’s saved JSON output. The harness writes result artifacts and a journal with hashes for resume.

We ran a two-step extraction and checking sequence against the mock. The initial run made two requests. The unchanged resume made none. After we changed the source, the harness reran extraction; its mock answer stayed identical, so the checking step reused its saved result.

The last run printed extract: saved and check: resumed. That matched the request count. Keep the journal with the output files after a later step fails; the supplied harness stops there and leaves retries to you. If you interrupt an in-flight request, resuming can send it again, with another charge from a paid provider.

Shape structured exports before asking for an explanation

The export contains records A, B and C. B is inactive; A has the title Cache, guide. Those details make a small test for two common processing mistakes: including the wrong record and splitting a quoted CSV field on its comma.

jq -r '["id","title"],
  (.records[] | select(.active) | [.id,.title]) | @csv' \
  records.json > active.csv
qsv join id active.csv id owners.csv > joined.csv
qsv select 1,2,4 joined.csv > owned-records.csv

Our joined header was id,title,id,owner. Selecting columns 1, 2 and 4 removed the repeated key. The first data row contained A, Cache, guide and docs; C belonged to platform. We read the file with a CSV parser for the assertion, so a misplaced comma would have failed it.

We also tried an owner file without C. The inner join produced one data row and omitted C. With two owner rows for A, it produced three data rows from the two active records. Choose an inner or left join to match your treatment of missing owners, and check duplicate keys before accepting a row count.

Our YAML fixture contains two services, api and worker, with different image versions. Python yq accepts a jq filter for the pair of fields:

yq -r '.services | to_entries[] | [.key, .value.image] | @tsv' services.yml

The output lists api with cache-api:2 and worker with cache-worker:1, separated by tabs. See Python yq’s options for this implementation, the jq manual for @csv, and qsv’s command documentation for joins.

So the model gets two active records and their owners. Do any sums with jq or qsv first so you can check them. Then ask for the migration note and tell it to name A or C whenever it mentions a record, which makes the note easy to check against the CSV. These rows check quoting and the join cases we exercised. They say nothing about speed on a big export.

Time the command, then test the wrapper

Use hyperfine when you’re deciding whether a repeated command is worth speeding up. Keep the input fixed and save the run data:

hyperfine --warmup 2 --runs 10 --export-json timings.json \
  'htmlmd --profile gfm captures/reference/cache.html' \
  'htmlmd --profile gfm --extract-selector article --remove-selectors .ad captures/reference/cache.html'

On our Raspberry Pi 5 both finished in about five milliseconds. Process startup and run-to-run variation made that a poor basis for choosing a winner. The published htmlmd benchmarks use bigger inputs. In July 2026, on the same kind of Pi, htmlmd 0.1.0 took 59 ms on the prose corpus against 174 ms for Go’s html-to-markdown v2, and lost 30 ms to 23 ms on the code-heavy one.

We ran ShellCheck on the review wrapper and fed it the failed responses above. The first catches quoting mistakes; the second catches a failure that wipes the result file. For an application change, add its build and a reproduction of the bug. Use actionlint if you touch a GitHub workflow.

Contribute a conversion that needs fixing

For htmlmd, a small HTML file with the exact options and expected result makes a useful bug report. You can use the code-brace finding above as that kind of example: record the installed version, show the input and compare GFM with mdx-safe output. Keep the original capture in case trimming the page changes the behavior.

To contribute a fix, follow the contributing guide: add a fixture, run the workspace tests, Clippy and formatting, and inspect changes to existing expected outputs. Include measurements from the same machine for a performance claim. Also update the option documentation and agent instructions if you change a flag or its behavior.

Next time you’re about to paste a docs page into a chat, save the HTML first and run the conversion. Check one code example and one table row against the original, and keep that check next to the command. When the output changes on a later capture, you’ll know exactly where to look.