Jul 21, 2026

LLM-assisted workflow for tracking drift in localized docs

We (the helm.sh maintainers) recently added a process for tracking drift in the localized Helm docs. It involves updating the front matter of each localized page with the latest commit from its English-language source.

So, for example, if you translate overview.md into Spanish at es/overview.md, you’d find the latest commit made to overview.md and add it to a default_lang_commit field in the front matter of es/overview.md. Then you can run a script that tells you when the default_lang_commit for es/overview.md differs from the latest commit for overview.md (i.e., that the page has drifted from its source content). Neat!

This process is a complete copy of how OpenTelemetry does it, on recommendation from the CNCF tech docs team.

Automating the tedious work of finding default_lang_commit

While the script for tracking drift is automated, there’s still the time-consuming and largely manual process of finding which English source commit maps to the localized page’s content.

We published the bulk of the localized docs in late 2025, and have several other localizations that were merged years before that. So, adding default_lang_commit isn’t as simple as finding the latest commit on each English page. For many pages that have already drifted, it would involve digging through the Git history on the English source to see which version matches.

To avoid spending an unreasonable amount of time manually wading through Git history page by page, I used Codex to write a deterministic script that does it for me (and probably more accurately, too).

This script checks:

if the localized page already has default_lang_commit
if the expected English source file exists
if the localized page structurally matches the latest English source (using markers such as headings, list items, code fences, admonitions, imports, links, images, and HTML anchors)
if the localized page structurally matches one recent historical version of the English source (I have it checking the most recent five commits)
if there are any differences that need to be flagged for manual review

Testing it on the Greek locale

There was a fair bit of back and forth with Codex to refine the script, mostly to reduce false positives. For example, the script should ignore the existence of explicit anchor IDs on headings (which tend to only be used in localized docs pages), and should also recognize when something like a link is present, but just split across multiple lines due to semantic line breaks. I didn’t see any cases where these issues were indicative of a semantic difference between the localized page and its English source, so I considered them safe for the script to ignore.

After several rounds of testing and refining, here’s the output of a dry run on the Greek locale:

node bin/i18n-add-default-lang-commit.mjs /Users/paigecalvert/Projects/github/paigecalvert/helm-www --locale el
Mode: dry-run

Locale el: scanning version-3 localized docs...
Locale el: found 112 Markdown/MDX file(s).
Locale el: deterministic pass 25/112
Locale el: deterministic pass 50/112
Locale el: deterministic pass 75/112
Locale el: deterministic pass 100/112
Locale el: deterministic pass 112/112
Locale el: writing reports...
Locale el: wrote .i18n-default-lang-commit/el/summary.md and .i18n-default-lang-commit/el/run.json
Report: .i18n-default-lang-commit/el/summary.md
  failed-source-missing: 1
  needs-review: 3
  would-update-historical: 4
  would-update-latest: 104
✨ Total elapsed: 1.4s

It processed 112 localized files from the Greek locale, and left me with only three for manual review!

It also looks like it would update the majority of the pages to use the latest commit from the English source (would-update-latest) for default_lang_commit, with a few more that it matched up with one of the historical commits (would-update-historical).

And, one “failed-source-missing” makes sense because I recently deleted a file in the English source as part of some reorg work.

Can an LLM help with the remaining manual review?

I was curious if an LLM might be able to do some of the remaining manual review for me. In this case, three pages left for review isn’t bad, but the Greek locale is unusually up-to-date since we had recently updated it. When I ran the script on some of the other locales, I was seeing 12 to 50 pages left for review, which is less ideal.

As a first step, I opened the auto-generated summary.md file from the dry run to review the specific differences flagged for each of the needs-review pages.

For example, for the topics/charts.md page, the summary describes an issue where the Greek version of the page is missing a line that appears in the English source:

## i18n/el/docusaurus-plugin-content-docs/version-3/topics/charts.md

Status: `needs-review`
Source: `versioned_docs/version-3/topics/charts.md`
Latest English commit: `f1c342d7bbd8fca5494262a93699b27012859e24`

Best candidates:
- `f1c342d7bbd8fca5494262a93699b27012859e24` (score 0.556, exact false)
- `6bf4b4cb62bb3952a17b68b3cd8b832bcede390c` (score 0.556, exact false)
- `07caa4dd6e58a47e79ac2ec7949e57157f1a2b2a` (score 0.556, exact false)
- `e86c42f154920b96754107e75387481a9b0de283` (score 0.556, exact false)
- `55b1d074fc1242558c6e49eec1b59fc0bfd6f204` (score 0.556, exact false)

### Specific Differences

#### sequence

- Missing from localized: `versioned_docs/version-3/topics/charts.md:182` line 182: 3. Remove the chart from the source repository (e.g. git)

I checked the Greek version of topics/charts.md, and that line does appear, so it’s a false positive. I copied this snippet of the summary report and shared it with Codex, prompting it to evaluate why the script marked the line as missing from the Greek version.

This was the response:

The localized sentence is there. The script is being fooled
earlier in the same section: the English source has a wrapped
prose line that begins with 3., and the fingerprint parser is
incorrectly counting it as an ordered-list item.

So, when there’s a line of prose that happens to begin with a number, the script identifies it as a list item, and looks for the same number of list items in both the English and localized version of the page. When it counted four list items in the English version and only three in the localized version, it flagged the fourth list item as “missing” from the localized page.

This felt like a pretty unusual edge case that I didn’t want to try to account for in the script’s logic. But, it is something that an LLM should be able to review, identify as a false positive, and then overrule.

Expected outcome

I reviewed the other two pages marked as needs-review as well. Both of those pages had legitimate semantic differences that weren’t represented in any of the five most recent commits on the English source.

Based on my review, these were my expectations for how the LLM should assess each of the pages following the deterministic pass (I’d use this as a baseline to judge the output):

File Differences Expected outcome

File	Differences	Expected outcome
`index.mdx`	Greek version is missing a handful of links in a bulleted list. While it has the same list items as the English version, they are plain text rather than links.	Leave this page for review.
`sdk/examples.mdx`	Various differences with import statements and code blocks.	Leave this page for review.
`topics/charts.md`	Missing “3. Remove the chart from the source repository (e.g. git)”. This difference is a false positive (see Testing it on the Greek locale).	The LLM should judge that this is a false positive, and set `default_lang_commit` to the latest commit from the English source.

index.mdx

Greek version is missing a handful of links in a bulleted list.

While it has the same list items as the English version, they are plain text rather than links.

Leave this page for review.

sdk/examples.mdx

Various differences with import statements and code blocks.

Leave this page for review.

topics/charts.md

Missing “3. Remove the chart from the source repository (e.g. git)”.

This difference is a false positive (see Testing it on the Greek locale).

The LLM should judge that this is a false positive, and set default_lang_commit to the latest commit from the English source.

Designing the workflow

My goal was for this to be an automated process where I’m not having to pass each page marked needs-review to an LLM, prompting it to review.

I wanted a single CLI command that would do the following:

Run the deterministic script, which either assigns a default_lang_commit based on the structural fingerprints of the localized and English versions of the page, or marks the page needs-review.
Pass only the pages marked needs-review to an LLM for review. (Use an optional --model flag to set which model to use. When --model is not set, the script doesn’t attempt the LLM pass).
The LLM either assigns a default_lang_commit, or leaves the page for manual review.
Generate a report with a summary of the changes.

Using local API calls with Ollama

Apart from the workflow itself, one other important consideration was that I wanted to avoid API billing. So, using a cloud-based provider where I’d need to supply an API key wasn’t going to work.

To get around that limitation, I decided to try Ollama to run the model locally.

Ollama comes with a built-in API that’s served at http://localhost:11434/api by default. For this workflow, I can use the API’s /api/generate endpoint to generate a response from a prompt.

To start, I downloaded the qwen3:0.6b model (ollama pull qwen3:0.6b). This is the smallest Qwen 3 model available on Ollama. I wasn’t sure how successful it would be, but figured it’s always good to start small and then scale up as needed.

Results with Qwen3:0.6b on the Greek locale

This is the output from another dry run against the Greek locale that includes the new LLM review step:

node bin/i18n-add-default-lang-commit.mjs /Users/paigecalvert/Projects/github/paigecalvert/helm-www --locale el --model qwen3:0.6b
Mode: dry-run

Locale el: scanning version-3 localized docs...
Locale el: found 112 Markdown/MDX file(s).
Locale el: deterministic pass 25/112
Locale el: deterministic pass 50/112
Locale el: deterministic pass 75/112
Locale el: deterministic pass 100/112
Locale el: deterministic pass 112/112
Locale el: deterministic results: failed-source-missing=1, needs-review=3, would-update-historical=4, would-update-latest=104
Locale el: writing reports...
LLM review: 1 eligible page(s) using qwen3:0.6b; 2 skipped.
  [1/1] reviewing i18n/el/docusaurus-plugin-content-docs/version-3/topics/charts.md
    accepted f1c342d7bbd8fca5494262a93699b27012859e24 (2728ms)
    report updated: .i18n-default-lang-commit/el/summary.md
Locale el: wrote .i18n-default-lang-commit/el/summary.md and .i18n-default-lang-commit/el/run.json
Report: .i18n-default-lang-commit/el/summary.md
  failed-source-missing: 1
  needs-review: 2
  would-update-historical: 4
  would-update-latest: 104
  would-update-llm: 1
✨ Total elapsed: 4.2s

The main difference between this pass and the earlier deterministic-only run is the LLM review line:

LLM review: 1 eligible page(s) using qwen3:0.6b; 2 skipped.
  [1/1] reviewing i18n/el/docusaurus-plugin-content-docs/version-3/topics/charts.md
    accepted f1c342d7bbd8fca5494262a93699b27012859e24 (2728ms)
    report updated: .i18n-default-lang-commit/el/summary.md

So, the LLM correctly recognized the false positive in topics/charts.md, and accepted it.

In the detailed summary of the LLM’s review of topics/charts.md, we can see that it would set default_lang_commit to the correct commit hash from the English source:

## i18n/el/docusaurus-plugin-content-docs/version-3/topics/charts.md

Status: `would-update-llm`
Source: `versioned_docs/version-3/topics/charts.md`
Assigned commit: `f1c342d7bbd8fca5494262a93699b27012859e24`
Latest English commit: `f1c342d7bbd8fca5494262a93699b27012859e24`

Best candidates:
- `f1c342d7bbd8fca5494262a93699b27012859e24` (score 0.556, exact false)
- `6bf4b4cb62bb3952a17b68b3cd8b832bcede390c` (score 0.556, exact false)
- `07caa4dd6e58a47e79ac2ec7949e57157f1a2b2a` (score 0.556, exact false)
- `e86c42f154920b96754107e75387481a9b0de283` (score 0.556, exact false)
- `55b1d074fc1242558c6e49eec1b59fc0bfd6f204` (score 0.556, exact false)
LLM elapsed: 2.7s

LLM decision:
- decision: `use_commit`
- recommendedCommit: `f1c342d7bbd8fca5494262a93699b27012859e24`
- confidence: ``
- reason: The candidate commit matches the localized page's content and purpose. The localized path is correctly referenced, and the commit hash aligns with the latest English commit provided.
- evidence:
  - (none)

### Specific Differences

#### sequence

- Unmatched source item: `versioned_docs/version-3/topics/charts.md:182` line 182: 3. Remove the chart from the source repository (e.g. git)

Huzzah!

In the summary report above, you can also see that the LLM didn’t fill out all the fields in the JSON response template (it left confidence and evidence blank). This isn’t super important to me since I can still see the specific differences identified by the deterministic run, with line numbers.

In fact, I ended up relaxing the validation for the LLM’s JSON response because the stricter requirements were was causing constant failures. More on that in Takeaways.

You also might notice LLM review: 1 eligible page(s) using qwen3:0.6b; 2 skipped. from the output. I ended up adding some eligibility criteria for which pages actually get passed to the LLM. Before these checks, the LLM kept messing up by accepting pages that had real semantic differences in reference-style components, including:

Links ([text](/some/path.md) or [text](https://example.com))
Images (![alt](/img/foo.png))
HTML anchor tags (<a id="anchor"></a>)
MDX tags (like <Install />, <CodeBlock>, <Tabs>)

For the Greek locale, this meant the LLM repeatedly, erroneously accepted the differences in the sdk/examples.mdx and index.mdx pages rather than leaving them for review. Since it was consistently bad at assessing these types of differences, rather than continuing to try to force it, I just changed the deterministic part of the script so that any pages with differences in those reference-style components never get passed to the LLM in the first place.

There are a few ideas I have for improving this in a follow-up, like rendering partials/imported components in the MDX pages before passing them to the LLM, or employing a stronger model in a separate step for assessing differences with links/images/anchor tags/MDX tags. But for now, the eligibility checks keep the LLM honest while still saving me time on manual reviews.

Results for other locales

After getting a successful result from the Greek locale, I ran the CLI command against several other locales to see how it held up.

This is where the potential value really started to shine. For example, for the French locale, there were 52 pages marked needs-review after the deterministic run, and the LLM got that down to seven:

// summary of dry run results on french locale

Locale: fr
Mode: dry-run
Model: qwen3:0.6b
Elapsed: 1m 15s

## Results

- failed-source-missing: 1
- needs-review: 7
- would-update-historical: 4
- would-update-latest: 57
- would-update-llm: 45

A spot-check review of the pages marked would-update-llm confirms that the LLM wasn’t just accepting all these pages without cause. So, the CLI is looking like a promising tool that I can use for assigning the default_lang_commit across all our locales, saving me a lot of time in manual review.

Takeaways

It was no small amount of effort to get the deterministic script, the LLM step, and the workflow mechanics in general to a place where it could be helpful for my use case.

Through this process of testing and failing and refining (and rinsing and repeating), I came away with a handful of learnings:

Originally, the CLI had a deterministic step that validated the JSON response from the LLM. The goal of this validation step was to ensure that the LLM provided its reasoning and evidence. But, trying to get a structured JSON response from the LLM was a pain.

I spent a lot of time editing things like the prompt and context passed to the LLM, but it wasn’t until I relaxed the strict LLM response validation that I actually saw the results I was looking for. A spot check after running the CLI against several other locales reveals that the those strict checks weren’t providing value anyway (the LLM’s assessments were accurate, it just couldn’t adhere to the JSON response requirements). I wonder if a different model or maybe using a different Ollama API endpoint would be better for getting structured responses. More to explore.
When I was first testing the LLM step, I was getting different outcomes every time in terms of which pages it accepted or left for review. Tweaking these model settings with Ollama helped me get consistent results:
- temperature: 0: Temperature controls how much randomness the model uses when choosing the next token. Lower temperature = more deterministic.
- top_p: 0.1: top_p (also called nucleus sampling) limits the model’s token choices to the smallest group of likely tokens. When top_p is 1, that means the model considers all possible tokens; when it’s 0.1, the model sorts possible next tokens from most likely to least likely, then considers the smallest group of likely tokens that have a combined probability of at least 0.1. (It’s not the simplest concept to understand, but I’m not sure what I was expecting from something called “nucleus sampling”).
- seed: 1: Initializes the runtime’s random number generator. When the model needs to use random sampling, a fixed seed makes the random sequence repeatable. When temperature is 0, there’s typically no need for the model to use randomness. But initializing a seed can still be useful for any edge cases (like for tie-breaking logic that uses random sampling).
I’ll need to try more of the available models on Ollama to get a better sense of the types of tasks where different models excel. For this project, I just tested the qwen3:0.6b model. I also briefly tested qwen3:4b and gemma3:4b, but the only difference I noticed was a longer run time, plus the soft whir of my laptop fan working harder.
Much of the meaningful payoff came from refining the deterministic script itself. The main challenge here was finding a balance between avoiding false positives, versus making the script overly permissive. But in general, I’d always try to default to fixing the problem programmatically first, then outsourcing to an LLM for the tasks that would truly benefit from AI.

Now, time to start updating some localized docs with default_lang_commit…