Every few days for a week, our server refused to update its own agent. The command was simple, the failure was consistent, and the first two fixes worked for exactly one day each. This is the full story of diagnosing a legacy source install of Hermes Agent, why the official update machinery fought us, and how we finally migrated to the architecture the updater was designed for. If you installed Hermes as a source checkout before the blobless-clone era and your updates now die with a merge-base timeout — this is your article.
The symptom
The command never changed:
hermes updateThe failure never changed either:
✗ Could not resolve the stable source channel: Command
['git', 'merge-base', '--is-ancestor', '71d34bad...', 'e06df9c2...']
timed out after 10 seconds. No update was applied.Five consecutive attempts, five identical timeouts. No update was ever half-applied — the updater’s safety gate works — but the server was now stuck on a version that was days behind stable, and the failure was deterministic: the updater gives every internal git operation a 10-second budget, and this particular merge-base --is-ancestor check needed 14 to 15 seconds. Every time. On a box with nothing else running.
The first instinct — run it again — was wrong. The second instinct — run it again faster — was wronger.
Why the naive fix worked for exactly one day
An ancestry check between two commits is not normally expensive. Ours crawled, and the reason was visible in the repository stats: the .git directory had fragmented into 43 separate packfiles carrying 1.3 million objects. Every failed update attempt fetches new objects before running the channel check, so each failure added more packs and made the next check slower. A loop, feeding itself.
The first repair was mechanical and satisfying:
git repack -adb
git commit-graph write --reachableThat collapsed 43 packs into 2, dropped the repository from 1.15 GB to 216 MB, and took the failing check from 14.4 seconds to 0.002 seconds. Declared victory. Updated successfully. Wrote a maintenance cron so it would never happen again.
For one day, it never did. Then the same timeout returned with a different commit pair, and the investigation got serious: something was actively undoing the fix.
Reading the updater’s source
Hermes is open source, so the answer was in the tree. Two files explained everything.
First, hermes_cli/git_pack_tidy.py — the updater’s own maintenance mechanism, written specifically for the disease we thought we had discovered. Its docstring describes a partial clone whose .git reached 39 GiB on a ~1 GiB repository, because partial clones re-download the same trees repeatedly, each download landing as its own packfile that nothing removes. The tidy is progressive: every update spends at most 60 seconds erasing redundant promisor packs and merging small ones, keeps its progress in a state file, and resumes next run.
Second, hermes_cli/gitlock.py — which re-applies three config keys on every update:
("maintenance.commit-graph.enabled", "false"),
("gc.writeCommitGraph", "false"),
("fetch.writeCommitGraph", "false"),Commented bluntly: "Keep git from writing a commit-graph in a partial clone." The reference is upstream issue #127711 — commit-graph writes caused problems in partial clones, so the updater deliberately disables them and re-disables them every run.
That is why my Oct 15 fix had evaporated. I had set fetch.writeCommitGraph=true at the repo level; the updater’s design considers commit-graph writes harmful in partial clones and re-applies the off switch each time. My "durable" configuration was a configuration the updater is programmed to erase.
There was one more subtlety in the pack-tidy design: it manages promisor packs only — the packs that a partial clone fetches into. Our repository was a full clone from the 2025 install era: fifteen months of complete history in plain, non-promisor packs that the tidy is structurally unable to merge. The progressive cleanup that keeps every fresh install healthy was blind to most of our weight.
The full audit we ran alongside
While the update question was open, we mapped the rest of the disk, because an agent server accumulates more than git objects. The inventory, with verdicts:
~/.npm— 2.3 GB of download cache andnpxtrees: regrows by design, since the agent’s TUI, web dashboard and browser tooling are Node applications rebuilt on every update~/.cache/ms-playwright— two complete Chromium builds (665 MB each), one per Playwright installation; Playwright never garbage-collects superseded revisions~/.cache/huggingface— 2.1 GB holding four Whisper speech-to-text model sizes while the configuration used one~/.cache/puppeteer— 653 MB of Chromium with zero references anywhere on the server, verified by grep before removal~/.cache/uvand the pip cache — another ~1.9 GB of pure package caches
All of it regenerable. About 10 GB came back in an afternoon. The lesson generalizes: du -xsh | sort -rh, then classify every heavy directory as disposable cache or load-bearing state before touching anything — and verify "probably unused" with a project-wide grep before it becomes "definitely broken".
The proper fix: migrate, don’t maintain
The updater’s architecture assumes a blobless partial clone — a checkout that carries commits and trees but fetches file contents on demand. Fresh installs since v0.22 are created exactly that way: small .git, progressive tidy converging over the first few updates, channel checks walking short deltas. Docker installs skip git entirely. Desktop installs use packaged updaters. Nobody in those worlds ever parses fifteen months of unindexed history at check time.
Our legacy install could be maintained forever with crons and repacks. The cleaner decision was to migrate it to the state the updater is designed for — and the official installer supports exactly that path: re-running it keeps your chosen channel and leaves your configuration alone, because everything that matters lives outside the repository.
The sequence, which took well under half an hour including verification:
# 1. Stop the gateway (system service — this is the one sudo step)
sudo systemctl stop hermes-gateway
# 2. Set the old checkout aside (instant rename, rollback insurance)
mv ~/.hermes/hermes-agent ~/.hermes/hermes-agent.bak
# 3. Re-run the official installer
cd ~ && curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
# 4. Verify the toolchain
hermes --version && hermes doctor
# 5. Back under systemd supervision
sudo systemctl start hermes-gateway
# 6. The proof: the check that failed for a week
hermes updateEverything that makes the server yours — config.yaml, .env with all API keys, skills, profiles, cron jobs, sessions, OAuth tokens — lives in ~/.hermes/ outside the source checkout. The installer replaces code, not identity.
The numbers
| Before | After | |
|---|---|---|
| Repository size | 2.9 GB | 993 MB |
| Packfiles | 12–43 | 2 |
| Clone type | full (2025-era) | blobless (blob:none) |
| Ancestry check | 15.2 s → timeout | 0.002 s |
| Update result | failed ×5 | success |
| Disk used | 69% | 47% |
The merge-base check that failed five times now completes in two milliseconds — and it does not need a commit-graph to do it, because the updater’s architecture was tuned for exactly this shape of repository.
Two honest footnotes. First, the update ledger recorded our detached verification run as still "running" — harmless bookkeeping from an invocation that outlived its shell; the checkout itself is the source of truth. Second, the gateway came back spawned by the updater rather than systemd; it serves traffic perfectly, but it will not survive a reboot in that mode, so the final housekeeping is a clean handover: stop the standalone process, sudo systemctl start hermes-gateway, and confirm the unit is active.
What we reported upstream
The legacy-install edge case is worth an issue: on pre-blobless source checkouts, the updater re-applies commit-graph-off keys (#127711’s design) and then runs a channel check that can exceed its own 10-second budget — the two mechanisms compound on old repositories. A migration hint in the updater’s error message ("your checkout predates partial-clone support; re-run the installer") would have turned our week into an hour. Upstream’s own recovery tooling — the closure-repair code that rebuilds a damaged checkout from saved recovery blobs — already shows how much thought went into surviving partial states; the legacy clone just predates all of it.
Takeaways
- A fix that changes state is weaker than a fix that changes the system. Repacking the repository worked for a day because the mechanism that caused the rot was still in place.
- Read the source of your updater. The re-applied config keys looked like a bug and were a design decision — with an issue number and a rationale.
~/.cacheis disposable by definition;~/.localis installed software. Ten gigabytes of the audit was caches; the pipeline-breaking dependencies lived in.local/liband were never touched.- Verify orphans before deleting them. A 653 MB Puppeteer cache with zero references is junk; the same directory with one reference is your browser stack.
- When your tooling and your installation disagree about architecture, migrate the installation. The supported installer path is shorter than any workaround you will maintain forever.
The server now updates in seconds, sleeps at 47% disk, and the only thing the migration cost was one afternoon and a backup directory we have not needed once.
Timeline of the actual week
| Date | Event |
|---|---|
| Oct 15 | First timeout; repack fix; 43→2 packs; 14.4 s → 0.002 s; declared victory too early |
| Oct 15 | Set writeCommitGraph=true — silently re-applied to false by every later update |
| Oct 17 | Timeout returns; stale locks found; second repack; cron workaround added |
| Oct 21 | Timeout again; source read: #127711 keys + promisor-only tidy identified as design |
| Oct 21 | Installer re-run; fresh blobless clone at 993 MB; update passes; cron deleted |
