Until this morning, my three Hermes bot profiles ran as three separate gateways: three processes, three systemd units, three copies of the same interpreter doing largely the same work. Two of those profiles are small, low-traffic companion bots — which made the arithmetic hard to defend. Every extra gateway was one more process to update, one more unit file to babysit, one more log to check after every hermes update.
So I merged them. One gateway process now serves all three profiles — the mode Hermes calls multiplexing. The migration itself was five commands and about a minute.
Then it broke.
Forty seconds after the restart, one of the bots started answering with short error notices instead of replies. It took most of the morning to understand why — and the answer turned out to be a real database bug, already reported upstream, with a fix in flight. Both halves of that sentence matter, so here is the whole story.
Why multiplex at all
The honest motivation is resource arithmetic. Three bots with the same job description do not need three Python processes, each carrying its own interpreter and heap. In multiplex mode a single gateway process serves every profile under profiles/ — the secondaries keep their own bot token, their own .env, their own sessions and their own state.db; what they give up is a private process.
The trade-off is worth stating plainly, because it decides whether multiplex is right for you:
- What you gain: one process, one unit, one log stream for every bot — and the cron scheduler ticks every served profile's job store in a single pass.
- What you give up: crash isolation. One restart reconnects every bot. If you want hard separation between profiles, keep one gateway per profile.
I host two small companion bots for other people, so shared fate is an acceptable price. And as the migration log later confirmed, the profiles stay properly isolated: namespaced sessions, per-profile secrets, per-profile databases. Profiles created after the migration are picked up by the running gateway automatically.
The migration, in five commands
The procedure is documented, and the order in it is load-bearing. The secondary gateways must stop before the flag takes effect, so the multiplexer never long-polls a bot token that another process still owns:
cp ~/.hermes/config.yaml ~/.hermes/backups/config.yaml.pre-multiplex-$(date +%Y%m%d)
sudo systemctl disable --now hermes-gateway-alpha.service hermes-gateway-beta.service
sudo mv /etc/systemd/system/hermes-gateway-alpha.service ~/.hermes/backups/ # and the beta unit
sudo systemctl daemon-reload
hermes config set gateway.multiplex_profiles true
sudo systemctl restart hermes-gateway.serviceTwo details worth keeping: disable matters as much as --now — an enabled unit comes back at boot and fights the multiplexer for the same token — and I moved the unit files instead of deleting them, which makes rollback a rename.
The proof showed up in the log within a minute:
✓ telegram connected (profile: alpha)
✓ telegram connected (profile: beta)
Gateway running with 3 platform(s)
Cron scheduler will tick 3 profile(s) under multiplex: ['default', 'alpha', 'beta']That is the whole migration. It works, exactly as advertised.
Forty seconds later
The bots' replies became short error notices while the log filled with this:
FATAL: a live process holds a deleted state.db-wal or state.db-shm inode
while the path names a different (or missing) generation.Plain-language translation. Hermes keeps sessions in a SQLite database in WAL mode — the write-ahead log journaling scheme, where every database is accompanied by two sidecar files: a log that buffers recent commits and a shared-memory index. If those sidecars disappear while a live process still holds them open, that process sits on an ambiguous, split view of the database — the exact class of situation that ends in corrupt pages. So Hermes ships a guard: refuse to write, refuse to open a second WAL, and say why. The agent could still compose a reply; it just could not persist the session, so the turn failed. Failing closed is a feature, and it is the reason this story has no corruption in it.
A restart cleared it — the guard is deliberately sticky for the life of a process, so the fix is "stop the holder, reopen". But within the hour it came back. Twice. After a hermes status command in one case, and around the top of the hour when cron jobs fired in the other. A fluke would not be that punctual.
What the forensics found
Three tools cracked it, and all three are boring.
lsof +L1 lists open files that no longer have a name in the filesystem. The deleted -wal and -shm were held open by my own gateway process. Nobody else — which instantly ruled out the usual suspects (a second writer, a rogue daemon) and pointed at a race instead of a rogue.
A capture folder. Hermes had quietly done something clever: before settling the failure, it captured the doomed WAL generation into a folder beside the database — state.db.retired-wal-<timestamp>-<pid>/ — with a manifest recording sizes, inode identities and checksums. The first capture's WAL was 0 bytes: nothing to lose. The second held about 4 MB of committed frames from the working window — preserved, not lost. The guard was protecting data, not just refusing to work.
Log timestamps. Lining up the two failures against everything else the box was doing produced the pattern: both followed a short-lived Hermes process — a status command, cron workers starting. One long-lived writer plus one short-lived reader/writer, sharing a WAL database.
The bug was not mine
I went looking for that signature upstream and found the exact case, reported by other users the same day: issue #109687, priority P0 — a single plain CLI invocation orphans the live gateway's state.db WAL generation. A maintainer reproduced it within hours; a fix, PR #109734, is in flight.
The mechanism, as the issue's analysis describes it: a helper that hardens file permissions walks the database and its sidecars with plain file descriptors — open, fchmod, close. Closing those descriptors cancels the process's own SQLite advisory locks. SQLite coordinates the "last connection" rule through those locks, so once they are gone, the next short-lived process to close the database believes it is the last one — and removes a WAL that a live gateway still holds open.
Two things are worth noting about that. It has nothing to do with multiplexing: any WAL-mode state.db with two processes on it was exposed, and the migration merely put me in the room the minute it happened. And it got caught by a custom guard, not by SQLite: my library version is outside the known WAL-reset vulnerability set, which is exactly why the guard exists as a second line of defence.
What I did about it
The fix is not merged yet, and the triggers are things I do every day anyway — CLI commands, cron jobs, updates. So I took the containment documented in the same issue: leave WAL mode until the fix ships.
The sequence, run with the gateway stopped (the switch needs exclusive access, and stopping also forces a full checkpoint first):
- switch every live database to
journal_mode=delete— a small script that refuses to run while a gateway is alive and prints, per file, the before → after mode plus an integrity check; - make it stick in config —
hermes config set database.journal_mode delete, once for the default profile and once per served profile, so the next open does not flip the files back to WAL; - start the gateway and watch the first replies.
Why it works is almost embarrassingly simple: with no WAL sidecars there is nothing to orphan. The cost is WAL's write concurrency — writes take a brief exclusive lock instead — and at the traffic of three chat bots and a handful of cron jobs, that is invisible. It is also reversible: when the fix ships, the same procedure with journal_mode=wal goes back.
Sixteen databases: all switched, all integrity checks clean. The first bot reply landed normally, and the cron schedule ran that hour like nothing had happened.
The part worth keeping
Four lessons, in descending order of usefulness.
Failing closed is a feature. The guard did not try to be clever and keep writing into a split view. It stopped, captured what could still be captured, and said exactly what was wrong. That converted silent data loss into a loud outage with preserved data — and loud outages are a much better failure mode, because you can fix them before the quiet version ruins a week. (It is the same principle as Nothing Fails Loudly — except this time the quiet failure was upstream's, and it did not stay quiet for long.)
The tools that solved it are boring. An lsof flag, a manifest file, log timestamps. No profiler, no debugger build. When a system fails weirdly, the first question is not "what is broken" but "what is actually happening" — and the answer usually fits in three terminal commands.
Never delete a WAL by hand. The error message says it plainly, and now I understand why: until a checkpoint folds its frames into the database, that sidecar file is the only copy of those commits. It looks like scratch space. It is not.
When you cannot prevent a class of failure, instrument it. I left a small passive watchdog watching the sidecar files, logging any change with a process snapshot. It cannot stop the bug — but it means the next time anything eats a journal file, I will know within seconds instead of noticing it through a confused person.
Where it stands
The fix PR is pending. When it merges I will stop the gateway, update, and decide whether to move back to WAL — for now the deleted-journal mode stays, because it makes this particular failure impossible while costing me nothing measurable.
And the migration itself? Still a win. One process, three bots, three isolated profiles, one place to look. The five commands were the easy part; the morning was the invoice for learning what was living under the floorboards. For a system that talks to people while I sleep, that is a trade I will make every time.
If you run several profiles under one gateway — or are thinking about it — the multiplex guide is accurate, and the bug above is already on the project's radar. Migrate anyway. Just read the error messages when the floor creaks.
