The Code Watching My Sump Pump Has Been Off the Books Since August

The best thing I read in the robotics press this week was not a product launch. It was an analysis of a quarterly filing. The Robot Report walked through Symbotic's relationship with Walmart, which supplied 90.5 percent of Symbotic's revenue in the quarter that ended in June. The interesting part is not the concentration. It is what Walmart wrote into the contracts anyway. Symbotic cannot afford to lose Walmart, and Walmart did not rely on that. Its 2022 master automation agreement gives it continuity rights: the right to embed its own engineers with Symbotic, and technical material held in escrow that it can use under specified conditions.

In plain words, the customer with the most leverage in warehouse automation asked one question before it trusted the system: if the vendor stops being there, what do we need in hand to keep this running? Then it got that in writing. The same article notes that about half of Symbotic's roughly 2,000 employees work at customer sites, installing and maintaining the robots. The other half of continuity is people who look at the machine.

I run monitoring on exactly one site I fully control, my own basement sump pump. So this morning I asked Walmart's question about it. If I vanished tonight, or the host it runs on rebooted, what would the person after me find?

The escrow copy is five months old

The monitor's code lives in a git repository, pushed to GitHub. That is my escrow. A successor would clone it. Here is what they would get, compared with what is actually running.

The repository's newest commit is from June 3. Origin matches it, so nothing is stuck on a laptop either. The monitor has been modified since, and the change was never committed. Neither was the guardian's. git status lists two modified files and sixteen untracked ones, including a deploy script, a watchdog script, two debugging loggers and five dated backups of the configuration file. The configuration's history is a row of file timestamps.

The edit that is off the books

Uncommitted edits are normal. What made me stop was which edit this one is.

In late August I found that the monitor had silently stopped sending its routine, low-priority alerts. Someone (me) had appended a blank recipient line to the configuration file on August 20, the code read the blank as "no recipients," and every routine alert since has gone nowhere. The count this morning is 8,386. I wrote about it at the time, and I have counted that line in nearly every post since.

The uncommitted change to the monitor was made at 9:16 am on August 20, the same minute as the configuration edit. Here is what it does, condensed:

if recipients:
    ... send through Gmail ...
else:
    log("LOG email skipped: no recipients configured")

The committed version has no such branch. It hands the empty list to Python's mail library, which raises an error when no recipient is accepted, and the monitor writes a log line that starts with ERROR: Failed to send email. So the code in escrow would have shouted. The code that is running writes a calm sentence that reads like a setting someone chose, and it has written it 8,386 times. The line that hid the problem exists on one disk in my basement.

I am not claiming the edit was a mistake. A clean "skipped" message is a reasonable thing to want. The point is that the decision to make a missing recipient list quiet, rather than loud, is recorded nowhere except the behavior of one running process. A successor reading the repository would be debugging a different program.

The fix that disappears on reboot

On October 5 the monitor spent eighteen hours unable to see the smart plug the pump is plugged into. The plug never moved; the host had lost its address on the pump's subnet. The fix that night was one command typed by hand, adding the address back to the network interface. It worked, and the outage ended.

That command is not in any file. The host's network configuration still says only "get an address by DHCP." The fix has held for 24 days because the host has not rebooted since September 2, 57 days ago. The next kernel update, power cut or restart quietly undoes it, and the monitor goes back to looking for a plug that is right where it left it. Every hand fix that is not written down is a scheduled repeat of the outage it fixed. The date is just unknown.

The AI watcher that reviews these logs four times a day has recommended a DHCP reservation for the plug 117 times since June. It was right. It was also writing into a log that nothing reads, which I have covered before.

The disk, and the visit

Two smaller findings from the same ten minutes:

• The host's disk is 97 percent full. 39 GB free of 1.1 TB. The pump's logs are about 78 MB; the rest is unrelated GPU and model work that shares the machine. A monitor that cannot write its log fails in the most boring way possible, and none of my alerts watch free space. The monitor shares a failure domain with whatever experiment fills that disk next.
• The site visit is on day 244. Symbotic keeps about half its people where the robots are. My equivalent is lifting the lid on the pit and watching one pump cycle, a two-minute job. The last time I did it was February 27. The pump has run 119 times a day through thirteen days without rain, and I still cannot say from the log whether it is moving water.

Five lines a successor needs

Walmart got continuity in a master agreement. A small building owner paying $99 a month for monitoring deserves a smaller version of the same thing, and it fits on one page. This is what I am adding to mine, and what I will hand to every monitoring client:

1. Where the running code is. A repository link, and a check that the newest commit is newer than every file the system runs. On my host that check is one git status, and today it fails.
2. What the configuration contains. The names of every setting, never the values, committed next to the code. Passwords stay out; the fact that a recipient line exists, and is blank, goes in.
3. What the host depends on. Network addresses written in the network config, not typed at a prompt. Outside services listed (mine: Gmail, ntfy, two weather APIs, the USGS water service). Free disk in the daily report.
4. Who owns each credential. The plug's password and the mail password live in one file on one machine. Someone other than me needs to know that, and how to rotate them.
5. When a person looks. A date on a calendar for the physical check, and a line in the daily digest saying how many days since the last one. Mine would say 244.

None of this is AI. All of it decides whether the AI parts are still running in a year, which is the question asked at a building-owner conference I wrote about two days ago. The honest answer for my own system this morning: it is running. If it stopped, the person who restarted it would get a different program, on a host that has forgotten how to reach the pump.