How to run an agent as a durable job
This week I built an agent that works from a chat channel. It reads the error tracker and the issue tracker, picks up a ticket, writes the fix, opens a pull request and posts the diff where the team can see it. A person approves the release. Always.
The chat part took an afternoon. Everything that made it usable took the rest of the week, and none of it was prompting. This is the part nobody writes down, so here it is.
A chat box over a long job is a lie
Type a request in a chat channel and you expect an answer in seconds. The work behind it takes between ten minutes and two hours. Build that as a normal request and response and the first timeout kills it, the user sees nothing, and you have no idea how far it got.
So: every skill is a separate function in a durable workflow engine, with its own retries and its own run history. The chat message is not the job. It is a receipt that points at the job.
Concretely, the pieces that mattered:
Durable state. Runs survive a restart of my server. I started self hosted with state in Postgres, then moved to a hosted engine over an outbound websocket because the inbound networking was a problem I did not want to own. Both took a day.
Absurd timeouts, deliberately. Two hours to start, three to finish. That feels wrong until you watch a single run genuinely take that long.
A cheap classifier in front. A short model decides what kind of request this is before anything expensive runs. This saves money, but the real benefit is that when a request goes to the wrong place I can see which step chose wrong instead of reading a transcript and guessing.
One job at a time, globally, for anything touching a checkout. Two runs in the same working copy produce a diff neither intended, and the commit looks deliberate.
The failure I did not predict, and you will hit it too
Long model sessions quietly broke the connection to the workflow engine. The run was alive, the model was working, and the orchestrator had already decided the worker was gone.
The cause was that the streaming loop never yielded, so the process could not answer liveness checks. The fix was four lines: yield to the event loop every few messages, plus a health endpoint and shorter poll intervals.
The agent and its watchdog were sharing one thread. That is a whole class of bug: when you put a long running model session inside a service, the service stops being able to say it is alive, and everything upstream concludes it died.
The two scheduled jobs that mattered more than the on demand ones
A patrol. It scans new errors on a schedule and files a ticket for anything that looks real. It needed a concurrency limit within a day, because two overlapping runs will happily file the same ticket twice and now you have a backlog problem instead of a bug problem.
A morning digest. What was fixed, what is waiting, what broke overnight. It is the least clever thing I built this month and the one people actually read.
The takeaway
The prompt is not where the difficulty lives. It lives in the boring infrastructure any long running job needs: durable state, honest timeouts, idempotency, a way to see what a run did, and a limit on how many things can touch the same code at once.
Get those right and a mediocre prompt still produces a useful agent. Get them wrong and no prompt saves you.