IT & operations
Runbooks, incidents and the fixes that worked, available to every on-call person and agent.
Who it's for
On-call engineers, site reliability and platform teams, and IT service desks. That is, anyone who has to answer "has this happened before, and what fixed it?" at short notice.
What Workstate does for you
Workstate puts your runbooks, tickets and incident history in one searchable place that every on-call agent can reach. The agent finds the right runbook and cites it. Before anyone starts from scratch, it searches the ledger for earlier incidents on the same system. During an incident, it keeps an incident topic current, so the record already exists when the postmortem starts.
The prompts below are illustrative. Change the system and project names to your own.
Find the runbook
text
The disk on the reporting database is at 95%. Find our runbook for this and list its steps, citing the page.Ask "has this failed before?"
text
Search the ledger for incidents and defects about TLS handshake failures on the edge proxy. What caused each one, and what fixed it?Keep the incident record current
text
Open an incident in project "vpn": users can't connect from outside the office. Current state: mitigated by restarting the gateway; root cause unknown.Later, the same person's agent or a colleague's moves it on:
text
Append the root cause to vpn-0007: the gateway's certificate expired. Set the status to resolved.Prepare the postmortem
text
Get vpn-0007 from the ledger with its full history. Find the related Jira tickets and the runbook it used, and draft a postmortem with citations.Sources to connect
Available now
- Confluence, for runbooks and postmortems.
- Jira, for incident and change tickets, with their comments.
- Upload files, for runbooks kept as PDF or Word documents, and for scripts and configuration.
- GitHub, for infrastructure-as-code repositories.
Coming soon
- Slack Coming soon, for incident channels
- Google Drive Coming soon
Instructions to give your agents
Add these rules to the shared agent instructions:
md
- Before you investigate, search the ledger for incidents, then defects, about the same system.
- Cite the runbook page for every step you suggest. If no runbook covers it, say so.
- When an incident starts, open an incident topic. Update its current state as things change, and resolve it when it's fixed.
- Record the underlying bug as a separate defect topic, and any dead end as an investigation.Ledger habits
- One incident topic per incident. Its summary is what people saw. Its current state is where things stand now.
- Update the current state as it changes. Anyone joining mid-incident then reads one field instead of a chat log.
- Record the root cause as a separate defect topic. A later search for the bug finds it, even if the symptom differs next time.
- Record an investigation for an alert that turned out to be noise, and why, so the next person doesn't chase it.
An illustrative example
Illustrative
At 3 a.m., an alert fires for a growing message queue backlog. The on-call engineer's agent searches the ledger and finds queue-0014, a resolved incident from two months earlier. After a deploy, one consumer had stalled, and restarting the consumers cleared it. A linked defect, queue-0015, records the client library bug behind it.
The agent finds the matching runbook in Confluence and lists its steps with a citation. It opens queue-0021 and notes that this looks like a repeat of queue-0014. The engineer starts from a known fix instead of a blank page.
Limits to know
- Workstate reads your sources on a schedule. Confluence and Jira sources sync once a day by default. An admin can choose every 5 minutes, 15 minutes, 1 hour, 6 hours, 1 day or 1 week. After someone edits a runbook, anyone can select Sync now on the Sources page.
- Uploaded files change only when an admin uploads a new version.
- Workstate doesn't receive alerts or run commands. It isn't an agent: your agents act, and Workstate gives them the record.
- Everyone with access to the namespace sees everything in it. Keep credentials out of the runbooks you index.