Backups run every hour: on Elasticsearch or OpenSearch through an external trigger such as a Kubernetes CronJob, and on an RDBMS through the Orchestration Cluster's internal scheduler. The last one completed at 1 a.m. UTC, and the next will run at 2 a.m. At 2 a.m. UTC, an operator starts a batch operation to clean up old test instances. The filter is wrong, and within minutes thousands of production process instances that should never have been touched are deleted. Deletion can't be undone, and the cluster has replicated it to every broker, exactly as designed. The only copy of those instances is in the backups, and the incident team decides to restore from it.
Whether that goes well depends on decisions made long before 2 a.m. UTC. Did the backup capture primary and secondary storage at a consistent point? Has anyone actually run the restore procedure?
The night above has two halves, the backup and the restore. Each half covers what changed in Camunda 8.9 and 8.10, and how to run it today.
What's at stake
The incident above is why backup and restore are a different discipline from high availability. High availability lets a correctly designed cluster ride through expected failures, such as a restarted container, a failed broker, or the loss of an availability zone. Backup and restore recover from failures the running cluster can't absorb: logical corruption, accidental deletion, operator error, or infrastructure loss beyond the cluster's fault tolerance.
The stakes rise as AI agents take real actions in production. A bad deletion can erase the record of what an agent decided, so restore has to work on the night it’s needed.
Every recovery plan comes down to two numbers:
- The recovery point objective (RPO) is how much recent data the organization can afford to lose.
- The recovery time objective (RTO) is how long the service can be unavailable.
They're easily confused. Backups every four hours can't meet a 15-minute RPO. But backup frequency says nothing about RTO. If provisioning infrastructure, restoring the database, restoring the runtime partitions, and validating business state together take three hours, your RTO is three hours. Backup frequency bounds the recovery point. The whole operational procedure determines the recovery time.
Where Camunda keeps its state
Camunda doesn't keep all relevant states in one place. A complete recovery can involve:
- Primary storage, which holds the Orchestration Cluster's authoritative runtime state: process instances, jobs, timers, variables, deployments, and partition state.
- Secondary storage, which holds the data behind the query APIs and the Applications (Operate, Tasklist, Optimize). Depending on the deployment, this can be Elasticsearch, OpenSearch, or a relational database.
The Orchestration Cluster captures primary storage with a runtime backup: one backup per partition, written to a blob backup store. Disk snapshots of each broker can't replace it. Brokers operate independently, so independent disk snapshots don't line up across partitions. Runtime backups are taken through the cluster so that the snapshots of all partitions are aligned. For more detail, see our earlier post on Zeebe hot backups.
A successful upload or database snapshot isn't enough, though. The recovery point has to be consistent across the storage layers, and how Camunda achieves that depends on the secondary storage backend.
Elasticsearch and OpenSearch use a coordinated recovery point
With Elasticsearch or OpenSearch, a recoverable backup is a coordinated set of artifacts: a runtime backup covering every partition, and an Application history backup. Every artifact in the set carries the same operator-supplied integer backup ID.
The shared ID defines which artifacts belong to the same recovery point. Restore primary storage from one ID and secondary storage from another, and the cluster ends up inconsistent: the Applications show instances the runtime has never heard of, or miss ones it's still running.
RDBMS decouples primary storage and the database
With a relational database as secondary storage, primary storage and the database are backed up independently, each on its own schedule. At restore time, Camunda aligns the selected runtime backup with the database's recovery point. Operators don't coordinate a shared ID or take both snapshots at the same moment. Your DBA or operator backs up and recovers the database with its own tooling, independently of the Orchestration Cluster. That recovery covers the Application data as well.
The difference comes from the database. Relational database backup mechanisms support point-in-time recovery. Elasticsearch and OpenSearch snapshots can't, so consistency has to be created at backup time, with a shared ID.
The table below summarizes the differences that matter for operations.
Backup coordination
Backups must be coordinated carefully via a backup ID
Primary storage and database backed up independently
Backup ID
Supplied by the operator
Generated by the cluster
Scheduled and continuous backups
Built-in scheduling isn't feasible (no point-in-time recovery)
Supported
Restore-point selection
A coordinated backup ID
A backup ID, or a point in time
Taking the backup
Recovery doesn't begin at 2 a.m. UTC, when things break. It begins long before, with preparation. Backups need a schedule. Someone has to know where they live and who can reach them. Failures have to get noticed. And the team needs to have drilled a restore at least once. A restore is only as good as the backup it starts from and the preparation around it.
The old way
Before 8.9, the building blocks for consistent backups existed, but operating them was manual work.
In our example scenario, the 1 a.m. UTC backup was a Kubernetes CronJob. It called the management API, a Spring Boot Actuator extension on port 9600 that wasn't part of the public API, so the job reached it through kubectl port-forward or kubectl exec. It generated a backup ID and passed it through every step. Nothing pruned last month's backups unless someone had written another job for that.
You built every piece yourself. Then you secured it and kept it running.
Camunda 8.9 makes recovery points cheaper
Camunda 8.9 focuses on the backup side. Retention applies to every backend, and the rest of the improvements apply to RDBMS-backed clusters:
- Backup retention periodically deletes runtime backups that fall outside a configured retention window.
- Scheduled backups (RDBMS) move the backup cadence into the Orchestration Cluster itself. No external cron job is needed, and metrics and audit logs cover every scheduled run. Existing backup commands keep working.
- Continuous backups (RDBMS) let the Orchestration Cluster take checkpoints continuously, giving you a range of restore points instead of a handful of discrete backup IDs.
- Decoupled backups (RDBMS) let the runtime and the database be protected independently, with Camunda aligning them at restore time.
- Time-based restore-point selection (RDBMS) lets you restore to a timestamp. Camunda picks the closest available checkpoint.
For RDBMS deployments, the RPO is now set by a cluster-managed schedule and continuous checkpoints rather than by how often an external job happens to run.
The restore itself doesn’t change in 8.9. That comes later with Camunda 8.10, where restore becomes an API operation.
Planning the backup
Start from the business requirement and design the schedule and storage around it. A backup policy should record the target RPO and RTO, backup frequency, retention window, etc.
Before automating anything, set up the backup stores and repositories, and check that they're reachable and secured. The backup and restore documentation covers the prerequisites.
Backing up Elasticsearch and OpenSearch
On Elasticsearch and OpenSearch you coordinate the backup yourself. With the Camunda 8.10 Orchestration Cluster API, you soft-pause exporting, back up the Elasticsearch (or OpenSearch) secondary storage, and then the runtime under the same backup ID, and resume exporting. The diagram below shows each step, the endpoint behind it, and the artifacts it leaves behind. Together, they form a single recovery point.
Backup requests are asynchronous, so poll for the backup state to know when one completes. Resume exporting even when a step fails. Otherwise a failed backup leaves exporting soft-paused while broker disks fill up. The Elasticsearch backup guide covers the full procedure and request details.
Backing up an RDBMS
For RDBMS deployments, there's nothing to script and nothing to trigger. The Orchestration Cluster takes its own scheduled and continuous backups, and native backup or point-in-time recovery protects the secondary storage database.. Your job shifts from running backups to watching them.
The two recovery windows have to stay aligned. A time is recoverable when both the database's point-in-time recovery window and the retained runtime backups cover it. If the Orchestration Cluster keeps seven days of backups but the database keeps three days of point-in-time recovery, you have three days of recovery, not seven. Set both retention periods to cover the same window.
The Orchestration Cluster's side of that window comes from the runtime backup state (GET /v2/backups/runtime/state), which returns the backup ranges of every partition. The runtime can restore to any point where those ranges overlap.
The RDBMS backup guide contains more details.
Monitoring recovery points, not jobs
On either backend, alert on the age of the newest completed recovery point as well as on failures. "No completed recovery point newer than the RPO threshold" catches what a failure alert never sees: an external scheduler that stopped running, a backup schedule that was changed or disabled, or a backup stuck IN_PROGRESS that never reports an error. On RDBMS, where there's no job of your own to fail, this alert and the backup ranges are your main signals.
Alongside age, track backup durations, the date and duration of the last successful restore exercise, and, depending on the backend, broker disk growth during the soft-pause window or the length of the database's recovery window.
With Camunda 8.10, backup state and every restore step are API calls with explicit outcomes, so recovery is observable instead of buried in broker logs.
Restoring the cluster
Back to the incident. It's just after 2 a.m., and the batch operation has been stopped.
The old way
Before 8.10, restoring was an offline operation. The on-call engineer opens the runbook and starts with a question: was last night's backup complete? There's no way to verify that it can be restored successfully before going through the full procedure.
Then comes the procedure. Scale the brokers down. Edit the Helm values to add a restore init container with the right backup ID. Restore secondary storage from the matching snapshots. Scale the brokers back up and watch each one repopulate its partitions. Progress lives in broker logs and pod states, so the engineer tails logs across a dozen pods to decide whether it worked.
That procedure still works, and it's still available. But every step is somewhere a runbook can go wrong, and every step happens under pressure in the middle of the night.
Camunda 8.10 makes restore an API operation
Camunda 8.10 moves the restore side onto the Orchestration Cluster REST API. Backup, exporting control, and restore become regular API operations, with the same authentication and authorization as the rest of the API, separate permissions for each, and no need for management-port access.
The restore itself now runs in place, inside the running cluster. Instead of stopping brokers and running a restore application on each one, you put the cluster into recovery mode, where brokers stop processing but backup queries remain available, and then ask it to restore. The cluster drops the local data of every partition, restores it from the selected backup, and switches the brokers back to PROCESSING.
Each request returns a changeId, so progress is tracked through the API instead of broker logs.
In place doesn't mean without downtime. While partitions are in recovery mode, they don't process anything. What 8.10 removes is the restart and redeployment work around the restore. The restore window itself remains.
Choosing the recovery point
Now the incident lead has to pick a recovery point, and this is where the backend matters. On Elasticsearch, the newest recovery point is the 01:00 backup, and an hour of work is lost. On an RDBMS-backed cluster, continuous checkpoints let the lead restore to 01:58, two minutes before the deletion began, so two minutes of work are lost. The 2 a.m. UTC scheduled backup completes after that checkpoint, which is what makes it a valid restore target. Either way, choosing a recovery point also decides which work runs twice: every job completed after it becomes active again, and your workers will pick it up a second time.
With 8.10, nothing needs scaling down or editing in a Helm chart. The restore takes five steps, and the downtime window runs from entering recovery mode in step 1 to the end of step 4.
While the cluster is in recovery mode, you can also dry-run the restore (POST /v2/restore?dryRun=true) to verify it first. The dry run checks that the backup contents are valid without applying them to the partitions.
The backup and restore documentation covers the full procedure and request details.
Restoring Camunda doesn't rewind the rest of the world. Every job completed between the recovery point and the incident is active again, and workers will repeat its side effects. A payment can go out twice. An email can go out again. A partner system gets called a second time. Whether that's harmless depends on your workers being idempotent. Check it in a drill, so that reconciliation holds no surprises.
The infrastructure reporting a healthy status is only a small part of the recovery process. The business, however, only recovers once the surrounding systems agree on what work should happen next.
Keeping people in charge
The old procedure had about as many steps. What went wrong was people doing the wrong thing, or the right things in the wrong order, under pressure. In 8.10, each step is an API call that validates the request and reports its outcome, so there are no Helm edits to get wrong and no logs to tail to see whether it worked. You can run the steps by hand from your incident runbook. Keep two decisions with people: approving the recovery point, and enabling business traffic once reconciliation is done. Everything else is mechanical.
From runbook to drill
Imagine the 2 a.m UTC incident again, but with a team that has done this before. Before 8.10, a restore meant stopping brokers, changing the deployment and reading logs, so most teams documented it and never ran it. Now every step is an API call with an explicit outcome, and the backup state and its ranges can be read at any time. That makes rehearsing cheap:
- Check the recoverable window on a schedule, so a gap shows up on a Tuesday afternoon and not during an incident.
- Run the full restore, dry run included, against a staging cluster, with the same steps you'd use in production, and record the recovery point, the changeId and the time each phase took. Your RTO becomes a measured number.
- Compare drills over time, so a restore that slows down as data grows shows up as a trend.
When 2 a.m. UTC comes, the team has run this before. They rehearsed it last week and checked the recovery point this morning.
Further your understanding
If you want to learn more about this subject, check out the links below to further your understanding:
