We Asked Jepsen to Break Orkes Conductor

What happens to a running workflow when the infrastructure breaks?

Years of experience designing and running Orkes Conductor gave us a reasonable idea of how it would hold up. But we decided to ask Kyle Kingsbury, the author of Jepsen, to test it and tell us where we actually stand.

Kyle has tested MongoDB, CockroachDB, and a couple of dozen other systems. He breaks distributed systems for a living. Check out his work.

His back catalog is public. As of today, Orkes Conductor is there too.

How this works

Kyle stands up a real cluster, runs a real workload against it, and then does unpleasant things to it: kills processes, pauses them, stops Postgres and Redis, and partitions the network. While that is happening he records every operation the client issued and what came back. Afterwards he checks whether the history he recorded is one the system was allowed to produce.

Anyone can break a cluster and watch what happens. The last step is the hard one: given everything the system promised, was this sequence of events even legal?

Orkes paid Kyle to do this. He keeps editorial control and publishes what he finds, whether or not it flatters whoever is paying. That is Jepsen’s ethics policy, not a promise from us.

The findings

unexpectedly frequent concurrent execution of individual tasks, one issue that could significantly delay workflow execution, four minor issues with startWorkflow errors or null IDs, and three serious problems which caused Conductor to permanently abandon or forget workflow executions, or to mark them completed even though their tasks were never run.

I would prefer not to read that about a system we built, but that is exactly what we were after. We went through every finding, and it holds.

The first item on that list, #1, is one I pushed back on at first. The same task can run on two workers at the same time, and that part is by design: Orkes Conductor delivers tasks at-least-once, and workers are expected to be idempotent. However, during the tests it appeared more often than our design predicted. There was a TOCTOU (Time Of Check Time Of Use) issue in the sweeper that re-queues stuck tasks, and it was causing extra deliveries. That is fixed in v5.5.6. The at-least-once part is not a bug and is not going away. Workers must be idempotent.

Every issue is public. They are worth reading. Most of them have the actual root cause written up.

The dual write (#5 and #7)

Starting a workflow used to be two separate writes. First we recorded the execution. Then we queued it for the decider to pick up.

Two writes, and no transaction between them. If a server crashed in the gap, or Postgres went away, or the network split, you ended up with a workflow sitting in the database in RUNNING that nothing would ever schedule. The execution was not lost, it was just never going to run, and nothing was looking for it, because recovery worked off the queue rather than off the store.

That is a dual write, and dual writes are a well-known way to lose things. We knew the pattern perfectly well. However, we had not noticed that we had built one. We should have caught this one ourselves.

There was a worse version. When the queueing step failed outright, the error handling deleted the execution it had already committed. A workflow that clients could see for a while simply stopped existing.

The fix is the transactional outbox. The two writes became one atomic write, so the execution and its queue entry commit together. A background process sweeps for anything that committed but never reached the queue. An interrupted start is now recovered instead of abandoned, and nothing gets deleted because a later step failed.

That shipped in v5.5.0. Kyle re-ran his tests against it during the engagement and stopped seeing the failure.

Where things stand

Seven of the nine issues are closed, across v5.5.0 and v5.5.6. Most of them need an infrastructure failure to appear: a killed Postgres, a killed Redis, a network partition. That is what Jepsen injects on purpose, and it is why some of these are rare in a healthy cluster. Two are still open: the null workflow ID and the misleading “Failed to connect” message.

The first one, #3 is a startWorkflow call that returned a null workflow ID during a Redis failure. We shipped a client-side fix so the SDK throws instead of handing you a null. Returning a null turns an indeterminate outcome into a false success. We have not reproduced the root cause, though, and we are not going to claim we have.

The other, #6, isn’t a bug in the usual sense. It’s about what an error means. When a server dies mid-request, the client reports “Failed to connect”, which sounds like the request never arrived. Quite often it did arrive, and the workflow was created: OkHttp had already delivered the body once, lost the connection, re-sent it on a fresh one, and then reported the failure of that second attempt. Kyle informally polled eleven engineers on that message and nine of them read it as a definite failure.

The re-sending part is fixed in this PR. The message itself has not changed, which is why #6 is still open.

Our verdict

Orkes Conductor kept executing while its infrastructure was torn apart. In our opinion that is reasonable behavior. The problems Kyle found were real and specific, and we fixed most of them.

We will keep designing and testing Orkes Conductor under failure to make it more reliable. This work is never finished.

Read the full report