The partially committed order is the part that sticks with me. Payment processed, inventory reserved, shipping failed, and a plain retry cannot tell you which of those already happened, so marking the timeout UNKNOWN and reconciling against the provider first is the move.
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
Exactly. A timeout tells us that we didn’t get a reliable response, not that the operation failed. That’s why retrying the entire workflow is dangerous. I’d rather preserve the last known state of each step, mark the uncertain operation as UNKNOWN, and reconcile it against the relevant provider before deciding whether to retry, continue, or compensate. The important distinction is between recovering the workflow and accidentally repeating the side effects that already succeeded.
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
Absolutely. UNKNOWN should never mean "we have no idea what happened." It should mean "we cannot yet confirm the outcome, but we have preserved enough execution history to investigate it."
I'd keep the last known state for each step, along with the operation ID, idempotency key, provider reference, last confirmed transition, timestamp, and any response or error received. That gives the recovery worker a concrete point from which to reconcile rather than starting from assumptions.
For example, if a payment was submitted but the response timed out, the ledger should retain SUBMITTED as the last confirmed state and mark the outcome UNKNOWN. Recovery can then check the provider against that exact operation before deciding whether to retry, compensate, or escalate. The important distinction is that UNKNOWN describes uncertainty about the outcome, not the absence of execution history.
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
Exactly. The recovery worker needs a reliable checkpoint, not just an UNKNOWN label. Keeping SUBMITTED as the last confirmed state, along with the operation ID, idempotency key, provider reference, and timestamp, gives reconciliation a concrete starting point. UNKNOWN should describe uncertainty about the outcome, not a lack of execution history. That’s what makes recovery evidence-driven rather than guesswork.
who owns the partial state is the real question here. if each agent only knows its own step then a retry from the second agent is basically a brand new order, does the orchestrator keep a ledger of which side effects already happened so the compensating step knows what to undo
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
Exactly. The orchestrator needs to own the workflow state, not leave individual agents responsible for remembering what happened across the entire process. I'd maintain a durable execution ledger that records each operation, its status, idempotency key, external reference, and any associated compensation. When another agent takes over, it should resume from that ledger rather than treat the task as a new order. The compensating step should also verify the actual external state before reversing anything, because a timeout doesn't necessarily mean the original action failed. Agents can own individual tasks, but the orchestrator must own the lifecycle and consistency of the overall workflow.
checking the real external state before compensating is the bit people forget, a timeout and a failure look the same from inside the agent. does the ledger write happen before the external call or after, because if its after a crash in between leaves a side effect nobody recorded
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
That's exactly the gap I'd want the execution ledger to handle. I'd write the operation intent to the ledger before making the external call, with a stable operation ID and idempotency key, but keep its status as PENDING or SUBMITTED, not completed. The tricky part is that a crash can happen on either side of the external call. We might record the intent and crash before sending it, or the provider might execute the operation, and our process might crash before recording the response.
So the ledger alone cannot tell us what actually happened. On recovery, I'd reconcile against the external provider using the same idempotency key or provider reference before retrying or compensating. If the outcome is still unclear, keep it as UNKNOWN rather than risk repeating a side effect. The ledger tells us what we intended and what we observed; the external system remains the source of truth for whether the operation actually happened.
One rollback case we discussed was an older version deleting a parent after a newer version had written a child that references it. We counted before deciding on a CI rule: 238 foreign-key clauses across 52 of 189 migrations, and 11 of the last 32, so on recent migrations a flag would have fired on about one in three. It became a written warning in the rollback section and the production steps instead. For a compensating transaction, we'd also want to specify which effects each step can reverse and what happens when compensation fails.
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
That’s a good distinction. We found that a blanket CI rule would have created too much noise, especially given how often foreign-key clauses appeared in recent migrations, so documenting the risk directly in the rollback guidance and production steps was more practical.
I think the same principle applies to compensating transactions. Each step should explicitly define what it can reverse, what it cannot, and what conditions must hold before compensation is attempted. And if compensation itself fails, that needs to remain visible as a separate recovery state with a clear escalation path. A compensation failure shouldn’t be treated as if the workflow has successfully rolled back.
A timeout is not a failure, it is an unknown, and the discussion under your post gets to the core of it: something has to own the state of the whole workflow, not each agent its own step.
We hit the same shape without any agents at Ascendra Ventures, in a candidate enrichment pipeline. Resumes, LinkedIn data, OpenAI batch jobs, embeddings and PostgreSQL writes were all involved, and any slow step left a person in an unclear state. The fix was to move the whole thing into Temporal. One workflow claims the person, processes urgent profiles immediately and batches the rest, waits durably for the parse and embedding jobs, falls back to polling when webhooks go quiet, and finalises every status so failed work is visible instead of silently stuck. The rule underneath was simple: every candidate either finishes or visibly fails.
Do you think the compensating steps belong inside the same durable workflow, or in a separate recovery worker that reads the ledger?
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
I’d keep the compensation logic within the same durable workflow, but separate it into explicit recovery activities rather than mixing it into the normal execution path.
The workflow already owns the sequence, dependencies, and last confirmed state, so it has the context needed to decide whether compensation is safe. A separate worker can still execute the compensation activities, but it should operate under the workflow’s recorded state and ownership, not independently guess what needs reversing.
For longer-running or repeatedly failing compensations, I’d make them resumable and independently observable, with their own status and retry history. That way, the original workflow remains the source of truth, while recovery can continue without losing track of what has already happened. The important thing is that there should be one owner of the overall workflow state, even if execution is distributed across multiple workers.
Who decides a workflow lands in MANUAL_REVIEW_REQUIRED though? If the system has to judge its own outcome as ambiguous first, haven't you just moved the trust problem rather than fixed it?
Hi, I'm Atul Kumar, an AI Engineer passionate about Generative AI, AI Agents, and building practical AI solutions. I enjoy sharing what I learn and exploring the future of intelligent technology.
That’s a fair challenge. I wouldn’t let the agent decide this based on its own confidence. The transition to MANUAL_REVIEW_REQUIRED should be triggered by predefined recovery rules: for example, the external provider cannot confirm whether the operation succeeded, the idempotency lookup returns an inconclusive result, or the operation remains in UNKNOWN beyond a defined recovery window.
The agent can provide context, but it shouldn’t have authority to declare its own outcome safe. The orchestrator evaluates the durable execution record against those rules, and if the external state still can’t be established, it escalates to a human. So I agree that we’ve moved the trust boundary, but the goal is to move it away from probabilistic agent judgement and into deterministic checks, independent state verification, and explicit human ownership when certainty isn’t possible.
We're a place where coders share, stay up-to-date and grow their careers.
The partially committed order is the part that sticks with me. Payment processed, inventory reserved, shipping failed, and a plain retry cannot tell you which of those already happened, so marking the timeout UNKNOWN and reconciling against the provider first is the move.
Exactly. A timeout tells us that we didn’t get a reliable response, not that the operation failed. That’s why retrying the entire workflow is dangerous. I’d rather preserve the last known state of each step, mark the uncertain operation as UNKNOWN, and reconcile it against the relevant provider before deciding whether to retry, continue, or compensate. The important distinction is between recovering the workflow and accidentally repeating the side effects that already succeeded.
preserving the last known state per step is what makes the UNKNOWN mark usable. without that, reconciling against the provider is guesswork.
Absolutely. UNKNOWN should never mean "we have no idea what happened." It should mean "we cannot yet confirm the outcome, but we have preserved enough execution history to investigate it."
I'd keep the last known state for each step, along with the operation ID, idempotency key, provider reference, last confirmed transition, timestamp, and any response or error received. That gives the recovery worker a concrete point from which to reconcile rather than starting from assumptions.
For example, if a payment was submitted but the response timed out, the ledger should retain SUBMITTED as the last confirmed state and mark the outcome UNKNOWN. Recovery can then check the provider against that exact operation before deciding whether to retry, compensate, or escalate. The important distinction is that UNKNOWN describes uncertainty about the outcome, not the absence of execution history.
SUBMITTED as the last confirmed state. Without that the recovery worker is just guessing.
Exactly. The recovery worker needs a reliable checkpoint, not just an UNKNOWN label. Keeping SUBMITTED as the last confirmed state, along with the operation ID, idempotency key, provider reference, and timestamp, gives reconciliation a concrete starting point. UNKNOWN should describe uncertainty about the outcome, not a lack of execution history. That’s what makes recovery evidence-driven rather than guesswork.
who owns the partial state is the real question here. if each agent only knows its own step then a retry from the second agent is basically a brand new order, does the orchestrator keep a ledger of which side effects already happened so the compensating step knows what to undo
Exactly. The orchestrator needs to own the workflow state, not leave individual agents responsible for remembering what happened across the entire process. I'd maintain a durable execution ledger that records each operation, its status, idempotency key, external reference, and any associated compensation. When another agent takes over, it should resume from that ledger rather than treat the task as a new order. The compensating step should also verify the actual external state before reversing anything, because a timeout doesn't necessarily mean the original action failed. Agents can own individual tasks, but the orchestrator must own the lifecycle and consistency of the overall workflow.
checking the real external state before compensating is the bit people forget, a timeout and a failure look the same from inside the agent. does the ledger write happen before the external call or after, because if its after a crash in between leaves a side effect nobody recorded
That's exactly the gap I'd want the execution ledger to handle. I'd write the operation intent to the ledger before making the external call, with a stable operation ID and idempotency key, but keep its status as
PENDINGorSUBMITTED, not completed. The tricky part is that a crash can happen on either side of the external call. We might record the intent and crash before sending it, or the provider might execute the operation, and our process might crash before recording the response.So the ledger alone cannot tell us what actually happened. On recovery, I'd reconcile against the external provider using the same idempotency key or provider reference before retrying or compensating. If the outcome is still unclear, keep it as
UNKNOWNrather than risk repeating a side effect. The ledger tells us what we intended and what we observed; the external system remains the source of truth for whether the operation actually happened.One rollback case we discussed was an older version deleting a parent after a newer version had written a child that references it. We counted before deciding on a CI rule: 238 foreign-key clauses across 52 of 189 migrations, and 11 of the last 32, so on recent migrations a flag would have fired on about one in three. It became a written warning in the rollback section and the production steps instead. For a compensating transaction, we'd also want to specify which effects each step can reverse and what happens when compensation fails.
That’s a good distinction. We found that a blanket CI rule would have created too much noise, especially given how often foreign-key clauses appeared in recent migrations, so documenting the risk directly in the rollback guidance and production steps was more practical.
I think the same principle applies to compensating transactions. Each step should explicitly define what it can reverse, what it cannot, and what conditions must hold before compensation is attempted. And if compensation itself fails, that needs to remain visible as a separate recovery state with a clear escalation path. A compensation failure shouldn’t be treated as if the workflow has successfully rolled back.
A timeout is not a failure, it is an unknown, and the discussion under your post gets to the core of it: something has to own the state of the whole workflow, not each agent its own step.
We hit the same shape without any agents at Ascendra Ventures, in a candidate enrichment pipeline. Resumes, LinkedIn data, OpenAI batch jobs, embeddings and PostgreSQL writes were all involved, and any slow step left a person in an unclear state. The fix was to move the whole thing into Temporal. One workflow claims the person, processes urgent profiles immediately and batches the rest, waits durably for the parse and embedding jobs, falls back to polling when webhooks go quiet, and finalises every status so failed work is visible instead of silently stuck. The rule underneath was simple: every candidate either finishes or visibly fails.
Do you think the compensating steps belong inside the same durable workflow, or in a separate recovery worker that reads the ledger?
I’d keep the compensation logic within the same durable workflow, but separate it into explicit recovery activities rather than mixing it into the normal execution path.
The workflow already owns the sequence, dependencies, and last confirmed state, so it has the context needed to decide whether compensation is safe. A separate worker can still execute the compensation activities, but it should operate under the workflow’s recorded state and ownership, not independently guess what needs reversing.
For longer-running or repeatedly failing compensations, I’d make them resumable and independently observable, with their own status and retry history. That way, the original workflow remains the source of truth, while recovery can continue without losing track of what has already happened. The important thing is that there should be one owner of the overall workflow state, even if execution is distributed across multiple workers.
Who decides a workflow lands in MANUAL_REVIEW_REQUIRED though? If the system has to judge its own outcome as ambiguous first, haven't you just moved the trust problem rather than fixed it?
That’s a fair challenge. I wouldn’t let the agent decide this based on its own confidence. The transition to MANUAL_REVIEW_REQUIRED should be triggered by predefined recovery rules: for example, the external provider cannot confirm whether the operation succeeded, the idempotency lookup returns an inconclusive result, or the operation remains in UNKNOWN beyond a defined recovery window.
The agent can provide context, but it shouldn’t have authority to declare its own outcome safe. The orchestrator evaluates the durable execution record against those rules, and if the external state still can’t be established, it escalates to a human. So I agree that we’ve moved the trust boundary, but the goal is to move it away from probabilistic agent judgement and into deterministic checks, independent state verification, and explicit human ownership when certainty isn’t possible.