Practical ERP guidance
ERP Integration Incident Recovery: A CIO's Runbook for Reconciling Unknown Outcomes
An ERP integration incident rarely ends as a clean failure. A request times out after the ERP has already committed the sales order. A queue redelivers a message after a worker crashes. A multi-step posting completes two steps and stops. The monitoring dashboard shows a mix of successes and errors, and the warehouse reports that one order shipped twice.
The hard part of ERP integration incident recovery is not restarting the pipeline. It is deciding, record by record, what actually happened in the system of record, and doing so before anyone retries, re-keys or deletes anything. Teams that skip this step often turn one incident into three: the original fault, the duplicates created by the cleanup, and the audit questions that follow.
This article is written for the CIO or IT head who owns integration architecture and will be asked, usually at short notice, whether it is safe to replay. It covers:
- Why integration incidents produce an unknown state, and why unknown is neither success nor verified failure.
- Why the common responses (blind replay, heuristic de-duplication, deleting rows) fail.
- A proposed five-step runbook: Contain, Classify, Reconcile, Repair, Prove.
- Ten mandatory go/no-go controls that must all hold before automated repair is allowed.
- Failure patterns, deployment implications, and how to evaluate any ERP platform, including ArionERP, against these controls.
The runbook is a proposed design drawn from widely documented engineering practice. It is not an industry standard, a certification, or a method we claim to have validated at any named customer. Code and scenarios are illustrative. Validate every step against your own ERP's documented behavior.
Key Takeaways
- Unknown is a real state. If completion could not be confirmed, the outcome is neither success nor verified failure. Treat it as unresolved until the authoritative system says otherwise.
- Never blindly retry an ambiguous timeout on a consequential mutation. Query the ERP's authoritative operation state first, then decide whether to replay, mark complete, or reverse.
- Bind each operation to a durable, tenant-scoped identity and an immutable approved payload hash. Reject reuse of an identity with a changed payload, and claim work atomically so concurrent workers cannot both act.
- Acknowledge delivery only after it is safe. Deduplication, the inbox or outbox record, and work scheduling must commit atomically, or be explicitly recoverable, before you tell the sender you have it.
- Repair with compensating documents, not deletes. Reversals need named business approvers and an audit trail.
- Privacy is part of recovery. Payloads, dead-letter queues and even identifiers can be sensitive data. Assign an owner, a retention rule and an access rule before incident responders start copying data around.
- Ten go/no-go controls, all required. There is no score. A pass means the recovery path has no known blocking gap, not that the integration is certified production-ready.
Why ERP Integration Incidents Leave Records in an Unknown State
🔑 Key takeaway: Distributed systems cannot always tell you whether an operation happened. Your recovery process must be designed for that, because the network will not cooperate.
An ERP rarely works alone. Orders arrive from eCommerce and CRM, stock moves are reported by warehouse or shop-floor systems, and postings flow to finance and BI. Each hop adds a place where a request can be sent but its outcome never confirmed. The common causes are:
- Timeout after commit. The ERP applied the change, but the response was lost or too slow. The caller sees a failure that did not happen.
- At-least-once delivery. Message brokers and webhook senders commonly redeliver when they do not receive an acknowledgment. Duplicates are normal behavior, not a bug.
- Partial completion. A posting that creates an order header, reserves stock and writes a financial entry can stop between steps.
- Human re-entry. Someone sees an error and re-keys the transaction while automation is still retrying.
- Mapping or schema changes. A transformation change mid-incident means two payloads can describe one business intent.
HTTP gives a useful but narrow vocabulary here. RFC 9110 on HTTP semantics classifies methods such as PUT and DELETE as idempotent and POST as not. That is a statement about the protocol method, not a guarantee that your business operation (create this order, post this receipt) will happen exactly once. Once-only business effect has to be designed on top of the transport.
The five outcome states you should name
| State | Evidence required | What it must not be called |
|---|---|---|
| Confirmed applied | Authoritative ERP record found, matching the approved payload hash or expected content | Failed, just because the caller saw a timeout |
| Confirmed not applied | Authoritative query shows no record for the operation identity or business key, and no in-flight work | Applied, just because the request was sent |
| Applied differently | Record exists but quantities, prices, dates or status differ from the approved payload | Applied (it needs a correction decision) |
| Duplicated | More than one record traces to one business intent | A data-quality nuisance (it has financial and fulfillment effects) |
| Unknown | Completion could not be confirmed | Success or verified failure |
The last row is the one most teams lack. An unknown result means completion could not be confirmed. It is neither success nor verified failure, and it cannot be closed by assumption.
How Most Teams Respond to an Integration Incident, and Why It Fails
🔑 Key takeaway: The usual responses optimize for restoring flow quickly. They trade away certainty about what is in the ledger.
Under pressure, teams reach for the actions that feel productive. Each has a reason to exist, and each has a specific way of making recovery worse.
| Common response | Why it is attractive | What breaks |
|---|---|---|
| Replay everything since the first alert | Fast, simple, restores the feed | Records that were applied are applied again. The true incident window usually starts before the first alert. |
| Retry on every timeout | Standard client behavior | An ambiguous timeout on a consequential mutation may duplicate it. Retrying is safe only when the operation is protected by a durable identity. |
| De-duplicate by heuristic (same customer, same amount, within N minutes) | Needs no new design | Legitimate repeat orders are removed, and true duplicates with a changed field survive. |
| Delete the extra rows | The data looks clean again | Downstream documents (shipments, invoices, stock moves) still reference them. Audit trails break. |
| Re-key manually while automation runs | Gets the customer an answer today | Two actors, one operation. Neither knows about the other. |
| Treat any 2xx as done | Matches the success path | A gateway or proxy can return success for work the ERP has not completed. |
Vendors who publish careful guidance on this tend to be explicit about it. Stripe's documentation, for example, says to treat the outcome of a request that returned a 500 as indeterminate, to avoid retrying with a new idempotency key until the original outcome is resolved, and to reconcile by retrieving the object and using webhook events. That is one payments vendor's API contract (see Stripe's advanced error handling documentation), and it should not be assumed to describe your ERP. The principle, however, transfers: reconcile the authoritative state before you repeat a consequential action.
For a deeper treatment of prevention, including safe retries, duplicates and timeouts at design time, see our companion guide on idempotent ERP integration. This article picks up where prevention has already failed.
A Proposed Design: The Contain, Classify, Reconcile, Repair, Prove Runbook
🔑 Key takeaway: Five ordered steps keep responders from repairing before they know what is broken. The order matters more than the speed of any single step.
The runbook below is a proposed design for CIOs to adapt, not an established proprietary method. It is consistent with the incident handling lifecycle emphasis in NIST SP 800-61 Revision 3, which frames incident response as part of broader risk management. NIST's guidance is about security incidents. We apply the same discipline of preparation, analysis, containment and recovery to integration data incidents, which is our adaptation rather than NIST's statement.
Step 1: Contain
- Pause automatic retry and replay for the affected operation classes (for example, order creation), not the whole ERP.
- Freeze dead-letter queue purging and log rotation for the affected window so evidence is not lost.
- Name an incident lead and the business owners who will approve repairs (finance for postings, operations for fulfillment and stock).
- Treat the incident window as a hypothesis. Start with the first alert, then look backward for the first anomalous record.
Step 2: Classify
Build an inventory of every operation in the window, keyed by tenant and durable operation identity. Assign each to one of the five states from the first section. At this stage nearly everything is unknown. That is expected and honest.
Step 3: Reconcile
- Query the ERP's authoritative state using the operation identity, or a business key you stored on the target document at creation. Do not infer from logs alone, and do not rely on timestamps.
- Compare what exists against the approved payload hash or the expected content.
- Check for in-flight work before declaring anything not applied. A queued job may still complete.
- If the ERP cannot be queried by operation identity or business key, automated repair is a no-go. Reconcile manually from documents and obtain business sign-off.
Step 4: Repair
| Reconciled state | Allowed repair | Not allowed |
|---|---|---|
| Confirmed applied | Mark complete. Do not replay. | Replay for safety |
| Confirmed not applied | Replay through the normal claim path with the same operation identity and payload hash | Replay under a new identity, which would hide a late-completing original |
| Applied differently | Business-approved correction, recorded as a new operation linked to the original | Reusing the original identity with the changed payload |
| Duplicated | Reverse with the ERP's compensating documents (cancel, credit note, stock adjustment with reason code) after business approval | Deleting rows or editing balances directly |
| Unknown | Hold in quarantine with an owner and a decision date | Closing as success or as failed |
Step 5: Prove
- Tie control totals (count and value by tenant and document type) between source system, ERP and downstream systems such as WMS and billing.
- Obtain named sign-off from each business owner.
- Leave any unresolved item open as unknown, with an owner, rather than closing it to make the dashboard green.
- Feed the lessons back into the go/no-go controls in the next section.
Illustrative code: durable identity, payload binding and atomic claim
The following SQL is illustrative only. It is not an ArionERP schema or API contract. It shows how the invariants fit together in an integration layer you control.
-- Illustrative only. Not an ArionERP schema or API. CREATE TABLE integration_operation ( tenant_id uuid NOT NULL, operation_id text NOT NULL, -- durable identity of one business intent payload_hash bytea NOT NULL, -- hash of the immutable approved payload state text NOT NULL DEFAULT 'pending', claimed_by text, lease_expires timestamptz, target_ref text, -- ERP document id, once confirmed PRIMARY KEY (tenant_id, operation_id) ); -- 1. Record intent. Same identity, different hash = conflict, not a retry. INSERT INTO integration_operation (tenant_id, operation_id, payload_hash) VALUES (:tenant, :op, :hash) ON CONFLICT (tenant_id, operation_id) DO NOTHING; -- Then read the row: if payload_hash differs from :hash, reject the request. -- 2. Atomic claim: exactly one worker can move pending to in_flight. UPDATE integration_operation SET state = 'in_flight', claimed_by = :worker, lease_expires = now() + interval '2 minutes' WHERE tenant_id = :tenant AND operation_id = :op AND payload_hash = :hash AND state = 'pending' RETURNING operation_id; -- zero rows = someone else owns it -- 3. An expired lease means UNKNOWN, never pending. The earlier worker may -- have reached the ERP. Reconcile before anything is re-sent. UPDATE integration_operation SET state = 'unknown' WHERE state = 'in_flight' AND lease_expires < now();Two details matter. The identity is derived from the business intent (for example, order version plus shipment line), not from the order number alone, so a legitimate second shipment against the same order gets its own identity. And an expired lease routes to unknown, not back to pending, because the previous worker's request may have succeeded.
Illustrative code: conditional mutation at the target
Where a correction changes an existing ERP record, the check and the write must happen together at the target. A read followed by a comparison in your application, followed by a write, is not compare-and-swap. Another actor can change the record between the read and the write.
-- Illustrative only. The version predicate makes check and write one atomic step. UPDATE sales_order SET quantity = :new_qty, version = version + 1 WHERE tenant_id = :tenant AND order_id = :order AND version = :expected_version; -- Rows affected = 0 means a conflict: re-read, re-decide, never overwrite blindly.If your ERP's API does not offer an equivalent conditional update (an ETag or version precondition, for example), disclose that limitation in the runbook and serialize writes to that record through a single owner. Do not present a read-compare-then-write sequence as safe.
Mandatory Go/No-Go Controls Before Any Automated Repair
🔑 Key takeaway: These ten controls are all required. Do not add them up into a readiness score. One missing control is a reason to repair manually, not an acceptable trade for the other nine.
A summed score invites teams to average a critical gap away. A mandatory control set does the opposite. It tells you precisely which repair actions are blocked until a specific gap is closed. Passing every control means the recovery path has no known blocking gap. It does not certify the integration as production-ready, and it is not a compliance attestation.
| # | Control | Go condition | No-go signal |
|---|---|---|---|
| 1 | Authoritative state lookup | You can query the ERP for what happened to an operation by identity or business key | Only logs or the sender's retry history tell you what happened |
| 2 | Tenant-scoped operation identity | The identity is durable and unique within the tenant, never shared across tenants | Keys are global, or derived from values that collide across tenants |
| 3 | Immutable approved payload hash | Each identity is bound to a hash. Reuse with a changed payload is rejected. | The same key can carry a different payload, with the latest one silently winning |
| 4 | Atomic claim | One worker wins a claim through a conditional database update or equivalent | Workers check status then act, with a gap in between |
| 5 | Verified acknowledgment | Deduplication, inbox or outbox record and work scheduling commit atomically, or are explicitly recoverable, before acknowledging the sender | The sender is acknowledged first and the record written afterward |
| 6 | Terminal failure state | Exhausted or rejected work reaches a named state with an owner and a deadline | Retries continue indefinitely, or failed items sit in a queue nobody owns |
| 7 | Uncertain-input handling | Malformed, ambiguous or unmappable input is quarantined for a human decision | The mapper guesses, defaults a field, or drops the record |
| 8 | Target-side conditional mutation | Updates carry a version or precondition enforced at the ERP, or serialization limits are disclosed | Read-compare-then-write in application code is described as safe |
| 9 | Compensating repair path | Reversals use documented ERP documents with named approvers | Repairs mean database edits or deletions |
| 10 | Privacy ownership | Payloads, logs, DLQs and exports each have an owner, retention rule and access rule. Identifiers are treated as potentially sensitive. | Responders copy payloads into tickets, chat or spreadsheets |
Scoping the standards you lean on
Idempotency guidance is easy to over-apply. Keep each reference inside its own scope:
- The Idempotency-Key HTTP header. The IETF HTTPAPI working group's Idempotency-Key header draft describes a client-generated key for making POST or PATCH requests fault-tolerant. It says a key must not be reused with a different payload, suggests a 409 response for a concurrent retry and a 422 response for payload mismatch, and says resources must publish their own idempotency specification. It is an Internet-Draft, not a published standard, and it covers HTTP requests only. It does not address message queues, file feeds or batch interfaces. Check its current status before citing it in a design document.
- Vendor-specific semantics. Stripe's documented behavior (key retention windows, saved responses, differences between API versions) belongs to Stripe. Your ERP's retention and replay rules may differ, and you must read them.
- Retry design in general. The AWS Builders' Library article Making retries safe with idempotent APIs is a well-regarded engineering explanation. It describes AWS's approach and is not a substitute for your ERP vendor's contract.
- Recording and scheduling together. The transactional outbox pattern describes writing state changes and outbound messages in a single local transaction. It is one way to satisfy control 5, not the only one.
Why identifiers belong in the privacy conversation
Control 10 surprises teams. An idempotency key, order number or customer reference can be personal or commercially sensitive data, especially when it embeds an email address, tax identifier or account number. Stripe's own guidance, for instance, advises against using email addresses or personal identifiers as idempotency keys. Our companion article on ERP integration data privacy walks through payload, log and dead-letter queue ownership in more detail. Your legal and privacy obligations depend on your jurisdiction and contracts, and your counsel should interpret them. This article does not.
Why This Fails in the Real World: Common Failure Patterns
🔑 Key takeaway: Capable teams fail here because of system gaps and split ownership, not carelessness. Each scenario below is illustrative and describes a pattern, not a specific customer's result.
Pattern 1: The replay that doubled the shipments
Illustrative scenario. An eCommerce platform posts orders to the ERP, which releases pick tasks to a warehouse system. During a database failover, the ERP commits a batch of orders but the responses never reach the platform. The platform's retry logic sees timeouts and resubmits the batch under new request identifiers. The ERP, with no durable identity linking the two submissions, creates a second set of orders. The warehouse picks both.
Why intelligent teams still fail: Retrying on timeout is textbook client behavior, and each component was working as designed. No component owned the question, "Did this business intent already happen?" The monitoring showed error rates falling after the retries, which looked like recovery.
Pattern 2: The corrected payload under the old key
Illustrative scenario. A pricing error is found, so the integration is fixed and the failed orders are replayed. The replay reuses the original operation keys, but the corrected payload has different line prices. Depending on the platform, the system either rejects the replay as a mismatch (good, but now nobody knows which orders were corrected) or silently returns the stored result of the first request (bad, because the corrected prices never arrive). Finance later finds invoices at the old prices and cannot tell which path each order took.
Why intelligent teams still fail: The team assumed a key identifies an order, when it should identify one approved payload. A correction is a new operation linked to the old one. Without payload binding, a success response proves nothing about content.
Pattern 3: The cleanup that created a privacy exposure
Illustrative scenario. Responders export the dead-letter queue to a spreadsheet to sort failures by hand, and paste samples into a chat channel. The payloads include customer names, delivery addresses and account references. No one owns the retention of those copies, and the access list on the channel is broader than the access list on the integration.
Why intelligent teams still fail: Incident urgency overrides data-handling habits. The runbook never said who owns incident evidence, so everyone assumed someone else did.
Pattern 4: Two workers, one operation
Illustrative scenario. A worker crashes mid-request, and a supervisor restarts the job after a timer expires. The first worker's request had in fact reached the ERP. Both operations complete. The check for existing work and the start of new work were separate steps, so there was a gap for the second worker to exploit.
Why intelligent teams still fail: Timer-based reassignment feels like resilience. It is safe only when the claim is atomic and an expired lease routes to unknown and reconciliation, not straight to a retry.
What the patterns share
- Nobody owned the question of authoritative state.
- A success signal (a 2xx, a falling error rate, a drained queue) was taken for proof of correct content.
- Ownership was split across integration, ERP and business teams with no single runbook.
- The shortcut was tolerable on an ordinary day and expensive on the worst one.
For the floor-level view of the same problem, see our plant-manager guide to shop floor transaction recovery. For the operations-side controls on order and inventory feeds, see our ERP integration exception handling guide.
Practical Implications for the CIO: Deployment, Evidence and Platform Questions
🔑 Key takeaway: Your deployment model decides what evidence you can reach during an incident, and how fast. Decide that before you need it.
How SaaS and on-premises deployments change recovery
ArionERP is offered as both Cloud (SaaS) and On-Premises, and the two models carry different recovery responsibilities. The same shape applies to most ERP platforms.
| Recovery need | Cloud (SaaS) deployment | On-premises deployment |
|---|---|---|
| Querying authoritative state | Through documented APIs, exports and vendor-provided audit views | Through APIs and, where policy allows, controlled direct database reads |
| Log and payload retention | Shared: vendor retention for platform logs, your retention for your integration layer | Yours end to end, including storage, backup and access control |
| Restore or point-in-time options | Defined by the service agreement and vendor procedure. Confirm the process and timeline in writing. | Defined by your own backup design. Test restores, not just backups. |
| Vendor support during an incident | Central to the response. Confirm channels and severity definitions in advance. | Depends on your support plan and your team's depth |
| Privacy and access ownership | Split between vendor and customer. Confirm the split contractually. | Primarily customer-owned |
Neither model is inherently safer for recovery. The risk lies in assuming the evidence will be there without checking. For a broader view of this trade-off, see the cloud vs on-prem ERP overview and the pages for ArionERP Cloud and ArionERP On-Premises.
Who decides what
| Decision | Accountable owner | CIO's role |
|---|---|---|
| Pausing and resuming integration flows | CIO or delegate | Decides |
| Whether a duplicate invoice or posting is reversed | Finance leader | Provides evidence and tooling |
| Whether a duplicate shipment is recalled or kept | Operations leader | Provides evidence and tooling |
| Retention of incident payloads and logs | Named data owner, with privacy or legal input | Enforces technically |
| Closing an unknown item | Business owner of the record | Confirms evidence is complete |
Comparing platform patterns on recovery questions
When you evaluate any ERP, include recovery questions in the demo, not just happy-path integration. The table below generalizes typical patterns and is not a claim about specific products or editions. Capabilities vary by vendor, edition, configuration and partner tooling, so verify against each vendor's primary documentation.
| Recovery question | Large suite pattern (varies) | Lightweight ERP pattern (varies) | What to verify with ArionERP |
|---|---|---|---|
| Can I look up an operation's outcome by identity or business key? | Often possible, sometimes through add-on tooling or partner work | Often limited to UI search or basic reports | Ask for a live lookup demonstration, and check the API documentation |
| How are concurrent or repeated requests handled? | Depends on module and interface type | Frequently left to the integrator | Ask what the documented contract is per interface, then test it with a repeated and a concurrent request |
| Are reversals handled by documented compensating documents? | Generally yes for core financial and logistics documents | Varies widely | Walk through a reversal of a posted order and its downstream documents |
| Who holds evidence in an incident? | Depends on hosting choice and contract | Usually the vendor, with limited export options | Compare Cloud and On-Premises evidence access in your scenario |
| What does recovery tooling cost in effort? | Typically partner-heavy | Typically in-house scripts | Ask for the integration scope in the integrations overview and request a written answer on each control above |
To keep the claims honest, we are not asserting here that a specific ArionERP interface offers idempotency keys, payload fingerprinting or conditional updates. That is what the proof test is for. For security and compliance scope, review the security and compliance page and ask for the specific attestation, its scope and its date, because a certification held by an organization does not automatically describe every product, hosting arrangement or your own obligations.
Risks, Trade-Offs and a Lower-Risk Approach
🔑 Key takeaway: Safer recovery costs design effort and some latency. The cheaper alternative is paying in reconciliation hours later, at a worse time.
Trade-offs you should accept knowingly
| Control | What it costs | Why it is usually worth it |
|---|---|---|
| Durable operation ledger | An extra write per operation, plus storage and retention rules | It is the only reliable way to answer "what did we send, and what was approved?" |
| Atomic claim and acknowledgment | Slightly higher latency and more careful transaction design | It removes the gap in which duplicates and lost messages occur |
| Quarantine for uncertain input | Some orders wait for a human | The alternative is guessing in the ledger |
| Reversal through compensating documents | More steps and approvals than a delete | It preserves the audit trail and downstream consistency |
| Retention limits on payloads and DLQs | Less history for forensic work | Smaller privacy footprint. Choose retention deliberately and document it. |
Limits you should disclose, not hide
- Not every interface can be made exactly-once. File drops, legacy batch jobs and some partner APIs offer no identity or lookup. For those, document the serialization limits and accept manual reconciliation as the control.
- Idempotency-key retention windows are finite. If a key expires, a late retry can execute again. Tie your retry limits to the documented window and reconcile once it closes.
- Reconciliation requires someone to decide. Tooling shortens the investigation, but a business owner still signs off on reversals.
A lower-risk approach in practice
- Choose the interfaces that can hurt you most. Start with order creation, goods receipt, invoice posting and stock adjustments, not every feed.
- Run the ten controls as a design review, with a named owner for each no-go.
- Rehearse. Inject a timeout after commit, a duplicate delivery, a worker crash and a changed-payload replay in a test environment, and run the five-step runbook against the result.
- Write the unknown policy down. Who may close an unknown, on what evidence, by when?
- Re-test after every material change to mappings, ERP upgrades, or hosting.
2027 update: what to re-verify, and what stays constant
As of late 2026, the IETF Idempotency-Key header work was still an active working-group draft and not a published standard. Cloud and ERP vendors also continue to revise their API retry and replay behavior, including between API versions. Re-check any standard's status and each vendor's retention windows when you review this runbook. The underlying logic does not depend on any of that: an unconfirmed outcome stays unresolved until authoritative state says otherwise, a key binds to one approved payload, and repairs go through compensating documents.
Can your ERP prove what happened after a timeout?
Ask for the proof test, not the feature list. Lookup by identity, repeated request, concurrent request, reversal.
Walk through these ten controls against ArionERP Cloud or On-Premises with our team.
Contact UsConclusion: What to Do Before the Next Incident
Integration incidents are certain. Duplicate or missing postings are not. The difference is whether your team can establish authoritative state quickly and repair through controlled steps. Five actions to take now:
- Add unknown to your incident vocabulary. Define the five outcome states and forbid closing an unknown as success or failure without evidence.
- Run the ten go/no-go controls against your three riskiest interfaces. Treat any no-go as a block on automated repair for that interface until it is closed or the limitation is disclosed and accepted by the business owner.
- Prove authoritative lookup. For each interface, show that you can find what the ERP did by operation identity or business key. If you cannot, document the manual reconciliation path.
- Assign owners for repairs and evidence. Name the approvers for reversals and the owner of payload, log and dead-letter queue retention, including how identifiers are handled.
- Rehearse once, then after every major change. Inject a post-commit timeout, a duplicate delivery and a worker crash, and run the five-step runbook end to end.
About ArionERP and this review
ArionERP is a modular, AI-enhanced ERP platform offered as Cloud (SaaS) and On-Premises deployments, built for SMBs and mid-market organizations with a focus on manufacturing. It is developed by Cyber Infrastructure (CIS), a software company operating since 2003. For the current scope, dates and holders of any certifications or attestations, consult the security and compliance page and request the underlying documents, since scope varies by entity, product and hosting arrangement.
Reviewed by ArionERP Expert Team. This article offers general architectural guidance and illustrative scenarios. It does not constitute legal advice or a compliance determination, and it does not describe results at any named customer.
Frequently Asked Questions
What is ERP integration incident recovery?
It is the process of working out what actually happened to business operations (orders, receipts, postings, stock moves) after an integration fault, and then repairing the differences safely. It centers on reconciling against the ERP's authoritative state before replaying, reversing or correcting anything.
Is it ever safe to retry after an ambiguous timeout?
Only when the operation is protected by a durable, tenant-scoped identity bound to the approved payload, and when the target's documented behavior makes a repeated request return the original outcome. Even then, retries should stay within the documented retention window. For consequential mutations without that protection, query the authoritative operation state first, then decide whether to replay, mark complete or reverse.
What does an unknown result mean?
It means completion could not be confirmed. It is not success, and it is not a verified failure. Keep the item quarantined, with an owner and a decision date, until the authoritative system supplies evidence.
How is changed-payload key reuse handled?
It should be rejected. A key or operation identity binds to one immutable approved payload hash. If the business needs a corrected payload, record it as a new operation with a new identity, linked to the original after the original's outcome is resolved. The IETF Idempotency-Key draft takes a similar position for HTTP requests, but remember it is a draft and covers HTTP only.
How do I tell a duplicate from a legitimate repeat order?
Derive operation identity from the business intent (for example, order version and shipment line), not from time proximity or customer and amount similarity. Heuristic matching removes legitimate repeats and misses true duplicates that differ in one field.
Should we delete duplicate records?
Generally no. Downstream documents may reference them, and deletion breaks audit trails. Use the ERP's compensating documents (cancellations, credit notes, stock adjustments with reason codes) with named business approval.
Can a read followed by a conditional write in my application serve as compare-and-swap?
No. Another actor can change the record between your read and your write. Use a conditional mutation enforced at the target (a version predicate or precondition), or disclose the serialization limits and route writes for that record through a single owner.
Why do identifiers matter for privacy during recovery?
Order numbers, account references and keys can themselves identify people or reveal commercial information, particularly if they embed email addresses or tax numbers. Define who owns payload, log and dead-letter queue data, how long it is kept, and who may access it. Your legal obligations depend on your jurisdiction and contracts, so involve your counsel.
Does passing all ten controls mean the integration is production-ready?
No. The controls are mandatory checks that expose known blocking gaps in the recovery path. They are not a score, a certification or a compliance attestation. Production readiness also depends on testing, monitoring, capacity, security review and business sign-off.
How does SaaS versus on-premises deployment affect recovery?
It changes who holds the evidence and who can act. In SaaS, you rely on documented APIs, exports and vendor procedures, so confirm them in writing. On-premises, you control logs, backups and database access, and you carry the responsibility to retain and test them. Neither is automatically safer for recovery.