ERP Integration Testing: A CIO's Go/No-Go Failure-Injection Plan Before Go-Live

Most ERP integrations pass their demos. They fail later, on an ordinary Tuesday, when a webhook arrives twice, a warehouse API times out after it has already committed a transfer, or a retry job runs while a second worker is halfway through the same order. ERP integration testing that only proves the happy path tells you almost nothing about those moments.

This article is written for the CIO or IT head who owns the go-live decision. It proposes a failure-injection test plan, built around a set of mandatory go/no-go gates, that you can apply to any ERP integration with CRM, eCommerce, BI, or warehouse systems. The plan is a proposed design. It is not a certification, and passing it does not prove production readiness on its own. It does give you evidence-based reasons to say yes or no.

The article is for the execution stage: you have chosen a platform, the integrations are built, and the question is what must be proven before real orders, inventory, and invoices flow through them. If you are still choosing an architecture, start with our guide on API-first, middleware, or point-to-point integration for modular ERP. If you already have an incident, see the runbook for reconciling unknown outcomes.

Key Takeaways

• Bottom line: An integration is ready for go-live when it has survived deliberate failures, not when it has passed a demo. Duplicates, timeouts, concurrent workers, and outages are normal operating conditions.
• Use all-required gates, not a score. The plan below has twelve go/no-go gates. A single failed gate blocks go-live. Averaging them hides the one failure that corrupts your ledger.
• Acknowledge only after durable recording. Return success to a sender only after the event, its deduplication record, and its follow-up work are committed together, or are explicitly recoverable.
• An unknown result is not a success and not a failure. After an ambiguous timeout, check the authoritative state in the target system before any retry of a consequential change.
• Name the owners. Terminal failures, reconciliation breaks, and sensitive data in logs and dead-letter queues each need a named owner before launch.
• Verify, do not assume. Vendor statements about API behavior, retries, and compliance are starting points. Confirm them in a sandbox and in writing.

Why ERP Integrations Pass Demos and Still Fail in Production

Key takeaway: Demos prove that a message can travel from A to B once. Production proves what happens when it travels twice, late, out of order, or not at all.

An ERP is a system of record. When an eCommerce platform, a CRM, or a warehouse system writes into it, the write becomes a financial or inventory fact: a sales order, a stock movement, an invoice. That is why integration defects are costlier than interface defects. A duplicated button click is an annoyance. A duplicated shipment posting is a stock variance and a revenue question.

The senders themselves tell you not to expect perfect delivery. Stripe's documentation says its webhook delivery retries for up to three days in live mode, does not guarantee event order, and that endpoints might occasionally receive the same event more than once. Shopify's guidance on verifying webhook deliveries says it minimizes duplicates but your app might still receive the same webhook more than once, and that it retries eight times over four hours before treating the delivery as failed. Shopify also advises that apps should not rely solely on webhooks and should run reconciliation jobs. Those are two mature platforms, describing normal behavior.

Most ERP integration test plans still look like the left column below.

What a typical test plan proves	What production actually does
One order arrives once and posts correctly	The same event arrives two or more times, sometimes concurrently
Events arrive in creation order	An update may arrive before its create event
The target API responds in time	The target commits, then the response is lost, so the sender sees a timeout
Bad data produces a visible error	Bad data is silently defaulted, truncated, or mapped to the wrong item
The system is up for the test window	A downstream system is down for hours, then replays a backlog
Test data belongs to one company	Multiple entities or tenants share identifiers and queues

The gap is not a lack of talent. It is that nobody was assigned to break the integration on purpose. The rest of this article closes that gap with a structure you can hand to your integration partner, your internal team, or the vendor's implementation group.

A Proposed Failure-Injection Model: Five Questions Every Integration Must Answer

Key takeaway: Every integration flow, whatever the systems, has to answer five questions under failure. If the team cannot answer one, that flow is not ready.

The model below is a proposed design for structuring tests. It is not an established standard, and it is not a method ArionERP claims to run on your behalf. It is a way to turn vague confidence into testable statements.

The five questions

1. Receipt: When we tell the sender "received," is the event durably recorded? If the receiver crashes one millisecond later, is the event still there?
2. Identity: What uniquely identifies this business operation, within which tenant or legal entity, and what approved payload does that identity bind to?
3. Effect: If the operation runs twice, or two workers try at once, does the business effect happen once?
4. Outcome: If we cannot confirm what happened, what do we record, and who or what resolves it?
5. Ownership: Who is accountable for terminal failures, reconciliation breaks, and the sensitive data that passes through payloads, logs, and dead-letter queues?

How the questions map to flows

Apply the five questions to each flow separately. A read-only BI feed has a lighter profile than a stock-transfer write. Classify each flow first.

Flow type	Example	Consequence of error	Test depth
Consequential mutation	Sales order creation, shipment posting, payment application, inventory adjustment	Financial or physical-inventory impact	All twelve gates
Reversible update	Customer address change, status sync	Rework, customer-facing error	Gates 1-5, 7, 9-11
Read-only extract	BI warehouse load, reporting feeds	Wrong dashboard, not wrong ledger	Gates 9-11, plus data-completeness checks

For the design behind these questions, our existing guide to idempotent ERP integration covers the reasoning. This article focuses on how to prove it before launch.

The Twelve Mandatory Go/No-Go Gates

Key takeaway: All gates that apply to a flow must pass. There is no summed score. A flow that passes eleven gates and fails duplicate handling is a no-go.

Scoring models invite trade-offs between unlike things. Ten excellent results do not offset a double-posted invoice. Treat the table below as a checklist where every applicable row is required, and treat the result as input to your own governance, not as a certificate. Your auditors, regulators, and contracts define what you owe; this table does not.

#	Gate	Failure to inject	Pass evidence required
1	Verified acknowledgment	Kill the receiver after it receives an event but before processing completes	Success is returned only after the event is durably committed. After restart the event is still present and gets processed. The test log shows no "acknowledged but lost" events.
2	Duplicate delivery	Send the same event many times, including concurrently	Exactly one business effect (one order, one posting). Duplicates are recorded and acknowledged without re-applying.
3	Changed payload, same identity	Reuse an operation identifier with a different amount, quantity, or item	The request is rejected and alerted. The original approved payload is unchanged. It is not silently accepted and not silently ignored.
4	Concurrent workers	Run several workers against the same queue, including after a worker stalls	Only one worker holds a claim at a time. An expired claim can be taken over, and the target-side effect remains single.
5	Out-of-order events	Deliver update before create, or cancel before ship	Documented rule per flow: hold, fetch current state from the source, or reject. No state regresses.
6	Ambiguous timeout	Let the target commit, then drop the response	The operation is marked unknown. Authoritative state is queried by business key before any retry. Outcome is resolved as applied, not applied, or escalated.
7	Terminal failure and uncertain input	Send malformed, unmapped, or policy-violating data, and a permanently failing event	Retries stop after a defined limit. The item goes to a dead-letter or review queue with a reason code and a named owner. Input is never guessed into a default.
8	Outage and backlog replay	Take the downstream system offline for a representative period, then restore it	Backlog drains without duplicates, respects rate limits, and prioritizes by business risk. Senders' retry windows are understood.
9	Reconciliation	Delete or alter a record on one side without an event	A scheduled comparison detects the break and repairs it through the same idempotent path used for live events. Overlap windows are used to avoid boundary gaps.
10	Tenant and entity isolation	Use identical identifiers in two tenants, companies, or sites	Deduplication keys are scoped by tenant. No collision, no cross-entity read or write, and no shared queue that leaks payloads.
11	Privacy ownership	Inspect payloads, application logs, traces, and dead-letter contents	Sensitive fields are minimized or masked. Retention is defined. A named owner exists. Identifiers are treated as potentially sensitive.
12	Cutover and rollback	Rehearse cutover, a mid-flight stop, and rollback	A written decision point, a named person who can stop, a defined way to reconcile in-flight items, and a tested rollback path.

Scoping notes that prevent misuse

• Signature checks are transport-specific. Shopify's HMAC verification, for example, applies to HTTPS deliveries and is not required for its Pub/Sub or EventBridge deliveries. A signature proves who sent a message. It does not prevent duplicates or replays by itself.
• Identifiers can be sensitive. Order numbers, customer emails, and delivery IDs can identify a person or a deal. Do not treat IDs as safe to scatter across logs just because they are not names.
• The sender's identifier is a starting point. A vendor's webhook ID is useful for deduplicating a delivery, but it may not equal your business operation identity. Shopify's own documentation distinguishes a per-delivery ID from an event ID shared across deliveries created by the same merchant action.

Building the Evidence: A Reference Pattern for Receipt, Claim, and Effect

Key takeaway: Deduplication, durable recording, and work scheduling must happen atomically, or be recoverable. Then claim work atomically, and make the target change conditional where the target allows it.

Below is an illustrative pattern, not an official API contract for any product. It shows what your tests should be able to observe. Table and column names are placeholders. Adapt them to your platform and verify the exact contract of each system you connect.

Step 1: Record receipt, identity, and follow-up work in one transaction

-- Illustrative only. Names and syntax vary by database. CREATE TABLE integration_inbox (   tenant_id      text NOT NULL,   source_system  text NOT NULL,   operation_id   text NOT NULL,   payload_hash   text NOT NULL,   -- hash of the approved, canonical payload   state          text NOT NULL DEFAULT 'received',   attempt_count  int  NOT NULL DEFAULT 0,   claimed_by     text,   claim_expires  timestamptz,   received_at    timestamptz NOT NULL DEFAULT now(),   PRIMARY KEY (tenant_id, source_system, operation_id) );  -- In ONE transaction: BEGIN;   INSERT INTO integration_inbox (tenant_id, source_system, operation_id, payload_hash)   VALUES ($1, $2, $3, $4)   ON CONFLICT (tenant_id, source_system, operation_id) DO NOTHING;    -- If a row was inserted: also insert the work item (outbox or job row)   -- in the same transaction so receipt and scheduling cannot diverge.    -- If nothing was inserted: read the stored payload_hash.   --   same hash      -> duplicate delivery: acknowledge, apply nothing   --   different hash -> changed payload under an existing identity:   --                     reject, alert, do not apply COMMIT; -- Return success to the sender only AFTER the commit succeeds.

Why this matters for your tests: Gate 1 asks you to crash the receiver between steps. Gate 2 and Gate 3 exercise the conflict branch. The primary key includes the tenant, which is what Gate 10 checks. If the receiver acknowledges before the commit, a crash loses the event. If it commits the event but schedules work in a separate, unrecoverable step, you get orphaned events that never run.

Step 2: Claim work atomically

-- Illustrative only. A worker claims one item; zero rows means another worker owns it. UPDATE integration_inbox SET state = 'claimed',     claimed_by = $worker_id,     claim_expires = now() + interval '2 minutes',     attempt_count = attempt_count + 1 WHERE tenant_id = $1 AND source_system = $2 AND operation_id = $3   AND (state = 'received'        OR (state = 'claimed' AND claim_expires < now())) RETURNING operation_id;

A claim that expires can be taken over, which means a second worker may re-run a step the first worker already performed. That is why the next step matters.

Step 3: Make the target-side change conditional

-- Illustrative only. A conditional mutation performed by the target system. UPDATE sales_order SET status = 'released', version = version + 1 WHERE order_id = $1 AND version = $expected_version; -- Zero rows updated = conflict. Treat as a conflict to resolve, never as success.

A read followed by a compare in your own code, followed by a write, is not compare-and-swap. Between the read and the write, another actor can change the record. Real protection requires the target to enforce the condition in the write itself, such as a version check, an entity tag precondition, or a unique constraint. If the ERP or partner API you integrate with offers no conditional write, say so in the test report. Then disclose the limitation, serialize changes to that record through a single worker, and keep the reconciliation job (Gate 9) as the safety net.

Step 4: Handle unknown outcomes

When a call to the target times out, completion could not be confirmed. Record the operation as unknown. Do not mark it succeeded and do not mark it failed. Query the authoritative state by business key or by the operation identity stored with the target record, then decide. Only retry blindly when the operation is demonstrably safe to repeat. For general retry mechanics, the AWS Builders' Library article on timeouts, retries, and backoff with jitter explains why uncontrolled retries amplify outages, which Gate 8 tests.

For HTTP request APIs specifically, an IETF HTTP API working group has published an Idempotency-Key header specification draft. Check its current status before depending on it, and remember it addresses the HTTP request transport only. It does not replace durable inbox recording, queue consumers, or file-based and message-broker flows.

For privacy-specific controls on payloads, logs, and dead-letter queues, see our CIO go/no-go controls for ERP integration data privacy.

Why This Fails in the Real World

Key takeaway: Capable teams still ship fragile integrations because ownership is split across vendors, partners, and departments, and nobody is paid to break things before launch.

The three scenarios below are illustrative composites of common patterns, not customer results. They focus on system and governance gaps, not on individuals.

Scenario 1: The acknowledged-but-lost order

A mid-market distributor connects its online storefront to the ERP. The receiver is built to respond quickly, because the sender enforces short response timeouts (Shopify documents a five-second limit for the full request). To stay fast, the handler returns success first and queues the work afterward. During a deployment, the process restarts between the two steps. The sender sees success and never retries. The order exists in the storefront, never in the ERP. Customer service finds out when the customer calls.

Why intelligent teams miss it: The speed requirement and the durability requirement were assigned to different people. Load tests measured response time. Nobody tested a restart mid-request. Gate 1 would have caught it.

Scenario 2: The timeout that was actually a success

A manufacturer's warehouse system posts a stock transfer into the ERP. The ERP commits the transfer but the response is dropped by a network device. The warehouse system sees a timeout and retries three times with no shared operation identity. The ERP, seeing four requests, books four transfers. The shortage appears at cycle count, days later, and is attributed to picking errors.

Why intelligent teams miss it: The retry logic was written as good practice. The team did not distinguish a safe-to-repeat read from a consequential mutation. Gate 6 requires reconciliation of authoritative state before retrying. Our guide for plant teams on recovering duplicate and missing shop floor postings covers the cleanup side.

Scenario 3: The shared key across entities

A group runs two legal entities on one integration layer. Both entities' source systems number orders from 1000. The deduplication table keys only on the order number, so the second entity's order 1001 is treated as a duplicate of the first entity's. It is acknowledged and dropped.

Why intelligent teams miss it: Testing used one company. The key design looked correct in a single-tenant world. Gate 10 forces the collision test.

Scenario 4: The dead-letter queue nobody owns

Bad records accumulate in a dead-letter queue. Nobody has been told it is theirs. Months later, the queue holds thousands of customer records in readable form, with no retention limit. Finance discovers unposted invoices; security discovers the privacy exposure in the same week.

Why intelligent teams miss it: "Handle errors" was a story point, not an operating responsibility. Gates 7 and 11 assign an owner and a retention rule before go-live.

Pattern	Governance gap	Gate that exposes it
Acknowledged but lost	Performance and durability owned separately	1
Ambiguous timeout retried	No distinction between safe and consequential operations	6, 2
Cross-entity key collision	Tests run in one tenant only	10
Orphaned dead-letter queue	No named owner or retention rule	7, 11

Tier-1, Mid-Market, and ArionERP: What Evidence to Demand From Each

Key takeaway: Deployment model and platform class change who owns each gate, not whether the gate applies. Ask for evidence, not assurances.

The comparison below describes the evidence to request in each category. It deliberately avoids claims about specific competitors' behavior. Confirm every item with the vendor in writing, in a sandbox that reproduces production limits. For ArionERP, the platform is available as Cloud (SaaS) and On-Premises, and its public API page and integrations page are the places to begin. Exact delivery semantics, idempotency support, rate limits, and any compliance scope should be confirmed during evaluation, not assumed from this article.

Dimension	Tier-1 ERP (large suite)	Mid-market ERP	ArionERP (verify in pilot)
Who builds and tests integrations	Often a systems integrator plus middleware licenses	Vendor, partner, or in-house	Vendor-supported modules and API; confirm which flows your team or a partner will own
Evidence to request	Documented retry, ordering, and error semantics per interface	Sandbox with production-like limits and published API behavior	Written API contract, sandbox access, and a run of the twelve gates against your flows
Customization risk	Heavy extension can bypass standard safeguards	Lighter, but custom scripts are common	Prefer configuration and the API over code changes to core; test any extensions against all gates
Deployment effect	Varies by edition and hosting	Often SaaS-only or partner-hosted	Cloud or On-Premises with the same functional scope; ownership of test, logging, and privacy duties differs by model
Compliance statements	Review the vendor's current attestations and their scope	Review the vendor's current attestations and their scope	Review the security and compliance page for current scope; map claims to your own legal obligations

How deployment choice shifts the work

• SaaS: The vendor operates the platform, so your tests must respect its rate limits, maintenance windows, and API versioning. Gate 8 (outage and replay) should include the vendor's throttling behavior. Privacy ownership is shared: you decide what your integration sends and logs, the vendor governs its own hosting.
• On-Premises: You own the test environment, the queues, the log pipeline, and the dead-letter storage. Gates 4, 7, 8, and 11 become your operational duties in full. You gain control over where data lives, and you carry the cost of proving it.
• Hybrid: Each boundary crossing is a place for duplicates and ambiguous timeouts. Test every boundary separately. Our guide on hybrid ERP architecture lays out the trade-offs.

Limits of this approach

• Failure injection takes time. A rough, illustrative planning range is a few days of dedicated effort per critical integration for a small team, but your number depends on flow count and environment quality.
• A sandbox rarely matches production volume or data quality. Passing in test lowers risk. It does not remove it.
• Some third-party systems do not support conditional writes or lookups by operation identity. In that case your design is limited by serialization and reconciliation, and you should state that openly.

Running the Plan: Sequence, Interpretation, and What to Do Next

Key takeaway: Run the gates in a fixed order, record evidence for each, and treat any failure as a block until it is fixed or formally risk-accepted by the right owner.

Recommended sequence

1. Inventory flows. List every integration, its direction, its sender and receiver, and its consequence class from the earlier table.
2. Name owners. For each flow, assign an owner for terminal failures, reconciliation breaks, and data privacy. Without names, Gates 7, 9, and 11 cannot pass.
3. Prepare a realistic sandbox. Include production-like rate limits, multiple tenants or entities, and representative master data.
4. Run Gates 1-4 (receipt, duplicates, changed payload, concurrency). These are the cheapest and catch the most damaging faults.
5. Run Gates 5-8 (ordering, ambiguity, terminal failure, outage). Record how long recovery takes.
6. Run Gates 9-11 (reconciliation, isolation, privacy). Involve security and, where relevant, your data protection function.
7. Rehearse Gate 12 (cutover and rollback) with the people who will actually be on call.

An illustrative calculation to size the risk

This is a hypothetical example, not a benchmark. Suppose an eCommerce integration handles 3,000 order events per day. Shopify's troubleshooting guidance describes a failed-delivery rate over 0.5% as higher than average. At 0.5%, that is 15 failed deliveries per day that must be retried or recovered. If your design cannot absorb retries, those 15 become 15 manual investigations or 15 silent losses every day. Even at a tenth of that rate, a weekly count is meaningful. Use your own measurements to size this for your flows.

Interpreting results

Result	Meaning	Action
All applicable gates pass with recorded evidence	No blocking weakness found in tested scenarios	Proceed to go-live under your normal change governance. Keep monitoring and scheduled reconciliation.
A gate fails on a consequential flow	Known path to duplicate, lost, or corrupt business data	No-go for that flow. Fix, retest, and re-run adjacent gates.
A gate is untestable (for example, the partner API lacks lookup by key)	Residual risk that you cannot measure	Disclose it, add compensating controls such as single-worker serialization and tighter reconciliation, and obtain explicit acceptance from the risk owner.
Evidence is missing	Unknown, not pass	Treat as fail until evidence exists.

Red flags during testing

• The team says "we retry until it works" without describing what happens if the first attempt actually worked.
• Duplicate protection depends on an in-memory set that disappears on restart.
• The same operation identifier is accepted with a different payload.
• Logs contain full customer payloads and nobody can say who approved that.
• "Reconciliation" means someone opening two spreadsheets monthly.

Planning note for the years ahead

As more organizations let automated or AI-assisted agents initiate ERP transactions, the volume and variety of callers grows, and duplicates and ambiguous outcomes become more likely, not less. The gates above stay valid because they test properties of the integration, not features of any one tool. For the controls specific to machine-initiated transactions, see the go/no-go control framework for AI-initiated ERP transactions. For the broader launch checklist, use the ERP implementation checklist.

Not sure which integration flows would fail a duplicate test?

Bring your flow list. We will walk through the gates against your CRM, eCommerce, BI, and warehouse connections.

Talk with ArionERP specialists about integration readiness for Cloud or On-Premises deployment.

Request a Consultation

Conclusion: Five Actions Before You Approve Go-Live

Integration readiness is a decision you can support with evidence. These five actions turn the plan into something your team can start this week.

1. Classify every flow. Mark each integration as a consequential mutation, a reversible update, or a read-only extract, and apply the matching gates.
2. Write down the owners. Assign a named person for terminal failures, reconciliation breaks, and sensitive data in payloads, logs, and dead-letter queues. If a gate has no owner, it cannot pass.
3. Run the failure injections in a production-like sandbox. Start with receipt, duplicates, changed payloads, and concurrency, and keep the evidence for each result.
4. Replace blind retries with reconciliation for consequential changes. After any ambiguous timeout, check authoritative state before attempting the operation again, and record unknown outcomes as unknown.
5. Get vendor claims in writing and scope them. Confirm API behavior, delivery semantics, and any compliance statements against primary documentation, and map them to your own legal and contractual obligations.

None of these actions require a specific vendor. They require the discipline to break your own integration before the business depends on it.

About the ArionERP Expert Team

ArionERP is an AI-enhanced, modular ERP platform available in Cloud (SaaS) and On-Premises models, built for SMBs and mid-market organizations with a primary focus on manufacturing. It is a product of Cyber Infrastructure (CIS), which has been delivering software since 2003. Details on security practices and current certification scope are published on the ArionERP security and compliance page; review them against your own requirements. Reviewed by ArionERP Expert Team.

Frequently Asked Questions

What is ERP integration testing, and how is it different from user acceptance testing?

ERP integration testing verifies that data moving between the ERP and connected systems behaves correctly, including under failure. User acceptance testing checks that business users can complete their tasks. UAT usually follows the happy path. Integration testing adds duplicates, out-of-order events, timeouts, outages, and concurrent workers.

Why do I need failure-injection tests if my vendor says the API is reliable?

Reliability claims describe the vendor's service, not your end-to-end flow. Mature senders document that duplicates, retries, and unordered delivery can occur, as Stripe and Shopify do in their webhook documentation. Your receiver and workers must handle those conditions, and only a deliberate test proves they do.

Should I use a scoring model to decide whether an integration is ready?

No. Use mandatory go/no-go gates. A weighted or summed score can let strong results in one area mask a serious failure in another, such as a duplicate posting defect. Any failed applicable gate should block that flow until it is fixed or formally accepted by the right risk owner.

What should we do after a timeout when calling an ERP API for a consequential change?

• Mark the operation as unknown, not successful and not failed.
• Query the authoritative state in the target by business key or stored operation identity.
• Retry only if the change is confirmed not to have been applied, or is safe to repeat.
• Escalate to a named owner if the state cannot be determined.

Is an idempotency key enough to prevent duplicate postings?

Not by itself. The key must be scoped to a tenant or legal entity, bound to the approved payload so reuse with different content is rejected, and backed by durable storage and atomic claiming of work. Where the target system cannot enforce a conditional write, you must disclose the limitation and use serialization and reconciliation to reduce the risk.

Does the Idempotency-Key HTTP header standard cover all integration types?

No. It addresses HTTP request transport and, at the time of writing, is described by an IETF working group draft whose status you should verify before relying on it. Message queues, file transfers, and webhook receivers still need their own durable recording and deduplication design.

How does SaaS versus on-premises deployment change integration testing?

The gates are the same. The ownership differs. With SaaS you must test against vendor rate limits, maintenance windows, and API versioning, and share privacy duties with the vendor. With on-premises you own the environment, queues, logs, and dead-letter storage, so more of the evidence is yours to produce.

Who should own dead-letter queues and reconciliation breaks?

A named operational owner for each flow, with a defined response time and retention rule. IT may run the tooling, but the business owner of the affected process, such as order management or finance, usually decides how to resolve the item. Without that assignment, failed items accumulate unseen.