Shop-Floor Transaction Recovery: A Plant Manager's Guide to Duplicate and Missing ERP Postings

When an ERP integration times out during a production confirmation, a material issue, or a receipt to stock, the plant does not know whether the posting landed. The safe response is to record the outcome as unknown, look up the authoritative state of that exact operation by its durable identifier, and only then decide whether to accept, reject, or resend. Retrying blindly is how a single 480-unit confirmation turns into 960 units of phantom finished goods.

This guide is written for the Manufacturing Head or Plant Manager who owns the consequences: stock that does not match the shelf, MRP that reacts to quantities that never existed, and lot genealogy that auditors cannot trust. It is not a developer tutorial. It explains what to demand from your ERP and MES integration, what to do on the day a network link drops mid-shift, and which controls must be in place before you let any shop-floor transaction post automatically.

It complements two existing ArionERP articles that approach the same boundary from other angles: the CIO-level guide to idempotent ERP integration and the COO-level decision on real-time MES and ERP integration. This piece focuses on recovery: what the plant does after something has already gone wrong. The frameworks below are proposed designs for you to adapt and test, not a certified method.

Key Takeaways

• Unknown is its own state. A timeout means completion could not be confirmed. It is not success and it is not verified failure. Screens, reports, and operators must say so.
• Reconcile before you resend. Query the authoritative state of the specific operation first. Never blindly retry a consequential posting such as a confirmation, issue, scrap, or release.
• Identity is the control. Each operation needs a durable, tenant-scoped ID bound to a hash of the approved payload. The same ID with a different payload is rejected, not silently accepted.
• Legitimate repeats exist. A second confirmation of 50 pieces next shift is a new operation. Deduplicating on content (work order plus quantity) will wrongly swallow real production.
• Acknowledge only what is durable. Receipt, scheduling, and posting must be recorded atomically, or be explicitly recoverable, before any acknowledgment goes back to the shop floor.
• Go/no-go, not a score. The control checklist in this article is all-required. One failed control blocks automation for that transaction type; no total can offset it.

Why a Lost Acknowledgment Is a Plant-Floor Problem, Not Just an IT Problem

🔑 Key takeaway: The physical world does not roll back. Parts are consumed and pallets move whether or not the ERP recorded it, so every ambiguous posting leaves a gap between reality and the system of record.

A software team sees a timeout as a failed request. A plant sees it as a work order that may be 480 units ahead or behind what the ERP believes. The operator has already scrapped the offcuts, the forklift driver has already staged the pallet, and the next shift is already planning against whatever the screen says.

Three outcomes are possible after any integration call, and most plants only design for two of them.

Outcome	What it means	Correct plant behavior
Confirmed success	The ERP durably recorded the intended effect and said so.	Show as posted. Release downstream dependencies such as shipping allocation.
Confirmed failure (terminal)	The ERP rejected the operation for a known reason, for example the work order is closed or the lot is on quality hold.	Do not retry. Route to a named owner in an exception queue with the reason attached.
Unknown	No acknowledgment arrived. The posting may have landed, may never have arrived, or may still be in flight.	Show as unconfirmed. Hold dependent decisions. Reconcile against authoritative state before any resend.

The third row is where money and traceability are lost. If your current process treats unknown as success, you will under-detect missing postings. If it treats unknown as failure and retries, you will create duplicates. Both errors are invisible until a cycle count, a customer shortage, or a recall exercise exposes them.

Lot and serial traceability raises the stakes. A duplicated genealogy link or a missing consumption record is not a rounding error; it can make a lot's history unprovable. For background on how ERP handles this discipline, see ArionERP's overview of batch and lot tracking in manufacturing ERP.

How Most Plants Handle Ambiguous Postings, and Why It Fails

🔑 Key takeaway: The four common approaches each fix one symptom and create another failure. Only a reconciliation-first approach respects the three-outcome model.

Approach	How it works	What it protects	What it breaks
Retry until it works	The MES or middleware resends on timeout, often several times.	Missing postings	Duplicates, because the first attempt may have succeeded.
Manual re-key	A supervisor enters the transaction directly in the ERP.	Speed, short-term	Duplicates if the original message arrives later. Leaves no link between the two entries.
Nightly clean-up	Someone compares totals each morning and adjusts inventory.	Period-end totals	Genealogy and lot-level truth. Adjustments hide the cause and erase the evidence.
Reconciliation first (proposed)	Record unknown, look up the operation's authoritative state by ID, then decide.	Both duplicates and gaps	Requires design work up front and a status lookup the target system can actually answer.

Each of the first three is rational under time pressure. The line has to run. The problem is that none of them has a way to answer the one question that matters: did this specific operation take effect? Without a durable operation identity on both sides, the plant can only guess from totals.

It also helps to be precise about what "retry" can safely mean. Under HTTP semantics, RFC 9110 defines which methods are idempotent, and POST is not among them. Creating a posting with POST therefore gives you no inherent protection against a repeated request. The protection has to be designed into the operation identity, which is the subject of the framework below.

Why This Fails in the Real World

🔑 Key takeaway: Capable teams still fail because the gaps sit between systems, shifts, and owners. No single person sees the whole transaction path. The scenarios below are illustrative composites, not customer results.

Scenario 1: The blind retry that doubled a backflush (illustrative)

Line 4 confirms 480 good units against a work order. The call to the ERP times out at the network edge, but the ERP has already committed. Middleware retries; the second request creates a second confirmation. The bill of materials needs two housings per unit, so the backflush consumes 1,920 housings instead of 960. MRP now sees a housing shortage and raises a purchase requisition for parts the plant actually owns. Nobody did anything careless. The retry logic was configured years earlier, for an integration that only moved master data, where repeating a request was harmless.

Scenario 2: The optimistic screen that hid a missing posting (illustrative)

The operator terminal shows "Confirmed" the moment the button is pressed, because the interface was built to feel responsive. The posting is lost during a switch failover. Finished goods never appear in the ERP, so shipping cannot allocate them. The gap surfaces only when a customer order slips. Root cause: the interface collapsed "sent" and "posted" into one status, so unknown was never visible.

Scenario 3: The correction that was ignored (illustrative)

An operator corrects a count from 480 to 408 and the MES resubmits using the same operation ID. The target recognizes the ID and returns the stored result for the first payload, silently ignoring the correction. Everyone sees a success response. The error persists. Root cause: the key identified the operation but was not bound to the payload, so a changed payload under a reused key was treated as a harmless duplicate.

Scenario 4: The manual fix that met the delayed message (illustrative)

During a link outage, a supervisor re-keys six confirmations directly in the ERP. Forty minutes after the link recovers, the queued originals arrive and post. There was no shared identifier to tell the two sets apart.

Why intelligent teams still miss it

• Ownership is split. IT owns the middleware, operations owns the work order, quality owns lot status, and nobody owns the end-to-end operation.
• The happy path dominates testing. Acceptance tests rarely include a lost acknowledgment after a successful commit.
• Shift pressure rewards speed. The fastest visible fix wins, and the evidence it erases is invisible until later.
• Vocabulary is missing. If the system has no "unknown" status, people cannot act on it.

A Proposed Design: The Operation Ledger and Reconciliation Protocol

🔑 Key takeaway: One durable record per operation, keyed by tenant and source, bound to a payload hash, claimed atomically, and updated in the same transaction as the business effect. This is a proposed design for you to evaluate, not a method ArionERP claims to already run in a particular way.

Design rules

1. Create the identity at the origin, before the first send. The operation ID is generated when the operator confirms, and it is persisted locally before any network call. A new ID created per retry defeats the purpose.
2. Scope it to the tenant and source. The uniqueness boundary is tenant (plant, legal entity, or customer account), source system, and operation ID. Two plants that both generate "1001" must never collide, and one tenant must never be able to see or replay another's operations.
3. Bind the ID to an immutable hash of the approved payload. Hash a canonical serialization of what was approved. On first sight the ledger stores ID and hash together. Same ID with the same hash returns the recorded state. Same ID with a different hash is rejected as a conflict and never applied.
4. Treat corrections as new operations. A correction gets its own ID and references the original. It is not a mutation of the original submission.
5. Claim work atomically. Multiple workers must not both process one operation. The first to win a conditional update owns it; others see that it is claimed.
6. Record, schedule, and acknowledge in one transaction. The inbox row (payload, hash, and the instruction to process it) is written in a single commit. Only after that commit does the system acknowledge receipt. If anything fails before commit, no acknowledgment is sent and the sender's retry is safe because the ledger deduplicates it.
7. Apply with target-side conditional mutation. The business effect, ledger update, and outbox event commit together, and the mutation itself carries its guard (for example, the work order must still be released).
8. Distinguish two acknowledgments. "Received durably" and "applied" are different. Do not let the first be displayed as the second.

Illustrative sketch

The SQL below is illustrative only. It is not ArionERP's schema or API; the exact contract of any system you integrate must come from its official documentation.

-- 1) Receive: record + schedule in ONE transaction, then acknowledge "received" BEGIN; INSERT INTO operation_ledger   (tenant_id, source_system, operation_id, operation_type,    payload_sha256, state) VALUES (:tenant, :source, :op_id, 'PROD_CONFIRM', :hash, 'RECEIVED') ON CONFLICT (tenant_id, source_system, operation_id) DO NOTHING; -- 0 rows inserted: read the stored hash. --   different hash -> reject as conflict, apply nothing --   same hash      -> return stored state (RECEIVED, APPLIED, ...) -- 1 row inserted: also insert the payload into the inbox here. COMMIT;  -- 2) Claim: only one worker can win UPDATE operation_ledger    SET state = 'CLAIMED', claim_token = :token,        claim_expires_at = now() + interval '2 minutes'  WHERE tenant_id = :tenant AND source_system = :source    AND operation_id = :op_id    AND (state = 'RECEIVED'         OR (state = 'CLAIMED' AND claim_expires_at < now())); -- affected rows = 1 means this worker owns the operation  -- 3) Apply: effect, ledger, and outbox commit together BEGIN; INSERT INTO stock_postings (tenant_id, source_system, operation_id,                             work_order, qty) VALUES (:tenant, :source, :op_id, :wo, :qty);  -- UNIQUE on the 3 key columns UPDATE work_order    SET qty_completed = qty_completed + :qty  WHERE tenant_id = :tenant AND wo_id = :wo    AND status = 'RELEASED'    AND qty_completed + :qty <= qty_planned + :tolerance; -- affected rows = 0 means a business rule failed: roll back and -- record REJECTED_TERMINAL with a reason, do not retry UPDATE operation_ledger SET state = 'APPLIED', result_ref = :posting_id  WHERE tenant_id = :tenant AND source_system = :source    AND operation_id = :op_id AND claim_token = :token; INSERT INTO outbox (tenant_id, event_type, ref) VALUES (:tenant, 'PROD_POSTED', :posting_id); COMMIT;

Why the guard lives in the target

Reading a work order, comparing its status in application code, and then writing is read-compare-then-write. Another worker can change the row between your read and your write. A compare-and-swap only exists when the target enforces the condition as part of the write itself, as in the WHERE status = 'RELEASED' clause above or a version check. If an ERP interface does not offer conditional writes, you must disclose that limitation and compensate by serializing all writes for a given work order through a single queue, accepting the throughput cost and the remaining conflict risk. The unique constraint on the posting table is a second line of defense: even if a claim expires and two workers race, the second insert fails.

Legitimate later operations

Deduplicate on identity, never on content. A second confirmation of 50 units for the same work order is valid when it comes from a different operator action, which means a different operation ID. Partial confirmations, rework loops, and reversals are all legitimate repeats. A content-matching rule (same work order, same quantity, within ten minutes) will eventually discard a real posting.

Practical Implications for the Plant: Transaction Treatment and an Outage Runbook

🔑 Key takeaway: Not every transaction deserves the same treatment. Decide per transaction type what can auto-post, what must wait for confirmation, and who owns each exception.

Treatment by transaction type

Transaction	Identity and guard	If outcome is unknown
Production confirmation (good quantity)	Unique operation ID. Conditional on work order status and quantity tolerance.	Show "unconfirmed." Hold shipping allocation of those units until resolved.
Backflush or material issue	Derived ID per component line (operation ID plus line), committed in the same transaction as the confirmation.	Never resend components separately from the confirmation.
Scrap and yield loss	Separate operation with a mandatory reason code.	Hold the reason code with the physical scrap tag until posted.
Quality hold or release	State transition guarded by the expected current status. A release applies only if the lot is still on hold.	Treat the lot as held. Unknown never releases stock.
Lot or serial genealogy link	Uniqueness enforced in the target on parent, child, and operation.	Reconcile by lot, not by quantity totals.
Receipt and put-away	Shared operation ID with the WMS or scanner event.	Keep the pallet in a "received, unposted" location.
Labor and time capture	Operator identifiers handled under a named privacy owner.	Queue locally, then reconcile against payroll rules.

An outage runbook

1. Pause automatic resend for affected transaction types. Pausing is cheaper than cleaning up duplicates.
2. List every operation in the unknown state from the sender's ledger, with ID, hash, and time of first attempt.
3. Query authoritative state per operation ID in the ERP: applied, rejected, in progress, or not found.
4. Classify. Applied: record the result and close. Rejected: move to the exception queue with the reason. In progress: wait and re-check.
5. Handle "not found" carefully. Not found now is not proof it will never land, because a request may still be in flight. Resend the identical payload under the identical ID only if the target is verified to enforce uniqueness on that ID. Otherwise hold it for supervised review.
6. Verify physically where the ledger cannot decide: count the pallet, check the lot label, confirm the work order output.
7. Post corrections as new, referenced operations. Do not overwrite history, and do not post supervisor re-keys without recording the original ID they replace.
8. Close with evidence. Keep the ledger entries, the physical count, and the approver for audit.

Identifiers can be sensitive

Operation IDs look harmless, but the payloads around them are not. Operator badge numbers, device identifiers, serial numbers, and supplier lot codes can identify individuals or reveal commercial relationships. Assign a named owner for the ledger, and decide up front: who can read payloads, how long they are retained, what is excluded from logs, and how the ledger is handled if an employee or supplier requests deletion. Hashing a low-entropy value such as a badge number does not make it anonymous, since it can be recovered by trying every plausible value. What applies to you depends on your jurisdiction and contracts, so have counsel confirm the specific obligations.

Deployment model changes who can do what

With an on-premises ERP you control the database and can add ledger tables, unique constraints, and reconciliation reports next to the data. With a SaaS ERP you work through the vendor's API and status endpoints, so the vendor's contract is the ceiling. ArionERP is available as both ArionERP Cloud and ArionERP On-Premises. Whichever you choose, confirm the specifics against the current API documentation and a proof of concept rather than assuming parity with any other platform.

What the Standards Do and Do Not Promise

🔑 Key takeaway: Protocol-level delivery guarantees are scoped to one hop. None of them, alone, guarantees that a business posting occurred exactly once in your ERP.

Mechanism	What it covers	What it does not cover
MQTT QoS 1 and 2 (MQTT 5.0 specification)	Delivery assurance between an MQTT sender and receiver. QoS 1 is "at least once," where the specification states that duplicates can occur.	The downstream ERP write performed by a bridge or consumer. Exactly-once at the transport hop does not make the posting exactly-once.
MQTT DUP flag (MQTT 3.1.1 specification)	Marks that a control packet might be a redelivery.	The specification notes it refers to the packet, not the application message, and a repeated application message can arrive with DUP set to 0 and a different packet identifier. It is not a business-level deduplication key.
HTTP method semantics (RFC 9110)	Defines which methods are idempotent, which informs when automatic retries are acceptable.	POST is not defined as idempotent, so a posting endpoint needs its own identity scheme.
Idempotency-Key header (IETF Internet-Draft)	A proposed HTTP request header for clients to supply a unique key so servers can recognize repeats.	It is a work-in-progress draft, not a published standard, and the IETF itself says drafts should be cited only as work in progress. It does not define your tenant scoping, payload binding, or reconciliation.

Industrial integration models such as ISA-95 help you decide which information crosses the enterprise-to-control boundary, such as production performance and material movements. They are not a substitute for defining how each of those messages is made safe to repeat or recover.

2026 Update, then the evergreen rule

As of this writing in late 2026, the Idempotency-Key header is still an active Internet-Draft in the IETF HTTP API working group, with revision 08 posted in October 2026. Check its current status before you cite it in a specification or contract. The durable principle does not depend on that outcome: whatever header, field, or protocol feature carries your key, the ledger design above supplies the tenant scope, payload binding, atomic claim, and reconciliation that no transport feature provides on its own.

Go/No-Go Controls, Trade-Offs, and a Lower-Risk Path

🔑 Key takeaway: Use all-required controls per transaction type. One "No-go" blocks automatic posting for that type until it is fixed. A completed checklist is not a certification that your integration is production-ready; it only means known blockers are cleared.

Mandatory controls (all required)

#	Control	Go only if	No-go if
1	Durable operation identity	ID is generated at the origin, persisted before first send, and reused on every attempt.	IDs are regenerated per retry or derived from content.
2	Tenant-scoped uniqueness	Uniqueness is enforced on tenant, source, and ID in the target.	Keys are global strings or only checked in application memory.
3	Payload binding	Stored hash is compared on every repeat and a mismatch is rejected.	Same key with a changed payload is silently accepted or ignored.
4	Authoritative status lookup	You can ask the ERP for the state of one operation by ID.	Reconciliation depends on comparing totals.
5	Atomic receipt and scheduling	Dedup record, payload, and work scheduling commit together before acknowledgment.	Acknowledgment can be sent before durable recording.
6	Atomic claim	Concurrent workers cannot both process one operation.	Claim is a read followed by a write.
7	Target-side conditional mutation	Guards (status, version, quantity limits) are enforced in the write, or the serialization limit is documented and accepted.	Read-compare-then-write in the client is the only guard.
8	Terminal failure path	Business rejections stop retries and go to a named owner.	Rejections are retried or discarded.
9	Unknown-state handling	Screens and reports show "unconfirmed." Unknown never releases stock or lots.	Unknown displays as success or is treated as failure.
10	Privacy ownership	A named owner, retention rule, and log-redaction rule exist.	Payloads containing operator or serial data sit in unmanaged logs.
11	Recovery rehearsal	You have simulated a lost acknowledgment after a successful commit and reconciled it.	Only the happy path has been tested.

Risks and trade-offs

• Latency and complexity. A ledger, status lookup, and exception queue add engineering effort and a small delay. For low-consequence data such as non-inventory notes, this may not be worth it.
• Operational friction. An "unconfirmed" state means supervisors must act on exceptions. Without staffing and ownership, the queue becomes the new backlog.
• Vendor limits. If an API cannot answer a status query or enforce a unique key, you cannot fully close the gap; you can only reduce it and disclose it.
• Not a replacement for physical control. Cycle counts and lot verification remain necessary. The ledger tells you what the system believes and why.

What to request from any ERP vendor, including ArionERP

The table lists evidence to request, not conclusions. Characteristics vary by product, edition, and version, so verify each item in writing and in a demonstration using your own transaction types.

Question	Large Tier-1 suite	Mid-market or lightweight ERP	ArionERP
Can I look up an operation's state by my own ID?	Verify per module and release.	Varies widely. Verify.	Request in API documentation and a scripted demo.
Does the target enforce uniqueness on my key and reject changed payloads?	Verify per interface.	Often left to the integrator. Verify.	Request a written answer per interface you plan to use.
Are conditional writes available for work order and lot status?	Verify.	Verify.	Review against manufacturing module scope and test.
What do Cloud and On-Premises each allow me to add?	Depends on the hosting model.	Depends on the hosting model.	Both models are offered; confirm differences in a proof of concept.
What security and compliance evidence covers this interface?	Obtain vendor reports and scope statements.	Obtain vendor reports and scope statements.	See the security page and request documentation. Scope applies to what the vendor states, not to your own legal obligations.

A lower-risk path

1. Start with one high-consequence transaction, usually production confirmation with backflush.
2. Run the full control table before enabling auto-posting; run rehearsals in a test tenant with injected timeouts.
3. Keep a human approval step for "not found" resends until the target's uniqueness enforcement is verified.
4. Expand type by type. Use the existing guidance on diagnosing ERP data integrity issues to clean master data before you add automation on top of it.

Do you know what your ERP does when the acknowledgment never arrives?

Bring one real transaction type. We will walk through identity, status lookup, and recovery against your plant's workflow.

Review your shop-floor integration with ArionERP specialists.

Contact Us

Conclusion: Five Actions for the Next 30 Days

A decision-oriented summary, not a recap. Your integration is only as safe as its worst unexamined timeout.

1. Add "unknown" to your vocabulary. Change operator screens, shift reports, and exception queues so that unconfirmed postings are visibly distinct from posted and rejected ones.
2. Inventory your consequential transaction types. List confirmations, issues, scrap, holds, releases, and genealogy links. For each, record who owns exceptions and whether the target can answer a status lookup by ID.
3. Audit your retry settings. Find every place a middleware, MES, or scanner gateway resends automatically. Disable blind retries on consequential types until identity and payload binding are verified.
4. Rehearse the lost acknowledgment. In a test tenant, let a posting commit, drop the response, and practice the runbook. Record what you could not determine.
5. Name the privacy and ledger owners. Decide retention, access, and log redaction for operator, device, and serial identifiers, and have counsel confirm obligations specific to your jurisdiction.

About the source of this guidance

ArionERP is a modular, AI-enhanced ERP platform available as Cloud (SaaS) and On-Premises. It was developed and launched in 2016 as a product of Cyber Infrastructure (CIS), a software development company operating since 2003. Certification and compliance details relevant to your deployment should be confirmed on the ArionERP security page and in vendor documentation; they do not replace your own legal and audit review. The designs in this article are proposed frameworks and illustrative scenarios, not customer results. Reviewed by the ArionERP Expert Team.

Frequently Asked Questions

What should we do when an ERP production confirmation times out?

Record it as unknown and stop automatic resend for that operation. Look up the operation's state in the ERP by its durable ID. If it is applied, record the result. If it was rejected, route it to the exception owner. If it is not found, resend the identical payload under the same ID only when the target is verified to enforce uniqueness; otherwise hold it for supervised review.

Why is retrying a timed-out posting risky?

A timeout means the sender did not receive confirmation, not that the posting failed. The first request may have committed. A second request without a shared identity can create duplicate finished goods, duplicate component consumption, and distorted MRP signals.

Is an idempotency key enough to prevent duplicate postings?

No. A key helps only if the target enforces it, scopes it by tenant and source, binds it to the approved payload, and records outcomes atomically with the business effect. The HTTP Idempotency-Key header is also still an Internet-Draft rather than a published standard, so check its status and do not rely on it as a guarantee.

Does MQTT QoS 2 guarantee my ERP posts a transaction exactly once?

No. MQTT delivery levels apply to the MQTT sender and receiver. A bridge or consumer that writes to the ERP afterward can still fail or repeat that write. QoS 1 is explicitly "at least once," which allows duplicates, so downstream deduplication by operation identity is still needed.

How do we tell a legitimate second confirmation from a duplicate?

Use identity, not content. A legitimate second confirmation comes from a distinct operator action and carries a new operation ID. A duplicate carries the same ID and the same payload hash. If the same ID arrives with a different payload, reject it and require the correction to be submitted as a new, referenced operation.

What is the difference between a terminal failure and an unknown outcome?

A terminal failure is a confirmed rejection with a known reason, such as a closed work order or a lot on hold. Retrying will not help. An unknown outcome means completion could not be confirmed and requires reconciliation of the authoritative state before any decision.

Does SaaS or on-premises ERP make reconciliation easier?

It depends on what you can control. On-premises gives you direct access to add ledger tables and constraints near the data. SaaS depends on the vendor's API, including status lookup and key enforcement. Verify the specific capabilities in documentation and a proof of concept for whichever model you choose.

Are serial numbers and operator IDs sensitive?

They can be. Operator and device identifiers may relate to individuals, and serial or lot codes may expose supplier or customer relationships. Assign a privacy owner, minimize what is logged, set retention rules, and confirm legal obligations for your jurisdiction with counsel. Hashing low-entropy identifiers does not reliably anonymize them.

Can a go/no-go checklist certify that our integration is production-ready?

No. It identifies blockers that must be cleared per transaction type. All controls must be Go, but passing them does not replace testing, security review, or audit. Treat it as a gate, not a score.