There is a point in the life of a software product where the meaning of a bug changes. A broken card layout is annoying. A duplicated journal entry, an incorrectly settled transfer, a stale balance or an unauthorized reversal is something else entirely. The code may still look like ordinary application code, but the system is now responsible for facts that people use to operate a business or make financial decisions.
This article is about engineering patterns and lessons from current production work. I have intentionally removed the names of the products, clients, infrastructure providers, financial providers and private integrations. The useful part is the system design, not the vendor list.
The architecture changes when the consequence changes
A lot of software begins as a collection of screens around CRUD operations. Create a record. Edit it. Delete it. List it. Add a dashboard. Add filters. Add permissions. That model can carry a product surprisingly far.
Then the records start meaning something outside the database.
A stock movement affects whether a branch can sell an item. A payment changes receivables. A journal entry changes the trial balance. A transfer changes a member's spendable funds. A reversal has to undo an earlier financial consequence without pretending the original event never happened. A retry after a network timeout cannot be allowed to create the same effect twice.
At that point, the system cannot be designed around forms alone. It needs invariants. It needs stable identities for operations. It needs database constraints that agree with application rules. It needs explicit state transitions, deterministic correction paths, and enough audit history to explain why the current number is what it is.
This is the same transition I wrote about in From CRUD App to Operating System, but the recent work made it much more concrete. Two systems in different domains kept failing or becoming risky at the same boundaries: concurrency, authority, reversibility and external truth.
Adding entry posting is not the same as adding an accounting form
One recent requirement sounded simple: finance users needed a proper manual entry-posting workflow. Not a loose "journal notes" screen, but a balanced multi-account voucher that could post real debits and credits into the same accounting backbone used by the rest of the system.
The visible interface was the easy part. Date, branch, reference, memo, account, description, debit and credit columns. The real work was deciding what the form was allowed to mean.
Every line had to contain exactly one side of the entry. Total debits had to equal total credits. The total had to be greater than zero. Accounts had to be active and manually postable at the moment of commit, not only when the page first loaded. The accounting period had to still be open. Posted vouchers had to become immutable. A correction could not silently edit historical accounting state.
Those rules also had to survive concurrency. If a period is being closed while somebody is posting into it, the system needs an ordering rule. If account state changes while a voucher is being prepared, the final commit needs to revalidate against current truth. If two identical submissions arrive because somebody double-clicked or a browser retried, they must not create two journals.
That is the difference between a form and a financial operation. The form collects intent. The posting service is responsible for turning that intent into one durable accounting fact.
The implementation therefore reused the existing ledger rather than introducing a second accounting path. Manual entries, sales, payments, returns, purchases and other financial events should converge on the same journal model. A financial system becomes harder to reason about every time a new feature invents its own version of money.
The race nobody sees in a demo is often the one that matters most
One of the most important fixes in the same system came after reviewing the posting path for concurrency rather than only correctness in a single request.
The application already used a source identity to prevent the same business event from being posted twice. Conceptually, a journal could say: this entry came from operation type X, record Y and event Z. A unique database constraint enforced that identity.
The problem was the classic check-then-insert race.
Request A checks whether a journal exists. It does not. Request B checks at nearly the same moment. It also sees nothing because A has not committed yet. Both requests attempt the insert. The database correctly allows one and rejects the other.
If the application treats the rejected insert as an unexpected database error, then the system is technically protected from duplication but operationally still broken. One user receives a raw failure for an operation that actually succeeded.
The stronger design is to treat the uniqueness conflict as part of the idempotency contract. Lock where appropriate. Attempt the write. If another request wins the race, load the winning record, verify that it represents the same operation, and return it as the result. The database remains the final arbiter of uniqueness, while the application converts a race into deterministic behaviour.
This is why I increasingly distrust idempotency implementations that live only in controller code. A request key is useful, but the durable guarantee belongs close to the data and has to account for concurrent execution. I explored the same principle in The Most Dangerous Payment Bug Is the One That Succeeds Twice.
Correction should add history, not rewrite it
Another important boundary was reversal.
In ordinary CRUD software, "fixing" a record often means editing it. In accounting and money movement, that can destroy the very evidence needed to understand what happened. Once an entry has produced a financial effect, silently changing the old rows makes the current state harder to explain.
The safer pattern is controlled reversal. The original journal remains intact. A new equal-and-opposite journal records the correction. The two are linked. Permissions determine who can perform the reversal. Repeating the reversal command does not create another reversal. The audit trail can answer both questions: what was posted originally, and how was it corrected later?
This sounds heavier than editing a row because it is. Financial history is not ordinary content. Reversibility is a domain feature.
The same principle applies beyond accounting. Refunds, voids, transfer failures and settlement corrections all become easier to reason about when correction is represented as an explicit event rather than an invisible mutation of the past.
Permissions are part of financial correctness, not an admin feature
A technically correct operation can still be a system failure if the wrong person can perform it or if it escapes the scope they are supposed to operate inside.
The manual posting work separated read, post and reverse permissions instead of treating accounting access as one broad capability. That matters because oversight and execution are different responsibilities. A manager may need to inspect entries without being able to create them. A finance user may be allowed to post but not reverse. A reversal may need narrower authority because it changes the interpretation of already-posted history.
Branch scope mattered too. A user operating inside one location should not be able to reverse or inspect a financial record from another simply because they can guess an identifier or construct a direct URL.
That last class of issue is easy to miss because the visible navigation can look correct. The real test is not whether the menu hides the button. It is whether the server refuses an out-of-scope operation even when the request is constructed directly.
This is one reason I think "enterprise software" is often described too superficially. More modules do not make a system enterprise. Authority, separation of duties, scope, auditability and controlled correction do. I wrote more about that in Your Business Software Is Not Enterprise Because It Has More Features.
Money movement needs one authoritative price and one authoritative debit
The second production system exposed the same engineering ideas through transfers rather than accounting vouchers.
A transfer flow can look simple from the user side: choose destination, enter amount, review fee, confirm. But there are several numbers hiding inside that journey. The requested amount, provider fee, platform fee, total debit, expected settlement amount and resulting balance may all differ.
If the browser calculates one version while the server calculates another, the system has created two competing truths. The review screen can show an amount that is not the amount actually authorized. A stale client can submit an old fee. A manipulated request can attempt to lower the debit. A configuration change between quote and confirmation can produce ambiguity.
The fix was to make transfer pricing server-authoritative. The server creates the quote, itemizes the components, and determines the exact total debit that authorization covers. The client displays that quote rather than inventing financial arithmetic independently.
That architecture sounds obvious after the fact, but it is a useful general rule: when a number affects money movement, the backend should be able to explain exactly where it came from and exactly which amount the user authorized.
Reconciliation is part of the transaction, not a cleanup job
External financial systems create a difficult category of state: the application sends a request, but local knowledge and external truth can temporarily diverge.
A timeout does not necessarily mean failure. A successful API response does not necessarily mean final settlement. A webhook can arrive after the original request path has already ended. An account-list endpoint can expose identity information without exposing balance fields. A later status check can contain more authoritative information than an earlier local projection.
The system therefore needs a hierarchy of evidence.
Provider-confirmed transaction state should be allowed to repair local uncertainty. Confirmed settlement should update the internal balance model exactly once. Reconciliation should be idempotent because the same external event may be delivered or checked repeatedly. The system should preserve enough identifiers to match an external event to the local operation it belongs to.
One of the recent bugs came from treating a stale relationship projection as if it were stronger evidence than a newly confirmed external balance. That blocked legitimate transfers even though the system had more recent proof that funds existed.
The fix was not to "trust the provider everywhere." It was to define which external observations are authoritative for which decisions, when they become durable local evidence, and how asynchronous projections are prevented from overriding fresher facts.
That distinction matters. Reconciliation is not a nightly cleanup script. It is one of the mechanisms by which a distributed financial system decides what is true.
Small numeric details become financial bugs very quickly
Another set of issues came down to minor units.
Financial APIs frequently represent amounts as integers in the smallest currency unit. That is a good storage and transport convention because it avoids floating-point ambiguity. It becomes dangerous when one layer forgets which unit it is holding.
A minimum transfer amount stored as 50,000 minor units can accidentally be shown to a user as "50,000" major units. A verification comparison can reject a valid transaction if one side is in major units and the other is in minor units. A notification can report the wrong amount even when the underlying ledger is correct.
These are not sophisticated algorithmic failures. They are representation failures. But financial engineering is full of cases where representation is the feature.
I prefer explicit names and conversion boundaries for this reason. If a value is in minor units, the code should make that hard to forget. Formatting belongs at the presentation edge. Ledger arithmetic should use exact integer values. Comparisons should happen only after both operands are known to share the same unit.
Correctness does not excuse a bad operational interface
The same weeks of work also included changes that look much smaller: showing product images in search results, making item identifiers visible in cart lines, adding controlled exports, improving report pagination, and correcting interface behaviour that hid information staff needed at the point of sale.
I do not see those as separate from the "serious" backend work.
Operational software is used under time pressure. A cashier trying to distinguish similar items benefits from an image and visible part number. A manager reviewing low stock needs bounded pagination rather than an overloaded page. An export action needs permission checks because data leaving the system is still an authorization event.
There was even a useful reminder in a small display bug. A newly added item identifier existed in the markup but remained invisible because an older shared CSS rule happened to match the same utility class. A regression test confirmed the text existed in the document, yet the browser still hid it.
That is another production lesson: tests have to verify the property users actually depend on. "The markup contains the value" is not the same claim as "the user can see the value."
The most valuable tests now sit at the boundaries
As these systems mature, the tests I care about most are increasingly the ones that force boundaries to disagree.
Post the same command twice. Submit the same idempotency key with a different payload. Close an accounting period while another process is preparing to post. Reverse the same journal twice. Attempt the action from the wrong branch. Make two processes settle the same operation concurrently. Deliver the same external event again. Return an external response without the field the happy path expected. Confirm settlement after the original request timed out. Format a minor-unit value at the UI boundary. Hide a visible element with the actual CSS cascade.
Those tests are less glamorous than a screenshot of a new feature, but they are much closer to the failures that production eventually creates on its own.
One concurrency hardening pass also exposed something unrelated but instructive: a local permissions problem in the test storage directory had been causing several suites to fail for reasons that looked like application defects. Once the environment issue was corrected, the full suite could finally run cleanly.
That was a good reminder that the test environment is part of the system too. A failing test is evidence, but it still has to be diagnosed.
What I am taking forward: preserve truth under pressure
The two systems are different enough that it would be easy to write separate stories about them. One deals with branches, inventory, sales, accounting and staff operations. The other deals with accounts, balances, transfers and asynchronous financial events.
But the engineering work converged.
Use one financial source of truth. Features should converge on the same ledger and accounting contracts rather than inventing parallel balance logic.
Make important operations idempotent at the database boundary. Retries and concurrent requests are normal behaviour, not exceptional behaviour.
Represent correction explicitly. Reversals and compensating events preserve history better than silent mutation.
Define authority. The browser, local projection, background worker and external system should not all be equally trusted for every decision.
Keep units exact. Money should not depend on ambiguous numeric representation.
Treat authorization as part of domain correctness. The right result produced for the wrong actor is still wrong.
Test the races and disagreements. The happy path proves very little about a system that has to survive retries, timeouts and asynchronous state.
Do not neglect the operator. Correct backend state still has to be understandable and usable by the person standing in front of the software.
The phrase I keep coming back to is simple: preserve truth under pressure. That is what production engineering becomes once software starts carrying real consequences. The system has to keep telling the truth when the network is unreliable, when two requests arrive together, when a user clicks twice, when an external service responds late, when a correction is needed and when the current number has to be explained months later.
That is the work behind the feature list, and increasingly, it is the part of software engineering I find most interesting.
Sources and further reading
- HTTP Semantics: idempotent methods
- Role-based access control model and resources
- Authorization design and verification guidance
- Currency representation and minor-unit standard
- Computer security log management guidance
- Why Every Serious Platform Eventually Needs a Ledger
- The Most Dangerous Payment Bug Is the One That Succeeds Twice
- Your Business Software Is Not Enterprise Because It Has More Features
- From CRUD App to Operating System: What Changes When Software Runs a Real Business
