Data Architecture

Code is relatively easy to change. Data is not. A botched function can be rewritten in an afternoon; a botched data model can haunt a system for years, because by the time you notice, there are terabytes of it and a dozen things reading it. This is why data decisions deserve more deliberation than almost any other technical call — they sit closest to the "irreversible" end of the spectrum we'll discuss in Chapter 6.

This chapter covers the ideas you'll lean on most: who owns data, how consistent it needs to be, the tradeoff the CAP theorem forces on you, and how to pick a store.

Data ownership

The first question for any piece of data is who owns it — which part of the system is the single source of truth and the only thing allowed to write it. Everyone else reads through that owner's interface or consumes copies it publishes.

This matters because shared write access is the tightest coupling there is. The moment two services both write the same table, neither can evolve the schema without the other, you get write conflicts, and the "boundary" between them is fiction. Clear ownership — one writer, well-defined read access for everyone else — is what makes boundaries real. (This is the data-side restatement of the coupling lesson from Chapter 2.)

Consistency models: strong vs eventual

When data lives in more than one place (replicas, caches, multiple services), you face a question that has no free answer: when I write, when does everyone else see it?

Strong consistency means every read returns the most recent write. After you update your profile, every subsequent read — anywhere — sees the new value. This is what a single relational database on one machine gives you, and it's the easiest model to reason about. The cost is coordination: to guarantee everyone agrees, the system must synchronize, which adds latency and limits availability.

Eventual consistency means replicas converge to the same value given enough time, but a read shortly after a write might return a stale value. This is the price of distributing data for scale and availability. It's perfectly acceptable for a huge class of problems — a like count that's off by one for two seconds harms nothing — and unacceptable for others (you can't be "eventually consistent" about whether a payment succeeded).

The lead's job is to ask, per piece of data, how much staleness the business can actually tolerate. The answer is rarely "none everywhere." Treating all data as needing strong consistency is a common and expensive over-engineering mistake; so is treating money or inventory as if eventual consistency were fine.

The CAP theorem

The CAP theorem formalizes the tradeoff for any distributed data store. It says that during a network partition — when nodes can't all talk to each other, which will happen — you must choose between two of three properties:

• Consistency — every read sees the latest write.
• Availability — every request gets a (non-error) response.
• Partition tolerance — the system keeps working despite dropped messages between nodes.

The catch people miss: in a distributed system, partitions are not optional — networks fail. So P is a given, and the real choice during a partition is between C and A. You can't have both.

CAP is a blunt instrument (the real world has a richer model called PACELC, which adds: even when there's no partition, you trade latency against consistency). But CAP is the right mental starting point: for this data, when the network breaks, would I rather return a possibly-stale answer or an error? Different data in the same system can answer differently.

Choosing a database

The "relational vs NoSQL" framing is mostly unhelpful. Better to match the shape and access pattern of your data to a store designed for it. A rough field guide:

Two pieces of practical guidance worth more than the table:

Start relational unless you have a concrete reason not to. Mature relational databases are flexible, transactional, well-understood, and scale far further than people assume. Reaching for an exotic store early is a frequent over-engineering trap. Choose a specialized store when you have a specific access pattern or scale that the relational option genuinely can't serve.

You'll likely end up with more than one (polyglot persistence): a relational store for core transactional data, a key-value store for caching, maybe a search index and a time-series store for metrics. That's fine — as long as each piece of data has one clear owner.

The pattern that bites everyone: the shared database

It's worth its own warning because it's so common. When you split a system into services but let them share one database, you have built a distributed monolith — the worst of both worlds. You pay the operational cost of distribution (network, multiple deploys) while keeping the tight coupling of a monolith (no one can change the schema independently). The whole benefit of splitting evaporates.

flowchart TB
  subgraph anti["❌ Distributed monolith"]
    SA[Service A] --> SDB[(Shared DB)]
    SB[Service B] --> SDB
    SC[Service C] --> SDB
  end
  subgraph good["✓ Service-owned data"]
    GA[Service A] --> DA[(DB A)]
    GB[Service B] --> DB2[(DB B)]
    GA -->|"API / events,<br/>never direct DB"| GB
  end

If you split, each service owns its data. Other services get to that data through the owner's API or through events it publishes — never by reaching into its tables.

What this means for you as a lead

Push hard on two questions in any data design review. First: what's the single owner of this data? Shared ownership is a smell that predicts pain. Second: how stale can this specific data be before the business is harmed? Forcing that question per-dataset prevents both the "everything must be strongly consistent" over-engineering and the "we'll be eventually consistent about money" disaster. And remember that data decisions are among the hardest to reverse — this is exactly where the deliberation budget from Chapter 6 should go.