Cross-Cutting Concerns

Features are what the system does. Cross-cutting concerns — often called non-functional requirements — are about how well it does them under real conditions: when traffic spikes, when a dependency dies, when an attacker probes it, when something breaks at 3am and someone has to figure out why. These are the qualities that separate a design that demos beautifully from one that survives contact with production.

They're called "cross-cutting" because they don't live in one module; they touch everything. That's exactly why they're a lead's responsibility — no single feature owner is accountable for them, so if you don't hold the thread, nobody does.

Scalability

Scalability is the ability to handle more load by adding resources. Two flavors:

• Vertical scaling (scale up) — a bigger machine. Simple, but there's a ceiling and a single point of failure.
• Horizontal scaling (scale out) — more machines behind a load balancer. Effectively unlimited, but it imposes a design constraint: your services must be able to run as many identical copies.

That constraint is statelessness. If a service holds user session state in its own memory, you can't freely add copies — requests must return to the same instance. Push that state out (to a database, cache, or token) and any instance can serve any request, so scaling out becomes trivial. Stateless services scale; stateful ones fight you.

The other two big levers:

Caching — keep a copy of expensive-to-compute or frequently-read data somewhere fast. The highest-leverage performance tool there is, and also a top source of bugs (the famous "two hard things": cache invalidation and naming). Cache deliberately, with a clear answer for when entries expire or get invalidated.

Load balancing — distribute requests across instances. Enables horizontal scaling and provides failover when an instance dies.

flowchart TB
  U[Clients] --> LB{{Load Balancer}}
  LB --> S1[Stateless<br/>instance 1]
  LB --> S2[Stateless<br/>instance 2]
  LB --> S3[Stateless<br/>instance 3]
  S1 --> CA[(Cache)]
  S2 --> CA
  S3 --> CA
  CA -. miss .-> DB[(Database)]

A caution: don't design for scale you don't have. Building for ten million users when you have ten thousand is a classic way to spend complexity you can't afford on a problem you don't have yet. Design so you can scale the parts that will need it (statelessness, clear bottleneck awareness), but don't pre-build it all.

Reliability

Reliability is the system continuing to work correctly in the presence of faults — and faults are not edge cases, they're the normal weather of distributed systems. Key tools:

Redundancy — no single point of failure. If losing one component takes the whole system down, that component needs a backup.

Retries with backoff — transient failures (a brief network blip) often succeed on retry. But naive retries can cause outages: if everything retries instantly and in lockstep, you hammer a struggling service into the ground (a "retry storm"). Retry with exponential backoff and jitter (increasing, randomized delays).

Idempotency — an operation you can safely repeat with the same result. This is what makes retries and at-least-once message delivery safe: if "charge this payment" might be delivered twice, it had better not charge twice. Designing operations to be idempotent (e.g. with a unique request key) is one of the most important reliability skills in distributed systems.

Circuit breakers — when a dependency is clearly failing, stop calling it for a while instead of piling on requests that will time out. The breaker "opens" after repeated failures, fails fast for a cooldown, then "half-opens" to test recovery. This prevents one sick service from cascading into a system-wide outage.

stateDiagram-v2
  [*] --> Closed
  Closed --> Open: failures exceed threshold
  Open --> HalfOpen: after cooldown timer
  HalfOpen --> Closed: trial request succeeds
  HalfOpen --> Open: trial request fails
  note right of Closed: normal — calls pass through
  note right of Open: fail fast, don't call dependency

Graceful degradation — when something fails, lose a feature, not the whole product. If recommendations are down, show the page without them rather than erroring out entirely.

Security

Security is a whole discipline, but a lead needs working fluency in a few essentials and the instinct to bring in expertise for the rest.

Authentication vs authorization — these are different and constantly confused. Authentication is "who are you?" (proving identity). Authorization is "what are you allowed to do?" (checking permissions). A bug in either is a breach; conflating them in your head leads to bugs in the code.

Least privilege — every user, service, and credential gets the minimum access needed to do its job, and no more. This limits the blast radius when (not if) something is compromised.

Secrets management — credentials, API keys, and tokens never live in source code or config files in the repo. They live in a dedicated secrets manager, are rotated, and are injected at runtime. A secret in git history is a secret that's compromised.

Know the common vulnerability classes. You don't need to be a security researcher, but you should recognize the recurring ways web systems get broken — injection, broken access control, exposed secrets, and the rest of the well-documented common categories (the OWASP Top Ten is the standard reference). Enough to spot them in review and to run a basic threat-modeling conversation: "who might attack this, what are they after, and where are we exposed?"

Treat all input — from users, from other services, from anywhere outside your trust boundary — as hostile until validated. Most security failures are a failure to do exactly that.

Observability

You cannot operate what you cannot see. Observability is your ability to understand what the system is doing from the outside, and it rests on three pillars:

On top of these sits alerting — automatically notifying a human when something needs attention. The hard part of alerting is not alerting on everything; an alert that fires constantly gets ignored, and then the real one gets ignored too. Alert on symptoms users feel (error rate, latency), not on every internal metric.

The cultural point: observability is not something you bolt on after an outage. "We'll add logging later" is how outages become unsolvable mysteries. Build it in from the start, even minimally.

Performance

Performance covers two related things: latency (how long one request takes) and throughput (how many requests per second you can handle). They're related but distinct, and improving one can hurt the other (e.g. batching boosts throughput but adds latency).

The single most important rule of performance work is measure first. Engineers' intuitions about where the time goes are wrong with remarkable consistency. Profile, find the actual bottleneck, fix that, measure again. Optimizing code that isn't the bottleneck adds complexity and risk for no user-visible gain — and premature optimization frequently makes the code harder to change later, which is its own cost.

When you do report or reason about latency, prefer percentiles over averages. The average hides the users having a terrible time; the 95th and 99th percentile (p95, p99) tell you about the slowest experiences, which are often where the real pain — and the real bugs — live.

What this means for you as a lead

These concerns have no natural owner, so they quietly become yours. In design reviews, the most valuable questions you ask are usually here: "What happens when this dependency is down?" "How do we know if this is working in production?" "What's the blast radius if this credential leaks?" "Have we measured, or are we guessing?" A team that ships features fast but can't answer these is accruing a debt that comes due during an incident — and incidents are far more expensive than the review conversation that would have prevented them.