Performance and Scale

Latency is elapsed time for one operation, measured from a defined start to a defined finish. Throughput is the number of completed operations per unit of time. They answer different questions: a service can complete many requests each second while some individual requests remain slow. Clear boundaries matter because browser latency includes network and rendering work that server processing latency does not.

As incoming work increases, resources such as CPU, connections, or workers become busier. Near capacity, requests wait in queues before processing, so latency can rise sharply even if each operation's service time is unchanged. Throughput eventually reaches a ceiling. Measuring offered requests, completed requests, errors, queue time, and active work reveals whether the system is keeping up or merely accumulating a backlog.

For example, an image service completes 500 images per minute with two-second median latency. A burst of 700 per minute grows its queue and pushes completion latency higher while throughput remains near 500. In practice, define user-visible latency goals, measure throughput by meaningful operation, test behavior near expected peaks, and include failures rather than counting only successful work.

A latency percentile is a value below which a percentage of observations fall. If the 95th percentile is 800 milliseconds, 95 percent of measured operations finished in 800 milliseconds or less, while 5 percent took longer. Percentiles exist because one average can hide a slow minority. They should always include a time window, operation, population, and measurement boundary.

A bottleneck is the resource or stage that currently limits throughput or dominates delay. To locate it, split a request into timed stages and correlate latency with resource saturation, queueing, errors, and workload type. Compare percentile distributions rather than one anecdote. Then change one likely constraint and repeat the same measurement; if the limit moves, the evidence supports the diagnosis.

An API has a 120-millisecond median but a two-second 99th percentile. Traces show that requests with a large account history perform repeated database calls and occupy all connection slots. Practical work segments those requests, measures database wait and query time separately, fixes the repeated access, and verifies both tail latency and resource utilization under the same load.

Vertical scaling gives a single node more resources, such as CPU cores, memory, faster storage, or network capacity. It exists because a larger node can handle more work without changing how requests are distributed. This is often the simplest response when one instance is resource-limited and a larger reliable size is available.

First identify the saturated resource. Choose a larger instance that increases that resource, verify software and license limits, then benchmark a representative workload. Plan the resize or replacement, observe capacity and latency afterward, and retain a rollback path. Added resources help only if the workload can use them; a single-threaded CPU path may not benefit from many additional cores.

A database repeatedly reaches its memory limit and reads hot pages from slow storage. Moving to a node with enough memory may sharply reduce read latency without partitioning data. In practice, estimate cost per unit of useful capacity, leave headroom for bursts and failover, test restart or migration time, and document the maximum viable node size so vertical scaling is not mistaken for an unlimited strategy.

Horizontal scaling adds service instances rather than enlarging one. A load balancer accepts client traffic and selects a healthy instance for each request. This exists to increase aggregate capacity, replace failed instances, and permit gradual deployment. It works best when instances are interchangeable and essential state is held in shared durable services.

Instances register with or sit behind the load balancer. Health checks remove instances that cannot safely serve traffic, and a routing algorithm spreads new requests. When load rises, automation can add instances; when it falls, instances can drain existing work and stop. The database or another downstream service must also support the resulting concurrency, or horizontal application scaling merely moves the bottleneck.

If one API instance safely handles 100 requests per second, four may approach 400 under a suitable workload, minus coordination and downstream limits. In practice, make shutdown graceful, separate process liveness from readiness to serve, set per-instance concurrency limits, test instance loss, and avoid session data that silently pins a user to one machine.

A cache stores a result so a later request can reuse it instead of repeating expensive work. A browser cache keeps responses near one user; an application cache keeps computed or fetched values near server code and may serve many users. Caches exist to lower latency, reduce downstream load, and sometimes continue serving safe older data during a brief dependency problem.

A request first constructs a cache key and checks for a valid entry. On a hit, it returns the stored value; on a miss, it computes or fetches the value, stores it with an expiration, and returns it. Browser response headers control reuse and revalidation. Application caches also need rules for invalidation, size eviction, concurrent misses, and failure. Every key must include all input that changes the result, including authorization scope.

A product catalog page can cache public item details for five minutes, while a user's private price agreement requires a user or account-specific key and possibly a shorter lifetime. Practically, choose data with high reuse and acceptable staleness, measure hit rate and miss cost, prevent a flood of identical misses, and treat the source store rather than the cache as the durable authority.

A content delivery network, or CDN, is a distributed set of edge servers that receives requests through a service's public host name. It exists to serve reusable content closer to users, reduce traffic to the origin server, and absorb large request volumes across many locations. A CDN is an independent network layer, not the same thing as a browser cache or an application cache.

DNS or network routing directs a user to a nearby edge. The edge evaluates its cache key and freshness rules. On a hit, it responds without contacting the origin; on a miss, it fetches from the origin, may store the response, and forwards it to the user. Cache-control headers, explicit purge operations, and versioned asset names govern freshness. Dynamic requests can pass through without being cached.

A site can publish app.4f2c.js with a content-based name and a one-year cache lifetime. New deployments use a new name, so no purge is needed, and users worldwide fetch it from nearby edges. In practice, cache public immutable assets first, keep authenticated or personalized responses private by default, monitor edge hit ratio and origin traffic, and test purge behavior before an emergency.

A measured performance plan connects a user outcome to a baseline, a diagnosed constraint, one proposed change, and a verification method. It prevents teams from applying fashionable techniques without knowing whether they address the limiting stage. The plan also states a capacity target and guardrails for correctness, cost, and reliability.

Start with a service-level goal and representative workload. Capture latency percentiles, throughput, errors, saturation, queueing, cache behavior, and request traces. Locate the current bottleneck, then choose the smallest fitting intervention: tune work, scale a constrained node vertically, add stateless instances horizontally, cache repeated computation, or move public reusable delivery to a CDN. Predict an outcome, test safely, compare against the baseline, and keep, revise, or roll back.

Suppose an international catalog has a 2.5-second 95th percentile against a one-second goal. Traces show static assets cross an ocean while API processing remains under 200 milliseconds; a CDN and versioned assets are the first experiment, not database sharding. In practice, assign owners and dates, estimate cost, protect data freshness and access control, and repeat measurements because removing one limit often exposes the next.