Scaling, Load Balancing, and Stateless Services

The short answer

Scale a system when a measured or well-supported requirement exceeds its current capacity. You might give one machine more resources (vertical scaling) or add instances that can share work (horizontal scaling). A load balancer can direct traffic to eligible instances. Neither option fixes a bottleneck elsewhere.

The previous lesson showed how to identify the limiting boundary. This lesson asks what changes when that boundary is the application tier.

Why systems scale

Growth may increase request rate, data size, background work, or the cost of individual operations. First determine whether the issue is application CPU, database time, a connection pool, network capacity, or a different boundary. Sometimes a query improvement or a larger single instance is enough. Scaling is a response to a constraint, not an architectural badge.

Vertical scaling

Vertical scaling gives an instance more CPU, memory, or faster storage. It can be the simplest valid move, especially when the application is hard to divide. Its limits are practical and economic: larger machines are finite and can cost disproportionately. Replacing an instance may involve downtime depending on the environment, and one larger instance is not automatically redundant.

Horizontal scaling

Horizontal scaling adds instances. It can increase capacity for work that can be distributed and can allow traffic to avoid a failed instance. It also introduces routing, health checks, shared-state decisions, deployment coordination, and cost. Ten identical application instances do not make a single database ten times faster.

What a load balancer does

A load balancer selects among eligible destinations. An HTTP-aware one may inspect request details; a transport-level one operates at a different layer. DNS may return different entry addresses, but a DNS answer is not the same as an HTTP routing decision. A reverse proxy can also balance traffic: NGINX's documentation describes its HTTP balancing behavior.The reverse-proxy lesson covers the proxy boundary in more detail.¹

Traffic-distribution strategies

Round robin cycles through destinations. Least connections considers active connections. Weighted routing gives destinations different shares. Hash-based or affinity routing uses a chosen key to keep related requests together. NGINX documents these as concrete HTTP-balancing methods, with different implementation details and availability by product edition.No algorithm is universally best; long requests, unequal machines, locality, and session behavior can change the choice.¹

Health checks

“Process is alive” and “ready to receive traffic” are not the same. A service can be running while still loading configuration, or alive but unable to serve its critical route. Kubernetes distinguishes liveness, readiness, and startup probes: failed readiness can remove a pod from service endpoints while failed liveness can trigger a restart.Those are Kubernetes mechanisms, not a requirement to use Kubernetes. An expensive or overly strict check can itself make an incident worse.²

Stateless services

A stateless application instance does not depend on its own process memory as the only authoritative place for information needed across independent requests. The application still has data; it may use a database, shared session store, object storage, or another external state system. Google Cloud describes statelessness as one way to make instances easier to replace and scale.Moving state outward does not remove its cost or failure modes.⁴

Stateful services

Some work genuinely holds local state, such as a long-running computation or an in-memory data partition. Those systems can scale, but may need placement, replication, recovery, or routing rules. “Stateful” is not a defect. The important question is where the authoritative state lives and what happens when an instance disappears.

Sessions and sticky routing

If login session data lives only in instance A's memory, a later request routed to B may not find it. Sticky routing can repeatedly send the same client to A, but may complicate balancing and failover if A dies. A shared session store is another option. Tokens have different revocation and security tradeoffs; a JWT is not an automatic scaling fix. The authentication and sessions lesson explains those models.

Shared state

Shared storage lets different instances see the state needed to serve a request, but it becomes a dependency with capacity, latency, and availability limits. Clarify which data must be shared, how current it must be, and what happens during a store outage. A local cache may be disposable while the database remains authoritative; another workload may require stronger coordination.

Autoscaling

Autoscaling changes capacity in response to a signal or schedule. It needs a relevant metric, minimum and maximum capacity, time for new instances to become ready, and controls against unstable oscillation. AWS's Application Auto Scaling concepts illustrate metric-driven policies, bounds, and cooldown behavior.Those details are provider-specific; the general lesson is that scaling is delayed, costs money, and cannot repair a saturated dependency automatically.³

Downstream limits

Each added instance may open more database connections, make more cache requests, or call an external API more often. A new bottleneck can appear downstream. Google SRE's overload guidance notes that balancing alone cannot prevent every overload condition.Capacity plans must include the dependency path, not only application instance count.⁵

Scaling reads and writes

Read and write workloads can differ. A cache might reduce some repeated reads, while writes still need their persistence and consistency rules. Read replicas, partitioning, and queues are possible later choices, not automatic additions to this diagram. They introduce freshness, ordering, and recovery questions that belong in later lessons.

Failure behavior

Removing a failed instance from routing helps only if another instance has capacity and required state. A balancer or shared store can itself fail. A design needs realistic failure boundaries, not just duplicate boxes. Check readiness, dependency health, traffic behavior during replacement, and whether users see errors while capacity changes.

Common misconceptions

Debugging scenario

Why this matters when reviewing AI-generated designs

An AI suggestion to “scale horizontally” may omit local session state, database connection limits, external quotas, startup time, health checks, and operational cost. Ask which tier is constrained, whether requests can be distributed, where state lives, and what becomes the next limit. If the answers are unknown, measure before adopting the diagram.

Knowledge check

What to learn next