Load balancer

Load balancers continuously monitor the health of backend servers. If a server becomes unresponsive, slow, or returns errors, the load balancer removes it from rotation and stops sending it traffic. This is how load balancers provide automatic failover: when a server fails, requests are rerouted to the remaining healthy instances with no visible impact on users.

Some applications require all requests from a particular user to be routed to the same backend server for the duration of a session — for example, applications that store session state in local memory rather than a shared store. Load balancers can achieve this using IP-based affinity (routing based on client IP address) or cookie-based tracking.

Handling SSL/TLS encryption and decryption is computationally expensive. A load balancer can offload this by decrypting incoming HTTPS traffic and forwarding unencrypted requests to the backend servers, reducing the burden on application code.

A load balancer itself introduces a potential [single point of failure]. To mitigate this, load balancers are typically deployed in redundant pairs or clusters — with standby instances ready to take over — often spread across multiple availability zones.

The simplest algorithm: requests are sent to each server in turn, cycling through the list. This works well when all servers are equally powerful and all requests are equally expensive. However, because it ignores the current state of each server, round robin can send requests to servers that are already overloaded, leading to poor tail latency (high 95th/99th percentile response times) even when the median is acceptable.

Round robin is still the default HTTP load balancing algorithm for Nginx.

An extension of round robin where each server is assigned a weight proportional to its capacity. More powerful servers receive more requests. Weights can be configured statically (by a human, based on known server specifications) or computed dynamically using a proxy metric such as observed response latency — if one server serves requests three times faster than another, it is assigned three times the weight. Dynamic weighting adapts to changes in server performance over time without manual configuration.

Rather than cycling through servers blindly, the load balancer tracks the number of active connections on each server and always sends the next request to the server with fewest. Because the load balancer sits between clients and servers, it has accurate real-time visibility into each server’s workload. This cuts through variance in both server power and request cost, and performs very well under overload — it only drops requests when there is literally no more queue space available. It is a great default choice for most workloads.

A variant that routes each request to the server with the lowest current response time. Similar in intent to least connections, but uses latency rather than connection count as the metric.

Hashes the client’s IP address to select a server, ensuring the same client is always directed to the same server. Useful for session persistence and stateful operations, though it can produce uneven distribution if client IP addresses are not uniformly distributed.

A more sophisticated algorithm that combines ideas from dynamic weighted round robin and least connections. For each server, it maintains a running weighted average of recent request latencies — where older measurements contribute exponentially less to the score — and multiplies this by the number of open connections. Lower scores are preferred. PEWMA can identify and stop sending requests to chronically slow servers entirely. It achieves better latency percentiles than least connections across the board, but at the cost of additional complexity and tuning parameters.