Circuit breaker

The circuit breaker [pattern] is a way to handle transient failures in distributed software.

When requests to services fail, a circuit breaker will automatically open, preventing any further requests from being sent to the failing service. The circuit breaker will remain open for a configurable amount of time, after which it will automatically close, allowing further requests to be sent again to the service.

Circuit breakers complement retry mechanism. Whereas a retry mechanism will keep retrying a failing service until it succeeds, a circuit breaker will trip when the error rate exceeds a configured threshold and it will prevent any further requests from being sent to the failing service for a configured period of time.

Both are solutions for improving fault tolerance, and the two solutions work particularly well in combination. If a service is failing, it is likely that it will continue to fail for a period of time, until its failure mode is resolved through disaster recovery mechanisms. While disaster recovery is ongoing, a circuit breaker will help to prevent the recovering system from being overwhelmed by retries.

Example implementation (pseudo-code)

public class CircuitBreaker
{
  private int failureCount = 0;

  /**
   * Number of failures before tripping.
   */
  private int failureThreshold = 5;

  /**
   * Time to wait before allowing requests again.
   */
  private TimeSpan timeout = TimeSpan.FromSeconds(30);

  private DateTime lastFailureTime;

  public bool execute(Func<bool> action)
  {
    if (isOpen())
    {
      /* Short-circuit the request. */
      return false;
    }

    try
    {
      bool result = action();
      reset();

      return result;
    }
    catch (Exception)
    {
      failureCount++;
      lastFailureTime = DateTime.Now;

      return false;
    }
  }

  private bool isOpen()
  {
    if (failureCount >= failureThreshold)
    {
      if (DateTime.Now - lastFailureTime > timeout)
      {
        /* Move to half-open state. */
        failureCount = 0;

        return false;
      }

      return true;
    }

    return false;
  }

  private void reset() => failureCount = 0;
}

public class OrderService
{
  private CircuitBreaker circuitBreaker = new CircuitBreaker();

  public void placeOrder(Order order)
  {
    // Logic to handle order placement...

    bool paymentSuccessful = circuitBreaker.execute(() =>
        paymentService.processPayment(order.paymentDetails));

    if (!paymentSuccessful)
    {
      // Handle payment failure, eg., notify user, log error, etc.
    }
    else
    {
      // Continue handling the order...
    }
  }
}