High Availability using Round Robin DNS

A couple of months ago I wrote about using Nginx as a load balancer to ensure uptime. However this still leaves a single point of failure e.g. if the load balancer goes down this means the client cannot reach your application servers. One solution to this is to have a secondary load balancer. How would such a setup work?

Round Robin DNS

To make use of this we would need to have multiple A records for a particular domain. For example say our endpoint is example.com we would need to have an A record pointing to load balancer 1's IP address and another A record pointing to load balancer 2's IP address. When a user types the domain name into the browser, DNS will return a list of IP addresses that point to the said resource, most Http clients will try the first IP in the list, if this is unavailable it will try the second one and so on. So if the first IP is unresponsive they will be directed to the second IP, all this of course is transparent to the user.

Sample Round Robin DNS Setup

Gotchas

Sometimes some http clients will not attempt to try the second IPs unless the user does a page refresh, other times because of a DNS cache the browser may not have the secondary IP available. DNS propagation is also something to consider. This method will not work for application level errors on your load balancer, thus it is important to have monitoring for the application health of your load balancers. Something else to consider is that you now have multiple servers to manage. For example now when generating SSL certificates you'll need to make them available to both load balancers.

Are there other techniques you are using to ensure HA for your applications?

Read this article

Hiking Mt. Satima

Dragon's Teeth

After months of lock-down in the city we got a chance to explore the outdoors. We settled on hiking Mt. Satima located in the Aberdares. Mt. Satima is the third highest point in the country standing tall at 4001 m. During this unique time we had to try and do what would be a 7-8 hour hike and be back in Nairobi before the 9 pm curfew. This meant we had to start early (4 am madness eh?).

There are two ways you can hike Mt. Satima one through Wandare gate or Shamata gate. Our contact person recommended the Shamata route as it was the more scenic option albeit a bit longer. We approached Shamata gate from Nyeri and entered the park through Rhino gate. To take this route travel up to Nyeri then take the Nyeri - Nyahururu road until a few kilometres before Wiyumiririe town. The distance from Rhino gate to the starting point for the hike (Twin Rocks) is about 12 km. We picked up our guide/ranger at the KWS outpost at Rhino gate.

The costs for such a trip is (2020 rates):

  • Park fee - Ksh. 250 (current discounted rate) per person
  • Guide fee - Ksh. 3015 per group
  • Vehicle fee - Ksh. 300 per vehicle (5 seater)

Suggested plan

  • Start trip at 4.00 am
  • Arrive at Rhino gate by 8.00 am
  • Start hike by 8.30 am from Twin Rocks
  • Summit by 2 pm or stop the hike at Dragon's Teeth if you hope to make it back before 9 pm
  • Finish descent by 4 pm to start trip back to Nairobi
  • Leave park by 4.30 pm for a 4 hour trip back to Nairobi
  • Alternatively plan to spend the night around Nyeri so there will be no mad rush to beat curfew

Stuff you need

  • A change of clothes (shoes, t-shirt, trousers, socks)
  • Warm clothes for the hike as it will get cold (several layers, plus covering for the head/face, gloves)
  • Hiking boots
  • Comfortable pair of trousers (avoid jeans)
  • Athletic t-shirts (the running ones)
  • Water (2 liters)
  • Snacks (sugary stuff to keep you going, trail mix)
  • A rain jacket
  • A camera (please carry an extra card don't be like me)
  • Identification documents

Pictures from the trip

Midway our journey I realized I carried a bad memory card meaning my camera was effectively useless (don't be like me) so I had to take the trip shots using my phone. In a way this was a good thing less weight plus no constant worry about missing the shot. All these shots were captured using my trusty Pixel 3a.

Mt Satima

Stuff to note

Nothing could have prepared us for how wet the ground was we literally at one point were wading through water. Our guide who had gum boots also got wet. I think the only way to prepare for this is to have an extra change of clothes including shoes and socks. It can also get really cold so dress warm without carrying too much weight. We couldn't get to the summit so because of time constraints so keep an eye out for your timing, if you realize you are running late you can terminate the hike at Dragon's Teeth. Don't carry unnecessary weight. When planning such we tend to overestimate what we will really need, keep it simple you'll have more fun that way. Don't forget to soak in the views.

Read this article

Load Balancing with Nginx

Nginx and Load Balancing

A lot can be said about Nginx (pronounced "engine X"). It is a web server that can be used as a reverse proxy or a load balancer. Basically a load balancer sits between the site visitor and the application servers (a pool of servers) acting like sort of cop directing traffic.

Load Balancing Algorithms on Nginx

There are different ways that traffic can be routed to your pool of servers:

  • Round Robin – Requests are distributed sequentially.
  • Least Connections – A new request is sent to the server with the fewest current connections to clients.
  • IP Hash – The IP address of the client is used to determine which server receives the request. Subsequent requests from the same IP will end at the same application server
  • Generic Hash - The server to which a request is sent is determined from a user‑defined key

There are other algorithms available for Nginx Plus which is the commercial version

Sample Load Balancer Setup

Sample Load Balancer Configuration

Using a round robin configuration:

http {
    upstream backend {
        server application.server.one;
        server application.server.two;
        server application.server.three backup;
    }

    server {
        location / {
            proxy_pass http://backend;
        }
    }
}

In this setup traffic will be routed this way assuming the visits are sequential:

  • Visitor one directed to application server one
  • Visitor two directed to application server two
  • Visitor three directed to application server one

In the event that both servers are unavailable traffic is directed to application server three. This allows for some high availability.

We can pass some extra headers in our location block for example X-Real-IP this will help our application servers to know the source traffic IP:

location / {
  proxy_pass http://backend;
  proxy_set_header        Host            $host;
  proxy_set_header        X-Real-IP       $remote_addr;
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
}

Sample Application Server Configuration

Suppose we are using Nginx on our Application servers we can get the source traffic IP and not the load balancer IP by using this sample configuration:

server {
    listen 80;
    .
    .
    location / {
        set_real_ip_from loadbalancer.server;
        real_ip_header X-Forwarded-For;
        .
        .
    }
}

Other considerations

What happens if our load balancer was to go down? How can we address having a single point of failure? How does our load balancer determine that an application server is unavailable not offline but for example if the server is returning an error 500? We will discuss possible solutions for that in future article.

Read this article