Docker Compose MySQL Keeps Restarting

Run into a strange bug where the MySQL container would keep restarting. The error I would see from the logs would be

[ERROR] --initialize specified but the data directory has files in it. Aborting.

You can view container logs by typing

docker logs -f <db_container_name>

I tried to rebuild the container by running

docker-compose build and docker-compose up -d but to no avail. Finally stumbled on a solution via a GitHub issue here. The solution was to run

docker system prune --volumes

which cleared out the said directory then after building the container worked.

Read this article

High Availability using Round Robin DNS

A couple of months ago I wrote about using Nginx as a load balancer to ensure uptime. However this still leaves a single point of failure e.g. if the load balancer goes down this means the client cannot reach your application servers. One solution to this is to have a secondary load balancer. How would such a setup work?

Round Robin DNS

To make use of this we would need to have multiple A records for a particular domain. For example say our endpoint is example.com we would need to have an A record pointing to load balancer 1's IP address and another A record pointing to load balancer 2's IP address. When a user types the domain name into the browser, DNS will return a list of IP addresses that point to the said resource, most Http clients will try the first IP in the list, if this is unavailable it will try the second one and so on. So if the first IP is unresponsive they will be directed to the second IP, all this of course is transparent to the user.

Sample Round Robin DNS Setup

Gotchas

Sometimes some http clients will not attempt to try the second IPs unless the user does a page refresh, other times because of a DNS cache the browser may not have the secondary IP available. DNS propagation is also something to consider. This method will not work for application level errors on your load balancer, thus it is important to have monitoring for the application health of your load balancers. Something else to consider is that you now have multiple servers to manage. For example now when generating SSL certificates you'll need to make them available to both load balancers.

Are there other techniques you are using to ensure HA for your applications?

Read this article

Load Balancing with Nginx

Nginx and Load Balancing

A lot can be said about Nginx (pronounced "engine X"). It is a web server that can be used as a reverse proxy or a load balancer. Basically a load balancer sits between the site visitor and the application servers (a pool of servers) acting like sort of cop directing traffic.

Load Balancing Algorithms on Nginx

There are different ways that traffic can be routed to your pool of servers:

  • Round Robin – Requests are distributed sequentially.
  • Least Connections – A new request is sent to the server with the fewest current connections to clients.
  • IP Hash – The IP address of the client is used to determine which server receives the request. Subsequent requests from the same IP will end at the same application server
  • Generic Hash - The server to which a request is sent is determined from a user‑defined key

There are other algorithms available for Nginx Plus which is the commercial version

Sample Load Balancer Setup

Sample Load Balancer Configuration

Using a round robin configuration:

http {
    upstream backend {
        server application.server.one;
        server application.server.two;
        server application.server.three backup;
    }

    server {
        location / {
            proxy_pass http://backend;
        }
    }
}

In this setup traffic will be routed this way assuming the visits are sequential:

  • Visitor one directed to application server one
  • Visitor two directed to application server two
  • Visitor three directed to application server one

In the event that both servers are unavailable traffic is directed to application server three. This allows for some high availability.

We can pass some extra headers in our location block for example X-Real-IP this will help our application servers to know the source traffic IP:

location / {
  proxy_pass http://backend;
  proxy_set_header        Host            $host;
  proxy_set_header        X-Real-IP       $remote_addr;
  proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
}

Sample Application Server Configuration

Suppose we are using Nginx on our Application servers we can get the source traffic IP and not the load balancer IP by using this sample configuration:

server {
    listen 80;
    .
    .
    location / {
        set_real_ip_from loadbalancer.server;
        real_ip_header X-Forwarded-For;
        .
        .
    }
}

Other considerations

What happens if our load balancer was to go down? How can we address having a single point of failure? How does our load balancer determine that an application server is unavailable not offline but for example if the server is returning an error 500? We will discuss possible solutions for that in future article.

Read this article