BIND DNS Server And Zone File Configuration

March 26, 2011 under The 100% Uptime Challenge

BIND is the standard for DNS services on Unix operating systems. It is somewhat complex and includes many useful features, but for this configuration I’m going to keep it really simple.

I already installed BIND using yum earlier on. On CentOS it’s installed from the ‘bind’ package, but the service is (somewhat illogically) called ‘named’. If you’re also going to use the server for local DNS resolution (by putting 127.0.0.1 in my /etc/resolv.conf) you’ll need a copy of the root server cache file, so BIND can look up external names. You’ll also need this if any records in your DNS zone refer to an external DNS name, for example a CNAME record pointing to another domain. Otherwise BIND will not be able to recurse and the lookup will fail.

So to get started:

# enable the service
chkconfig named on

# save the DNS root server info to /etc/db.cache:
wget -O /etc/db.cache http://www.internic.net/zones/named.cache

# create a config file for named:
# see my config file for an example
# at a minimum, edit the zone entry to match your domain
vi /etc/named.conf

# create the DNS zone file
# see my zone file for an example
# edit to match your domain and server IPs
vi /var/named/data/cwik.ch.hosts

# start the server
service named start

My sample config files are a very simple, minimal configuration. You may want to elaborate further. For further reading, check out the BIND documentation and/or the excellent DNS and BIND from O’Reilly.

In my setup, I have configured the 3 DNS servers in master/master/master configuration, because I wanted full control of the zone file on each server to be able to test different configurations. I also want to experiment later on with putting some intelligent health checks into the DNS system, whereby each server will check the availability of the other servers, and only return the IPs of those servers it can reach. The hope is that using this technique I can prevent the IP of an unreachable server being returned to the client, thus further increasing uptime.

Back to the BIND configuration, a little explanation on my config files:

DNS recursion: I’ve only listed the loopback interface under allow-recursion{} in the options{} block of named.conf, so only my local system can issue recursive queries. You may want to add the IPs of each of your servers so they can use each other as backup resolvers. In my case I have used the DNS resolvers provided by my upstream providers as secondaries in my /etc/resolv.conf file.

Zone configuration: I have instructed each server to be a master for my zone file. I will therefore have to propagate updates to my zone file to all my DNS servers manually. Another (more usual) way to configure BIND is to nominate one server as your primary and instruct the other servers to slave your zone. In this configuration, zone updates are propagated automatically, but you lose the ability to modify the zone file on each resolver independently, which is why I have used 3 masters.

Zone file: My zone file is for the most part pretty standard: a default TTL, an SOA record, NS and A records for each of my nameservers and an MX record. I’ve also set up an SPF record to help ensure my mail gets through the more aggressive spam filters.

The interesting part is the configuration for www.cwik.ch, this blog. I’ve created 3 A records at the root of the domain, ie. http://cwik.ch/ – one for each of my servers. I’ve then created www.cwik.ch as a CNAME (Canonical NAME – ie. an alias) of cwik.ch, so any DNS query for www.cwik.ch will return the 3 IPs from the root of the domain. This is the foundation on which this whole 100% uptime project is based.

A test query shows it is working perfectly:

$ host -t A www.cwik.ch
www.cwik.ch has address 67.202.99.74
www.cwik.ch has address 80.93.25.175
www.cwik.ch has address 83.96.156.169

On the left hand side of this page, under MySQL replication status, you should find a small indicator showing which server you are currently connected to. It’ll be one of the above IPs, depending on which one your browser picked.

The glue: Once I had all 3 nameservers set up correctly, I went to SWITCH, the Swiss domain registry, and changed the listed nameservers for cwik.ch to my new setup. The process to do this varies by the registry you use, but all registries offer the facility. There is an extra step involved which is not necessary if you are using someone else’s nameservers: setting up the glue record. This is the mapping of your DNS server names to their IPs, which must be done statically at your registrar. In my case I could do it via a simple web interface as depicted here:

DNS glue

Each of the 3 servers need to be defined and the IP hard-coded. This IP is stored in the .ch root so other resolvers know how to recurse to the cwik.ch domain. If I ever change the IP of one of my servers, I’ll also need to update it at the registry.

Propagating zone updates: I’ve already explained why I chose not to configure a master/slave setup, choosing instead to have 3 masters. However this means I need to update the zone file manually on all 3 servers in order to make any changes. That’s not very convenient, so I wrote a small shell script to do it for me. Feel free to copy/use it if you find it’s useful.

comments: 0 » tags: , , , , ,

Setting Up Multi-Master Circular Replication with MySQL

March 22, 2011 under The 100% Uptime Challenge

Circular replication diagramThe goal of this configuration is to allow each server to operate autonomously for a period of time should it ever lose contact with the other 2 servers, while doing everything I can to make sure there will be no conflicting database updates once connectivity resumes. Each server will act as both a Master and a Slave with replication going in a circle. The binary log, which is written by the master and read by the slave, contains the name of the MySQL server which wrote it. Therefore a loop will never happen, as once the query comes back around to its original master, the master will know that it has already been executed.

I’ll attempt to prevent key conflicts by using the auto_increment_offset and auto_increment_increment configuration variables, to ensure AUTO_INCREMENT values will be unique to each server. This may also prove handy as I’ll be able to tell which server created a record, just by looking at the key value.

There is one remaining potential problem which I can’t guarantee won’t happen, that is the problem of the order of updates. Based on my lovely sketch of the replication circle, consider this scenario:

  • nl.cwik.ch loses connectivity to us.cwik.ch, interrupting replication
  • user A updates a post on nl.cwik.ch: UPDATE posts SET title=’Some new title’ WHERE id=123;
  • the post does not get replicated to us.cwik.ch because the link is down
  • later, user B updates the same post on us.cwik.ch: UPDATE posts SET title=’Some other title’ WHERE id=123;
  • user B’s update gets replicated around the circle to nl.cwik.ch
  • the connection between nl and us comes back up
  • the original update statement from nl.cwik.ch gets replicated to us and nl servers
  • we now have inconsistent data:
    • title on nl: Some other title
    • title on us: Some new title
    • title on ie: Some new title

It’s an unlikely scenario, but plausible. One potential workaround would be to design your application such that records never get updated, but instead create a new record for every change. You can then always select the newest record based on a timestamp column, so in the above scenario all servers would show ‘Some other title’ as the newest blog post title.

Another possibility, should you consider the likelihood of this scenario limited for your application, would be to store a checksum of each row and compare the checksums across all the servers. Any inconsistencies could be flagged and/or an administrator notified.

I already installed mysql-server earlier, so let’s get down to the configuration!

Step 1: Enable the MySQL service and create a config file. I’ve posted my config as an example. This will need to be done on all 3 servers.

chkconfig mysqld on
vi /etc/my.cnf (use my config as a starting point, or create your own - not forgetting to set
server_id and auto_increment_offset appropriately)
service mysqld start

Step 2: Log in to your new MySQL server using ‘mysql -u root’

Step 3: Set your root password and configure your replication user (do this on each server):

USE mysql
DELETE FROM user WHERE user='';
UPDATE user SET password=PASSWORD('your new root password') WHERE user='root';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'80.93.25.175' IDENTIFIED BY 'somethinglongandrandom';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'83.96.156.169' IDENTIFIED BY 'somethinglongandrandom';
GRANT REPLICATION SLAVE ON *.* TO 'repl'@'67.202.99.74' IDENTIFIED BY 'somethinglongandrandom';
FLUSH PRIVILEGES;
SHOW MASTER STATUS;
+---------------------+----------+--------------+------------------+
| File                | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+---------------------+----------+--------------+------------------+
| mysql-bin-ie.000003 |      685 |              |                  |
+---------------------+----------+--------------+------------------+

Now on each server, use CHANGE MASTER to start slaving the next server in the circle, until each server is both master and slave. Use the log file name and position as shown in SHOW MASTER STATUS.

On nl.cwik.ch: CHANGE MASTER TO MASTER_HOST='ie.cwik.ch', MASTER_USER='repl',
  MASTER_PASSWORD='somethinglongandrandom', MASTER_LOG_FILE='mysql-bin-ie.000003', MASTER_LOG_POS=685;

On us.cwik.ch: CHANGE MASTER TO MASTER_HOST='nl.cwik.ch', MASTER_USER='repl',
  MASTER_PASSWORD='somethinglongandrandom', MASTER_LOG_FILE='mysql-bin-nl.000003', MASTER_LOG_POS=685;

On ie.cwik.ch: CHANGE MASTER TO MASTER_HOST='us.cwik.ch', MASTER_USER='repl',
  MASTER_PASSWORD='somethinglongandrandom', MASTER_LOG_FILE='mysql-bin-us.000003', MASTER_LOG_POS=685;

Step 4: Replication should now be up and running in a multi-master, circular configuration. Use ‘SHOW SLAVE STATUS\G’ on each server to check that it’s working properly. Check that Slave_IO_Running and Slave_SQL_Running both report Yes. Once it’s all working properly, create a test table to check that the AUTO_INCREMENT values are being created properly (with unique values per server):

mysql ie> USE test;
mysql ie> CREATE TABLE test (id INT NOT NULL AUTO_INCREMENT PRIMARY KEY, val VARCHAR(255) NOT NULL);
mysql ie> INSERT INTO test (val) VALUES ('I am Ireland');
mysql nl> INSERT INTO test (val) VALUES ('I am the Netherlands');
mysql us> INSERT INTO test (val) VALUES ('I am the USA');
mysql ie> INSERT INTO test (val) VALUES ('I am Dublin');
mysql nl> INSERT INTO test (val) VALUES ('I am Amsterdam');
mysql us> INSERT INTO test (val) VALUES ('I am Chicago');

Check the table contents and note the id values:

mysql> SELECT * FROM test;
+----+----------------------+
| id | val                  |
+----+----------------------+
|  1 | I am Ireland         |
|  2 | I am the Netherlands |
|  3 | I am the USA         |
| 11 | I am Dublin          |
| 12 | I am Amsterdam       |
| 13 | I am Chicago         |
+----+----------------------+
6 rows in set (0.00 sec)

Looking good! Note the records created by the Irish server always end in 1, the Dutch server always end in 2 and the American server always will end in 3.

comments: 12 » tags: , , ,
Subscribe