Configuring File Replication

March 23, 2011 under The 100% Uptime Challenge

This is probably one of the easiest tasks. I already installed the rsync program earlier, and I’ve already got key based SSH logins configured, so all I have to do now is install a cron job to sync my webroot from my nominated master ( to the other 2 servers:

crontab -e
* * * * * rsync -e ssh -a --delete ie:/var/www/html /var/www/

And we’re done! ┬áTo do a manual sync, just run

rsync -e ssh -a -v ie:/var/www/html /var/www/

I’ve installed this crontab in my own user’s crontab, making sure my user account has the necessary permissions to read and write all the files in the webroot. While probably best not to run the sync from root’s crontab, it can be run from any user who has the necessary privileges.

Multi-way file replication

I have only accounted for file writes on one master server, with updates being synchronised one-way to the other servers in my pool. Situations may also arise whereby file uploads need to be processed and the files stored on the servers. Here are two possible solutions to this problem:

Storing files in blobs: The blob type allows binary data to be stored in a MySQL table. Taking advantage of this, you could store uploaded files in a MySQL table instead of writing them to disk, letting MySQL replication (which is already running) take care of propagating the data across all servers.

Unique filenames: Taking a lesson from our multi-master MySQL configuration, the same technique we used to ensure key uniqueness across the server pool could equally be applied as a file naming convention. In my example, file uploads could be prefixed with the location of the server. So a file uploaded to the Irish server would have an ie_ prefix, to the Dutch server would have nl_, or to the American server, us_. rsync could then be configured on each server to replicate the uploads directory to all the other servers in the pool.

comments: 0 » tags: , ,