Lightning fast synch with csync2 and lsyncd _ Axivo Community.pdf

11/7/13

Lightning fast synch with csync2 and lsyncd | Axivo Community

Log in or Sign up

Home

Forums

Search Forums

Home

Members

Recent Posts

Forums

Developer Zone

Tutorials and Review s

Lightning fast synch with csync2 and lsyncd Introduction Credits: Axel Kittenberger, Gordan Bobic Support: Ask your questions related to this tutorial into Performance Drive forum If you find this tutorial useful, please post a link to it in your site. Floren Axivo Developer

We recently had a client who hired us to build from scratch his web site cluster. Among many performance problems, they were struggling with in their vBulletin based site, one issue being the file syncronization between the web nodes. The site does not contain any physical attachments and all the static files were manually uploaded by the owner through all 8 web servers. They tried in the past to synchronize the data through most popular solutions used by web hosts but they encountered performance issues and a SAN based solution was not in their budget. As result, they were forced to store the avatars into MySQL database, creating unnecessary stress at database level while affecting the overall site performance. So I had to come up with a solution that does not eat server resources like an elephant, while providing instant synchronization through an entire cluster. After some research done, I discarded instantly similar solutions like drbd, rsync and unison based on a variety of tests performed. The performance was not at the level I wanted to provide. The only client that performed flawlessly was csync2. I selected csync2 as prime synchronization candidate for the following reasons: 1. 2. 3. 4. 5.

ease of configuration setup and the large variety of automated tasks unlimited number of nodes to be synchronized great flexibility how the synchronization is performed level of security used for external nodes data storage of currently synced files in each node, for fast file comparison

https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/

1/11

11/7/13

Lightning fast synch with csync2 and lsyncd | Axivo Community

Now that the synchronization part was covered, I had to find a solution that allows csync2 to trigger a sync event, once a new file is present into any cluster node. Most Linux based operating systems have installed inotify in their kernel. Basically, inotify is a kernel subsystem that acts to extend filesystems to notice changes to the filesystem and reports those changes to an application, in our case csync2. With that established, I had to find a robust C based application that properly handled the inotify events and also could be daemonized, so I run it as Linux service. Again, I quickly discarded all Python or Perl based scripts, for the same reasons I mentioned earlier: speed and performance. The main candidates I selected were inotify-tools and lsyncd. They were both written on C and had options to run as daemon. Based on preliminary tests, I discovered inotify-tools would not satisfy my selection criteria, as its daemon features contained bugs. I even tried to daemonize myself inotifywait with a shell wrapper but the performance was just not there. The server load generated was too high, so I was forced to discard it and noticed Radu Voicila (inotifytools developer) about the issue. I selected lsyncd as prime inotify candidate for the following reasons: 1. runs as daemon, so it can be started as Linux service 2. does not hamper local filesystem performance like other similar solutions 3. aggregates and combines events in batch, avoiding unnecessary server load 4. uses lua as configuration scripting tool Once the solution logic was finished, it was time to transform everything into reality. First, I created all necessary rpm's allowing me to install csync2 and lsyncd into any cluster in a matter of seconds. CentOS 5.6 lacked several important packages, so I had to write everything from scratch except for xinetd who was available into base repository. All packages are currently available into Axivo repository. If you are a host provider, please contact us if you want to help us host our packages on your servers. The dependencies I wrote/used for csync2.x86_64 1.34 rpm are: xinetd.x86_64 2:2.3.14+ (CentOS repo) gnutls.x86_64 1.4.1+ (Axivo repo) librsync.x86_64 0.9.7+ (Axivo repo) libtasn1.x86_64 2.9+ (Axivo repo) sqlite2.x86_64 2.8.17+ (Axivo repo)

https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/

2/11

11/7/13

Lightning fast synch with csync2 and lsyncd | Axivo Community

The dependencies I wrote/used for lsyncd.x86_64 2.04 rpm are: rsync.x86_64 2.6.8+ (CentOS repo) lua.x86_64 5.1.4+ (Axivo repo) Note: lsyncd was designed mainly for rsync, so I had to implement it as requirement into Axivo rpm. Next, it was time to actually install all packages and test how well they perform. For that, I've setup a small 3 nodes cluster on an internal network and installed all of the above listed rpm's in each node: # yum --enablerepo= axivo install csync2 lsyncd

I will explain in detail the logic, because it is important to understand how the actual setup will work. For simplicity, I will call the 3 nodes apollo, chronos and hermes, as my test nodes I used into cluster. See below the detailed setup steps I used for csync2 and lsyncd. Important: Please be aware that I refer to several file locations in the next steps. These locations are unique to Axivo rpm builds and will probably not match the actual source build locations. If you install the Axivo rpm's, you will not encounter any setup difficulties. Floren, May 16, 2011

Floren

#1

Csync2 Setup Csync2 is very straight forward, but it required some precise steps to perform, once I installed all Axivo rpm's. The csync2 rpm automates several tasks that are required to perform manually, once you install the software from source. For example, it adds the default csync2 TCP port to /etc/services file, creates the SSL certificates (in case you connect to an external node) and manages the sqlite database needed to store the sync information for each node.

Axivo Developer

To avoid frustration, it is important that you study the Csync2 documentation and understand how its process works, before you proceed with the tutorial. 1) Generate the csync2.key, into apollo node. To do that, I simply issued the command: # csync2 -k /etc/csync2/csync2.key

It will require to generate some additional entropy, in order to properly generate https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/

3/11

11/7/13

Lightning fast synch with csync2 and lsyncd | Axivo Community

the key. To do that, I simply opened a new terminal window and ran the usual commands like: # du -h /

If is not enough, run top and wait until enough random data is created. Y ou will notice when the process is complete once your shell prompt is returned into the window where you started the csync2 key generation. 2) Edit the default /etc/csync2/csync2.cfg configuration file and insert into the following content: Code: nossl * *; group web { host apollo; host chronos; host hermes; key /etc/csync2/csync2.key; include /var/www/html; exclude *~ .*; auto none; }

I used this file for the initial syncronisation step on all nodes, explained into final setup steps. Personally, I did not wanted the encrypted overhead on an internal network, so I disabled the SSL with the nossl directive you noticed into above csync2 configuration file. 3) Create a custom configuration file called csync2_apollo.cfg for apollo node. This part is the key of success for our solution, I will explain the logic in detail at the end of this article. Code: nossl * *; group apollo { host apollo; host (chronos); key /etc/csync2/csync2.key; include /var/www/html; exclude *~ .*; auto none; }

4) On each additional node, copy all files present into apollo:/etc/csync2 directory. To be save, I ran the following commands on chronos and hermes: https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/

4/11

11/7/13

Lightning fast synch with csync2 and lsyncd | Axivo Community

# rm -f /etc/csync2/* # scp root@apollo:/etc/csync2/* /etc/csync2/

I did that because I wanted to have the SSL certificates and csync2.key identical on all nodes. That is the condition needed by csync2, in case you decide to use an SSL connection while syncing the files. Once the file transfer completed on each node, rename accordingly the csync2 configuration files to reflect the name of each node. On chronos node, the configuration file was renamed csync2_chronos.cfg and had the content: Code: nossl * *; group chronos { host chronos; host (hermes); key /etc/csync2/csync2.key; include /var/www/html; exclude *~ .*; auto none; }

while on hermes node, the configuration file was renamed csync2_hermes.cfg and had the content: Code: nossl * *; group hermes { host hermes; host (apollo); key /etc/csync2/csync2.key; include /var/www/html; exclude *~ .*; auto none; }

As detailed into Csync2 documentation, nodes will be able to communicate with eachothers if they have the chained configuration files present in each node. For example, the csync2_apollo.cfg file should be present in both apollo and chronos nodes. Y ou will probably say: "Floren, what is this non-sense, simply use the global csync2 configuration file across the entire network!" "Patience, Obiwan. I will explain in detail the logic at the end of this article." 5) On each node, edit the xinetd csync2 service file: https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/

5/11

11/7/13

Lightning fast synch with csync2 and lsyncd | Axivo Community

Code: service csync2 { flags = REUSE socket_type = stream wait = no user = root group = root server = /usr/sbin/csync2 server_args = -i port = 30865 type = UNLISTED #log_on_failure += USERID disable = no #only_from = 192.168.199.3 192.168.199.4 }

Basically, all you have to do is change the disable setting from yes, to no. Y ou could also run the command: # chkconfig csync2 on

but I wanted you to familiarize with the options inside the service starter, in case you want to customize some settings in the future. Floren, May 16, 2011

#2

Lsyncd Setup On each node, edit the /etc/lsyncd/lsyncd.conf file and insert the following lua script: Code:

Floren Axivo Developer

https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/

6/11

11/7/13

Lightning fast synch with csync2 and lsyncd | Axivo Community

------ User configuration file for lsyncd. --- This example synchronizes one specific directory through multiple nodes, -- by combining csync2 and lsyncd as monitoring tools. -- It avoids any race conditions generated by lsyncd, while detecting and -- processing multiple inotify events in batch, on each node monitored by -- csync2 daemon. --- @author Floren Munteanu -- @link http://www.axivo.com/ ----settings = { logident = "lsyncd", logfacility = "user", logfile = "/var/log/lsyncd/lsyncd.log", statusFile = "/var/log/lsyncd/status.log", statusInterval = 1 }

initSync = { delay = 1, maxProcesses = 1, action = function(inlet) local config = inlet.getConfig() local elist = inlet.getEvents(function(event) return event.etype ~= "Blanket" end) local directory = string.sub(config.source, 1, -2) local paths = elist.getPaths(function(etype, path) return "\t" .. config.syncid .. ":" .. directory .. path end) log("Normal", "Processing syncing list:\n", table.concat(paths, "\n")) spawn(elist, "/usr/sbin/csync2", "-C", config.syncid, "-x") end, collect = function(agent, exitcode) local config = agent.config if not agent.isList and agent.etype == "Blanket" then if exitcode == 0 then log("Normal", "Startup of '", config.syncid, "' instance finish elseif config.exitcodes and config.exitcodes[exitcode] == "again" then Rename the corresponding source key and accordingly. ou can specify log(" Norma l", value "Retryi ng startup ofY '" , co nfig .syncid, "' instan return "again" several sources, using this format: else log("Error", "Failure on startup of '", config.syncid, "' insta terminate(-1) Code: end return local sources = { ew n/ dhtml/images"] ["/var/ww = "apollo", ["/var/wl wo wc /a hl tmr lc /c= usc to on mf ai vg a. te ax ri st "c ]od =es "aa pn od llc oo "nfig.exitcodes[exitcode] i f r c = = " d i e " t h e n } return rc end if agent.isList then The sources array will allow to create if rc you == "a gai n" thenspecific csync2 configurations for l o g ( " N ormal", "Retrying events list on exitcode = ", exitcode) each directory you monitor. else The key allows you to define a l specific og("Normdirectory al", "Finis to hed beev synced, ents list while = ", the exitc value ode) is end the csync2 configuration file id: else if rc == "again" then log("Normal", "Retrying ", agent.etype, " on ", agent.sourcePat /etc/csync2/csync2_apollo.cfg else log("Normal", "Finished ", agent.etype, " on ", agent.sourcePat end end specify whatever values you want. Y ou can even define a Obviously, you can return rc specific syncid end, per directory, if your configuration need to have specific actions init = function(inlet) local config = inlet.getConfig() https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/

7/11

11/7/13

Lightning fast synch with csync2 and lsyncd | Axivo Community

processed only for your defined source key. Floren, May 16, 2011

#3

Final Setup Steps So, the hardcore stuff is done. All we have to do left is start the daemons on each node and let the sync beast unleash its power through out the entire network.

Floren Axivo Developer

1) We need to prepare the csync2 sqlite database on each node. On apollo, run the csync2 initialization command. It will detect all new files, mark them as dirty and then sync them through the rest of nodes: # csync2 -xv

Based on your /var/www/html directory size, it will take csync2 from seconds to minutes replicating the entire list of files. That includes not only the file content, but also its permissions. Don't worry, next syncs will be very fast as csync2 stores each file information into its database and syncs only the files that changed, resulting in dramatic performance gains over a large cluster. Once the sync was completed on apollo, I performed the same command on each other node, in our case chronos and hermes. That resulted on a global sync, allowing lsynd to start with fresh data, perfectly syncronized. 2) Start the daemons on each node: # chkconfig xinetd on # service xinetd start # chkconfig lsyncd on # service lsyncd start

Y ou can see if everything is OK, by consulting the /var/log/lysncd/lsyncd.log file. Final Test Results I created a very simple shell script that generates 100 .html files into a /test directory. The goal was to see how fast the files are replicated through entire network and how much resources were consumed: Code:

https://www.axivo.com/community/threads/lightning-fast-synch-with-csync2-and-lsyncd.121/

8/11

11/7/13

Lightning fast synch with csync2 and lsyncd | Axivo Community

#!/bin/sh for ((i=0; i