This time the source RPM is included as well.

You can find them here

These systems run Slackware Linux, which has historically been my distro of choice for critical systems. When deploying Heartbeat on Slackware i ran into some issues which i’d like to share here. I won’t go into basic stuff like actually compiling and installing DRBD and Heartbeat, since that is pretty well documented in various other places, for starters the Linux-HA site (home of Heartbeat).

Heartbeat starts HA services as defined in the file ‘haresources’, but in doing so, Heartbeat seems a bit SysV-init biased. SysV init based systems only start certain services when they are told to do so, by linking them to a certain runlevel. So on a Sysv-style system, if you don’t want to start NFS services at boot time, you just remove them from their runlevels, but leaving the init.d script intact.

Because Slackware has more of a BSD-style init, it starts most of its service daemons from /etc/rc.d/rc.M, including RPC services and NFS. We don’t want these services started by rc.M at boot time, because we want Hearbeat to manage these services. Normally on Slackware, if you don’t want a certain service started at boot-time, you would simply ‘chmod a-x’ the specific rc script in /etc/rc.d, but that is not an option now, since heartbeat still needs to be able to start/stop the service from its ‘haresources’. My solution was to leave all scripts executable, and rename the rc scripts of the services i wanted to be managed by Heartbeat:

cd /etc/rc.d
mv rc.rpc rc.rcp-hb
mv rc.nfsd rc.nfsd-hb
mv rc.samba rc.samba-hb

Now the RPC services, NFS and Samba will no longer be started at boot time, since rc.M only looks for the existence of the rc scripts without the added ‘-hb’ part.

Next we tell Heartbeat the names of the rc scripts to start/stop by putting them in ‘haresources’. My ‘haresources’ file looks like this:

fs1 drbddisk::shared drbddisk::backups \
Filesystem::/dev/drbd0::/var/nfsroot/shared::reiserfs \
Filesystem::/dev/drbd1::/var/nfsroot/backups::xfs \
Delay::2::0 \
rc.samba-hb \
rc.rpc-hb \
rc.nfsd-hb \
Delay::3::0 \
IPaddr::10.0.0.150/16/eth0

As you can see i have Hearbeat managing two DRBD volumes (’shared’ and ‘backups’), NFS and Samba, and one shared IP address.

To have DRBD and Heartbeat started at boot time, i added the following to ‘rc.local’:

/etc/rc.d/drbd start
/etc/rc.d/heartbeat start

And to stop them at reboot, i added this to ‘rc.local_shutdown’:

/etc/rc.d/heartbeat stop
/etc/rc.d/drbd stop

Also, don’t forget to move /var/lib/nfs to your drbd volume for proper locking, and alter your rc.rpc-hb to add a cluster name to the startup of statd. More background info on this at Linux-HA (step 4e and 4f)

Now for all this to work properly, there is one more thing, which I found out the hard way When Heartbeat releases its resources, it stops all services mentioned in ‘haresources’ by calling their related rc scripts with ’stop’. I ran into some strange failover behaviour, and found the following in my logs:

heartbeat: 2007/10/01_13:16:42 info: Running /etc/rc.d/rc.samba stop
heartbeat: 2007/10/01_13:16:42 ERROR: Return code 1 from /etc/rc.d/rc.samba
heartbeat: 2007/10/01_13:16:42 ERROR: Resource script for rc.samba probably not LSB-compliant.
heartbeat: 2007/10/01_13:16:42 WARN: it (rc.samba) MUST succeed on a stop when already stopped
heartbeat: 2007/10/01_13:16:42 WARN: Machine reboot narrowly avoided!


Apparently, Heartbeat will sometimes call an rc script with ’stop’ while the services is already in “stopped state”. Now, looking at our rc.samba-hb, we see the following stop function:

samba_stop() {
killall smbd nmbd
}


This call to killall will return with a non-zero exit code when there are no processes to kill, which results in the rc script exiting with a non-zero exit code. This makes Heartbeat think something failed, resulting in the above error messages. The fix for this is rather simple, though maybe a bit hackish. Change the samba_stop function by adding a ‘exit 0′:

samba_stop() {
killall smbd nmbd
exit 0
}


This should make Heartbeat happy. There are probably other rc scripts around that do not comply to this, so check your startup scripts.

Finally, watch out with DRBD-0.7.24 on a system with a kernel >= 2.6.22. I still use the DRBD 0.7 branch, and when i deployed 0.7.24 on kernel 2.6.22.9, i ran into a whole bunch of trouble. The load average would suddenly spike enormously, and my systems went unresponsive to shutdown or reboot commands. I found a drbd-user mailinglist posting from someone with similar issues. Apparently it’s a known issue with drbd-0.7.24 on kernel >= 2.6.22 and XFS, which is fixed only in subversion!
After fetching and installing the latest subversion revision of the drbd-0.7 branch as described here, the problem is solved.

After some searching on Google i found an excellent post on the weblog of a CentOS developer named Karanbir Singh. To make everything work, you need to poison your 64-bit system with some 32-bit libraries

I had to install the following packages to make things work (some are sucked in by dependancies):

• xorg-x11-libs.i386
• expat.i386 (dep. of xorg-x11-libs)
• fontconfig.i386 (dep. of xorg-x11-libs)
• freetype.i386 (dep. of xorg-x11-libs)
• xorg-x11-Mesa-libGL.i386 (dep. of xorg-x11-libs)
• zlib.i386 (dep. of xorg-x11-libs)
• glibc.i686
• gcc.x86_64

After this vmware-config.pl ran flawlessly and VMWare Server is up and running!

See the result at http://www.flickr.com/photos/hrak/map/

You can get them here

At the moment, we mostly run Slackware and Gentoo based servers at my workplace. Since the number of servers is steadily growing (currently 30-something servers), so is the maintenance. This is why i’ve been doing some research on more maintenance-friendly distros, one of which is CentOS, the community release of Red Hat Enterprise Linux.

So far i am really impressed by CentOS, and i’m getting more into the whole RPM buzz. I have always been a ‘compile-it-yourself’ person, and never really been a fan of RPM, but i’m starting to see the benefits now

The only thing i was a bit bummed about, was the relatively old version of php that came with CentOS 4.3. Since we run the latest version of PHP4 on most of our servers, i’ve been looking into rolling my own RPMs. After a bit of research on rpmbuild and related tools, the result is a bunch of PHP-4.4.2 RPMs for both 64-bit (x86_64) and 32-bit (i386) servers. And since CentOS is the community version of RHEL4, i thought i’d share these with the world.

You can find them here. Ofcourse the usual disclaimers apply

Ik ben vorig weekend samen met Jasper op pad geweest om wat te experimenteren met IR fotografie. Hierbij een van de eindresultaten. Even snel gepost, eigenlijk meer als smerig excuus om de integratie van Flickr met Wordpress eens te checken

Moet nog wel ff een nieuwe (eigen) look en een fatsoenlijke naam tegenaan in ieder geval!