Linux/Unix/BSD

Normally I used to have a backup-retention-script in place that will create a TAR-ball of the backup data (using Herakles). But this way I was not able to have a retention of longer then 3 days

So I had to look into another solution, I could add a new harddrive in the server... but there should be something else possible. So I ended up by using LVM snapshots. So I created a Volume group of about 100GB. In that volume group I created a logical volume of about 30GB, which is enough (and if not, we can 'grow' the Filesystem thanks to LVM )

After having all that done, I've created a script located in /root/scripts/lvm-snapshot. This script runs every midnight and creates a snapshot.

#!/bin/bash
#
# Create LVM Snapshots
#
#
#---------------------------------------------------------------------------------------------------------------
CURRENT_SNAPNAME="snap-"$(date "+%Y%m%d%H%M%S")
VOLUME2SNAPSHOT="/dev/vol_backup/lvm0"
LVMSNAPSHOTCMD="/usr/sbin/lvcreate -L 2G -s -n $CURRENT_SNAPNAME $VOLUME2SNAPSHOT"
LINE="---------------------------------------------------------------------------------------------------------------------"

echo $LINE
df -h /mnt/data
echo $LINE
$LVMSNAPSHOTCMD 2> /dev/null
#---------------------------------------------------------------------------------------------------------------
SNAPSHOT_RETENTION=15
CURRENT_SNAPSHOT_COUNT=$(lvdisplay | grep "^  LV Name                /dev/vol_backup/snap" | sort | awk '{ print $3 }' | wc -l)

OVERFLOW=$(echo $CURRENT_SNAPSHOT_COUNT - $SNAPSHOT_RETENTION | bc)
if [ $OVERFLOW -gt 0 ];
then
        echo $LINE
        for files in  $(lvdisplay | grep "^  LV Name                /dev/vol_backup/snap" | sort | awk '{ print $3 }' | head -n$OVERFLOW);
        do
                 /usr/sbin/lvremove -f $files 2> /dev/null
        done
fi
#---------------------------------------------------------------------------------------------------------------
echo $LINE
/usr/sbin/vgdisplay vol_backup
echo $LINE
/usr/sbin/lvdisplay $VOLUME2SNAPSHOT

And the crontab entry is:

# crontab -l
0 0 * /root/scripts/lvm-snapshot

This week I had to set up puppet with mongrel and apache for the balancing. To be honest... I never heard aboutthe puppet tool at all before (sorry puppet-developers ). As far as I understood, the puppetmaster is not really doing well with handling multiple concurrent requests and we need to provision about 128 compute nodes from a HPC cluster ...

On how I set up all the stuff in the test environment/production environment.

First I identified the right RPMs from EPEL, because I had no connection to the Internet:

augeas-0.5.3-1.el5.x86_64.rpm
augeas-libs-0.5.3-1.el5.x86_64.rpm
facter-1.5.7-1.el5.noarch.rpm
puppet-0.24.8-4.el5.noarch.rpm
puppet-server-0.24.8-4.el5.noarch.rpm
ruby-augeas-0.3.0-1.el5.x86_64.rpm
rubygem-daemons-1.0.7-2.el5.noarch.rpm
rubygem-fastthread-1.0.1-1.el5.x86_64.rpm
rubygem-gem_plugin-0.2.2-2.el5.noarch.rpm
rubygem-mongrel-1.0.1-6.el5.x86_64.rpm
rubygem-rake-0.8.3-1.el5.noarch.rpm
rubygems-1.3.1-1.el5.noarch.rpm
ruby-shadow-1.4.1-7.el5.x86_64.rpm

You can install them with 'yum --nogpgcheck localinstall *.rpm'.

You must als make sure that apache with mod_ssl is installed as well (yum install httpd mod_ssl).

Configuring Apache for load-balancing

As mentioned before, apache and puppet master must be installed.

First I copied and modified the /etc/init.d/httpd script to use it for puppet loadbalancing. Find my copy of the script on:

http://www.xs4all.nl/~paderijk/blog/puppet-balancer

So you can do:

# cd /etc/init.d/
# wget http://www.xs4all.nl/~paderijk/blog/puppet-balancer
# chmod 755 ./puppet-balancer

Now we have to create the some additional directories:

# mkdir -p /var/log/puppet-balancer
# mkdir -p /etc/puppet-balancer/{conf,conf.d}

I used the configuration file initially set up by from Jeff McCune and modified it for RHEL5. This file can be found on:

http://www.xs4all.nl/~paderijk/blog/puppet-balancer.conf

So you can put it into place like this:

# cd /etc/puppet-balancer/conf
# wget http://www.xs4all.nl/~paderijk/blog/puppet-balancer.conf

And finally we need to create a symlink name /usr/sbin/puppet-balancer which links to /usr/sbin/httpd

# ln -s /usr/sbin/httpd /usr/sbin/puppet-balancer

I've done this, because somehow the init-scripts are having some inconstancy.

And we also need a /etc/sysconfig/puppet-balancer file

# cd /etc/sysconfig
# wget http://www.xs4all.nl/~paderijk/blog/sysconfig-puppetbalancer
# mv sysconfig-puppetbalancer puppet-balancer

For so far the apache configuration, still we don't start the puppet-balancer 'service'. First we need to configure and start the puppet master.

Configuration puppetmaster

The configuration of the puppetmaster was actually very easy.

*UPDATE* A colleague pointed me on the fact that the CA wasn't set up properly going into mongrel mode. So first start puppetmaster normally:

# service puppetmaster start ; sleep 5 ; service puppetmaster stop

Make sure the following line is 'enabled' to /etc/sysconfig/puppetmaster:

PUPPETMASTER_PORTS=( 18140 18141 18142 18143 )

This will enable puppetmaster with mongrel and starts on the ports 18140 to 18143

I also noticed that the RPM for puppetmaster didn't set up all the required directories, so I need to add them manual:

# mkdir -p /var/lib/puppet/yaml/{facts,nodes}
# chown puppet:puppet /var/lib/puppet/yaml/{facts,nodes}

Start the puppetmaster and Apache (puppet-balancer)

Now we can start the puppetmaster:

# service puppetmaster start

But we want to survive reboots as well, so we do also:

# chkconfig puppetmaster on

Now the Puppet CA is also set up. So now we can start the puppet-balancer (apache) as well. But first we need to symlink the key-file and the certificate file:

# ln -s /var/lib/puppet/ssl/private_keys/{$(hostname -f).pem,puppet-balancer.pem}
# ln -s /var/lib/puppet/ssl/certs/{$(hostname -f).pem,puppet-balancer.pem}

And now we can start the puppet-balancer:

# service puppet-balancer start

And get it also up and running after a reboot:

# chkconfig puppet-balancer on

More details on how to do further configuration/setting up manifest files et cetera can be found on the Puppet wiki:

http://reductivelabs.com/trac/puppet/

In one of the last Linux Magazine issues, there was an article with the title "Untangling the Web with Mozilla Weave". I really recognized the issue of having several Firefox instances with their own bookmarks/tabs/et cetera.

So I thought... let's give it a try, and so far... it works fine for me. I've now set up Weave on my Linux Laptop and Linux workstation at home. The upcoming week I will set up weave for my Linux workstation at work and on my Portable Apps Firefox on ahum Windows Vista workstation.

Recently I need to create a report about utilization of an HPC Cluster that uses Grid Engine, but we didn't had ARCO and so not running yet for that cluster

So I digged into my brain on how to load data from a "RAW" format into a database... it's something I did when I worked for PricewaterhouseCoopers Advisory, but then I used financial data.

Please press the continue reading link below... to read more

A user group we support uses an application and they have the license to use 20 concurrent runs. For them we've implement very recently the Sun Grid Engine job scheduler. Although they recently start complaining that jobs didn't run.

Their cluster exists out of 8 nodes with each 2 cores, so they've 16 slots in the job scheduler. We've set up two queues, the suspendable.q and the unsuspendable.q. Jobs in the suspendable.q queue can get suspended by jobs in the unsuspendable.q queue. So we can have in total 32 concurrent jobs (where 16 will be suspended).

Once their cluster is really busy, some jobs will not run... and after some investigation we found out why. The jobs that are suspended don't release their license to the license server. So we can have a total of 16 jobs in the suspendable.q and 4 in the unsuspendable.q. So we start limiting the number of unsuspendable jobs to 4, because the 21nd job that will start running will fail because it won't get a license.

Yesterday I thought let's play with FC12 (aka Rawhide, aka FC11.90). So I enabled the Rawhide-repositories on my FC11 laptop and entered "yum -y update". And after a while it was there... bleeding edge kernel and other packages.

The first issue I run into, was that Firefox 3.5 was not able to run, it caused a segfault. There seems to be a bug in the xulrunner package. So I was able to fix it, by "downgrading" Firefox to 3.0.11, but that one crashed on pages using "Adobe Flash plugin". So I removed the flash plugin, because I wanted bleeding edge Fedora. So having that "sort out" I wanted to suspend my laptop, and guess what... It didn't want to suspend... so after some hacking around... it still didn't work.

So my final decision was Go back to FC11. I was able to "downgrade" my system in about 60 minutes. At home I've a mirror repository with al the backups, so during installation I added these repositories, so I also had all the updates in one go.

Lesson learned: "Bleeding edge... is indeed bleeding edge!"

I need my work for my daily work... If I won't need it for my daily work I would have keep FC12 (aka Rawhide, aka FC11.90) on it to participate in developing FC12.

On a kind of "intranet" website, which is secured with username/password combinations and HTTPS I've implemented the next feature:

- Authorized users can read everything on the website

- Files with in their filename "classified" requires a valid SSL-Client certificate...

Here is the output of my apache config:

<Directory /usr/sites/ssl-site/intranet/htdocs>
  Options Indexes MultiViews
  AllowOverride Authconfig
  Order allow,deny
  Allow from all
  AuthName "intranet"
  AuthType "Basic"
  AuthUserFile /usr/sites/ssl-site/intranet/etc/users.pwl
  require valid-user
</Directory>

<LocationMatch .*(c|C)(l|L)(a|A)(s|S)(s|S)(i|I)(f|F)(i|I)(e|E)(d|D).+>
  SSLVerifyClient require
  SSLVerifyDepth 1
  SSLOptions +OptRenegotiate
</LocationMatch>

I still have to sort out some issues, like directories having a directory with the name "classified" in them.

Recently I got a iPhone, but I have multiple mails coming into my mailbox (private/business/sysop). I use maildrop to put them into the right folders. I want share my business (sub)folder(s) with my special iPhone-account... but how could we do that... (please note you should have admin-privileges)

Step 1 - Create new user and put the "source" mailbox user in the right group

Create a iPhone user on your server (in my case user is iphone) and add the user (in my case pieter) to the iphone group (created during the creation of the iphone user).

Step 2 - Set permissions correct of the source mailbox

Make sure the world can access ~pieter/Maildir, set this by entering:

[ root@server ~]# chmod o+x ~pieter/Maildir

New we also have to set the grouppermissions correct of the source sub-folders:

[ root@server ~]# chown -R pieter:iphone ~pieter/Maildir/.Business*

Set groupbit and grouppermissions on the folders you want to share:

[ root@server ~]# find ~pieter/Maildir/.Business* -type d -exec chmod 2770 {} \;

Set the grouppermissions on the current messages"

[ root@server ~]# find ~pieter/Maildir/.Business* -type f -exec chmod  0660 {} \;

Step 3 - Setup the functional account and mailstructure

Become that user (can be done via sudo).
[ pieter@server ~]$ sudo su - iphone
Password: ****
[ iphone@server ~]$

Create the maildir structure:

[ iphone@server ~]$ maildirmake ~/Maildir

Remove the cur, new and tmp folders:

[ iphone@server ~]$ rm -rf ~/Maildir/[cnt]*

Now link them to the source:

[ iphone@server ~]$ for x in cur new tmp; do ln -s /home/pieter/Maildir/.Business/$x ~iphone/Maildir/$x; done

Step 4 - Share the subfolders as well

[ iphone@server ~]$ cd ~/Maildir
[ iphone@server Maildir]$ maildirmake .Archive
[ iphone@server Maildir]$ rm -rf ~/.Archive/[cnt]*
[ iphone@server Maildir]$ for x in cur new tmp; do ln -s /home/pieter/Maildir/.Business.Archive/$x ~iphone/Maildir/.Archive/$x; done

Perform step 4 for al the other subfolders you would like to share (Please note that you've to set the permissions in step 2 as well). This was done on a FreeBSD6.3 system, I don't know what the impact might be on Linux systems with SELinux... nor I don't know what the impact might be of the chmod o+x on Maildir... we wil investigate. Initially I did a chown pieter:iphone on the source maildir... but my imap-server refused connection due to wrong gid.

Also keep in mind to put in your procmail/maildrop filter a umask of 007!

But... conclusion... it works cool.

Last Friday I was playing around with one of my FreeBSD production servers. On that server I've a number of users for e-mail and other services.

I was playing around as root, because I wanted to update/install some new stuff. But at a certain moment I found out that I was not able to login as a non-root user (nor as root). So first I've changed the root-password and allowed root to login via SSH. Because I had a running session to that box via a screen-session I was able to do so.

But, still needed to figure out what went wrong.

Some while ago, I've start using subversion to make backups of my config-files. And as a standard procedure I make sure I've an up to date version of the config-repository on my laptop and workstation.

I found out that the next files were modified:

• /etc/passwd
• /etc/master.passwd
• /etc/pwd.db
• /etc/spwd.db

After having these files restored, normal users were able to login in again

There were also some other files modified, but by using diff and creating a patch file I was able to restore them very quick.

So lessons learned for me about this... is... make sure you've backups, and do read the messages which pops up!

It can happen, you have sudo-access to another account (most of the time it will be access to the root account). But most of the time the NOPASSWD option is not used due to security reasons. But there are moments you want to have sudo-credentials available, think about a script or something else.... I had the same issue, so I found the next "hack" to get the timestamp refreshed every 60 seconds.

(Please note the script will use user "root" but it can be another user, please modify the scripts so it fits your needs).

Step 1)

Create a script in you $HOME/bin with the next content (I call it sudo-hack.sh):

---

#!/bin/bash 
while [ true ]; 
do 
  sudo -u root /bin/true > /dev/null 2> /dev/null 
  sleep 60 
done 

---

Step 2)

Get a valid sudo-timestamp:

$ sudo -u root /bin/true
Password:
$

Step 3)

Start sudo-hack.sh in the background:

$ $HOME/bin/sudo-hack.sh &
$

That's all!

This week I had the "Red Hat Enterprise Directory Services and Authentication" course and exam in Amsterdam.

In the course we had some very nice stuff, like Red Hat DS and at the end Red Hat Enterprise IPA... all very cool... but today I had the exam (due to the RedHat NDA I am not allowed to say anything about the exam, so I won't do it)... but a few hours after the exam I received my results... and I passed the exam

For a project I am working on migrating UNIX applications to Linux. Most of the scripting work supposed to be done in India, and that is where the issues came in. First you have a developer who knows how to work with M$ Technet and never worked with PERL before (at least 80% of the scripts is written in PERL).

First of all I introduced the user Net::LDAP within PERL, because they first did a ldapsearch, put the output into a ASCII file... and with a PERL script they structured the data... and loaded it into a Oracle database... so that was the first improvement.

Next there were several issues, like not good reading or understanding LDAP/PERL at all...

But at a certain moment, they start complaining about the fact that one of the scripts was slow... on the old system the script had a run time of 4 hours... and now it is up to 28 hours(!!!) So they requested me to investigate this.

First I found a 'main' kornshell script doing the next thing:

---

for VAR in a b d e f g i j k m n o p q r s t u v w x y z
do
   for NAME in "'" a b c d e f g h i j k l m n o p q r s t u v w x y z
   do
     ldap_script.pl $NAME $VAR
   done
done

---

The content of the ldap_script.pl was something like:

---

#!/usr/bin/perl
use Net::LDAP;
$ldap = Net::LDAP->new($LDAP_SERVER);
$ldap->bind($LDAP_DN, password=>$LDAP_PASSWD) or die "Cannot connect";
$LDAP_FILTER="(&(sn=$ARGV[0]*)(OfficeName=$ARGV[1]*))";
$mesg = $ldap->search(base=>$LDAP_BASE,
                      filter=>$LDAP_FILTER,
                     ) or die "Cannot connect";
push(@ENTRIES,$mesg->entries);
$ldap->unbind;

---

I thought that this costs a lot... loading PERL script, connecting to server, binding to it... et cetera... And this was done in the original script > 2000 times

So... I removed the loop out of the mainscript... and implemented it into the PERL-script, like this:

---

#!/usr/bin/perl

use Net::LDAP;

$ldap = Net::LDAP->new($LDAP_SERVER);
@LOOP=("a","b","c","d","e","f","g","h","i","j","k","l","m","n","o",
       "p","q","r","s","t","u","v","w","x","y","z", "'");


$ldap->bind($LDAP_DN, password=>$LDAP_PASSWD) or die "Cannot connect";

foreach $LOOP1 (@LOOP)
{
  foreach $LOOP2 (@LOOP)
  {
     $LDAP_FILTER="(&(sn=$LOOP1*)(OfficeName=$LOOP2*))";
     $mesg = $ldap->search(base=>$LDAP_BASE,
                           filter=>$LDAP_FILTER,
                          ) or die "Cannot connect";
     push(@ENTRIES,$mesg->entries);
  }
}

$ldap->unbind;

---

And this runs within 3 hours!!! And it is flying!

There can be done more performance tuning... but that will be another project!

Currently I own and maintain some domains. For these domains you need a DNS-server to make a proper translation from hostname to ip-adress.

blog.adslweb.net -> 80.126.215.23

But to keep redundancy it is smart to have some extra DNS-servers configured. So I setup a 'master' DNS and a 'slave' DNS, but both of them were connected to a server on a ADSL line... This might not be a big problem, but due to some problems recently, I preferred to have something somewhere else

After some Google-ing, I found Twisted4Life, if you create an account, they provide you for free a 'slave' DNS.

NOTE: Some of the software is very experimental!!! I had some issues that my .ics was totally removed when I added a new 'task' via lightning, please make a backup of your ics file!

Some while ago I wrote a small article about WebDAV and Calendar files. Until so far no results on reading the files into my Nokia , but... I found an extension for thunderbird to handle meeting-requests from Outlook clients

You can find information on the project website of 'lightning'. To configure your webdav, you need to click on the calendar button

Click in the left pane with the right button en select 'New Calendar', now you can use the wizard to select your calendar.

And now let the invitations come...

Yesterday evening I start playing with Fedora Directory Server

So first I setup Fedora Core 8 as a VMWare-instance... But after some playing around, I had the next message:

"Server failed to start !!! Please check errors log for problems"

And guess what... no information at all in the logs So removed the packages and the next directories:

/etc/dirsrv
/etc/sysconfig/dirsrv
/var/lock/dirsrv
/var/lib/dirsrv

So no information... then strace will be your best friend

So I started:

[root@fedora-ds debug]# strace -o ~/debug/setup -ff /usr/sbin/setup-ds.pl

And guess what... I had the error again... So I went to the ~/debug folder on another terminal and did:

[root@fedora-ds debug]# grep "failed" *
setup.31676:read(4, "Server failed to start !!! Pleas"..., 4096) = 64
setup.31676:write(2, "Server failed to start !!! Pleas"..., 64) = 64
setup.31711:write(1, "Server failed to start !!! Pleas"..., 64) = 64
[root@fedora-ds debug]#

When I digged into setup.31711 I found:
read(255, "if test ! -f $STARTPIDFILE ; the"..., 2220) = 663
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat64("/var/run/dirsrv/slapd-fedora-ds.startpid", 0xbfcd8eb8) = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_SETMASK, [], NULL, = 0
fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f7c000
write(1, "Server failed to start !!! Pleas"..., 64) = 64

So this is a nice clue... /var/run/dirsrv... and guess what... the owner of this directory was fedora-ds (a user I set up initially for testing purposes for the Directory Server ) With the comment chown I corrected the owner of this folder... and now service dirsrv start works

Conclusion... strace is your best friend