Work

Normally I used to have a backup-retention-script in place that will create a TAR-ball of the backup data (using Herakles). But this way I was not able to have a retention of longer then 3 days

So I had to look into another solution, I could add a new harddrive in the server... but there should be something else possible. So I ended up by using LVM snapshots. So I created a Volume group of about 100GB. In that volume group I created a logical volume of about 30GB, which is enough (and if not, we can 'grow' the Filesystem thanks to LVM )

After having all that done, I've created a script located in /root/scripts/lvm-snapshot. This script runs every midnight and creates a snapshot.

#!/bin/bash
#
# Create LVM Snapshots
#
#
#---------------------------------------------------------------------------------------------------------------
CURRENT_SNAPNAME="snap-"$(date "+%Y%m%d%H%M%S")
VOLUME2SNAPSHOT="/dev/vol_backup/lvm0"
LVMSNAPSHOTCMD="/usr/sbin/lvcreate -L 2G -s -n $CURRENT_SNAPNAME $VOLUME2SNAPSHOT"
LINE="---------------------------------------------------------------------------------------------------------------------"

echo $LINE
df -h /mnt/data
echo $LINE
$LVMSNAPSHOTCMD 2> /dev/null
#---------------------------------------------------------------------------------------------------------------
SNAPSHOT_RETENTION=15
CURRENT_SNAPSHOT_COUNT=$(lvdisplay | grep "^  LV Name                /dev/vol_backup/snap" | sort | awk '{ print $3 }' | wc -l)

OVERFLOW=$(echo $CURRENT_SNAPSHOT_COUNT - $SNAPSHOT_RETENTION | bc)
if [ $OVERFLOW -gt 0 ];
then
        echo $LINE
        for files in  $(lvdisplay | grep "^  LV Name                /dev/vol_backup/snap" | sort | awk '{ print $3 }' | head -n$OVERFLOW);
        do
                 /usr/sbin/lvremove -f $files 2> /dev/null
        done
fi
#---------------------------------------------------------------------------------------------------------------
echo $LINE
/usr/sbin/vgdisplay vol_backup
echo $LINE
/usr/sbin/lvdisplay $VOLUME2SNAPSHOT

And the crontab entry is:

# crontab -l
0 0 * /root/scripts/lvm-snapshot

This week I had to set up puppet with mongrel and apache for the balancing. To be honest... I never heard aboutthe puppet tool at all before (sorry puppet-developers ). As far as I understood, the puppetmaster is not really doing well with handling multiple concurrent requests and we need to provision about 128 compute nodes from a HPC cluster ...

On how I set up all the stuff in the test environment/production environment.

First I identified the right RPMs from EPEL, because I had no connection to the Internet:

augeas-0.5.3-1.el5.x86_64.rpm
augeas-libs-0.5.3-1.el5.x86_64.rpm
facter-1.5.7-1.el5.noarch.rpm
puppet-0.24.8-4.el5.noarch.rpm
puppet-server-0.24.8-4.el5.noarch.rpm
ruby-augeas-0.3.0-1.el5.x86_64.rpm
rubygem-daemons-1.0.7-2.el5.noarch.rpm
rubygem-fastthread-1.0.1-1.el5.x86_64.rpm
rubygem-gem_plugin-0.2.2-2.el5.noarch.rpm
rubygem-mongrel-1.0.1-6.el5.x86_64.rpm
rubygem-rake-0.8.3-1.el5.noarch.rpm
rubygems-1.3.1-1.el5.noarch.rpm
ruby-shadow-1.4.1-7.el5.x86_64.rpm

You can install them with 'yum --nogpgcheck localinstall *.rpm'.

You must als make sure that apache with mod_ssl is installed as well (yum install httpd mod_ssl).

Configuring Apache for load-balancing

As mentioned before, apache and puppet master must be installed.

First I copied and modified the /etc/init.d/httpd script to use it for puppet loadbalancing. Find my copy of the script on:

http://www.xs4all.nl/~paderijk/blog/puppet-balancer

So you can do:

# cd /etc/init.d/
# wget http://www.xs4all.nl/~paderijk/blog/puppet-balancer
# chmod 755 ./puppet-balancer

Now we have to create the some additional directories:

# mkdir -p /var/log/puppet-balancer
# mkdir -p /etc/puppet-balancer/{conf,conf.d}

I used the configuration file initially set up by from Jeff McCune and modified it for RHEL5. This file can be found on:

http://www.xs4all.nl/~paderijk/blog/puppet-balancer.conf

So you can put it into place like this:

# cd /etc/puppet-balancer/conf
# wget http://www.xs4all.nl/~paderijk/blog/puppet-balancer.conf

And finally we need to create a symlink name /usr/sbin/puppet-balancer which links to /usr/sbin/httpd

# ln -s /usr/sbin/httpd /usr/sbin/puppet-balancer

I've done this, because somehow the init-scripts are having some inconstancy.

And we also need a /etc/sysconfig/puppet-balancer file

# cd /etc/sysconfig
# wget http://www.xs4all.nl/~paderijk/blog/sysconfig-puppetbalancer
# mv sysconfig-puppetbalancer puppet-balancer

For so far the apache configuration, still we don't start the puppet-balancer 'service'. First we need to configure and start the puppet master.

Configuration puppetmaster

The configuration of the puppetmaster was actually very easy.

*UPDATE* A colleague pointed me on the fact that the CA wasn't set up properly going into mongrel mode. So first start puppetmaster normally:

# service puppetmaster start ; sleep 5 ; service puppetmaster stop

Make sure the following line is 'enabled' to /etc/sysconfig/puppetmaster:

PUPPETMASTER_PORTS=( 18140 18141 18142 18143 )

This will enable puppetmaster with mongrel and starts on the ports 18140 to 18143

I also noticed that the RPM for puppetmaster didn't set up all the required directories, so I need to add them manual:

# mkdir -p /var/lib/puppet/yaml/{facts,nodes}
# chown puppet:puppet /var/lib/puppet/yaml/{facts,nodes}

Start the puppetmaster and Apache (puppet-balancer)

Now we can start the puppetmaster:

# service puppetmaster start

But we want to survive reboots as well, so we do also:

# chkconfig puppetmaster on

Now the Puppet CA is also set up. So now we can start the puppet-balancer (apache) as well. But first we need to symlink the key-file and the certificate file:

# ln -s /var/lib/puppet/ssl/private_keys/{$(hostname -f).pem,puppet-balancer.pem}
# ln -s /var/lib/puppet/ssl/certs/{$(hostname -f).pem,puppet-balancer.pem}

And now we can start the puppet-balancer:

# service puppet-balancer start

And get it also up and running after a reboot:

# chkconfig puppet-balancer on

More details on how to do further configuration/setting up manifest files et cetera can be found on the Puppet wiki:

http://reductivelabs.com/trac/puppet/

Recently I need to create a report about utilization of an HPC Cluster that uses Grid Engine, but we didn't had ARCO and so not running yet for that cluster

So I digged into my brain on how to load data from a "RAW" format into a database... it's something I did when I worked for PricewaterhouseCoopers Advisory, but then I used financial data.

Please press the continue reading link below... to read more

A user group we support uses an application and they have the license to use 20 concurrent runs. For them we've implement very recently the Sun Grid Engine job scheduler. Although they recently start complaining that jobs didn't run.

Their cluster exists out of 8 nodes with each 2 cores, so they've 16 slots in the job scheduler. We've set up two queues, the suspendable.q and the unsuspendable.q. Jobs in the suspendable.q queue can get suspended by jobs in the unsuspendable.q queue. So we can have in total 32 concurrent jobs (where 16 will be suspended).

Once their cluster is really busy, some jobs will not run... and after some investigation we found out why. The jobs that are suspended don't release their license to the license server. So we can have a total of 16 jobs in the suspendable.q and 4 in the unsuspendable.q. So we start limiting the number of unsuspendable jobs to 4, because the 21nd job that will start running will fail because it won't get a license.

On a kind of "intranet" website, which is secured with username/password combinations and HTTPS I've implemented the next feature:

- Authorized users can read everything on the website

- Files with in their filename "classified" requires a valid SSL-Client certificate...

Here is the output of my apache config:

<Directory /usr/sites/ssl-site/intranet/htdocs>
  Options Indexes MultiViews
  AllowOverride Authconfig
  Order allow,deny
  Allow from all
  AuthName "intranet"
  AuthType "Basic"
  AuthUserFile /usr/sites/ssl-site/intranet/etc/users.pwl
  require valid-user
</Directory>

<LocationMatch .*(c|C)(l|L)(a|A)(s|S)(s|S)(i|I)(f|F)(i|I)(e|E)(d|D).+>
  SSLVerifyClient require
  SSLVerifyDepth 1
  SSLOptions +OptRenegotiate
</LocationMatch>

I still have to sort out some issues, like directories having a directory with the name "classified" in them.

It can happen, you have sudo-access to another account (most of the time it will be access to the root account). But most of the time the NOPASSWD option is not used due to security reasons. But there are moments you want to have sudo-credentials available, think about a script or something else.... I had the same issue, so I found the next "hack" to get the timestamp refreshed every 60 seconds.

(Please note the script will use user "root" but it can be another user, please modify the scripts so it fits your needs).

Step 1)

Create a script in you $HOME/bin with the next content (I call it sudo-hack.sh):

---

#!/bin/bash 
while [ true ]; 
do 
  sudo -u root /bin/true > /dev/null 2> /dev/null 
  sleep 60 
done 

---

Step 2)

Get a valid sudo-timestamp:

$ sudo -u root /bin/true
Password:
$

Step 3)

Start sudo-hack.sh in the background:

$ $HOME/bin/sudo-hack.sh &
$

That's all!

This week I had the "Red Hat Enterprise Directory Services and Authentication" course and exam in Amsterdam.

In the course we had some very nice stuff, like Red Hat DS and at the end Red Hat Enterprise IPA... all very cool... but today I had the exam (due to the RedHat NDA I am not allowed to say anything about the exam, so I won't do it)... but a few hours after the exam I received my results... and I passed the exam

For a project I am working on migrating UNIX applications to Linux. Most of the scripting work supposed to be done in India, and that is where the issues came in. First you have a developer who knows how to work with M$ Technet and never worked with PERL before (at least 80% of the scripts is written in PERL).

First of all I introduced the user Net::LDAP within PERL, because they first did a ldapsearch, put the output into a ASCII file... and with a PERL script they structured the data... and loaded it into a Oracle database... so that was the first improvement.

Next there were several issues, like not good reading or understanding LDAP/PERL at all...

But at a certain moment, they start complaining about the fact that one of the scripts was slow... on the old system the script had a run time of 4 hours... and now it is up to 28 hours(!!!) So they requested me to investigate this.

First I found a 'main' kornshell script doing the next thing:

---

for VAR in a b d e f g i j k m n o p q r s t u v w x y z
do
   for NAME in "'" a b c d e f g h i j k l m n o p q r s t u v w x y z
   do
     ldap_script.pl $NAME $VAR
   done
done

---

The content of the ldap_script.pl was something like:

---

#!/usr/bin/perl
use Net::LDAP;
$ldap = Net::LDAP->new($LDAP_SERVER);
$ldap->bind($LDAP_DN, password=>$LDAP_PASSWD) or die "Cannot connect";
$LDAP_FILTER="(&(sn=$ARGV[0]*)(OfficeName=$ARGV[1]*))";
$mesg = $ldap->search(base=>$LDAP_BASE,
                      filter=>$LDAP_FILTER,
                     ) or die "Cannot connect";
push(@ENTRIES,$mesg->entries);
$ldap->unbind;

---

I thought that this costs a lot... loading PERL script, connecting to server, binding to it... et cetera... And this was done in the original script > 2000 times

So... I removed the loop out of the mainscript... and implemented it into the PERL-script, like this:

---

#!/usr/bin/perl

use Net::LDAP;

$ldap = Net::LDAP->new($LDAP_SERVER);
@LOOP=("a","b","c","d","e","f","g","h","i","j","k","l","m","n","o",
       "p","q","r","s","t","u","v","w","x","y","z", "'");


$ldap->bind($LDAP_DN, password=>$LDAP_PASSWD) or die "Cannot connect";

foreach $LOOP1 (@LOOP)
{
  foreach $LOOP2 (@LOOP)
  {
     $LDAP_FILTER="(&(sn=$LOOP1*)(OfficeName=$LOOP2*))";
     $mesg = $ldap->search(base=>$LDAP_BASE,
                           filter=>$LDAP_FILTER,
                          ) or die "Cannot connect";
     push(@ENTRIES,$mesg->entries);
  }
}

$ldap->unbind;

---

And this runs within 3 hours!!! And it is flying!

There can be done more performance tuning... but that will be another project!

Currently I own and maintain some domains. For these domains you need a DNS-server to make a proper translation from hostname to ip-adress.

blog.adslweb.net -> 80.126.215.23

But to keep redundancy it is smart to have some extra DNS-servers configured. So I setup a 'master' DNS and a 'slave' DNS, but both of them were connected to a server on a ADSL line... This might not be a big problem, but due to some problems recently, I preferred to have something somewhere else

After some Google-ing, I found Twisted4Life, if you create an account, they provide you for free a 'slave' DNS.

NOTE: Some of the software is very experimental!!! I had some issues that my .ics was totally removed when I added a new 'task' via lightning, please make a backup of your ics file!

Some while ago I wrote a small article about WebDAV and Calendar files. Until so far no results on reading the files into my Nokia , but... I found an extension for thunderbird to handle meeting-requests from Outlook clients

You can find information on the project website of 'lightning'. To configure your webdav, you need to click on the calendar button

Click in the left pane with the right button en select 'New Calendar', now you can use the wizard to select your calendar.

And now let the invitations come...

Yesterday evening I start playing with Fedora Directory Server

So first I setup Fedora Core 8 as a VMWare-instance... But after some playing around, I had the next message:

"Server failed to start !!! Please check errors log for problems"

And guess what... no information at all in the logs So removed the packages and the next directories:

/etc/dirsrv
/etc/sysconfig/dirsrv
/var/lock/dirsrv
/var/lib/dirsrv

So no information... then strace will be your best friend

So I started:

[root@fedora-ds debug]# strace -o ~/debug/setup -ff /usr/sbin/setup-ds.pl

And guess what... I had the error again... So I went to the ~/debug folder on another terminal and did:

[root@fedora-ds debug]# grep "failed" *
setup.31676:read(4, "Server failed to start !!! Pleas"..., 4096) = 64
setup.31676:write(2, "Server failed to start !!! Pleas"..., 64) = 64
setup.31711:write(1, "Server failed to start !!! Pleas"..., 64) = 64
[root@fedora-ds debug]#

When I digged into setup.31711 I found:
read(255, "if test ! -f $STARTPIDFILE ; the"..., 2220) = 663
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat64("/var/run/dirsrv/slapd-fedora-ds.startpid", 0xbfcd8eb8) = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_SETMASK, [], NULL, = 0
fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f7c000
write(1, "Server failed to start !!! Pleas"..., 64) = 64

So this is a nice clue... /var/run/dirsrv... and guess what... the owner of this directory was fedora-ds (a user I set up initially for testing purposes for the Directory Server ) With the comment chown I corrected the owner of this folder... and now service dirsrv start works

Conclusion... strace is your best friend

As I wrote before, I did pass RHCE, RHCT and LPIC1. After a few weeks of study I also passed LPIC2 and ITIL

The results for ITIL:

---

Passing score: 65
Your score...: 67
Grade........: PASSED

Section Title                          Score
-------------------------------------- -----
General                                 66
Service Desk                            66
Incident Management                     25
Problem Management                      80
Change Management                       80
Configuration Management                75
Release Management                     100
Service Level Management                66
Availability Management                  0
Capacity Management                     50
IT Service Continuity Management       100
Financial Management for IT Services   100
Other ITIL Topics                      100
Relationships                           50

---

During the exam I start mixing up 'Availability management' and 'IT Service Continuity Management', which resulted into a score of 0 for Availability management

But I also passed LPIC 201 en LPIC 202.

LPIC 201 results:

---

Required score: 500
Your score....: 750
Status........: PASS

Section                              Percent Correct
------------------------------------ ---------------
Linux Kernel                               90%
System startup                             87%
Filesystem                                100%
Hardware                                   75%
File and Service Sharing                   87%
System Maintenance                         83%
System Customization & Automation          66%
Troubleshooting                            50%

---

LPIC 202 results:

---

Required score: 500
Your score....: 740
Status........: PASS

Section                              Percent Correct

------------------------------------ ---------------
Networking configuration                 85%
Mail & News                              92%
DNS                                      80%
Web Services                            100%
Network Client Management                66%
System Security                          80%
Network Troubleshooting                 100%

---

Every sysop might recognize this... I changed a config file some while ago... but what did I change and what is the history of that file. Besides of the changes, I want to have them in a backup

So I decided to setup a config-file repository, where the servers can commit automaticly their changes using SVN over SSH. I use SSH to have no password prompts but certificates. Only the user svn-backup can commit files to the repository. On both server I only checked out the repository-part concerning them.

[root@tank] svn co \
svn+ssh://svn-backup@svnserver.example.com/repos/config-files/tank \
/usr/data/svn-config

And in the crontab for the user root, we have the next entry:

45 * * * */usr/local/backup/svn-backup.sh

And in the svn-backup script we have the next lines:

---

#!/bin/bash
#
# Subversion script to backup configfiles
#
# Written by Pieter de Rijk <pieter -at- de-rijk.com>

SVN_BACKUP_PATH="/usr/local/svn-backup"

cp -R /usr/local/etc/ $SVN_BACKUP_PATH/usr/local/etc/ > /dev/null 2> /dev/null
cp -R /etc/ $SVN_BACKUP_PATH/etc/ > /dev/null 2> /dev/null
rm -rf $SVN_BACKUP_PATH/usr/local/etc/squid/errors
rm -rf $SVN_BACKUP_PATH/usr/local/etc/squid/icons/*.jpg
rm -rf $SVN_BACKUP_PATH/usr/local/etc/squid/icons/*.gif
cd $SVN_BACKUP_PATH \
/usr/local/bin/svn up > /dev/null 2> /dev/null
for files in "`/usr/local/bin/svn status $SVN_BACKUP_PATH | grep ^\?`";
do
   ADD_FILE=`echo $files | awk '{ print $2 }'`
   if [ ! -z $ADD_FILE ];
   then
     /usr/local/bin/svn add $ADD_FILE
   fi
done
/usr/local/bin/svn commit -m "[`hostname -s`] Config changes `date`"

---

And when something change I receive a message

Only users in the wheel group can checkout the repositories, but are not allowed to commit

As I wrote before some vendor promissed me a 16-core machine.... and indeed we received it. Now I have my 'toy' placed in the basement. Besides of some trouble putting a modified version of RHEL 4.5 on it, it works fine. At this moment I've installed RHEL 4.6 and it works fine.

But to test the 16-cores I decide to compile a brand new kernel:

[root@cool-system]$ mkdir -p /tmp/kernel-test; cd /tmp/kernel-test/
[root@cool-system]$ tar -xjvf ~/downloads/linux-2.6.24.tar.bz2
[root@cool-system]$ cd linux-2.6.24
[root@cool-system]$ make mrproper
[root@cool-system]$ cp /boot/config-2.6.9-67.ELlargesmp ./.config
[root@cool-system]$ echo y | make oldconfig
HOSTCC scripts/basic/fixdep
HOSTCC scripts/basic/docproc
HOSTCC scripts/kconfig/conf.o
[..snip..]
#
# configuration written to .config
#
[root@cool-system]$ time make -j 32 bzImage
scripts/kconfig/conf -s arch/x86/Kconfig
CHK include/linux/version.h
UPD include/linux/version.h
[..snip..]
Setup is 11352 bytes (padded to 11776 bytes).
System is 1835 kB
Kernel: arch/x86/boot/bzImage is ready (#1)

real 0m40.887s
user 5m15.688s
sys 2m1.029s
[root@cool-system]$ _

During the compile, all the 16-cores are working hard

For a project at work, I did some benchmark tests of the impact of SSH-tunnels on HTTP-traffic. And as we should suspect is true, on this test the traffic slows down with a factor 6. I did this, because some off-shore company out of India came up with some bad results from a benchmark, but they used SSH-tunnels.

So time for some testing... I did 6 types of tests using ApacheBench:

1. 10.000 requests with 200 concurrent connections without HTTP-keepalive via direct request to
   port 80 on the server
2. 10.000 requests with 200 concurrent connections with HTTP-keepalive via direct request to
   port 80 on the server
3. 10.000 requests with 200 concurrent connections without HTTP-keepalive via SSH-tunnel to
   port 80 on the server
4. 10.000 requests with 200 concurrent connections with HTTP-keepalive via SSH-tunnel to port
   80 on the server
5. 10.000 requests with 200 concurrent connections without HTTP-keepalive via SSH-tunnel with
   compression to port 80 on the server
6. 10.000 requests with 200 concurrent connections with HTTP-keepalive via SSH-tunnel with
   compression to port 80 on the server



For details check the report.




Phase Short Descr                         # Request/sec  Avg tim per request 
===== =================================== ============== ===================

   01 Direct                                     6323.94            0.158 ms
   02 Direct + Keepalive                         6319.27            0.158 ms
   03 SSH                                        1047.83            0.954 ms
   04 SSH + Keepalive                            1138.74            0.878 ms
   05 SSH + Compression                           824.09            1.213 ms
   06 SSH + Compression + Keepalive              1012.70            0.987 ms
===== =================================== ============== ===================