System Op stuff

This week I had to set up puppet with mongrel and apache for the balancing. To be honest... I never heard aboutthe puppet tool at all before (sorry puppet-developers ). As far as I understood, the puppetmaster is not really doing well with handling multiple concurrent requests and we need to provision about 128 compute nodes from a HPC cluster ...

On how I set up all the stuff in the test environment/production environment.

First I identified the right RPMs from EPEL, because I had no connection to the Internet:

augeas-0.5.3-1.el5.x86_64.rpm
augeas-libs-0.5.3-1.el5.x86_64.rpm
facter-1.5.7-1.el5.noarch.rpm
puppet-0.24.8-4.el5.noarch.rpm
puppet-server-0.24.8-4.el5.noarch.rpm
ruby-augeas-0.3.0-1.el5.x86_64.rpm
rubygem-daemons-1.0.7-2.el5.noarch.rpm
rubygem-fastthread-1.0.1-1.el5.x86_64.rpm
rubygem-gem_plugin-0.2.2-2.el5.noarch.rpm
rubygem-mongrel-1.0.1-6.el5.x86_64.rpm
rubygem-rake-0.8.3-1.el5.noarch.rpm
rubygems-1.3.1-1.el5.noarch.rpm
ruby-shadow-1.4.1-7.el5.x86_64.rpm

You can install them with 'yum --nogpgcheck localinstall *.rpm'.

You must als make sure that apache with mod_ssl is installed as well (yum install httpd mod_ssl).

Configuring Apache for load-balancing

As mentioned before, apache and puppet master must be installed.

First I copied and modified the /etc/init.d/httpd script to use it for puppet loadbalancing. Find my copy of the script on:

http://www.xs4all.nl/~paderijk/blog/puppet-balancer

So you can do:

# cd /etc/init.d/
# wget http://www.xs4all.nl/~paderijk/blog/puppet-balancer
# chmod 755 ./puppet-balancer

Now we have to create the some additional directories:

# mkdir -p /var/log/puppet-balancer
# mkdir -p /etc/puppet-balancer/{conf,conf.d}

I used the configuration file initially set up by from Jeff McCune and modified it for RHEL5. This file can be found on:

http://www.xs4all.nl/~paderijk/blog/puppet-balancer.conf

So you can put it into place like this:

# cd /etc/puppet-balancer/conf
# wget http://www.xs4all.nl/~paderijk/blog/puppet-balancer.conf

And finally we need to create a symlink name /usr/sbin/puppet-balancer which links to /usr/sbin/httpd

# ln -s /usr/sbin/httpd /usr/sbin/puppet-balancer

I've done this, because somehow the init-scripts are having some inconstancy.

And we also need a /etc/sysconfig/puppet-balancer file

# cd /etc/sysconfig
# wget http://www.xs4all.nl/~paderijk/blog/sysconfig-puppetbalancer
# mv sysconfig-puppetbalancer puppet-balancer

For so far the apache configuration, still we don't start the puppet-balancer 'service'. First we need to configure and start the puppet master.

Configuration puppetmaster

The configuration of the puppetmaster was actually very easy.

*UPDATE* A colleague pointed me on the fact that the CA wasn't set up properly going into mongrel mode. So first start puppetmaster normally:

# service puppetmaster start ; sleep 5 ; service puppetmaster stop

Make sure the following line is 'enabled' to /etc/sysconfig/puppetmaster:

PUPPETMASTER_PORTS=( 18140 18141 18142 18143 )

This will enable puppetmaster with mongrel and starts on the ports 18140 to 18143

I also noticed that the RPM for puppetmaster didn't set up all the required directories, so I need to add them manual:

# mkdir -p /var/lib/puppet/yaml/{facts,nodes}
# chown puppet:puppet /var/lib/puppet/yaml/{facts,nodes}

Start the puppetmaster and Apache (puppet-balancer)

Now we can start the puppetmaster:

# service puppetmaster start

But we want to survive reboots as well, so we do also:

# chkconfig puppetmaster on

Now the Puppet CA is also set up. So now we can start the puppet-balancer (apache) as well. But first we need to symlink the key-file and the certificate file:

# ln -s /var/lib/puppet/ssl/private_keys/{$(hostname -f).pem,puppet-balancer.pem}
# ln -s /var/lib/puppet/ssl/certs/{$(hostname -f).pem,puppet-balancer.pem}

And now we can start the puppet-balancer:

# service puppet-balancer start

And get it also up and running after a reboot:

# chkconfig puppet-balancer on

More details on how to do further configuration/setting up manifest files et cetera can be found on the Puppet wiki:

http://reductivelabs.com/trac/puppet/

Recently I need to create a report about utilization of an HPC Cluster that uses Grid Engine, but we didn't had ARCO and so not running yet for that cluster

So I digged into my brain on how to load data from a "RAW" format into a database... it's something I did when I worked for PricewaterhouseCoopers Advisory, but then I used financial data.

Please press the continue reading link below... to read more

A user group we support uses an application and they have the license to use 20 concurrent runs. For them we've implement very recently the Sun Grid Engine job scheduler. Although they recently start complaining that jobs didn't run.

Their cluster exists out of 8 nodes with each 2 cores, so they've 16 slots in the job scheduler. We've set up two queues, the suspendable.q and the unsuspendable.q. Jobs in the suspendable.q queue can get suspended by jobs in the unsuspendable.q queue. So we can have in total 32 concurrent jobs (where 16 will be suspended).

Once their cluster is really busy, some jobs will not run... and after some investigation we found out why. The jobs that are suspended don't release their license to the license server. So we can have a total of 16 jobs in the suspendable.q and 4 in the unsuspendable.q. So we start limiting the number of unsuspendable jobs to 4, because the 21nd job that will start running will fail because it won't get a license.

Yesterday I thought let's play with FC12 (aka Rawhide, aka FC11.90). So I enabled the Rawhide-repositories on my FC11 laptop and entered "yum -y update". And after a while it was there... bleeding edge kernel and other packages.

The first issue I run into, was that Firefox 3.5 was not able to run, it caused a segfault. There seems to be a bug in the xulrunner package. So I was able to fix it, by "downgrading" Firefox to 3.0.11, but that one crashed on pages using "Adobe Flash plugin". So I removed the flash plugin, because I wanted bleeding edge Fedora. So having that "sort out" I wanted to suspend my laptop, and guess what... It didn't want to suspend... so after some hacking around... it still didn't work.

So my final decision was Go back to FC11. I was able to "downgrade" my system in about 60 minutes. At home I've a mirror repository with al the backups, so during installation I added these repositories, so I also had all the updates in one go.

Lesson learned: "Bleeding edge... is indeed bleeding edge!"

I need my work for my daily work... If I won't need it for my daily work I would have keep FC12 (aka Rawhide, aka FC11.90) on it to participate in developing FC12.

On a kind of "intranet" website, which is secured with username/password combinations and HTTPS I've implemented the next feature:

- Authorized users can read everything on the website

- Files with in their filename "classified" requires a valid SSL-Client certificate...

Here is the output of my apache config:

<Directory /usr/sites/ssl-site/intranet/htdocs>
  Options Indexes MultiViews
  AllowOverride Authconfig
  Order allow,deny
  Allow from all
  AuthName "intranet"
  AuthType "Basic"
  AuthUserFile /usr/sites/ssl-site/intranet/etc/users.pwl
  require valid-user
</Directory>

<LocationMatch .*(c|C)(l|L)(a|A)(s|S)(s|S)(i|I)(f|F)(i|I)(e|E)(d|D).+>
  SSLVerifyClient require
  SSLVerifyDepth 1
  SSLOptions +OptRenegotiate
</LocationMatch>

I still have to sort out some issues, like directories having a directory with the name "classified" in them.

Recently I got a iPhone, but I have multiple mails coming into my mailbox (private/business/sysop). I use maildrop to put them into the right folders. I want share my business (sub)folder(s) with my special iPhone-account... but how could we do that... (please note you should have admin-privileges)

Step 1 - Create new user and put the "source" mailbox user in the right group

Create a iPhone user on your server (in my case user is iphone) and add the user (in my case pieter) to the iphone group (created during the creation of the iphone user).

Step 2 - Set permissions correct of the source mailbox

Make sure the world can access ~pieter/Maildir, set this by entering:

[ root@server ~]# chmod o+x ~pieter/Maildir

New we also have to set the grouppermissions correct of the source sub-folders:

[ root@server ~]# chown -R pieter:iphone ~pieter/Maildir/.Business*

Set groupbit and grouppermissions on the folders you want to share:

[ root@server ~]# find ~pieter/Maildir/.Business* -type d -exec chmod 2770 {} \;

Set the grouppermissions on the current messages"

[ root@server ~]# find ~pieter/Maildir/.Business* -type f -exec chmod  0660 {} \;

Step 3 - Setup the functional account and mailstructure

Become that user (can be done via sudo).
[ pieter@server ~]$ sudo su - iphone
Password: ****
[ iphone@server ~]$

Create the maildir structure:

[ iphone@server ~]$ maildirmake ~/Maildir

Remove the cur, new and tmp folders:

[ iphone@server ~]$ rm -rf ~/Maildir/[cnt]*

Now link them to the source:

[ iphone@server ~]$ for x in cur new tmp; do ln -s /home/pieter/Maildir/.Business/$x ~iphone/Maildir/$x; done

Step 4 - Share the subfolders as well

[ iphone@server ~]$ cd ~/Maildir
[ iphone@server Maildir]$ maildirmake .Archive
[ iphone@server Maildir]$ rm -rf ~/.Archive/[cnt]*
[ iphone@server Maildir]$ for x in cur new tmp; do ln -s /home/pieter/Maildir/.Business.Archive/$x ~iphone/Maildir/.Archive/$x; done

Perform step 4 for al the other subfolders you would like to share (Please note that you've to set the permissions in step 2 as well). This was done on a FreeBSD6.3 system, I don't know what the impact might be on Linux systems with SELinux... nor I don't know what the impact might be of the chmod o+x on Maildir... we wil investigate. Initially I did a chown pieter:iphone on the source maildir... but my imap-server refused connection due to wrong gid.

Also keep in mind to put in your procmail/maildrop filter a umask of 007!

But... conclusion... it works cool.

It can happen, you have sudo-access to another account (most of the time it will be access to the root account). But most of the time the NOPASSWD option is not used due to security reasons. But there are moments you want to have sudo-credentials available, think about a script or something else.... I had the same issue, so I found the next "hack" to get the timestamp refreshed every 60 seconds.

(Please note the script will use user "root" but it can be another user, please modify the scripts so it fits your needs).

Step 1)

Create a script in you $HOME/bin with the next content (I call it sudo-hack.sh):

---

#!/bin/bash 
while [ true ]; 
do 
  sudo -u root /bin/true > /dev/null 2> /dev/null 
  sleep 60 
done 

---

Step 2)

Get a valid sudo-timestamp:

$ sudo -u root /bin/true
Password:
$

Step 3)

Start sudo-hack.sh in the background:

$ $HOME/bin/sudo-hack.sh &
$

That's all!

This week I had the "Red Hat Enterprise Directory Services and Authentication" course and exam in Amsterdam.

In the course we had some very nice stuff, like Red Hat DS and at the end Red Hat Enterprise IPA... all very cool... but today I had the exam (due to the RedHat NDA I am not allowed to say anything about the exam, so I won't do it)... but a few hours after the exam I received my results... and I passed the exam

Currently I own and maintain some domains. For these domains you need a DNS-server to make a proper translation from hostname to ip-adress.

blog.adslweb.net -> 80.126.215.23

But to keep redundancy it is smart to have some extra DNS-servers configured. So I setup a 'master' DNS and a 'slave' DNS, but both of them were connected to a server on a ADSL line... This might not be a big problem, but due to some problems recently, I preferred to have something somewhere else

After some Google-ing, I found Twisted4Life, if you create an account, they provide you for free a 'slave' DNS.

NOTE: Some of the software is very experimental!!! I had some issues that my .ics was totally removed when I added a new 'task' via lightning, please make a backup of your ics file!

Some while ago I wrote a small article about WebDAV and Calendar files. Until so far no results on reading the files into my Nokia , but... I found an extension for thunderbird to handle meeting-requests from Outlook clients

You can find information on the project website of 'lightning'. To configure your webdav, you need to click on the calendar button

Click in the left pane with the right button en select 'New Calendar', now you can use the wizard to select your calendar.

And now let the invitations come...

Yesterday evening I start playing with Fedora Directory Server

So first I setup Fedora Core 8 as a VMWare-instance... But after some playing around, I had the next message:

"Server failed to start !!! Please check errors log for problems"

And guess what... no information at all in the logs So removed the packages and the next directories:

/etc/dirsrv
/etc/sysconfig/dirsrv
/var/lock/dirsrv
/var/lib/dirsrv

So no information... then strace will be your best friend

So I started:

[root@fedora-ds debug]# strace -o ~/debug/setup -ff /usr/sbin/setup-ds.pl

And guess what... I had the error again... So I went to the ~/debug folder on another terminal and did:

[root@fedora-ds debug]# grep "failed" *
setup.31676:read(4, "Server failed to start !!! Pleas"..., 4096) = 64
setup.31676:write(2, "Server failed to start !!! Pleas"..., 64) = 64
setup.31711:write(1, "Server failed to start !!! Pleas"..., 64) = 64
[root@fedora-ds debug]#

When I digged into setup.31711 I found:
read(255, "if test ! -f $STARTPIDFILE ; the"..., 2220) = 663
rt_sigprocmask(SIG_BLOCK, NULL, [], 8)  = 0
stat64("/var/run/dirsrv/slapd-fedora-ds.startpid", 0xbfcd8eb8) = -1 ENOENT (No such file or directory)
rt_sigprocmask(SIG_SETMASK, [], NULL, = 0
fstat64(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f7c000
write(1, "Server failed to start !!! Pleas"..., 64) = 64

So this is a nice clue... /var/run/dirsrv... and guess what... the owner of this directory was fedora-ds (a user I set up initially for testing purposes for the Directory Server ) With the comment chown I corrected the owner of this folder... and now service dirsrv start works

Conclusion... strace is your best friend

Some while ago, I submitted a bug to the Gentoo Bug-trac. I had trouble with Firefox 2.0.0.2 when switching from non-proxy to a proxy-network.

http://bugs.gentoo.org/show_bug.cgi?id=169155

I found that when starting firefox (note: not firefox-bin) the proces wants to connect to www.gentoo.org

Now someone found the issue:

I have the same problem with mozilla-firefox-2.0.0.13.
Firefox tries to connect to the page mentioned in
/usr/lib/mozilla-firefox/defaults/pref/all-gentoo.js:
pref("browser.startup.homepage",           "http://www.gentoo.org/");
If DNS or the page itself is unreachable firefox waits until the request
timeouts.

I haven't test it yet... but it might solve the issue...

Yesterday evening I completed for 50% an entry from my wish-list regarding my servers. On the backup-MTA I now have TLS and authenication enabled on Sendmail

I wanted to have this enabled, because from my laptop I often setup an SSH-tunnel, but on other devices it wasn't really possible

After some Google-ing I found this page which was really helpfull in setting the stuff up on sendmail. The next step is to use TLS between the MTA's I own and TLS between other parties who provide TLS on their MTA.

At this moment, I have at home a FreeBSD server named tank.adslweb.net. This Server is acting as backup dns/mta/backup for the fileserver. But this 'server' is actually my oldlaptop

So recently it came up into my mind to virtualize some services... so I requested a quote of my local computer reseller (A-SYS Computers) for the next components:

1x AMD Athlon 64X2 4400+ dual-core

1x ASUS M2N

1x Kingston 2GB

1x El-Cheapo videokaart

1x Aopen QF50C (Black of Silver)

1x 160GB Harddisk SATA

2x 500GB Harddisk SATA

I will install Gentoo 64bit on it with VMWare-server. Once that is all setup and running, I will move all my current virtual systems to that box... and virtualize tank.adslweb.net as I did before But this won't be done while the system is running... I will turn it off and boot it with a knoppix live-cd.

As I wrote before, I did pass RHCE, RHCT and LPIC1. After a few weeks of study I also passed LPIC2 and ITIL

The results for ITIL:

---

Passing score: 65
Your score...: 67
Grade........: PASSED

Section Title                          Score
-------------------------------------- -----
General                                 66
Service Desk                            66
Incident Management                     25
Problem Management                      80
Change Management                       80
Configuration Management                75
Release Management                     100
Service Level Management                66
Availability Management                  0
Capacity Management                     50
IT Service Continuity Management       100
Financial Management for IT Services   100
Other ITIL Topics                      100
Relationships                           50

---

During the exam I start mixing up 'Availability management' and 'IT Service Continuity Management', which resulted into a score of 0 for Availability management

But I also passed LPIC 201 en LPIC 202.

LPIC 201 results:

---

Required score: 500
Your score....: 750
Status........: PASS

Section                              Percent Correct
------------------------------------ ---------------
Linux Kernel                               90%
System startup                             87%
Filesystem                                100%
Hardware                                   75%
File and Service Sharing                   87%
System Maintenance                         83%
System Customization & Automation          66%
Troubleshooting                            50%

---

LPIC 202 results:

---

Required score: 500
Your score....: 740
Status........: PASS

Section                              Percent Correct

------------------------------------ ---------------
Networking configuration                 85%
Mail & News                              92%
DNS                                      80%
Web Services                            100%
Network Client Management                66%
System Security                          80%
Network Troubleshooting                 100%

---