Just my Blog

use subversion to publish websites

nospam@example.com (Pieter de Rijk) — Wed, 04 Aug 2010 19:54:30 +0200

Sometimes I'm really surprised about myself... especially how lazy I am.

I'm currently playing around with one of my private websites, and to improve developing I decided to use subversion. So far so good, but I wanted that the committed subversion code was automatically online on the webserver. So I did the following very simple trick.

First I check out the code (subtree) from the subversion server (which is using https):

$ cd /sites
$ mv dev.adslweb.net{,-backup}
$ svn co https://svn.adslweb.net/svn/websites/dev.adslweb.net

Next step was to commit the current content of the website into subversion:

$ cd /sites/dev.adslweb.net
$ cp -Rv /sites/dev.adslweb.net-backup/* ./
$ svn add *
$ svn commit -m "Initial commit of ADSLWEB.net dev env"

Now download the simple script I created for making sure that subversion doesn't fire off twice for updating the same tree.

Download svn-update.sh via this link.

So something like this:

$ mkdir ~/scripts/
$ cd ~/scripts
$ wget http://www.xs4all.nl/~paderijk/pics/svn-update.sh
$ chmod 700 svn-update.sh

Now... the last step... create a crontab entry with the following content:

*/1 * * * * /home/pieter/scripts/svn-update.sh /sites/dev.adslweb.net 2>&1 > /dev/null

And guess... and it works like a charm, on every new commit done by whoever... you get your online site updated within 1 minute without the need log in into the website/webserver using ftp/ssh/whatever.

More flexible yum-repo sync script

nospam@example.com (Pieter de Rijk) — Tue, 01 Jun 2010 08:30:40 +0200

In the past I started syncing every night the Updates-repositories from Fedora and CentOS on a local server, just to speed up updates/kick starts et cetera... The first version of the script was very quick and dirty, now I've a more decent script that allow you to add/remove very quick new versions of CentOS and Fedora.

You can find the script here.

How full are your snapshot volumes in LVM?

nospam@example.com (Pieter de Rijk) — Thu, 22 Apr 2010 08:29:38 +0200

As I mentioned in my previous post, which is already 2 months old :(, I'm using snapshots for data retention.

Now I run up in the situation, that I wanted to know how full the snapshots are. A 'normal' df will not work... but I figured it out! The command lvs is willing to do the work:

# lvs --aligned --separator \| vol_backup
LV                |VG        |Attr |LSize |Origin|Snap% |Move|Log|Copy% |Convert
lvm0              |vol_backup|owi-ao|40.00G|      |      |    |   |      |
snap-20100412_2350|vol_backup|swi-a-| 4.00G|lvm0 | 23.71|    |   |      |
snap-20100413_2350|vol_backup|swi-a-| 4.00G|lvm0 | 21.70|    |   |      |
snap-20100414_2350|vol_backup|swi-a-| 4.00G|lvm0 | 19.52|    |   |      |
snap-20100415_2350|vol_backup|swi-a-| 4.00G|lvm0 | 17.53|    |   |      |
snap-20100416_2350|vol_backup|swi-a-| 4.00G|lvm0 | 15.54|    |   |      |
snap-20100417_2350|vol_backup|swi-a-| 4.00G|lvm0 | 13.56|    |   |      |
snap-20100418_2350|vol_backup|swi-a-| 4.00G|lvm0 | 11.56|    |   |      |
snap-20100419_2353|vol_backup|swi-a-| 4.00G|lvm0 | 9.02|    |   |      |
snap-20100420_2353|vol_backup|swi-a-| 4.00G|lvm0 | 6.76|    |   |      |
snap-20100421_2350|vol_backup|swi-a-| 4.00G|lvm0 | 2.79|    |   |      |

In the 'Snap%' column you can see how full your snapshot volume is!

Creating Snapshots of a backup using LVM snapshot

nospam@example.com (Pieter de Rijk) — Mon, 22 Feb 2010 09:18:43 +0100

Normally I used to have a backup-retention-script in place that will create a TAR-ball of the backup data (using Herakles). But this way I was not able to have a retention of longer then 3 days

So I had to look into another solution, I could add a new harddrive in the server... but there should be something else possible. So I ended up by using LVM snapshots. So I created a Volume group of about 100GB. In that volume group I created a logical volume of about 30GB, which is enough (and if not, we can 'grow' the Filesystem thanks to LVM )

After having all that done, I've created a script located in /root/scripts/lvm-snapshot. This script runs every midnight and creates a snapshot.

#!/bin/bash
#
# Create LVM Snapshots
#
#
#---------------------------------------------------------------------------------------------------------------
CURRENT_SNAPNAME="snap-"$(date "+%Y%m%d%H%M%S")
VOLUME2SNAPSHOT="/dev/vol_backup/lvm0"
LVMSNAPSHOTCMD="/usr/sbin/lvcreate -L 2G -s -n $CURRENT_SNAPNAME $VOLUME2SNAPSHOT"
LINE="---------------------------------------------------------------------------------------------------------------------"

echo $LINE
df -h /mnt/data
echo $LINE
$LVMSNAPSHOTCMD 2> /dev/null
#---------------------------------------------------------------------------------------------------------------
SNAPSHOT_RETENTION=15
CURRENT_SNAPSHOT_COUNT=$(lvdisplay | grep "^ LV Name                /dev/vol_backup/snap" | sort | awk '{ print $3 }' | wc -l)

OVERFLOW=$(echo $CURRENT_SNAPSHOT_COUNT - $SNAPSHOT_RETENTION | bc)
if [ $OVERFLOW -gt 0 ];
then
        echo $LINE
        for files in $(lvdisplay | grep "^ LV Name                /dev/vol_backup/snap" | sort | awk '{ print $3 }' | head -n$OVERFLOW);
        do
                 /usr/sbin/lvremove -f $files 2> /dev/null
        done
fi
#---------------------------------------------------------------------------------------------------------------
echo $LINE
/usr/sbin/vgdisplay vol_backup
echo $LINE
/usr/sbin/lvdisplay $VOLUME2SNAPSHOT

And the crontab entry is:

# crontab -l
0 0 * /root/scripts/lvm-snapshot