Zach Burlingame
Programming, Computers, and Other Notes on Technology

May, 2011Archive for

HOWTO: Upgrade from Subversion 1.4 to 1.6 on CentOS 5

Friday, May 27th, 2011

How to upgrade the packages and existing repositories from Subversion 1.4 to 1.6.6 on CentOS 5.

# File: Subversion_1.6_Upgrade.notes
# Auth: burly
# Date: 12/01/2009
# Refs:
#       http://dev/
# Desc: Upgrading from subversion 1.4 to 1.6.6 on CentOS 5
#       NOTE:These instructions are actually fairly generic 
#       in regards to the version of SVN you are upgrading
#       from/to. At the time of writing, it just happened
#       to be 1.4 -> 1.6.6

# Backup each repository
svnadmin dump /srv/svn/<repo> > /backup/svn/<Repo>_20091201_rXXXX.dump

# Backup any hooks or configuration files in 
# /srv/svn/<repo>/hooks and /srv/svn/conf

# Setup yum to allow the package to come in from
# RPMforge (must setup RPMforge repo first).
vim /etc/yum.repos.d/Centos-Base.repo

# Add the following line at the end of each section
# in the Centos-Base.repo
exclude=subversion mod_dav_svn

# Restart the yum update daemon
service yum-updatesd restart

# Upgrade subversion
yum upgrade subversion

# For each repository
#    delete the existing repo
rm -rf /srv/svn/<repo>

# Create a new repo
svnadmin create /srv/svn/<repo> --fs-type fsfs

# Import the data
svnadmin load /srv/svn/<repo> < /backup/srv/<Repo>_20091201_rXXXX.dump

# Restore any hooks or configuration files in 
# /srv/svn/<repo>/hooks and /srv/svn/<repo>/conf

# If you are using Trac, you'll need to resync the repo
trac-admin /srv/trac/<repo> resync

HOWTO: Migrate an Existing RAID Array to a New Array

Wednesday, May 25th, 2011

How to migrate from an existing software RAID 1 array to a new RAID 1 array on CentOS 5.5

# File: Migrate_to_new_RAID_Array_on_CentOS_5.5.notes
# Auth: burly
# Date: 11/20/2010
# Refs: 
# Desc: How migrate from one RAID 1 array to a new one
#       on CentOS 5.5

# I booted from a Knoppix CD to do this. In retrospect,
# I should have used a CentOS LiveCD because the
# tooling, versions, and layout of Knoppix are different 
# which caused some issues. Also, because my OS is x86-64
# but Knoppix is x86, I could not chroot into my system 
# environment, which are ultimately required to create the
# initrd files.

# Boot from the Knoppix CD and drop to a shell

# Start up the existing RAID Array (one of the 2 drives
# from the existing RAID 1 array was on sdc for me)
mdadm --examine --scan /dev/sdc1 >> /etc/mdadm/mdadm.conf
mdadm --examine --scan /dev/sdc2 >> /etc/mdadm/mdadm.conf
mdadm --examein --scan /dev/sdc3 >> /etc/mdadm/mdadm.conf
/etc/init.d/mdadm start
/etc/init.d/mdadm-raid start

# Partition first SATA drive in whatever partition numbers
# and sizes you want. Make sure all partitions that 
# will be in an RAID array use ID type "fd" for RAID 
# autodetect and type "82" for swap. Make sure /boot
# is marked with the bootable flag
fdisk /dev/sda
# Repeat for the other disks OR if you are using the
# identical setup on each, you can use sfdisk to 
# simplify your life.
sfdisk -d /dev/sda | sfdisk /dev/sdb

# Create the new boot array
# NOTE: If you don't use metadata 0.90 (but instead 
#       1.0 or 1.1) you'll run into problems with grub.
#       In RAID 1, with metadata 0.90, you can mount
#       the fs on the partition without starting RAID.
#       With newer versions of metadata the superblock
#       for RAID gets written at the beginning of the 
#       partition where the filesystem superblock
#       normally would go. This results in the inability
#       to mount the filesystem without first starting
#       RAID. In the case of your boot partition, this 
#       results in the inability to setup grub and thus boot.
mdadm --create --verbose --metadata=0.90 /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1

# Copy everything over for /boot
mkdir /mnt/oldBoot
mkdir /mnt/newBoot
mkfs.ext3 /dev/md0
mount --options=ro /dev/md0 /mnt/oldBoot
cd /mnt/oldBoot
find . -mount -print0 | cpio -0dump /mnt/newBoot

# Make the new swap
mkswap /dev/sda2
mkswap /dev/sdb2

# Create the new array for LVM. I used metadata
# 0.90 again for consistency AND because I believe
# the version of mdadm in CentOS won't handle newer
# versions of it
mdadm --create --verbose --metadata=0.90 /dev/md1 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3

# Setup LVM2
pvcreate /dev/md1
vgcreate vg /dev/md1
lvcreate -L8G -nroot vg
lvcreate -L10G -nhome vg
lvcreate -L250G -nvm vg

# Format the filesystems.
# NOTE: I fixed the reserved space to 1% (default is 5%)
#       for the VM LV to save some space and 
#       because in larger, non-root partitions, you
#       don't need all that reserved space.
mkfs.ext3 /dev/vg/root
mkfs.ext3 /dev/vg/home
mkfs.ext3 -m 1 /dev/vg/vm

# Copy everything over for /
mkdir /mnt/oldRoot
mkdir /mnt/newRoot
mount --options=ro /dev/vgOS/lvRoot /mnt/oldRoot
mount /dev/vg/root /mnt/newRoot
cd /mnt/oldRoot
find . -mount -print0 | cpio -0dump /mnt/newRoot

# Copy everything over for /home
mkdir /mnt/oldHome
mkdir /mnt/newHome
mount --options=ro /dev/vgOS/lvHome /mnt/oldHome
mount /dev/vg/home /mnt/newHome
cd /mnt/oldHome
find . -mount -print0 | cpio -0dump /mnt/newHome

# Copy everything over for /boot
mkdir /mnt/oldVM
mkdir /mnt/newVM
mount --options=ro /dev/vgOS/lvVM /mnt/oldVM
mount /dev/vg/vm /mnt/newVM
cd /mnt/oldVM
find . -mount -print0 | cpio -0dump /mnt/newVM

# Remove any existing/stale lines in the mdadm.conf file

# Setup the mdadm config on the new /
mdadm -Esb /dev/sda1 >> /mnt/newRoot/etc/mdadm.conf
mdadm -Esb /dev/sda3 >> /mnt/newRoot /etc/mdadm.conf

# Update fstab on the new machine to use the new 
# mount points (e.g. if you changed VolumeGroup or 
# LogicalVolume names)
vim /mnt/newRoot/etc/fstab

# REBOOT TO A CENTOS LIVECD (if you weren't already on one)

# First we chroot
mkdir /mnt/sysimage
mount /dev/vg/root /mnt/sysimage
mount /dev/vg/home /mnt/sysimage/home
mount /dev/md0 /mnt/sysimage/boot
mount --bind /dev /mnt/sysimage/dev
mount -t proc none /mnt/sysimage/proc
mount -t sysfs none /mnt/target/sys
chroot /mnt/sysimage

# Make a new initrd to boot from
cd /boot
mv initrd-2.6.18-194.26.1.el5.img initrd-2.6.18-194.26.1.el5.img.bak
mkinitrd initrd-2.6.18-194.26.1.el5.img  2.6.18-194.26.1.el5

# Setup grub on both of the drives

# Reboot!

HOWTO: Create a Local Repository Mirror on Ubuntu

Monday, May 23rd, 2011

How to create and use a local repository mirror on Ubuntu 9.10. These instructions should work with minor modifications for other versions of Ubuntu.

# File: HOWTO Create a Local Repository Mirror on Ubuntu.notes
# Date: 2010/03/17
# Refs:
# Desc: How to create a local repository for 
#       Ubuntu 9.10 Karmic Koala.

# -------------------------------------
#           Setup the Server
# -------------------------------------
# Install Ubuntu (I used 9.10) on a machine with plenty of 
# free storage space (I used an 8GB OS vmdk and an 80GB data 
# vmdk used through LVM so that I could easily add/grow to
# it in the future if necessary).

# Create the mirror user, I'll be using ubuntu.
# NOTE: You don't have to add this user to the wheel but if you don't, the steps below that require sudo
#       will require you to run them from an account with root or wheel access and may also require
#       that you change the ownership/group of files/directories afterwards.
sudo useradd -m ubuntu -Gusers,wheel
sudo password ubuntu

# UPDATE 2012/01/30: As Dave points out below, you'll need to create your mirrorkeyring folder with the correct user account.
#                    If you aren't already running as that user, you can change your shell using su
su - ubuntu

# Update your apt-get package listing
sudo apt-get update

# Install debmirror
sudo apt-get install debmirror

# Create the location for the repo data to live
sudo mkdir -P /mirror/ubuntu

# Set the permissions for the repo data
sudo chown -R ubuntu:ubuntu /mirror/ubuntu
sudo chmod -R 771 /mirror/ubuntu

# Setup the keyring for correctly verifying Release signatures
gpg --no-default-keyring --keyring /home/ubuntu/mirrorkeyring/trustedkeys.gpg --import /usr/share/keyrings/ubuntu-archive-keyring.gpg

# Create the script
vim /home/ubuntu/

# NOTE: The ubuntu community documentation has you using 
#       the HTTP protocol for the mirror build script
#       however, I prefer rsync because we can rate limit.
#       When the download is going to take days,
#       I'd like to be able to use my connection in
#       the interim.

# --------------------------------------------
# --------------------------------------------


## Setting variables with explanations.

# Don't touch the user's keyring, have our own instead
export GNUPGHOME=/home/ubuntu/mirrorkeyring

# Arch=         -a      # Architecture. 
# For Ubuntu can be i386, amd64, powerpc and/or sparc (sparc support begins with dapper)
# Comma separated values

# Minimum Ubuntu system requires main, restricted
# Section=      -s      # Section
# (One of the following - main/restricted/universe/multiverse).
# You can add extra file with $Section/debian-installer.
# ex: main/debian-installer,universe/debian-installer,multiverse/debian-installer,restricted/debian-installer

# Release=      -d      # Release of the system
# (Dapper, Edgy, Feisty, Gutsy, Hardy, IntrepidJaunty, Karmic), 
# and the -updates and -security ( -backports can be added if desired)

# Server=       -h      # Server name,
# minus the protocol and the path at the end
# CHANGE "*" to equal the mirror you want to create your
# mirror from. au. in Australia  ca. in Canada. This can be 
# found in your own /etc/apt/sources.list file, 
# assuming you have Ubuntu installed.

# Dir=          -r      # Path from the main server,
# so http://my.web.server/$dir, Server dependant
# Lead with a '/' for everything but rsync,
# where we lead with a ':'

# Proto=        -e      # Protocol to use for transfer
# (http, ftp, hftp, rsync)
# Choose one - http is most usual the service, and the
# service must be availabee on the server you point at.
# NOTE: debmirror uses -aIL --partial by default.
#       However, if you provide the --rsync-options
#       paramter (which we do) then you HAVE to provide 
#       it -aIL --partial in addition to whatever You
#       want to add (e.g. --bwlimit) If you don't
#       debmirror will exit with thousands of files
#       missing.
rsyncoptions="-aIL --partial --bwlimit=100"

# Outpath=              # Directory to store the mirror in
# Make this a full path to where you want to mirror the material.

# The --nosource option only downloads debs and not deb-src's
# The --progress option shows files as they are downloaded
# --source \ in the place of --no-source \ if you want sources also.
# --nocleanup  Do not clean up the local mirror after mirroring
# is complete. Use this option to keep older repository
# Start script
debmirror       -a $arch \
                --no-source \
                -s $section \
                -h $server \
                -d $dist \
                -r $inPath \
                --progress \
                -e $proto \
                --rsync-options="$rsyncoptions" \

# -----------------------------------------------------
# -----------------------------------------------------

# Add execute permissions on the script
chmod +x

# Run the script

# Go home, kick back, have a beer while it downloads 43GBs 
# (in the case of karmic, karmic-update, karmic-securty for
# i386 and amd64)

# --------------------------------------
#          Setup the mirror
# --------------------------------------
# Install apache2
sudo apt-get install apache2

# Symlink the mirror data into the web root
sudo ln -s /mirror/ubuntu /var/www/ubuntu

# Point your browser at http://localhost/ubuntu and
# you should see your pool!

# -------------------------------------
#        Updating the Repo Mirror
# -------------------------------------
# To update the repo mirror, just execute the
# script used to initially build it.

# -------------------------------------
#   Configure Clients to Use this Repo
# -------------------------------------
# Update the apt sources list
cd /etc/apt
sudo mv sources.list sources.list.orig
sudo sensible-editor sources.list

# Replace 'mirrorbox' with your server's DNS name 
# (e.g.
# -----------------------------------------------------------------------------
# -----------------------------------------------------------------------------
# Local network mirror sources.
deb http://mirrorbox/ubuntu karmic main restricted universe multiverse
deb http://mirrorbox/ubuntu karmic-updates main restricted universe multiverse
deb http://mirrorbox/ubuntu karmic-security main restricted universe multiverse
# -----------------------------------------------------------------------------
# -----------------------------------------------------------------------------

# Test to see if you are able to pull down updates 
# from the new mirror
sudo apt-get update

Erasing a hard drive using a Linux LiveCD

Friday, May 20th, 2011

Normally when I need to erase a hard drive, I use dban. Recently however, I’ve run into issues with dban not detecting disks (I’m guessing it doesn’t support the I/O controller/drivers). While it isn’t as secure, a decent and easy way is to just zero out the hard drive using a Linux LiveCD (besides, if you really want it done securely, physically destroy the drive). Ubuntu is my usual distro of choice but there are tons out there that will work.

dd if=/dev/ of=/dev/sda bs=4096

To get an update on it’s progress, you can signal it from another terminal using

pkill -USR1 ^dd

HOWTO: Setup a DHCP Server on Ubuntu 9.10

Friday, May 20th, 2011

Setting up a DHCP server on a LAN with Ubuntu 9.10. These instructions should also basically work on Ubuntu 10.x.

# File:	HOWTO Configure a DHCP Server on Ubuntu.notes
# Date:	2010/03/24
# Refs:
# Desc:	Setting up a DHCP server on a LAN with Ubuntu 9.10

# Install DHCP3
sudo apt-get install dhcp3-server

# Specify the interface(s) that dhcp3-server should manage
# in /etc/default/dhcp3-server

# Set a static IP for the DHCP server itself on the
# interfaces that it will manage in /etc/network/interfaces
auth eth0
iface eth0 inet static

# Edit the etc/dhcp3/dhcpd.conf configuration file
# I'm going to be running a host-only 
# vmnet on eth0 with fixed addresses for several machines 
# in my example here
ddns-update-style none;
log-facility local7;

subnet netmask {

    option routers    ;
    option subnet-mask;
    option broadcast-address;
    option domain-name-servers;
    option ntp-servers;
    default-lease-time          7200;
    max-lease-time              86400;

    host helium {
            hardware ethernet 00:0c:29:c6:de:09;
    host lithium {
            hardware ethernet 00:0c:29:d8:d5:7f;
    host beryllium {
            hardware ethernet 00:0c:29:b6:93:41;
    host boron {
            hardware ethernet 00:0c:29:3f:c6:f3;

HOWTO: Setup a Locally Authoritative DNS Server on Ubuntu 9.10

Wednesday, May 18th, 2011

How to setup a locally authoritative DNS server on a LAN with Ubuntu 9.10. These instructions should work almost entirely on Ubuntu 10.x without modification.

# File:	HOWTO Setup a DNS Server on Ubuntu 9.10.notes
# Date:	2010/03/25
# Refs:
# Desc:	Setting up a locally authoritative DNS server on a LAN with Ubuntu 9.10

# Install bind9 (the actual DNS server) and the dnsutils package
# (useful for testing and troubleshooting DNS issues). We 
# also install resolvconf so that we can manage the static
# IP DNS settings in /etc/network/interfaces since NetworkManager
# will stomp on any changes we make to /etc/resolv.conf
sudo apt-get install bind9 dnsutils resolvconf

# Establish a static IP for the nameserver by editing
# /etc/network/interfaces. In our setup, we have a multihomed
# machine with eth0 static on the trusted LAN and eth1 DHCP
# to the internet
auto eth0
iface eth0 inet static
    # dns-options are implemented by the resolvconf package

# Make the static IP changes take affect
# NOTE: You should be able to use 
# ifconfig <interface> down/up, but I found rebooting to be
# the only reliable way. Also, when these changes take effect,
# you'll have NO internet DNS yet, so beware.
sudo shutdown -r now

# For our setup here, I want to be an authoritative nameserver 
# on the local LAN, so I need at least 1 Forward Zone and
# 1 Reverse Zone for it. Our domain is and our 
# subnet is First we add the zones to
# /etc/bind/named.conf.local
zone "" {
        type master;
        file "/etc/bind/";

zone "" {
        type master
        notify no;
        file "/etc/bind/db.192";

# Use an existing zone file as a template
sudo cp /etc/bind/db.local /etc/bind/

# Edit the /etc/bind/ file to fit our zone requirements
; BIND data file for
$TTL    604800
@       IN      SOA (
                       20100325         ; Serial (YYYYMMDD)
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
@       IN      NS
@       IN      A
ns      IN      A

; Hostname entries
hydrogen    IN  A
helium      IN  A
lithium     IN  A
beryllium   IN  A
boron       IN  A

# Use an existing reverse zone file as a template
sudo cp /etc/bind/db.127 /etc/bind/db.192

# Edit the Reverse Zone file for our reqs
; BIND reverse data file for
$TTL    604800
@       IN      SOA (
                       20100325         ; Serial (YYYYMMDD)
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
@       IN      NS      ns.
1       IN      PTR

; Hostname RNL pointers
2       IN      PTR
3       IN      PTR
4       IN      PTR
5       IN      PTR

# Start the bind9 daemon
sudo /etc/init.d/bind9 start

# Check the status of the server. You should get a big 
# printout from the local nameserver
dig @localhost

Resolving Redefinition Errors Betwen ws2def.h and winsock.h

Tuesday, May 17th, 2011

The Problem

This particular issue I run into much more frequently than I would like (full pathnames removed for brevity):

ws2def.h(91): warning C4005: ‘AF_IPX’ : macro redefinition
winsock.h(460) : see previous definition of ‘AF_IPX’
ws2def.h(131): warning C4005: ‘AF_MAX’ : macro redefinition
winsock.h(479) : see previous definition of ‘AF_MAX’
ws2def.h(168): warning C4005: ‘SO_DONTLINGER’ : macro redefinition
winsock.h(402) : see previous definition of ‘SO_DONTLINGER’
ws2def.h(212): error C2011: ‘sockaddr’ : ‘struct’ type redefinition
winsock.h(485) : see declaration of ‘sockaddr’
ws2def.h(390): error C2059: syntax error : ‘constant’
ws2def.h(524): warning C4005: ‘IN_CLASSA’ : macro redefinition
winsock.h(287) : see previous definition of ‘IN_CLASSA’

The Cause

The problem is due to an incompatibility between winsock.h and WinSock2.h. This usually occurs when something has included Windows.h (which includes winsock.h) before WinSock2.h, like this:

#include <Windows.h>
#include <WinSock2.h>

int main( int argc, char* argv[] )
  return 0;

Of course, finding where this include ordering has occurred is often more complex than this example. Turn on the “Show Includes” compiler option and look in the output window of your build to trace the occurrence to your source files.

The Solution

You basically have three options to fix this:


Defining WIN32_LEAN_AND_MEAN will reduce the size of the Windows headers by excluding several APIs, including Windows Sockets.

#include <Windows.h>
#include <WinSock2.h>

int main( int argc, char* argv[] )
  return 0;

Option 2: Explicitly include WinSock2.h before Windows.h

By explicitly including WinSock2.h before every place that you Windows.h, you prevent the collision. However, this can be quite cumbersome at times.
UPDATE 2011-11-12:This method can cause issues with struct packing and alignment if WIN32 isn’t defined before including WinSock2.h (see Oren’s comment below). To resolve this issue, either define WIN32 as a preprocessor flag in your project or explicitly define #WIN32 prior to the WinSock2.h include as I’ve done below.

#ifndef WIN32
  #define WIN32
#include <WinSock2.h>
#include <Windows.h>

int main( int argc, char* argv[] )
  return 0;

Option 3: Create a WinSock Wrapper Header

This option creates a header file that prevents the collision in the first place and then you include this at (or nearest as possible) to the top of every file that needs WinSock2.h.

The following is based on code from this stackoverflow answer.

#if _MSC_VER > 1000
#pragma once

#ifndef _WINDOWS_
#include <windows.h>

#include <winsock2.h>

#pragma comment(lib, "ws2_32.lib")

If the #pragma option above is unfamiliar to you, the compiler passes it along to the linker telling it to include the lib rather than having to set it in the project settings. Theres more on #pragma comment options over on MSDN.

HOWTO: Configure a Local NTP Server on Ubuntu 9.10

Monday, May 16th, 2011

Setting up a Network Time Protocol (NTP) server and configuring NTP clients on a LAN with Ubuntu 9.10. These instructions should also work for Ubuntu 10.x but may require slight tweaking for use with upstart.

# File:	HOWTO Configure a Local NTP server on Ubuntu.notes
# Date:	2010/03/16
# Refs:
# Desc:	Setting up an NTP server and configuring NTP clients
#       on a LAN with Ubuntu 9.10

# -------------------------------------
#           Configure the Server
# -------------------------------------
# We can't uninstall ntpdate like the ubuntugeek reference 
# suggest because it will remove ubuntu-minimal along with it,
# which has a bunch of stuff we want. So, we just have to
# disable the ntpdate service
sudo update-rc.d -f ntpdate remove

# Install NTP
sudo apt-get install ntp

# Edit the ntp configuration file: /etc/ntp.conf
# Change the server lines to point to the pool servers
# you want to use. 
# In my case, we are on a LAN with no Internet access, 
# so I will be the master time keeper, so I commented
# out all server lines and added the following:

# Fudge your local server's local clock to a lowish stratum 
# so that other local computers will still use it as a time 
# source, but will resync to a better time source when
# they're able to.
# Using a stratum of 9 means it's considered better than the
# computer's own time (usually stratum 16 for unsynced, often 
# stratum 12 for stand-alone time servers), but will be
# considered a poor source of time compared to most other 
# servers (usually stratums ranging from 1 to 4), and will 
# be ignored in those cases (when better sources are available).
fudge stratum 9

# Restart the ntp service
sudo service ntp restart

# -------------------------------------
#           Configure the Clients
# -------------------------------------
# Install NTP
sudo apt-get install ntp

# Edit the ntp configuration file: /etc/ntp.conf
# Change the server lines to point to the IP (or DNS record
# if you created one) of the LAN server created above
# Bonus Points if you have a DNS entry for the NTP server

# Restart the ntp service
sudo service ntp restart

HOWTO: Using cpio to Copy a Partition in Linux

Sunday, May 15th, 2011

Using cpio to copy everything from one partition to another in Linux, maintaining all file permissions, symlinks, and timestamps.

# File: copy_partition.notes
# Auth: burly
# Date: 03/20/05
# Desc: A method of copying everything from one 
#      partition to another

# Change directories to the top level of the 
# partition you want to copy
cd /part

# Use find to locate all objects and cpio to copy them
find . -mount -print0 | cpio -0dump /path/to/new/part

HOWTO: Rebuild a Software RAID 5 Array After Replacing a Disk

Sunday, May 15th, 2011

How to rebuild a software RAID 5 array after replacing a failed hard disk on CentOS linux.

# File: rebuild_RAID5.notes
# Auth: burly
# Date: 2005/08/09
# Ref: 
# Desc: Rebuild a degraded RAID 5 array w/ a new HDD

# Assumptions: 
	Failed drive is /dev/sda
	Good drives are /dev/sdb, /dev/sdc
	RAID array(s) are /dev/md3
# Copy the partition table from one of the existing
# drives over to the new drive
sfdisk -d /dev/sdb | sfdisk /dev/sda

# Rebuild the array
mdadm --manage --add /dev/md3 /dev/sda1

# Check mdstat for progress
watch -n 60 cat /proc/mdstat

	md3 : active raid5 sda[3] sdc1[1] sdb1[0]
	      490223104 blocks level 5, 128k chunk, algorithm 2 [3/2] [UU_]
	      [>....................]  recovery =  0.1% (380160/245111552) finish=64.3min speed=63360K/sec

# That's it!