Zach Burlingame
Programming, Computers, and Other Notes on Technology

Archive for the ‘Linux’ Category

Setting the sticky bit recursively on directories only

Wednesday, August 31st, 2011

This is more of a reminder for me.

Several times recently I’ve run into problems where files in a MultiUser Mercurial repository on a linux host are getting the wrong group permissions. If you properly set the group sticky bit when you first setup the repo, you won’t have this issue. To fix the issue, I needed to set the sticky bit on every directory in the .hg/store directory recursively.

find /path/to/.hg/store/ -type d -exec chmod g+s {} \;

HTTP File Download Reassembly in WireShark with Chunked Transfer Encoding

Wednesday, August 24th, 2011

I was having problems with binaries I was downloading with a particular application the other day. As part of the debugging process at one point, I was taking packet captures with Wireshark inside the client LAN, at the client router’s WAN, and tcpdump from the server. I was then reassembling the file from the stream in each packet capture and comparing them to see where the corruption was occurring relative to the copy that resided on the server.

To accomplish this, I was going to the HTTP GET message packet in Wireshark. Then I would right-click on the packet and select Follow Stream. Next I would select only the direction of traffic from the server to the client (since this was a download). Then I would make sure RAW was selected and save the file. Finally I would open the file up in a hex editor, remove the HTTP header that winds up prepended to the file, and save it. Annnnd then the file was corrupted.

Doing a binary diff of a valid copy of the file with the reconstructed file using 010 Editor I could see that the only differences were several small sections of the file with values like these spaced throughout the file:

Hex: 0D 0A 31 30 30 30 0D 0A
ASCII: \r\n1000\r\n

and one of these at the end of the file:

Hex: 0D 0A 00 00 0D 0A
ASCII: \r\n00\r\n

I confirmed that each of the packet captures at the various points along the way all had the same result. Where the heck was this random data getting injected into my stream and better still, why?!

The first clue that it wasn’t truly random data was the \r&#92n values. Carriage Return – Line Feed (CRLF) is a staple demarcation value in the HTTP protocol. My second clue was that the values were typically 1000 and 0. Although respresented with ASCII codes in the file, if you interpret them as hex they are 4096 and 0. When doing buffered I/O a 4K buffer is very common as is getting a 0 back from a read function when you reach EOF.

As it turns out, the particular behavior I was seeing was a feature of the HTTP/1.1 Protocol called Chunked Transfer Encoding. The wikipedia article does a great job explaining it, but basically it allows for content to be sent prior to knowing the exact size of that content. It does this by prepending the size to the each chunk:

The size of each chunk is sent right before the chunk itself so that a client can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero.

Ah-ha! So my naïve manual file reconstruction from the Wireshark packet capture of the HTTP download was flawed. Or was it? I checked the file on disk and sure enough it too had these extra data values present.

Once again, Wikipedia to the rescue (emphasis mine):

For version 1.1 of the HTTP protocol, the chunked transfer mechanism is considered to be always acceptable, even if not listed in the TE request header field, and when used with other transfer mechanisms, should always be applied last to the transferred data and never more than one time

The server was utilizing chunked transfer encoding but the application I was using wasn’t fully HTTP/1.1 compliant and was thus doing a naïve reconstruction just like me! So, if you find yourself doing file reconstruction from packet captures of HTTP downloads, make sure you take chunked transfer encoding into account.

HOWTO: Enable Wireless Networking on Boot in Ubuntu Linux without NetworkManager

Monday, July 25th, 2011

Building on my previous post, this is how to enable wireless networking on boot without NetworkManager.

I’m using WPA in this example, but the setup is similar for WEP and WPA2 using wpa_supplicant.

Remove NetworkManager (Optional)

sudo apt-get remove network-manager

Setup WPA Supplicant

To convert the WPA passphrase into the appropriate form (which is salted with the SSID), you need to use wpa_passphrase. For example:

wpa_passphrase my_ssid my_secret_password



Using this information, you need to setup /etc/wpa_supplicant.conf like so:


Enable Wireless Interface

Put an entry in /etc/network/interfaces for wlan0 (or wlan1, or whatever your wireless interface is).

NOTE: I’ve put the DHCP option here for completeness, but I ran into problems with a Belkin USB F5D9050 wireless adapter not getting an IP successfully, even after it associated with the AP. I’m not sure if this was a problem with the device, the linux driver, or the AP. I ended up adding a DHCP reservation on the AP and then using a static IP configuration on the server.

Option 1: DHCP

auto wlan0
iface wlan0 inet dhcp

Option 2: Static IP

auto wlan0
iface wlan0 inet static
wpa-driver wext
wpa-conf /etc/wpa_supplicant.conf


If you are having issues getting this to work, one debugging trick is to start up wpa_supplicant directly in the foreground and checking the output of dmesg and /var/log/syslog for additional details.

sudo wpa_supplicant -Dwext -iwlan0 -c/etc/wpa_supplicant.conf -dd

HOWTO: Enable Wired Networking on Boot in Ubuntu Linux without NetworkManager

Thursday, July 21st, 2011

A lot of Linux distros are going to applet-based management of their network connections in their desktop flavors. For example, Ubuntu Linux Desktop Edition has been using the Gnome applet NetworkManager since at least 9.10 Karmic Koala. While it works great most of the time, I’ve run into issues with it several times.

UPDATE:I believe this issue may have gone away with recent versions of NetworkManager.
The first was that (at least with 9.10) while NetworkManager was running from boot, it didn’t start receiving commands to connect until the user initiated their Gnome session by logging in. If you wanted to run an SSH server on the machine, you wouldn’t be able to connect to it until a local user logged in.

The second issue is that I often times end up using the Desktop Edition in a server-like capacity and turn gdm/X off entirely. The Desktop Edition has a shorter-lead time for package updates (which can be both a blessing and a curse). In my experience it’s also easier to find help/info on it versus the Server Edition. I recently setup a machine to act as a server for my dad, connecting to his weather station’s base station and uploading the results online. I ended up using the Desktop Edition of 11.04 because the server version didn’t have support out-of-the-box for some of his hardware.

Anyways, while I found it maddening to find a solution to initially, like many things Linux, once you know the magic incantation to recite, it’s cake.

Remove NetworkManager

This is optional and many of you may want or need to keep it around. For me, in the cases where I need to use this at all, I find it easier just to completely remove NetworkManager from the picture.

sudo apt-get remove network-manager

Enable Wired Interface

Put an entry in /etc/network/interfaces for eth0 (or eth1, or whatever your wired interface is).

Option 1: DHCP

auto eth0
iface eth0 inet dhcp

Option 2: Static IP

auto eth0
iface eth0 inet static

Now your network interface should come up on boot, without NetworkManager!

HOWTO: Disable IPv6 in Ubuntu Linux

Thursday, July 21st, 2011

Although we are edging closer to wide-spread IPv6 adoption with milestones such as World IPv6 Day, we aren’t quite there yet. Since I don’t use IPv6 on my LAN, I prefer to disable it. These instructions were written with Ubuntu 11.04, but it should work for 9.x,10.x, and probably many other distros as well.

Check if IPv6 is enabled

cat /proc/sys/net/ipv6/conf/all/disable_ipv6

0 means IPv6 is Enabled while 1 indicates that IPv6 is Disabled

Disable IPv6

Add the following to /etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1


Ubuntu 11.04 Natty Narwhal Upgrade – Grub Prompt on First Reboot

Wednesday, June 29th, 2011

I just updated one of my VMs from Ubuntu 10.10 to 11.04 Natty Narwhal using the Update Manager. All seemed to go well during the upgrade process. When it rebooted for the first time however, I was left with a grub prompt rather than a booting system. Grrrrrr.

NOTE: The following assumes the default disk layout. If you installed to a different disk or partition, you’ll have to adjust the steps below accordingly.

The fix is to manually boot the system at the grub prompt by typing

set root=(hd0,1)
linux /boot/vmlinux-2.6.38-8-generic root=/dev/sda1 ro
initrd /boot/initrd.img-2.6.38-8-generic

Then once you are successfully booted, re-install grub like this:

sudo grub-install /dev/sda
sudo update-grub

Thanks to Rob Convery for the tip!

HOWTO: Upgrade from Subversion 1.4 to 1.6 on CentOS 5

Friday, May 27th, 2011

How to upgrade the packages and existing repositories from Subversion 1.4 to 1.6.6 on CentOS 5.

# File: Subversion_1.6_Upgrade.notes
# Auth: burly
# Date: 12/01/2009
# Refs:
#       http://dev/
# Desc: Upgrading from subversion 1.4 to 1.6.6 on CentOS 5
#       NOTE:These instructions are actually fairly generic 
#       in regards to the version of SVN you are upgrading
#       from/to. At the time of writing, it just happened
#       to be 1.4 -> 1.6.6

# Backup each repository
svnadmin dump /srv/svn/<repo> > /backup/svn/<Repo>_20091201_rXXXX.dump

# Backup any hooks or configuration files in 
# /srv/svn/<repo>/hooks and /srv/svn/conf

# Setup yum to allow the package to come in from
# RPMforge (must setup RPMforge repo first).
vim /etc/yum.repos.d/Centos-Base.repo

# Add the following line at the end of each section
# in the Centos-Base.repo
exclude=subversion mod_dav_svn

# Restart the yum update daemon
service yum-updatesd restart

# Upgrade subversion
yum upgrade subversion

# For each repository
#    delete the existing repo
rm -rf /srv/svn/<repo>

# Create a new repo
svnadmin create /srv/svn/<repo> --fs-type fsfs

# Import the data
svnadmin load /srv/svn/<repo> < /backup/srv/<Repo>_20091201_rXXXX.dump

# Restore any hooks or configuration files in 
# /srv/svn/<repo>/hooks and /srv/svn/<repo>/conf

# If you are using Trac, you'll need to resync the repo
trac-admin /srv/trac/<repo> resync

HOWTO: Migrate an Existing RAID Array to a New Array

Wednesday, May 25th, 2011

How to migrate from an existing software RAID 1 array to a new RAID 1 array on CentOS 5.5

# File: Migrate_to_new_RAID_Array_on_CentOS_5.5.notes
# Auth: burly
# Date: 11/20/2010
# Refs: 
# Desc: How migrate from one RAID 1 array to a new one
#       on CentOS 5.5

# I booted from a Knoppix CD to do this. In retrospect,
# I should have used a CentOS LiveCD because the
# tooling, versions, and layout of Knoppix are different 
# which caused some issues. Also, because my OS is x86-64
# but Knoppix is x86, I could not chroot into my system 
# environment, which are ultimately required to create the
# initrd files.

# Boot from the Knoppix CD and drop to a shell

# Start up the existing RAID Array (one of the 2 drives
# from the existing RAID 1 array was on sdc for me)
mdadm --examine --scan /dev/sdc1 >> /etc/mdadm/mdadm.conf
mdadm --examine --scan /dev/sdc2 >> /etc/mdadm/mdadm.conf
mdadm --examein --scan /dev/sdc3 >> /etc/mdadm/mdadm.conf
/etc/init.d/mdadm start
/etc/init.d/mdadm-raid start

# Partition first SATA drive in whatever partition numbers
# and sizes you want. Make sure all partitions that 
# will be in an RAID array use ID type "fd" for RAID 
# autodetect and type "82" for swap. Make sure /boot
# is marked with the bootable flag
fdisk /dev/sda
# Repeat for the other disks OR if you are using the
# identical setup on each, you can use sfdisk to 
# simplify your life.
sfdisk -d /dev/sda | sfdisk /dev/sdb

# Create the new boot array
# NOTE: If you don't use metadata 0.90 (but instead 
#       1.0 or 1.1) you'll run into problems with grub.
#       In RAID 1, with metadata 0.90, you can mount
#       the fs on the partition without starting RAID.
#       With newer versions of metadata the superblock
#       for RAID gets written at the beginning of the 
#       partition where the filesystem superblock
#       normally would go. This results in the inability
#       to mount the filesystem without first starting
#       RAID. In the case of your boot partition, this 
#       results in the inability to setup grub and thus boot.
mdadm --create --verbose --metadata=0.90 /dev/md0 --level=1 --raid-devices=2 /dev/sda1 /dev/sdb1

# Copy everything over for /boot
mkdir /mnt/oldBoot
mkdir /mnt/newBoot
mkfs.ext3 /dev/md0
mount --options=ro /dev/md0 /mnt/oldBoot
cd /mnt/oldBoot
find . -mount -print0 | cpio -0dump /mnt/newBoot

# Make the new swap
mkswap /dev/sda2
mkswap /dev/sdb2

# Create the new array for LVM. I used metadata
# 0.90 again for consistency AND because I believe
# the version of mdadm in CentOS won't handle newer
# versions of it
mdadm --create --verbose --metadata=0.90 /dev/md1 --level=1 --raid-devices=2 /dev/sda3 /dev/sdb3

# Setup LVM2
pvcreate /dev/md1
vgcreate vg /dev/md1
lvcreate -L8G -nroot vg
lvcreate -L10G -nhome vg
lvcreate -L250G -nvm vg

# Format the filesystems.
# NOTE: I fixed the reserved space to 1% (default is 5%)
#       for the VM LV to save some space and 
#       because in larger, non-root partitions, you
#       don't need all that reserved space.
mkfs.ext3 /dev/vg/root
mkfs.ext3 /dev/vg/home
mkfs.ext3 -m 1 /dev/vg/vm

# Copy everything over for /
mkdir /mnt/oldRoot
mkdir /mnt/newRoot
mount --options=ro /dev/vgOS/lvRoot /mnt/oldRoot
mount /dev/vg/root /mnt/newRoot
cd /mnt/oldRoot
find . -mount -print0 | cpio -0dump /mnt/newRoot

# Copy everything over for /home
mkdir /mnt/oldHome
mkdir /mnt/newHome
mount --options=ro /dev/vgOS/lvHome /mnt/oldHome
mount /dev/vg/home /mnt/newHome
cd /mnt/oldHome
find . -mount -print0 | cpio -0dump /mnt/newHome

# Copy everything over for /boot
mkdir /mnt/oldVM
mkdir /mnt/newVM
mount --options=ro /dev/vgOS/lvVM /mnt/oldVM
mount /dev/vg/vm /mnt/newVM
cd /mnt/oldVM
find . -mount -print0 | cpio -0dump /mnt/newVM

# Remove any existing/stale lines in the mdadm.conf file

# Setup the mdadm config on the new /
mdadm -Esb /dev/sda1 >> /mnt/newRoot/etc/mdadm.conf
mdadm -Esb /dev/sda3 >> /mnt/newRoot /etc/mdadm.conf

# Update fstab on the new machine to use the new 
# mount points (e.g. if you changed VolumeGroup or 
# LogicalVolume names)
vim /mnt/newRoot/etc/fstab

# REBOOT TO A CENTOS LIVECD (if you weren't already on one)

# First we chroot
mkdir /mnt/sysimage
mount /dev/vg/root /mnt/sysimage
mount /dev/vg/home /mnt/sysimage/home
mount /dev/md0 /mnt/sysimage/boot
mount --bind /dev /mnt/sysimage/dev
mount -t proc none /mnt/sysimage/proc
mount -t sysfs none /mnt/target/sys
chroot /mnt/sysimage

# Make a new initrd to boot from
cd /boot
mv initrd-2.6.18-194.26.1.el5.img initrd-2.6.18-194.26.1.el5.img.bak
mkinitrd initrd-2.6.18-194.26.1.el5.img  2.6.18-194.26.1.el5

# Setup grub on both of the drives

# Reboot!

HOWTO: Create a Local Repository Mirror on Ubuntu

Monday, May 23rd, 2011

How to create and use a local repository mirror on Ubuntu 9.10. These instructions should work with minor modifications for other versions of Ubuntu.

# File: HOWTO Create a Local Repository Mirror on Ubuntu.notes
# Date: 2010/03/17
# Refs:
# Desc: How to create a local repository for 
#       Ubuntu 9.10 Karmic Koala.

# -------------------------------------
#           Setup the Server
# -------------------------------------
# Install Ubuntu (I used 9.10) on a machine with plenty of 
# free storage space (I used an 8GB OS vmdk and an 80GB data 
# vmdk used through LVM so that I could easily add/grow to
# it in the future if necessary).

# Create the mirror user, I'll be using ubuntu.
# NOTE: You don't have to add this user to the wheel but if you don't, the steps below that require sudo
#       will require you to run them from an account with root or wheel access and may also require
#       that you change the ownership/group of files/directories afterwards.
sudo useradd -m ubuntu -Gusers,wheel
sudo password ubuntu

# UPDATE 2012/01/30: As Dave points out below, you'll need to create your mirrorkeyring folder with the correct user account.
#                    If you aren't already running as that user, you can change your shell using su
su - ubuntu

# Update your apt-get package listing
sudo apt-get update

# Install debmirror
sudo apt-get install debmirror

# Create the location for the repo data to live
sudo mkdir -P /mirror/ubuntu

# Set the permissions for the repo data
sudo chown -R ubuntu:ubuntu /mirror/ubuntu
sudo chmod -R 771 /mirror/ubuntu

# Setup the keyring for correctly verifying Release signatures
gpg --no-default-keyring --keyring /home/ubuntu/mirrorkeyring/trustedkeys.gpg --import /usr/share/keyrings/ubuntu-archive-keyring.gpg

# Create the script
vim /home/ubuntu/

# NOTE: The ubuntu community documentation has you using 
#       the HTTP protocol for the mirror build script
#       however, I prefer rsync because we can rate limit.
#       When the download is going to take days,
#       I'd like to be able to use my connection in
#       the interim.

# --------------------------------------------
# --------------------------------------------


## Setting variables with explanations.

# Don't touch the user's keyring, have our own instead
export GNUPGHOME=/home/ubuntu/mirrorkeyring

# Arch=         -a      # Architecture. 
# For Ubuntu can be i386, amd64, powerpc and/or sparc (sparc support begins with dapper)
# Comma separated values

# Minimum Ubuntu system requires main, restricted
# Section=      -s      # Section
# (One of the following - main/restricted/universe/multiverse).
# You can add extra file with $Section/debian-installer.
# ex: main/debian-installer,universe/debian-installer,multiverse/debian-installer,restricted/debian-installer

# Release=      -d      # Release of the system
# (Dapper, Edgy, Feisty, Gutsy, Hardy, IntrepidJaunty, Karmic), 
# and the -updates and -security ( -backports can be added if desired)

# Server=       -h      # Server name,
# minus the protocol and the path at the end
# CHANGE "*" to equal the mirror you want to create your
# mirror from. au. in Australia  ca. in Canada. This can be 
# found in your own /etc/apt/sources.list file, 
# assuming you have Ubuntu installed.

# Dir=          -r      # Path from the main server,
# so http://my.web.server/$dir, Server dependant
# Lead with a '/' for everything but rsync,
# where we lead with a ':'

# Proto=        -e      # Protocol to use for transfer
# (http, ftp, hftp, rsync)
# Choose one - http is most usual the service, and the
# service must be availabee on the server you point at.
# NOTE: debmirror uses -aIL --partial by default.
#       However, if you provide the --rsync-options
#       paramter (which we do) then you HAVE to provide 
#       it -aIL --partial in addition to whatever You
#       want to add (e.g. --bwlimit) If you don't
#       debmirror will exit with thousands of files
#       missing.
rsyncoptions="-aIL --partial --bwlimit=100"

# Outpath=              # Directory to store the mirror in
# Make this a full path to where you want to mirror the material.

# The --nosource option only downloads debs and not deb-src's
# The --progress option shows files as they are downloaded
# --source \ in the place of --no-source \ if you want sources also.
# --nocleanup  Do not clean up the local mirror after mirroring
# is complete. Use this option to keep older repository
# Start script
debmirror       -a $arch \
                --no-source \
                -s $section \
                -h $server \
                -d $dist \
                -r $inPath \
                --progress \
                -e $proto \
                --rsync-options="$rsyncoptions" \

# -----------------------------------------------------
# -----------------------------------------------------

# Add execute permissions on the script
chmod +x

# Run the script

# Go home, kick back, have a beer while it downloads 43GBs 
# (in the case of karmic, karmic-update, karmic-securty for
# i386 and amd64)

# --------------------------------------
#          Setup the mirror
# --------------------------------------
# Install apache2
sudo apt-get install apache2

# Symlink the mirror data into the web root
sudo ln -s /mirror/ubuntu /var/www/ubuntu

# Point your browser at http://localhost/ubuntu and
# you should see your pool!

# -------------------------------------
#        Updating the Repo Mirror
# -------------------------------------
# To update the repo mirror, just execute the
# script used to initially build it.

# -------------------------------------
#   Configure Clients to Use this Repo
# -------------------------------------
# Update the apt sources list
cd /etc/apt
sudo mv sources.list sources.list.orig
sudo sensible-editor sources.list

# Replace 'mirrorbox' with your server's DNS name 
# (e.g.
# -----------------------------------------------------------------------------
# -----------------------------------------------------------------------------
# Local network mirror sources.
deb http://mirrorbox/ubuntu karmic main restricted universe multiverse
deb http://mirrorbox/ubuntu karmic-updates main restricted universe multiverse
deb http://mirrorbox/ubuntu karmic-security main restricted universe multiverse
# -----------------------------------------------------------------------------
# -----------------------------------------------------------------------------

# Test to see if you are able to pull down updates 
# from the new mirror
sudo apt-get update

HOWTO: Setup a DHCP Server on Ubuntu 9.10

Friday, May 20th, 2011

Setting up a DHCP server on a LAN with Ubuntu 9.10. These instructions should also basically work on Ubuntu 10.x.

# File:	HOWTO Configure a DHCP Server on Ubuntu.notes
# Date:	2010/03/24
# Refs:
# Desc:	Setting up a DHCP server on a LAN with Ubuntu 9.10

# Install DHCP3
sudo apt-get install dhcp3-server

# Specify the interface(s) that dhcp3-server should manage
# in /etc/default/dhcp3-server

# Set a static IP for the DHCP server itself on the
# interfaces that it will manage in /etc/network/interfaces
auth eth0
iface eth0 inet static

# Edit the etc/dhcp3/dhcpd.conf configuration file
# I'm going to be running a host-only 
# vmnet on eth0 with fixed addresses for several machines 
# in my example here
ddns-update-style none;
log-facility local7;

subnet netmask {

    option routers    ;
    option subnet-mask;
    option broadcast-address;
    option domain-name-servers;
    option ntp-servers;
    default-lease-time          7200;
    max-lease-time              86400;

    host helium {
            hardware ethernet 00:0c:29:c6:de:09;
    host lithium {
            hardware ethernet 00:0c:29:d8:d5:7f;
    host beryllium {
            hardware ethernet 00:0c:29:b6:93:41;
    host boron {
            hardware ethernet 00:0c:29:3f:c6:f3;