Zach Burlingame
Programming, Computers, and Other Notes on Technology

Archive for the ‘Continuous Integration’ Category

Using Mercurial over HTTPS with TeamCity

Wednesday, November 9th, 2011

Uh oh, it’s b0rked

I use Mercurial as my VCS for all my personal projects and JetBrains TeamCity for my build server. Naturally, I need TeamCity to talk to the VCS. There are two basic ways you can serve Mercurial repos: over HTTP(S) using something like hgweb.cgi and over SSH. I use SSH with public key authentication for all of my development boxes and it works great. However, SSH public key auth requires that I have a full-blown shell account on the VCS server. I really didn’t want to have a shell account dedicated for the TeamCity user, so I preferred using HTTPS. Starting with 1.6.4, Mercurial began (smartly) verifying SSL certificates. This coupled with my use of self-signed certificates caused me to get errors in TeamCity from Mercurial when it was trying to pull from the VCS server:

‘cmd /c hg pull’
command failed.
stderr: abort: error: _ssl.c:490: error: 14090086:SSL
routines:SSL2_GET_SERVER_CERTIFICATE:certificate verify failed

Teamcity Mercurial Error

Teamcity Mercurial Error

Ahh, I think I know what’s going on here…

The fix for this actually fairly simple: add the self-signed cert to the trusted chain. The tricky bit however, is that Mercurial doesn’t use the Windows certificate store so adding an entry like you would for say, Internet Explorer, won’t work. Instead, Mercurial uses a cacert.pem file. For these instructions, I’m using TortoiseHg as my Mercurial client on the build server. The basic concept however, applies regardless of the specific client so it should be fairly easy to adapt to your environment.

A Walk-through the park

The first step is to get the necessary certificate information. I did this by browsing to the URL of one of the repositories in Internet Explorer. For example:

Once there, I clicked on the “Security Report” lock icon next to the URL and selected “View Certificates”.

Which brings up a window like this:
View Certificate

View Certificate

You then click on the “Details” tab and select “Copy to File”:
View Certificate - Copy to File

View Certificate - Copy to File

In the “Certificate Export Wizard”, it’s important that you select the “Base-64 encoded X.509(.CER)” format as this is the format used by the cacert.pem file.
Certificate Export Wizard

Certificate Export Wizard

Then it’s simply a matter of going to the TeamCity build server and opening the cacert.pem located in

C:\Program Files\TortoiseHg\hgrc.d\cacert.pem

and adding a name for the cert followed by the contents of the .cer saved in the previous step. For example:

Save the file and then in a minute or so (by default the VCS check interval for TeamCity is 60s) you should see big smiles from TeamCity (or at least no more VCS errors)!

Teamcity Mercurial over HTTPs

Teamcity Mercurial over HTTPs

HOWTO: Generate and Publish Doxygen Documentation in TeamCity

Sunday, June 12th, 2011

I’ve started using Doxygen and JavaDoc style comments on my native C/C++ applications for documentation generation. In keeping with my goal to have everything “just work” on checkout with minimal dependencies (ideally just Visual Studio and version control) I wanted to get it integrated directly into the project. That way anyone can generate the latest version of the documentation from their working copy whenever they need it. Since I use TeamCity for my continuous integration server, it was natural to have the CI server generate and publish the latest documents during the build process.

Setup Doxygen

Setup Doxygen in a Working Copy of your Project

  1. Download the Windows Doxygen zip package
  2. Create build/doxygen/bin
  3. Place DoxyFile in build/doxygen/
  4. Create build.xml in build/doxygen/
  5. +---build
    |   \---doxygen
    |       \---bin


    <?xml version="1.0" encoding="utf-8"?>
    <Project ToolsVersion="3.5" DefaultTargets="Doxygen" 
      <Target Name="Doxygen">
        <MakeDir Directories="$(OutputDirectory)" />
        <Exec Command="bin\doxygen.exe" />
  6. Test the script using MSBuild in a Visual Studio Command prompt
  7. [batch]
    cd /path/to/project/build/doxygen
    msbuild build.xml

  8. Add the documentation/doxygen output folder to your VCS ignore pattern. (e.g. for Mercurial, add the following to the .hgignore file)
  9. glob:documentation/doxygen/*
  10. Add the all the files (including doxygen.exe, build.xml, and DoxyFile) to version control, commit the changes and publish them to the master repo (e.g. for Mercurial)
  11. [batch]
    hg add
    hg commit -m"Added Doxygen"
    hg push ssh://user@reposerver://srv/hg/project

Setup Public Key Authentication

The following steps must be done on the BUILD SERVER. If you have multiple build servers/build agents for TeamCity, then you’ll need to duplicate most of these steps on each one. Alternatively, you can use a shared RSA key.

Generate an RSA public/private key pair

  1. Download and Install PuTTY.Look under “A Windows installer for everything except PuTTYtel”
  2. Open puttygen
  3. Click Generate
  4. Click Save Private Key
  5. Choose a location to save (I’ll use C:\keys for this example)
  6. Name the file (I’ll use buildserver.ppk for this example)
  7. Click Save Public Key
  8. Choose the same location used for the private key
  9. Name the file (I’ll use for this example)

The following steps must be done on the WEB SERVER

Add an account for the buildserver

sudo useradd buildserver

Setup public key authentication for the user

  1. Setup the necessary files, directories, and permissions
  2. su buildserver
    mkdir ~/.ssh
    chmod 700 ~/.ssh
    touch ~/.ssh/authorized_keys
    chmod 600 ~/.ssh/authorized_keys
    vim ~/.ssh/authorized_keys
  3. In PuTTYGen, copy the entire contents of the box labeled: “Public key for pasting into OpenSSH authorized_keys file:”
  4. Paste the contents into authorized_keys on the Web Server and save the file.

Setup Rsync

The following steps must be done on the BUILD SERVER.
Since Windows doesn’t natively have rsync, I use the cygwin packaged cwrsync.

  1. Download cwrsync
  2. I ran into problems with cwrsync being used in conjunction with plink where I received the following error:

    “Unable to read from standard input: The parameter is incorrect.”

    The problem apparently is when cygwin programs use redirected stdin/stdout handles. The solution I found to this was to use cygnative. From their website:

    Cygnative.exe is a wrapper for cygwin programs which call native Win32 programs with stdin and/or stdout handle redirected. Without cygnative.exe it is possible that the native program can not read from the handle and receives a “invalid handle” error.

  3. Download cygnative
  4. Create a script which will call cwrsync and pipe the rsync over an SSH connection from the build server to the web server.
  5. I placed this script during testing in project\build\doxygen\cwrsync.cmd I was only using it for testing before I put it into a TeamCity build step so I had no plans of commiting it to source control since it ultimately needs the private key which I don’t want in version control. If you aren’t going to use a TeamCity build step to publish the documentation, you can use this script as a starting point for your solution.

    REM *****************************************************************
    REM CWRSYNC.CMD – Batch file template to start your rsync command (s).
    REM By Tevfik K. (
    REM *****************************************************************
    REM Make environment variable changes local to this batch file

    SET LOCAL_DIR=../../documentation/doxygen/html
    SET REMOTE_SERVER=yer_remote_machine_name
    SET REMOTE_USER=yer_remote_user_name
    SET REMOTE_DIR=/var/cache/doxygen/yer_project_name
    SET SSH_KEY=yer_ssh_key
    SET PLINK_CMD=cygnative plink -i %SSH_KEY% -batch

    REM Specify where to find rsync and related files (C:\CWRSYNC)

    REM *****************************************************************
    REM Don’t Change Below This Line
    REM *****************************************************************

    REM Set HOME variable to your windows home directory. That makes sure
    REM that ssh command creates known_hosts in a directory you have access.

    REM Make cwRsync home as a part of system PATH to find required DLLs

    REM Publish the files

    REM Fix the permissions on the files

    In a command prompt, cd to the directory that the cwrsync.cmd script is and run it

    cd /path/to/cwrsync/script/

    It should ‘just work’. If you get an error running the script or your Web Server isn’t serving up the content, try turning up the verbosity of plink and and rsync by adding -v like this:

    SET RSYNC_ARGS=-arzvvvv
    SET PLINK_CMD=cygnative plink -v -i %SSH_KEY% -batch

Configure TeamCity

  1. Create a build step to generate the documentation using the build.xml file created earlier in your project’s build configuration.
  2. Runner type: MSBuild
    Build file path: build\doxygen\build.xml
    Working Directory:
    MSBuild version: Microsoft .NET Framework 4.0
    MSBuild ToolsVersion: 4.0
    Run platform: x86
    Targets: Doxygen
    Command line parameters:
    Reduce test failure feedback time:
    .NET Coverage tools:
    Report type:

    Click “Save”

  3. Create a build step to publish the documentation to the web server.
  4. Rather than use a CMD file in version control or pushing it out to all the build agents, I prefer to use a build step in the build configuration for the project in TeamCity. To use the script in the TeamCity build step, you have to use %% rather than % because TeamCity will treat the % as a TeamCity build property.

    Runner type: Command Line
    Working directory:
    Run: Custom Script
    Custom script: < the contents of your cwrsync.cmd from earlier, with every '%' replaced with '%%' >
    Report type:

    Click “Save”

  5. Run your build and verify that everything works!