Zach Burlingame

Programming, Computers, and Other Notes on Technology

BSOD When Installing Windows 7 Checked Build with VMWware Workstation 7

January 21st, 2012

The Blue Screen of Death

I was creating a Windows 7 VM using the checked build and during the OS installation process I was treated to the following BSOD: STOP: 0x0000008E (0xC0000420,0x8CB513E6,0x8C3D3A10,0×0000000)

Windows 7 x86 Checked Build VMWware BSOD

The TLDR Fix

Add the following to the virtual machine’s configuration file (.vmx):

piix4pm.smooth_acpi_timer = "TRUE"

Also, when creating the virtual machine using the “New Virtual Machine Wizard”, be sure to uncheck the box on the last step called “Power on this virtual machine after creation” so that you have the opportunity to edit the vmx file before installation begins.

VMWare - New Vitual Machine Wizard

Digging Deeper

I tried rebooting a few times and each time resulted in BSOD with the same stop code and exception code. The stop code 0x0000008E corresponds to the Bug Check code 0x8E which is KERNEL_MODE_EXCEPTION_NOT_HANDLED. From the MSDN article we can see the 4 values after the stop code are:

  1. the exception code that was not handled
  2. the address where the exception occurred
  3. the trap frame
  4. and a reserved parameter

Looking in the Ntstatus.h from the WDK we can see that exception code 0xC0000420 is STATUS_ASSERTION_FAILURE.

So why is a checked build of Windows throwing an assertion when installed inside of VMWare Workstation? A quick google searched turned up this recommendation. I was a bit curious as to what the piix4pm.smooth_acpi_timer option was and why an ACPI timer would be causing kernel driver crashes on checked builds but not free builds of Windows. I found this VMWare Knowledge Base article on this issue for Windows Vista and Server 2008 which sheds some light on it. The PIIX4 acronym refers to the Intel PCI ISA IDE Xcelerator (which wikipedia calls the Intel IDE ISA Xcelerator for some reason). From page 2 of this this Intel datasheet:

The 82371AB PCI ISA IDE Xcelerator (PIIX4) is a multi-function PCI device implementing a PCI-to-ISA bridge
function, a PCI IDE function, a Universal Serial Bus host/hub function, and an Enhanced Power Management
function.

It goes on to say (emphasis mine):

PIIX4 supports Enhanced Power Management, including full Clock Control, Device Management for up to
14 devices, and Suspend and Resume logic with Power On Suspend, Suspend to RAM or Suspend to Disk. It
fully supports Operating System Directed Power Management via the Advanced Configuration and Power
Interface (ACPI) specification. PIIX4 integrates both a System Management Bus (SMBus) Host and Slave
interface for serial communication with other devices.

So it appears that the default behavior of the ACPI abstraction in VMWare workstation occasionally violates timer reads which doesn’t play nicely with the hal!HalpGenerateConsistentPmTimerRead assertion in checked builds of Windows. To understand why the guest OS relies on these timer reads and the challenges a virtual machine faces in providing them see “Timekeeping in VMWare Virtual Machines”. The fix is to turn on the smooth_acpi_timer option which I can’t seem to find anymore documentation on.

For what it’s worth, I am running VMware Workstation 7.1.5 build-491717 and using the en_windows_7_debug_checked_build_dvd_x86_398742.iso from MSDN.

Tags:
Posted in VMware, Windows | No Comments »

Customizing your WinDbg Workspace and Color Scheme

December 12th, 2011

My Eyes! The Goggles Do Nothing!

WinDbg is a great tool for kernel and application debugging. Unfortunately, out of the box it also has a pretty useless workspace layout and (imho) a terrible color scheme. There are several themes that ship with it, if you look in the themes subdirectory of where the Debugging Tools are installed (e.g. C:\Program Files\Debugging Tools for Windows (x64)\themes). There is a also a themes.doc file that provides a bit of background on loading workspaces as well as providing screenshots of the provided workspaces.

Default WinDbg Workspace

I’m personally a big fan of dark color schemes as I find black-on-white strains my eyes after hours of staring at the monitor, even with the brightness and contrast levels adjusted. What I really want in WinDbg is a color scheme closer to what I use for Visual Studio with a layout most similar to another great debugger, OllyDbg. None of the provided workspaces come even close to what I want. I did some searching around but other than this article, I came up empty on any helpful content.

VS2010 Color Scheme - Debugging

Understanding WinDbg Workspaces

The workspace includes the displayed windows, the window positions, font, color scheme, open files, register order, source file directory, symbol file directory, image file directory, and probably a few other tidbits that I’m forgetting.

WinDbg breaks the workspaces out into four types – Dump, Explicit, Kernel, and User. Dump is used when you are debugging a dump file, explicit is when you create an custom workspace for a specific project (i.e. explicitly), kernel is for kernel debugging, and User is for …. regular user applications. WinDbg stores the settings for each of these types of workspaces in a matching registry key located under HKCU\Software\Microsoft\Windbg\Workspaces. In each of these keys is a binary value named “Default”. Additionally, there is an over-arching binary value named “Default” in the top-level Workspaces key. As far as I can tell, the top-level Default values are used for any workspace type but are superseded by any values in the nested keys. In practice however, I found it infuriating trying to determine exactly which inheritance style would be used between the two levels. As a result, I filled in values for everything in one of the workspaces and then just copied the “Default” value to each of the other four locations so that they were identical.

Customizing Your Workspaces

I first start by deleting any existing HKCU\Software\Microsoft\Windbg\Workspaces key (back it up first if you want to be able to restore what you had). Then I open WinDbg and add all the windows I wish to see. Once I get them laid out how I want, I edit the Font and Colors. Since I was basing the color scheme off an existing one, I found it easiest to take the 10 or so different colors and add them all to the custom colors palette for the process first. NOTE: The custom color palette does NOT persist beyond a restart of the application! Once you have the color palette, it’s just a matter of walking through each of the objects and selecting one of the existing colors. Once I’ve settled on the layout and color scheme, I select File->Save Workspace As… and use the name Default. Finally, I copy the Default binary value from the sub-key that was just created under HKCU\Software\Microsoft\WinDbg\Workspaces and copy it to the remaining four locations. Then it’s just a matter of exporting the Workspaces key to a file. It’s rather tedious but once you get this done once, you just import the registry key on any machine where you want to use WinDbg.

Final Results

This is ultimately what I came up with. It’s not perfect but it “feels” familiar enough to me that I’m happy with it. Getting here though, was a real chore. If you want my registry key for this settings, you can get it here. Note that this registry key contains some path information that is specific to my machine (e.g. source path) that’ll need to update once you install the key on your machine.

WinDbg Dark Color Scheme

Tags: , , , , ,
Posted in Native, WinDbg, Windows | No Comments »

Using Mercurial over HTTPS with TeamCity

November 9th, 2011

Uh oh, it’s b0rked

I use Mercurial as my VCS for all my personal projects and JetBrains TeamCity for my build server. Naturally, I need TeamCity to talk to the VCS. There are two basic ways you can serve Mercurial repos: over HTTP(S) using something like hgweb.cgi and over SSH. I use SSH with public key authentication for all of my development boxes and it works great. However, SSH public key auth requires that I have a full-blown shell account on the VCS server. I really didn’t want to have a shell account dedicated for the TeamCity user, so I preferred using HTTPS. Starting with 1.6.4, Mercurial began (smartly) verifying SSL certificates. This coupled with my use of self-signed certificates caused me to get errors in TeamCity from Mercurial when it was trying to pull from the VCS server:

‘cmd /c hg pull https://mercurial.mydomain.com/hg/ModuleUtilities’
command failed.
stderr: abort: error: _ssl.c:490: error: 14090086:SSL
routines:SSL2_GET_SERVER_CERTIFICATE:certificate verify failed

Teamcity Mercurial Error

Teamcity Mercurial Error

Ahh, I think I know what’s going on here…

The fix for this actually fairly simple: add the self-signed cert to the trusted chain. The tricky bit however, is that Mercurial doesn’t use the Windows certificate store so adding an entry like you would for say, Internet Explorer, won’t work. Instead, Mercurial uses a cacert.pem file. For these instructions, I’m using TortoiseHg as my Mercurial client on the build server. The basic concept however, applies regardless of the specific client so it should be fairly easy to adapt to your environment.

A Walk-through the park

The first step is to get the necessary certificate information. I did this by browsing to the URL of one of the repositories in Internet Explorer. For example:

https://mercurial.mydomain.com/hg/myrepo

Once there, I clicked on the “Security Report” lock icon next to the URL and selected “View Certificates”.


Which brings up a window like this:
View Certificate

View Certificate


You then click on the “Details” tab and select “Copy to File”:
View Certificate - Copy to File

View Certificate - Copy to File


In the “Certificate Export Wizard”, it’s important that you select the “Base-64 encoded X.509(.CER)” format as this is the format used by the cacert.pem file.
Certificate Export Wizard

Certificate Export Wizard


Then it’s simply a matter of going to the TeamCity build server and opening the cacert.pem located in
C:\Program Files\TortoiseHg\hgrc.d\cacert.pem

and adding a name for the cert followed by the contents of the .cer saved in the previous step. For example:

mercurial.mydomain.com
=======================
—–BEGIN CERTIFICATE—–
MIICWjCCAcMCAgGlMA0GCSqGSIb3DQEBBAUAMHUxCzAJBgNVBAYTAlVTMRgwFgYDVQQKEw9HVEUg
Q29ycG9yYXRpb24xJzAlBgNVBAsTHkdURSBDeWJlclRydXN0IFNvbHV0aW9ucywgSW5jLjEjMCEG
A1UEAxMaR1RFIEN5YmVyVHJ1c3QgR2xvYmFsIFJvb3QwHhcNOTgwODEzMDAyOTAwWhcNMTgwODEz
MjM1OTAwWjB1MQswCQYDVQQGEwJVUzEYMBYGA1UEChMPR1RFIENvcnBvcmF0aW9uMScwJQYDVQQL
Ex5HVEUgQ3liZXJUcnVzdCBTb2x1dGlvbnMsIEluYy4xIzAhBgNVBAMTGkdURSBDeWJlclRydXN0
IEdsb2JhbCBSb290MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCVD6C28FCc6HrHiM3dFw4u
sJTQGz0O9pTAipTHBsiQl8i4ZBp6fmw8U+E3KHNgf7KXUwefU/ltWJTSr41tiGeA5u2ylc9yMcql
HHK6XALnZELn+aks1joNrI1CqiQBOeacPwGFVw1Yh0X404Wqk2kmhXBIgD8SFcd5tB8FLztimQID
AQABMA0GCSqGSIb3DQEBBAUAA4GBAG3rGwnpXtlR22ciYaQqPEh346B8pt5zohQDhT37qw4wxYMW
M4ETCJ57NE7fQMh017l93PR2VX2bY1QY6fDq81yx2YtCHrnAlU66+tXifPVoYb+O7AWXX1uw16OF
NMQkpw0PlZPvy5TYnh+dXIVtx6quTx8itc2VrbqnzPmrC3p/
—–END CERTIFICATE—–

Save the file and then in a minute or so (by default the VCS check interval for TeamCity is 60s) you should see big smiles from TeamCity (or at least no more VCS errors)!

Teamcity Mercurial over HTTPs

Teamcity Mercurial over HTTPs

Tags:
Posted in Continuous Integration, SysAdmin, Version Control, Windows | 2 Comments »

Setting a Debug Breakpoint in a Win32 API in Visual Studio

September 25th, 2011

Recently, I was stress-testing an application of mine and noticed that the handle count kept growing. After a quick check in Process Explorer I saw that the handle was for a registry key. I was curious as to where the key was being opened in the first place but, it wasn’t in my code so I couldn’t just set a breakpoint on a line in my source. Wouldn’t it be nice if you could set a breakpoint on a function in Visual Studio but that’s not in your code?

You can! If you have the NT symbols loaded for your system, then you can set a breakpoint on any public function. In my particular case, I set a breakpoint on RegOpenKey and RegOpenKeyEx for both ANSI and UNICODE versions of the function.

{,,Advapi32.dll}_RegOpenKeyA@12
{,,Advapi32.dll}_RegOpenKeyW@12
{,,Advapi32.dll}_RegOpenKeyExA@20
{,,Advapi32.dll}_RegOpenKeyExW@20

Note that you have to use the decorated name of the function. For Win32 API functions, you have to put an underscore before the function name and remember that many functions are macro redirected to ANSI or UNICODE versions with an A or W extension. Since most functions use the __stdcall calling convention the @<number> is generally 4x the number of arguments. So for RegOpenKey for example, it’s redirected to RegOpenKeyA or RegOpenKeyW and has 3 arguments, so it’s decorated name is _RegOpenKeyA@12 and _RegOpenKeyW@12.

Similarly, you can set breakpoints in third-party libraries for public functions which assumes that you have a PDB for it.

UPDATE 2011-11-13: You only have to use the decorated name of the function for 32-bit applications. For 64-bit applications, the names are undecorated, like this:

{,,Advapi32.dll}RegOpenKeyA
{,,Advapi32.dll}RegOpenKeyW
{,,Advapi32.dll}RegOpenKeyExA
{,,Advapi32.dll}RegOpenKeyExW

Related: Can I Add a Breakpoint on CreateProcess in VS

Posted in Native, Visual Studio | No Comments »

Setting the sticky bit recursively on directories only

August 31st, 2011

This is more of a reminder for me.

Several times recently I’ve run into problems where files in a MultiUser Mercurial repository on a linux host are getting the wrong group permissions. If you properly set the group sticky bit when you first setup the repo, you won’t have this issue. To fix the issue, I needed to set the sticky bit on every directory in the .hg/store directory recursively.

find /path/to/.hg/store/ -type d -exec chmod g+s {} \;

Tags: ,
Posted in Linux, SysAdmin | No Comments »

HTTP File Download Reassembly in WireShark with Chunked Transfer Encoding

August 24th, 2011

I was having problems with binaries I was downloading with a particular application the other day. As part of the debugging process at one point, I was taking packet captures with Wireshark inside the client LAN, at the client router’s WAN, and tcpdump from the server. I was then reassembling the file from the stream in each packet capture and comparing them to see where the corruption was occurring relative to the copy that resided on the server.

To accomplish this, I was going to the HTTP GET message packet in Wireshark. Then I would right-click on the packet and select Follow Stream. Next I would select only the direction of traffic from the server to the client (since this was a download). Then I would make sure RAW was selected and save the file. Finally I would open the file up in a hex editor, remove the HTTP header that winds up prepended to the file, and save it. Annnnd then the file was corrupted.

Doing a binary diff of a valid copy of the file with the reconstructed file using 010 Editor I could see that the only differences were several small sections of the file with values like these spaced throughout the file:

Hex: 0D 0A 31 30 30 30 0D 0A
ASCII: \r\n1000\r\n

and one of these at the end of the file:

Hex: 0D 0A 00 00 0D 0A
ASCII: \r\n00\r\n

I confirmed that each of the packet captures at the various points along the way all had the same result. Where the heck was this random data getting injected into my stream and better still, why?!

The first clue that it wasn’t truly random data was the \r\n values. Carriage Return – Line Feed (CRLF) is a staple demarcation value in the HTTP protocol. My second clue was that the values were typically 1000 and 0. Although respresented with ASCII codes in the file, if you interpret them as hex they are 4096 and 0. When doing buffered I/O a 4K buffer is very common as is getting a 0 back from a read function when you reach EOF.

As it turns out, the particular behavior I was seeing was a feature of the HTTP/1.1 Protocol called Chunked Transfer Encoding. The wikipedia article does a great job explaining it, but basically it allows for content to be sent prior to knowing the exact size of that content. It does this by prepending the size to the each chunk:

The size of each chunk is sent right before the chunk itself so that a client can tell when it has finished receiving data for that chunk. The data transfer is terminated by a final chunk of length zero.

Ah-ha! So my naïve manual file reconstruction from the Wireshark packet capture of the HTTP download was flawed. Or was it? I checked the file on disk and sure enough it too had these extra data values present.

Once again, Wikipedia to the rescue (emphasis mine):

For version 1.1 of the HTTP protocol, the chunked transfer mechanism is considered to be always acceptable, even if not listed in the TE request header field, and when used with other transfer mechanisms, should always be applied last to the transferred data and never more than one time

The server was utilizing chunked transfer encoding but the application I was using wasn’t fully HTTP/1.1 compliant and was thus doing a naïve reconstruction just like me! So, if you find yourself doing file reconstruction from packet captures of HTTP downloads, make sure you take chunked transfer encoding into account.

Posted in Linux, SysAdmin, Windows | No Comments »

Getting the Load Count for a DLL

August 17th, 2011

Recently I was trying to unload a DLL from a running process so that I could delete it from the disk but it just wouldn’t delete. Looking at the Modules pane in Visual Studio, I could see that the DLL was still loaded. I doubled and tripled check all of my calls to LoadLibrary for a corresponding call to FreeLibrary, and everything checked out. I needed to figure out what was loading it and where. One of the things that I wanted to know was, “What is the current load count for the DLL?”

Windows maintains a load count for each module on a per-process basis. When the load count reaches zero, the module will be unloaded. The problem is that this load count is not accessible through documented API calls. To get it, you need to use some undocumented structures and API calls from ntdll.dll. Fortunately, like so many other issues you run into, someone else has already run into it and Google knows where they are at. In this case there is a great article here (unfortunately I couldn’t figure out who specifically was the contributing author for that article so that I could give them due props).

The securityxploded article did such a great job of explaining it that I won’t bother to duplicate it here. I based my implementation on theirs however I updated it to support both MBCS and Unicode via the TCHAR macros. It’s available as part of my ModuleUtilities library.

Posted in Uncategorized | No Comments »

How to Get the Exit Code of a Windows Command Line Application

July 29th, 2011

Since I regularly forget the exact name and end up having to look this up, I’m documenting it here.

If you run an application from the command-line and want to get it’s exit status code, you use:

echo %ERRORLEVEL%

Beware however, if there is an environment variable called errorlevel, because it will override the exit code.

See this SO question for more information.

Posted in Windows | No Comments »

HOWTO: Enable Wireless Networking on Boot in Ubuntu Linux without NetworkManager

July 25th, 2011

Building on my previous post, this is how to enable wireless networking on boot without NetworkManager.

I’m using WPA in this example, but the setup is similar for WEP and WPA2 using wpa_supplicant.

Remove NetworkManager (Optional)

sudo apt-get remove network-manager

Setup WPA Supplicant

To convert the WPA passphrase into the appropriate form (which is salted with the SSID), you need to use wpa_passphrase. For example:

wpa_passphrase my_ssid my_secret_password

Generates:

network={
ssid=”my_ssid”
#psk=”my_secret_password”
psk=6bea99c21cff6002adc637d93a47fba760ec5e6326cb41784c597b6691ed700d
}

Using this information, you need to setup /etc/wpa_supplicant.conf like so:

ap_scan=1
network={
ssid=”my_ssid”
#psk=”my_secret_password”
psk=6bea99c21cff6002adc637d93a47fba760ec5e6326cb41784c597b6691ed700d
}

Enable Wireless Interface

Put an entry in /etc/network/interfaces for wlan0 (or wlan1, or whatever your wireless interface is).

NOTE: I’ve put the DHCP option here for completeness, but I ran into problems with a Belkin USB F5D9050 wireless adapter not getting an IP successfully, even after it associated with the AP. I’m not sure if this was a problem with the device, the linux driver, or the AP. I ended up adding a DHCP reservation on the AP and then using a static IP configuration on the server.

Option 1: DHCP

auto wlan0
iface wlan0 inet dhcp

Option 2: Static IP

auto wlan0
iface wlan0 inet static
address 192.168.0.20
gateway 192.168.0.1
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
wpa-driver wext
wpa-conf /etc/wpa_supplicant.conf

Debugging

If you are having issues getting this to work, one debugging trick is to start up wpa_supplicant directly in the foreground and checking the output of dmesg and /var/log/syslog for additional details.

sudo wpa_supplicant -Dwext -iwlan0 -c/etc/wpa_supplicant.conf -dd

Tags: ,
Posted in Linux, SysAdmin | No Comments »

HOWTO: Enable Wired Networking on Boot in Ubuntu Linux without NetworkManager

July 21st, 2011

A lot of Linux distros are going to applet-based management of their network connections in their desktop flavors. For example, Ubuntu Linux Desktop Edition has been using the Gnome applet NetworkManager since at least 9.10 Karmic Koala. While it works great most of the time, I’ve run into issues with it several times.

UPDATE:I believe this issue may have gone away with recent versions of NetworkManager.
The first was that (at least with 9.10) while NetworkManager was running from boot, it didn’t start receiving commands to connect until the user initiated their Gnome session by logging in. If you wanted to run an SSH server on the machine, you wouldn’t be able to connect to it until a local user logged in.

The second issue is that I often times end up using the Desktop Edition in a server-like capacity and turn gdm/X off entirely. The Desktop Edition has a shorter-lead time for package updates (which can be both a blessing and a curse). In my experience it’s also easier to find help/info on it versus the Server Edition. I recently setup a machine to act as a server for my dad, connecting to his weather station’s base station and uploading the results online. I ended up using the Desktop Edition of 11.04 because the server version didn’t have support out-of-the-box for some of his hardware.

Anyways, while I found it maddening to find a solution to initially, like many things Linux, once you know the magic incantation to recite, it’s cake.

Remove NetworkManager

This is optional and many of you may want or need to keep it around. For me, in the cases where I need to use this at all, I find it easier just to completely remove NetworkManager from the picture.

sudo apt-get remove network-manager

Enable Wired Interface

Put an entry in /etc/network/interfaces for eth0 (or eth1, or whatever your wired interface is).

Option 1: DHCP

auto eth0
iface eth0 inet dhcp

Option 2: Static IP

auto eth0
iface eth0 inet static
address 192.168.0.10
gateway 192.168.0.1
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255

Now your network interface should come up on boot, without NetworkManager!

Tags: ,
Posted in Linux, SysAdmin | No Comments »

« Older Entries