Unity TDD with MPLAB C18

I'm lucky enough to be involved in a project at work that requires some microcontroller development. Being a convert to Test Driven Development for enterprise-style code, I was keen to take advantage of a testing framework for my embedded code. I bought James Grenning's book Test-Driven Development for Embedded C to get myself started. The Unity framework seems to be just what I need.

I chose the Microchip PIC family of microcontrollers because they appear to be very popular, flexible, easy to program, there are evaluation boards, and there are so many different variants that it should be possible to find just the right one for any occasion. The MPLAB IDE doesn't come close to the IntelliJ or Eclipse benchmarks, but I'm comfortable doing TDD in a simple text editor (Notepad++), so that's ok.

I downloaded Unity and installed the three source files into my MPLAB project. I wrote a simple test:

    #include "unity.h"

    void setUp(void) { }

    void tearDown(void) { }

    void test_demo()
    {
        TEST_ASSERT_EQUAL_INT(2,3);   
    }

and a simple test runner:

    #include "unity.h"
    void test_demo(void);

    void main(void)
    {
        UnityBegin();
        RUN_TEST(test_demo,1);
        UnityEnd();   
    }

Compiling showed that unity_internals.h was looking for a stdint.h header file, but not finding it. I had to define the constant UNITY_EXCLUDE_STDINT_H in my build. To do that, I used Project > Build Options > Project, and on the MPLAB C18 tab, clicked "Add..." to add it as a preprocessor macro.

The next problem was the lack of a putchar() function. I added this function to my test runner:
int putchar(int c)
{
    putc((char)c,stdout);
}

I selected the MPLAB debugger (Debugger > Select Tool > MPLAB Sim), and enabled the UART output tab (Debugger > Settings > Uart 1 IO > Enable, and choose Window for output). When I clicked Run, my SIM Uart1 window filled up with trash. I decided to check putc:

     void main(void)
    {
         putc('a',stdout);  // try this one
         while(1);
         UnityBegin();
         RUN_TEST(test_demo,1);
         UnityEnd();   
     }

Yes, when that runs, the SIM Uart1 window shows an 'a'. Each time I press reset, I get an extra 'a'. Try puts:

    void main(void)
    {
        putc('a',stdout);
        puts("abcde"); // try this one
        while(1);
        UnityBegin();
        RUN_TEST(test_demo,1);
        UnityEnd();   
    }

Yes, that works too. I looked into UnityBegin() (not much there) and then UnityEnd(). That starts with a UnityPrint. Let's try that one.

    void main(void)
    {
        putc('a',stdout);
        puts("abcde");
        UnityPrint("Does this print?");
        while(1);
        UnityBegin();
        RUN_TEST(test_demo,1);
        UnityEnd();   
    }

Looks like there's a problem with UnityPrint(). After some searching, I discovered (in the C18 C Compiler Getting Started manual) that the C18 compiler puts string constants in the code section, so they are const rom char*, rather than just const char *.

After a lot of experimentation, I decided that I needed to have two versions of the UnityPrint() function: one with a const rom char* parameter, and the original one with const char *. The two versions only differ in the signature and the first line:

    void UnityPrint(const char* string)
    {
        const char* pch = string;

    void UnityPrintRom(const rom char* string)
    {
        const rom char* pch = string;
   
When I changed my code to use UnityPrintRom, it worked. By the way, I don't really understand why it should help to have that apparently unused char c. It seems to be necessary to convince the compiler to use the right addressing.

Now I had to make Unity use UnityPrintRom at the appropriate times. I did this by two global search and replaces in unity.c. The first was to change all occurrences of UnityPrint(" to UnityPrintRom(" (there were 8 of these) and the second was to change UnityPrint(UnityStr to UnityPrintRom(UnityStr (there were 50 of these). The last two to change were UnityPrintRom(file) and UnityPrintRom(Unity.CurrentTestName). And I added a couple of prototypes into unity.h:

    void UnityPrintRom(const rom char* string);
    int putchar(int c);

Now, it works.

    testDemo.c:8:test_demo:FAIL: Expected 2 Was 3
    -----------------------
    1 Tests 1 Failures 0 Ignored
    FAIL

And when I fix up the assertion, I get:

    testDemo.c:1:test_demo:PASS
    -----------------------
    1 Tests 0 Failures 0 Ignored
    OK

The last step was to surround these changes with some #ifdef UNITY_MPLAB directives. The resulting code is now in a fork of the original repository at https://github.com/johnyesberg/Unity.

Windows Development VMs - a long adventure through Virtualbox, Vagrant, VeeWee, Ruby, RVM, Cygwin

It's a deep rabbit hole, I admit. But the prospects of gold at the end of the mixed-metaphor rainbow are enough to make it worth burrowing all the way. Here's the story so far:

In the last five years or so, I've been enjoying a journey towards continuous delivery, with a number of tour-guides and companions (Erik Dörnenburg, Glennn Moy, Steve Dalton, Craig Aspinall, Rob Nielsen, various Yow! conference attendees and presenters, and many bloggers, especially Uncle Bob and lots of Thoughtworkers). It started with pair programming, and then progressed to test-driven development, and most recently, continuous integration, test, and deployment. At each stage, I found that I had acquired additional armour that  provided increasing levels of courage, confidence, freedom, and protection. It's now hard to go back.

My latest adventure is towards DevOps. The gradual accretion of software packages into my development environment can make it unclear what our actual dependencies are. Rather than creating (or building, or deploying) new projects on existing machines, I want to be able to spin up a shiny new vanilla virtual machine, and install a clean set of known software on it. In the Windows world, I don't want to forget to install the right version of the .Net Framework, or the ASP.NET component, or msdeploy, or all the other little bits that seem to be needed.

I've been running a couple of VMs on Virtualbox for some time. Vagrant is a very neat way to start, configure, and stop various VMs - especially when you need multiple VMs to work together. (I also like cumberbatch for testing such configurations.) All I need is a way to create the right base VM. All that interactive clicking to install Windows, SQLServer, SharePoint, etc is ok, but it's hard to document. A script would be better. And VeeWee is the tool for that. It's been working well on linux-related machines for some time, and support has recently been added for Windows. The VeeWee installation guide recommends using RVM (Ruby Version Manager). So to build RVM on Windows, I first had to install (well, add a few packages to) Cygwin.

So how did it all go? Well, there were a few errors. I'm running on a Windows 7 (64 bit) machine, in case that helps anyone.

First, the RVM on Cygwin script did cause a couple of dialog boxes "expr.exe has stopped working" to pop up for each version of Ruby that it installed (1.9.3-p194 and 1.9.2-p320). The log file suggested problems with the compiler, but it seems to have finished...

John@margaux ~/.rvm/log/ruby-1.9.3-p194/yaml
$ cat configure.log
]  ./configure --prefix="/cygdrive/c/Users/John/.rvm/usr"
checking for a BSD-compatible install... config/install-sh -c
checking whether build environment is sane... yes
./configure: line 2562: /cygdrive/c/Program: No such file or directory
configure: WARNING: `missing' script is too old or missing
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/c/Users/John/.rvm/src/yaml-0.1.4':
configure: error: C compiler cannot create executables
See `config.log' for more details
c:\Users\John\Downloads\UnxUtils\usr\local\wbin\sed.exe: -e expression #1, char 1: Unknown command: ``C''

John@margaux ~/.rvm/log/ruby-1.9.3-p194/yaml

I configured RVM to use 1.9.2 as the default, which is what the VeeWee installation suggested.

rvm --default use 1.9.2

 The second error was in the VeeWee installation "bundle install" step.

John@margaux ~/repos/veewee
$ bundle install
Fetching gem metadata from http://rubygems.org/......
Using rake (0.9.2.2)
Installing CFPropertyList (2.1.1)
Installing Platform (0.4.0)
Installing ansi (1.3.0)
Installing archive-tar-minitar (0.5.2)
Installing builder (3.0.0)
Using bundler (1.1.3)
Installing ffi (1.0.11) with native extensions
Installing childprocess (0.3.2)
Installing diff-lcs (1.1.3)
Installing json (1.5.4) with native extensions
Installing gherkin (2.10.0) with native extensions
Installing cucumber (1.2.0)
Installing erubis (2.7.0)
Installing excon (0.9.6)
Installing formatador (0.2.1)
Installing mime-types (1.18)
Installing multi_json (1.0.4)
Installing net-ssh (2.2.2)
Installing net-scp (1.0.4)
Installing nokogiri (1.5.2) with native extensions
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

        /cygdrive/c/Users/John/.rvm/rubies/ruby-1.9.2-p320/bin/ruby.exe extconf.rb
checking for libxml/parser.h... no
-----
libxml2 is missing.  please visit http://nokogiri.org/tutorials/installing_nokogiri.html for help with installing dependencies.
-----
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

Provided configuration options:
        --with-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/cygdrive/c/Users/John/.rvm/rubies/ruby-1.9.2-p320/bin/ruby
        --with-zlib-dir
        --without-zlib-dir
        --with-zlib-include
        --without-zlib-include=${zlib-dir}/include
        --with-zlib-lib
        --without-zlib-lib=${zlib-dir}/lib
        --with-iconv-dir
        --without-iconv-dir
        --with-iconv-include
        --without-iconv-include=${iconv-dir}/include
        --with-iconv-lib
        --without-iconv-lib=${iconv-dir}/lib
        --with-xml2-dir
        --without-xml2-dir
        --with-xml2-include
        --without-xml2-include=${xml2-dir}/include
        --with-xml2-lib
        --without-xml2-lib=${xml2-dir}/lib
        --with-xslt-dir
        --without-xslt-dir
        --with-xslt-include
        --without-xslt-include=${xslt-dir}/include
        --with-xslt-lib
        --without-xslt-lib=${xslt-dir}/lib


Gem files will remain installed in /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/nokogiri-1.5.2 for inspection.
Results logged to /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/nokogiri-1.5.2/ext/nokogiri/gem_make.out
An error occured while installing nokogiri (1.5.2), and Bundler cannot continue.
Make sure that `gem install nokogiri -v '1.5.2'` succeeds before bundling.

So I had to install libxml2 to get nokogiri working. I went back to my cygwin setup, added libxml2 and libxml2-devel, and tried again. A similar error, reporting that libxslt is missing. In installed libxslt and libxslt-devel with cygwin setup. Then, the nokogiri & the rest of VeeWee installation went smoothly.

At this point, I was ready to start VeeWee-ing for Vagrant, using these directions. I started by creating a new directory ~/boxes to put my VMs in. But veewee wouldn't work from there. I looked into RVM a little, but didn't find anything obvious, so I decided to leave that as an exercise for later. I could make boxes inside ~/repos/veewee for now.

John@margaux ~/repos/veewee
$ veewee vbox define '2008r2' 'windows-2008R2-serverstandard-amd64'
The basebox '2008r2' has been succesfully created from the template 'windows-2008R2-serverstandard-amd64'
You can now edit the definition files stored in definitions/2008r2 or build the box with:
veewee vbox build '2008r2'
$ veewee vbox build  '2008r2'
Error: We executed a shell command and the exit status was not 0
- Command :VBoxManage -v.
- Exitcode :127.
- Output   :
sh: VBoxManage: command not found

Wrong exit code for command VBoxManage -v

John@margaux ~/repos/veewee
$ VBoxManage
bash: VBoxManage: command not found

John@margaux ~/repos/veewee
$

Looks like my virtualbox directory isn't in the path. Added it to the windows environment variable, and restarted my Cygwin bash. Hmm - not there. C:\Program Files\Oracle\Virtualbox shows up in a Cmd window echo %PATH%, but not in a Cygwin bash echo $PATH. How do I get Cygwin to reload the windows environment? Hooray for Stackoverflow, which already had the answer.

John@margaux ~
$ export PATH="$PATH:$(cygpath -pu "`reg query 'HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Environment' /v PATH| grep PATH | cut -c23-`")"

John@margaux ~
$ VBoxManage -v
4.1.12r77245

John@margaux ~
$

So would my build work now? No.

$ veewee vbox build  '2008r2'
/cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh/transport/openssl.rb:1:in `require': no such file to load -- openssl (LoadError)
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh/transport/openssl.rb:1:in `'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh/buffer.rb:2:in `require'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh/buffer.rb:2:in `'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh/transport/algorithms.rb:1:in `require'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh/transport/algorithms.rb:1:in `'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh/transport/session.rb:7:in `require'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh/transport/session.rb:7:in `'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh.rb:10:in `require'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/net-ssh-2.2.2/lib/net/ssh.rb:10:in `'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/helper/ssh.rb:20:in `require'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/helper/ssh.rb:20:in `'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/helper/ssh.rb:18:in `'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/helper/ssh.rb:4:in `'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/helper/ssh.rb:3:in `'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/helper/ssh.rb:2:in `'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/helper/ssh.rb:1:in `'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/box.rb:2:in `require'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/box.rb:2:in `'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/virtualbox/box.rb:1:in `require'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/virtualbox/box.rb:1:in `'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/provider.rb:34:in `require'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/provider/core/provider.rb:34:in `get_box'
        from /cygdrive/c/Users/John/repos/veewee/lib/veewee/command/virtualbox.rb:17:in `build'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/thor-0.14.6/lib/thor/task.rb:22:in `run'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/thor-0.14.6/lib/thor/invocation.rb:118:in `invoke_task'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/thor-0.14.6/lib/thor.rb:263:in `dispatch'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/thor-0.14.6/lib/thor/invocation.rb:109:in `invoke'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/thor-0.14.6/lib/thor.rb:205:in `block in subcommand'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/thor-0.14.6/lib/thor/task.rb:22:in `run'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/thor-0.14.6/lib/thor/invocation.rb:118:in `invoke_task'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/thor-0.14.6/lib/thor.rb:263:in `dispatch'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/gems/thor-0.14.6/lib/thor/base.rb:389:in `start'
        from /cygdrive/c/Users/John/repos/veewee/bin/veewee:18:in `'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/bin/veewee:23:in `load'
        from /cygdrive/c/Users/John/.rvm/gems/ruby-1.9.2-p320@veewee/bin/veewee:23:in `

'

How to install openssl for ruby? I tried a few things, but I think what I needed to do was install openssl-devel (as well as the openssl I already had) using Cygwin setup. Then I had to

rvm remove 1.9.2
rvm install 1.9.2
rvm --default use 1.9.2
gem install bundler
bundle install

Finally, I'm at the point where running veewee vbox build '2008r2' will actually download the iso from the Internet!

John@margaux ~/repos/veewee
$ veewee vbox build  '2008r2'
Downloading vbox guest additions iso v 4.1.12 - http://download.virtualbox.org/virtualbox/4.1.12/VBoxGuestAdditions_4.1.12.iso
Creating an iso directory
Checking if isofile VBoxGuestAdditions_4.1.12.iso already exists.
Full path: /cygdrive/c/Users/John/repos/veewee/iso/VBoxGuestAdditions_4.1.12.iso
Moving /tmp/open-uri20120513-4468-1cp4y7g to /cygdrive/c/Users/John/repos/veewee/iso/VBoxGuestAdditions_4.1.12.isooooooooooooooooooooooooooooooooooooooooooooooo|  48.4MB 357.7KB/s ETA:   0:00:00
Building Box 2008r2 with Definition 2008r2:
- postinstall_include : []
- postinstall_exclude : []

We did not find an isofile in /iso.

The definition provided the following download information:
- Download url: http://care.dlservice.microsoft.com//dl/download/7/5/E/75EC4E54-5B02-42D6-8879-D8D3A25FBEF7/7601.17514.101119-1850_x64fre_server_eval_en-us-GRMSXEVAL_EN_DVD.iso
- Md5 Checksum: 4263be2cf3c59177c45085c0a7bc6ca5


Download? (Yes/No) Yes
Checking if isofile 7601.17514.101119-1850_x64fre_server_eval_en-us-GRMSXEVAL_EN_DVD.iso already exists.
Full path: /cygdrive/c/Users/John/repos/veewee/iso/7601.17514.101119-1850_x64fre_server_eval_en-us-GRMSXEVAL_EN_DVD.iso
|  34.9MB 306.6KB/s ETA:   2:46:10

I decided interrupt this download, and to modify my definitions/2008r2/definition.rb by commenting out the iso filename line and adding one I had prepared earlier.

    #:iso_file => "7601.17514.101119-1850_x64fre_server_eval_en-us-GRMSXEVAL_EN_DVD.iso",
    :iso_file => "en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso",

The next problem seems to be that when running VBoxManage, veewee is passing a Cygwin path, rather than a Windows path.

John@margaux ~/repos/veewee
$ veewee vbox build  '2008r2'
Downloading vbox guest additions iso v 4.1.12 - http://download.virtualbox.org/virtualbox/4.1.12/VBoxGuestAdditions_4.1.12.iso
Checking if isofile VBoxGuestAdditions_4.1.12.iso already exists.
Full path: /cygdrive/c/Users/John/repos/veewee/iso/VBoxGuestAdditions_4.1.12.iso

The isofile VBoxGuestAdditions_4.1.12.iso already exists.
Building Box 2008r2 with Definition 2008r2:
- postinstall_include : []
- postinstall_exclude : []

The isofile en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso already exists.
Received port hint - 59856
Found port 59856 available
Creating vm 2008r2 : 384M - 1 CPU - Windows2008_64
Creating new harddrive of size 10140
Mounting cdrom: /cygdrive/c/Users/John/repos/veewee/iso/en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso
Error: We executed a shell command and the exit status was not 0
- Command :VBoxManage storageattach "2008r2" --storagectl "IDE Controller" --type dvddrive --port 0 --device 0 --medium "/cygdrive/c/Users/John/repos/veewee/iso/en_windows_server_2008_r2_standard
_enterprise_datacenter_web_x64_dvd_x15-50365.iso".
- Exitcode :1.
- Output   :
VBoxManage.exe: error: Could not find file for the medium 'C:\cygdrive\c\Users\John\repos\veewee\iso\en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso' (VERR_PATH
_NOT_FOUND)
VBoxManage.exe: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component Medium, interface IMedium, callee IUnknown
Context: "OpenMedium(Bstr(pszFilenameOrUuid).raw(), enmDevType, AccessMode_ReadWrite, fForceNewUuidOnOpen, pMedium.asOutParam())" at line 210 of file VBoxManageDisk.cpp
VBoxManage.exe: error: Invalid UUID or filename "/cygdrive/c/Users/John/repos/veewee/iso/en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso"

Wrong exit code for command VBoxManage storageattach "2008r2" --storagectl "IDE Controller" --type dvddrive --port 0 --device 0 --medium "/cygdrive/c/Users/John/repos/veewee/iso/en_windows_server
_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso"

John@margaux ~/repos/veewee
$

At this point, I googled to see what could be done, but found little in the way of advice. Maybe next time, this blog entry will come up.I didn't see anything that talked about VeeWee converting from the Cygwin path to a windows path. So I decided to take matters into my own hands, and modify the VeeWee code. I've read a little about ruby, but I'm not an experienced ruby programmer. I wasn't sure how to write code to detect whether it was running under Cygwin, so someone else may have to help with that soon, and perhaps contribute the results back to the project. But my changes were limited to two files.

In the file ~/repos/veewee/lib/veewee/provider/virtualbox/box/helper/create.rb I added a new method at the beginning of the BoxCommand module, i.e. line 6:

        def cygpath(s)
          `/bin/cygpath -w "#{s}"`.chomp
        end

This was inspired by Kevin Kleinfelter's code in this thread, although I had to add the -w flag to make sure the result was a windows path, not a cygwin path, and add quotes to ensure that it would work when there were spaces in the pathname (virtualbox puts spaces in the "Virtualbox VMs" folder name). I then had to call this method a four times: in the attach_disk, attach_isofile, attach_guest_additions, and attach_floppy methods, so that the resulting construction of the command variable looked like this:

command ="#{@vboxcmd} storageattach \"#{name}\" --storagectl \"SATA Controller\" --port 0 --device 0 --type hdd --medium \"#{cygpath(location)}\""

I modified some of the ui.info lines (including moving them below the command assignment) to print the whole command, rather than just the filename.

The second file that needed changing was ~/repos/veewee/lib/veewee/provider/core/box/floppy.rb. It seemed that the construction of the image of the floppy was not working properly for the same reason as above. So I added the same cypath(s) method. But that wasn't enough either. I saw errors like this:

Unable to access jarfile C:UsersJohnreposveeweelibjavadir2floppy.jar

It seemed that Java was devouring an extra backslash. So I added another method escape to floppy.rb.

       def escape(s)
         s.gsub('\\','\\\\\\\\')
       end

And I added a new line assigning jar_file, and updated the assignment to command, and creating a log message:

           jar_file=File.join(javacode_dir,"dir2floppy.jar")
           command="java -jar #{escape(cygpath(jar_file))} \"#{escape(cygpath(temp_dir))}\" \"#{escape(cygpath(floppy_file))}\""
           ui.info("Creating floppy: #{command}")
           shell_exec("#{command}")

Getting all this right took several iterations of running

$ veewee vbox build  '2008r2' --force

I had to add the --force to make veewee really construct the box again. I also had to manually delete the folder C:\Users\John\VirtualBox VMs\2008r2 each time before running that command. But after all these modifications, I eventually saw a Virtualbox window come up, and install windows (with no user intervention), rebooting a few times along the way. My VeeWee seemed to be waiting for sshd to work, but there seemed to be nothing happening on the VM:

$ veewee vbox build  '2008r2' --force
Downloading vbox guest additions iso v 4.1.12 - http://download.virtualbox.org/virtualbox/4.1.12/VBoxGuestAdditions_4.1.12.iso
Checking if isofile VBoxGuestAdditions_4.1.12.iso already exists.
Full path: /cygdrive/c/Users/John/repos/veewee/iso/VBoxGuestAdditions_4.1.12.iso

The isofile VBoxGuestAdditions_4.1.12.iso already exists.
Building Box 2008r2 with Definition 2008r2:
- postinstall_include : []
- postinstall_exclude : []
- force : true

The isofile en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso already exists.
Received port hint - 59856
Found port 59856 available
Creating vm 2008r2 : 384M - 1 CPU - Windows2008_64
Creating new harddrive of size 10140
Mounting cdrom: C:\Users\John\repos\veewee\iso\en_windows_server_2008_r2_standard_enterprise_datacenter_web_x64_dvd_x15-50365.iso
Mounting guest additions: C:\Users\John\repos\veewee\iso\VBoxGuestAdditions_4.1.12.iso
Attaching disk: C:\Users\John\VirtualBox VMs/2008r2/2008r2.vdi
Creating floppy: java -jar C:\\Users\\John\\repos\\veewee\\lib\\java\\dir2floppy.jar "C:\\cygwin\\tmp\\d20120513-6476-np2ws7" "C:\\Users\\John\\repos\\veewee\\definitions\\2008r2\\virtualfloppy.v
fd"
Received port hint - 59856
Found port 59856 available
Changing ssh port from 5985 to 59856
Waiting 1 seconds for the machine to boot

Typing:[1]:
Done typing.

Skipping webserver as no kickstartfile was specified
Starting a webserver :
Waiting for ssh login on 127.0.0.1 with user vagrant to sshd on port => 59856 to work, timeout=10000 sec
........................

I noticed that there was a C:\cygwin folder on my new VM, but it was empty. No wonder sshd wasn't connecting. The floppy (A:) contained install-cygwin-sshd.bat which was supposed to install cygwin. But it didn't contain cygwin-setup.exe like it should. (I ran the bat file inside a command prompt so that I could see where the first error was occurring.) So I copied that file to C:\Users\John\repos\veewee\definitions\2008r2, added a reference to that file in the ~/repos/veewee/definitions/2008r2/definition.rb file as below.

    :floppy_files => [
      "Autounattend.xml", # automate install and setup winrm
      "install-cygwin-sshd.bat",
      "install-winrm.bat",
      "cygwin-setup.exe",
      "oracle-cert.cer"],

Then shutdown & deleted my VM, and re-ran veewee build. Now I see Cygwin downloading various packages. It will be nice when it can use the repository from my host machine, rather than downloading from the Internet again. I know it should only have to do this once, but if we're developing...

So now I have a server sitting there with Cygwin installed, but it's not answering the sshd request that my host is trying to make. It seems the install-cygwin-sshd.bat may still not be completing. I don't want to run the whole thing, since it would install cygwin again. So I checked parts individually.

• The cygrunsrv command worked. 
• The /etc/group and /etc/passed files are there. 
• The ssh-host-config command seems to run ok.
• Although the ssh rule is in the firewall configuration, the SSHD one isn't. When I try to run the command manually, the SSHD one complains. It seems the word "SSHD" near the end might be causing problems, since it works correctly if I leave that bit out. Must look into that a bit more.
• Running "net start sshd" returns " "System error 1069 has occurred. The service did not start due to a logon failure". Checked in the services control panel, and saw that there was an entry for Cygwin sshd, that indeed was not running. Trying to start it caused the same error.

And I'm out of time for today. I hope I'll be able to finish this journey soon, and describe further experience. I'd be pleased to hear from anyone who can suggest better ways to do all this.

St Hallett Old Block Shiraz

Three of us visited Craig's place last night. It was supposed to be night for board games, but since it had been a fair while since the last one, we had a bit of chat to get through. And the tour of his house. The tour ended with a visit to the wine fridge. Then we had to pore over the wine list and choose something.

Craig was fairly confident that the 1991 St Hallett Old Block shiraz (acquired at auction 6 years ago) would be past its best. That was just the excuse I needed to encourage him to choose it - no point waiting longer!

The cork was very soft, and less than half came out with the corkscrew. After some digging, the only option was to push the cork in, and to sieve and decant. The bottle was fairly free of lees, the wine was on the brickish side of red, and the scent was a cause for optimism. After decanting, we left it for about 30 mins.

The first tastes revealed a rather solid depth with central smoothness, although there were still some aromatics and a hint of acetic acid. But after it had warmed slightly in the glass, and perhaps with slightly recalibrated palates thanks to the Mersey Valley cheese, the tannic complexities began to reveal themselves, and worries about the wine being too old seemed to evaporate. No unripe peppery flavours here; it was just plums with a hint of barnyard. The only things I could have asked for more of would have been some touches of tobacco and a bit more length.

We concluded in the end that the wine may well have been better a few years ago, but it was definitely still on the delicious side of good last night.

Thanks Craig!

Binary Polictical Prisoners

I get jealous when I see that other governments are making all their data freely available. I can understand why, in the old economy, it could make sense to have user-pays access to all that data. But these days, it's so much less appropriate to keep that data locked up - political prisoner. I've written to my state Member of Parliament and the relevant minister to ask LNP and Labor positions.

Dear ...
I believe that as much state government information as possible (such as property sales data) should be free. While the government raises some revenue from selling access to such information, I think that making such information publicly available would reduce the bureaucracy.  Most importantly though, it would create an environment that would allow people to create useful innovative services for the public. There are precedents:

• In 2005 Peter Costello made all the Australian Bureau of Statistics information freely available. http://www.abs.gov.au/websitedbs/d3310114.nsf/4a256353001af3ed4b2562bb00121564/83b66e9ffafd6dabca257140007575b5!OpenDocument
• The UK has recently decided to take this path on an even bigger scale. http://www.cabinetoffice.gov.uk/news/public-data-corporation-free-public-data-and-drive-innovation

Does (your party) support such a position?

Best regards,

John.

If I get any answers, I'll put them here.

Monad tutorials

I studied some computer science when I was at uni. It was pretty interesting. Perhaps I didn't do enough, though. I didn't learn about compilers, nor much about functional languages. I've been redressing both of these deficits a little, recently. I've been reacquainting myself with the Antlr compiler compiler (20 years ago it was yacc and lex!), and looking at monads - not because I have an urgent need, or because I've had a Haskell conversion, but because it seems like an interesting intellectual challenge. I've even started reading SICP.

I'm not tempted to tell you that a Monad is like a Burrito, nor to write my own monad tutorial. I can't say I've put in the effort to write a historical analysis of a whole bunch of them. Instead, all I can do is tell you that the three best descriptions I've found are:

• Brian Beckman's video
• Mike Vanier's tutorial
• Jeff Newbern's guide

Using Apache httpclient through an NTLM authenticating proxy with ftp

I needed to (programmatically) retrieve a file from an FTP server out in the internet. In this example, the URL is ftp://site.com/dir/file.txt. My computer can only access the Internet through proxies. There is an HTTP proxy called web-proxy.local, and an FTP proxy called ftp-proxy.local.

I noticed that I could retrieve the file using my browser, but not using command-line ftp. I determined that the ftp-proxy was slightly mis-configured, and didn't believe that my host was a legitimate user. But how did the browser fetch the file, using that URL above? A little work with wireshark showed that the browser makes an HTTP connection to the proxy, and passes the HTTP command:

GET ftp://site.com/dir/file.txt HTTP/1.1

When I tried to use the java.net.URLConnection with this URL, it wouldn't connect to the web-proxy. That seemed reasonable - it was probably trying to connect to it with FTP. but somehow I needed to create an HTTP connection to a URL starting with FTP.

I decided to try Apache HttpComponents HttpClient - Apache code is always great. After a little difficulty getting the right version (eventually 4.1), I found that I was getting

java.lang.IllegalStateException: Scheme 'ftp' not registered.
at org.apache.http.conn.scheme.SchemeRegistry.getScheme(SchemeRegistry.java:71)
at org.apache.http.impl.conn.DefaultHttpRoutePlanner.determineRoute(DefaultHttpRoutePlanner.java:111)
at org.apache.http.impl.client.DefaultRequestDirector.determineRoute(DefaultRequestDirector.java:710)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:356)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
...

I peered into the source and javadoc of the DefaultHttpRoutePlanner and SchemeRegistry, and discovered that I could add the 'ftp' scheme like this:

HttpClient hc = new DefaultHttpClient()

// Register a scheme so that we can ask the proxy to use ftp
Scheme ftp = new Scheme("ftp", 80, new PlainSocketFactory())
hc.getConnectionManager().getSchemeRegistry().register(ftp)

In this case, I don't think it matters what port number (80) I use, or even which type of socket factory, since the connection will be sent via the proxy anyway - the system doesn't really need to know how to create an ftp socket, or which port to use.

The next issue I had was making it work with the proxy. I had copied the example HttpClient code for using authenticating proxies, but it didn't work. Again, wireshark helped. When the browser fetched the file, I could see the 3-phase NTLM negotiation. But not when my software ran. A spot of googling showed me that instead of using the UsernamePasswordCredentials, it would be better to use NTCredentials. And now it works. The final code looks like this:

HttpClient hc = new DefaultHttpClient()

// Register a scheme so that we can ask the proxy to use ftp
Scheme ftp = new Scheme("ftp", 80, new PlainSocketFactory())
hc.getConnectionManager().getSchemeRegistry().register(ftp)

// Set up NT(LM)Credentials for use with the proxy.
hc.getCredentialsProvider().setCredentials(AuthScope.ANY, new NTCredentials("myUsername", "myPassword", "", ""));

// Set up the proxy
HttpHost proxy = new HttpHost("web-proxy.local", 8080);
hc.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy)

// Set up the URL to fetch
HttpGet hg = new HttpGet(ftp://site.com/dir/file.txt)
HttpResponse hr = hc.execute(hg)

HttpEntity entity = hr.getEntity()
InputStream instream = entity.getContent()
...

Grails proxy settings

I always forget these.

1. When setting the details from the (Windows) command line, use quotes everywhere:
> grails add-proxy someProxyName "--host=the.host.name" "--port=8080" "--username=myUserName" "--password=myPassword"

2. Don't forget to set-proxy:
> grails set-proxy someProxyName

3. The ProxySettings.groovy file is stored in the top level of .grails - not in the 1.3.5 subdirectory etc.

Online book shopping

For some professional development, and to see what Uncle Bob is on about, I decided to buy Structure and Interpretation of Computer Programs (SICP).

I was impressed that QBD Online had a price (including delivery) of $53.95. This was comparable to Amazon ($53.18), and Book Depository ($57.54). (It's available for free online, but I think I'd rather pay for a hard copy.) I ordered through QBD. Eleven days later (admittedly some were public holidays), they informed me that they were "temporarily out of stock", and the publisher couldn't advise when more stock would be available. I wasn't happy, and asked them to cancel the order. I was pleasantly surprised to see that they responded very quickly (cancel requested 9pm, response at 6am). It means that I'm more likely to go back to them in future.

It was simple to order the book at Amazon. They said that the book was in stock. Twelve hours later I have an email saying that the book has been shipped. Nice contrast.

Setting up Scholastic "I Spy Fantasy" for non-administrative users

My children have non-administrator accounts on their computers, one of which uses Windows XP. Maybe that doesn't encourage them to learn, but that's how it is at the moment. They wanted to install (again) I Spy Fantasy, an old game, but it required administrative access. That was ok. But then we found that administrative access was required not just to install, but also to play the game. That's not ok.

A short web search didn't yield any answers, so I delved. The error message reported that it couldn't save game details to the disk. Probably because it stores the saved games in C:\Program Files\Scholastic...

I tried to move the saved games folder, and make a shortcut, but that seemed fruitless. So I moved the whole installation folder from Program Files to daughter's Documents and Settings folder. But when running the application, it still complained about the same problem. I wondered if the registry was pointing to the old location. Sure enough - Local Computer/Software/Scholastic inc/I Spy Fantasy/install-dir pointed to Program Files... . So I changed that, switched users back to the non-administrative account, and tried again. This time, the error message was new: the program was trying to modify the registry! Well, it looks like in regedit, you can set permissions for different entries. So I gave the non-privileged user full control for just the I Spy Fantasy folder. That's better.

Oops. One last problem. Don't forget to adjust where the start menu shortcut points to!

That seems to have solved the problem. I wonder if anyone else cares about such things. Maybe you'll leave me  a comment if this helps.

Prime factors coding kata using transformations.

Inspired by Uncle Bob's Transformation Priority Premise, Glennn and I tried a Groovy kata.

  @Test
  void shouldReturnPrimeFactors(){
    assertThat getPrimeFactors(1), equalTo([])
    assertThat getPrimeFactors(2), equalTo([2])
    assertThat getPrimeFactors(3), equalTo([3])
    assertThat getPrimeFactors(4), equalTo([2,2])
    assertThat getPrimeFactors(5), equalTo([5])
    assertThat getPrimeFactors(6), equalTo([2,3])
    assertThat getPrimeFactors(7), equalTo([7])
    assertThat getPrimeFactors(8), equalTo([2,2,2])
    assertThat getPrimeFactors(9), equalTo([3,3])
    assertThat getPrimeFactors(10), equalTo([2,5])
    assertThat getPrimeFactors(11), equalTo([11])
    assertThat getPrimeFactors(12), equalTo([2,2,3])
    assertThat getPrimeFactors(13), equalTo([13])
    assertThat getPrimeFactors(14), equalTo([2,7])
    assertThat getPrimeFactors(15), equalTo([3,5])
    assertThat getPrimeFactors(16), equalTo([2,2,2,2])
    assertThat getPrimeFactors(17), equalTo([17])
    assertThat getPrimeFactors(18), equalTo([2,3,3])

    assertThat getPrimeFactors(25), equalTo([5,5])

    assertThat getPrimeFactors(64), equalTo([2,2,2,2,2,2])

    assertThat getPrimeFactors(74), equalTo([2,37])
}


We didn't record everything as we went. It would have been good to look back at our mistakes. But here's the steps I took when re-doing it later.

Step 1. "{} -> nil" transformation. Suceeds for 1, fails at 2.
  static def getPrimeFactors(int number) {
    return []
  }


Step 2. "unconditional -> if" transformation. Succeeds up to 3, fails at 4.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    return [number]
  }


Step 3. "unconditional -> if" transformation and "statement-> recursion" transformation. Fails at 9.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    if (number%2 == 0) {
      return [2]+getPrimeFactors(number.intdiv(2))
    }
    return [number]
  }


Step 4. "unconditional -> if" transformation. Fails at 25.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    if (number%2 == 0) {
      return [2]+getPrimeFactors(number.intdiv(2))
    }
    if (number%3 == 0) {
      return [3]+getPrimeFactors(number.intdiv(3))
    }
    return [number]
  }


Step 5. Refactor to remove duplication. No change to behaviour.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    for (int divisor=2; divisor<=3; divisor++) {
      if (number % divisor == 0) {
        return [divisor] + getPrimeFactors(number.intdiv(divisor))
      }
    }
    return [number]
  }


Step 6. "constant -> scalar" transformation. Success for all cases.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    for (int divisor=2; divisor*divisor<=number; divisor++) {
      if (number % divisor == 0) {
        return [divisor] + getPrimeFactors(number.intdiv(divisor))
      }
    }
    return [number]
  }

One slightly undesirable characteristic is that this method attempts to use composite divisors, although these can never succeed. Some may consider it better to construct a list of primes as potential divisors. I'll look into this another time, but I suspect that the additional work to construct the list of primes will add complexity, and not improve performance too much (unless the primes are calculated just once, and there are many calls to getPrimeFactors).