Using Apache httpclient through an NTLM authenticating proxy with ftp

I needed to (programmatically) retrieve a file from an FTP server out in the internet. In this example, the URL is ftp://site.com/dir/file.txt. My computer can only access the Internet through proxies. There is an HTTP proxy called web-proxy.local, and an FTP proxy called ftp-proxy.local.

I noticed that I could retrieve the file using my browser, but not using command-line ftp. I determined that the ftp-proxy was slightly mis-configured, and didn't believe that my host was a legitimate user. But how did the browser fetch the file, using that URL above? A little work with wireshark showed that the browser makes an HTTP connection to the proxy, and passes the HTTP command:

GET ftp://site.com/dir/file.txt HTTP/1.1

When I tried to use the java.net.URLConnection with this URL, it wouldn't connect to the web-proxy. That seemed reasonable - it was probably trying to connect to it with FTP. but somehow I needed to create an HTTP connection to a URL starting with FTP.

I decided to try Apache HttpComponents HttpClient - Apache code is always great. After a little difficulty getting the right version (eventually 4.1), I found that I was getting

java.lang.IllegalStateException: Scheme 'ftp' not registered.
at org.apache.http.conn.scheme.SchemeRegistry.getScheme(SchemeRegistry.java:71)
at org.apache.http.impl.conn.DefaultHttpRoutePlanner.determineRoute(DefaultHttpRoutePlanner.java:111)
at org.apache.http.impl.client.DefaultRequestDirector.determineRoute(DefaultRequestDirector.java:710)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:356)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
...

I peered into the source and javadoc of the DefaultHttpRoutePlanner and SchemeRegistry, and discovered that I could add the 'ftp' scheme like this:

HttpClient hc = new DefaultHttpClient()

// Register a scheme so that we can ask the proxy to use ftp
Scheme ftp = new Scheme("ftp", 80, new PlainSocketFactory())
hc.getConnectionManager().getSchemeRegistry().register(ftp)

In this case, I don't think it matters what port number (80) I use, or even which type of socket factory, since the connection will be sent via the proxy anyway - the system doesn't really need to know how to create an ftp socket, or which port to use.

The next issue I had was making it work with the proxy. I had copied the example HttpClient code for using authenticating proxies, but it didn't work. Again, wireshark helped. When the browser fetched the file, I could see the 3-phase NTLM negotiation. But not when my software ran. A spot of googling showed me that instead of using the UsernamePasswordCredentials, it would be better to use NTCredentials. And now it works. The final code looks like this:

HttpClient hc = new DefaultHttpClient()

// Register a scheme so that we can ask the proxy to use ftp
Scheme ftp = new Scheme("ftp", 80, new PlainSocketFactory())
hc.getConnectionManager().getSchemeRegistry().register(ftp)

// Set up NT(LM)Credentials for use with the proxy.
hc.getCredentialsProvider().setCredentials(AuthScope.ANY, new NTCredentials("myUsername", "myPassword", "", ""));

// Set up the proxy
HttpHost proxy = new HttpHost("web-proxy.local", 8080);
hc.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY, proxy)

// Set up the URL to fetch
HttpGet hg = new HttpGet(ftp://site.com/dir/file.txt)
HttpResponse hr = hc.execute(hg)

HttpEntity entity = hr.getEntity()
InputStream instream = entity.getContent()
...

Grails proxy settings

I always forget these.

1. When setting the details from the (Windows) command line, use quotes everywhere:
> grails add-proxy someProxyName "--host=the.host.name" "--port=8080" "--username=myUserName" "--password=myPassword"

2. Don't forget to set-proxy:
> grails set-proxy someProxyName

3. The ProxySettings.groovy file is stored in the top level of .grails - not in the 1.3.5 subdirectory etc.

Online book shopping

For some professional development, and to see what Uncle Bob is on about, I decided to buy Structure and Interpretation of Computer Programs (SICP).

I was impressed that QBD Online had a price (including delivery) of $53.95. This was comparable to Amazon ($53.18), and Book Depository ($57.54). (It's available for free online, but I think I'd rather pay for a hard copy.) I ordered through QBD. Eleven days later (admittedly some were public holidays), they informed me that they were "temporarily out of stock", and the publisher couldn't advise when more stock would be available. I wasn't happy, and asked them to cancel the order. I was pleasantly surprised to see that they responded very quickly (cancel requested 9pm, response at 6am). It means that I'm more likely to go back to them in future.

It was simple to order the book at Amazon. They said that the book was in stock. Twelve hours later I have an email saying that the book has been shipped. Nice contrast.

Setting up Scholastic "I Spy Fantasy" for non-administrative users

My children have non-administrator accounts on their computers, one of which uses Windows XP. Maybe that doesn't encourage them to learn, but that's how it is at the moment. They wanted to install (again) I Spy Fantasy, an old game, but it required administrative access. That was ok. But then we found that administrative access was required not just to install, but also to play the game. That's not ok.

A short web search didn't yield any answers, so I delved. The error message reported that it couldn't save game details to the disk. Probably because it stores the saved games in C:\Program Files\Scholastic...

I tried to move the saved games folder, and make a shortcut, but that seemed fruitless. So I moved the whole installation folder from Program Files to daughter's Documents and Settings folder. But when running the application, it still complained about the same problem. I wondered if the registry was pointing to the old location. Sure enough - Local Computer/Software/Scholastic inc/I Spy Fantasy/install-dir pointed to Program Files... . So I changed that, switched users back to the non-administrative account, and tried again. This time, the error message was new: the program was trying to modify the registry! Well, it looks like in regedit, you can set permissions for different entries. So I gave the non-privileged user full control for just the I Spy Fantasy folder. That's better.

Oops. One last problem. Don't forget to adjust where the start menu shortcut points to!

That seems to have solved the problem. I wonder if anyone else cares about such things. Maybe you'll leave me  a comment if this helps.

Prime factors coding kata using transformations.

Inspired by Uncle Bob's Transformation Priority Premise, Glennn and I tried a Groovy kata.

  @Test
  void shouldReturnPrimeFactors(){
    assertThat getPrimeFactors(1), equalTo([])
    assertThat getPrimeFactors(2), equalTo([2])
    assertThat getPrimeFactors(3), equalTo([3])
    assertThat getPrimeFactors(4), equalTo([2,2])
    assertThat getPrimeFactors(5), equalTo([5])
    assertThat getPrimeFactors(6), equalTo([2,3])
    assertThat getPrimeFactors(7), equalTo([7])
    assertThat getPrimeFactors(8), equalTo([2,2,2])
    assertThat getPrimeFactors(9), equalTo([3,3])
    assertThat getPrimeFactors(10), equalTo([2,5])
    assertThat getPrimeFactors(11), equalTo([11])
    assertThat getPrimeFactors(12), equalTo([2,2,3])
    assertThat getPrimeFactors(13), equalTo([13])
    assertThat getPrimeFactors(14), equalTo([2,7])
    assertThat getPrimeFactors(15), equalTo([3,5])
    assertThat getPrimeFactors(16), equalTo([2,2,2,2])
    assertThat getPrimeFactors(17), equalTo([17])
    assertThat getPrimeFactors(18), equalTo([2,3,3])

    assertThat getPrimeFactors(25), equalTo([5,5])

    assertThat getPrimeFactors(64), equalTo([2,2,2,2,2,2])

    assertThat getPrimeFactors(74), equalTo([2,37])
}


We didn't record everything as we went. It would have been good to look back at our mistakes. But here's the steps I took when re-doing it later.

Step 1. "{} -> nil" transformation. Suceeds for 1, fails at 2.
  static def getPrimeFactors(int number) {
    return []
  }


Step 2. "unconditional -> if" transformation. Succeeds up to 3, fails at 4.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    return [number]
  }


Step 3. "unconditional -> if" transformation and "statement-> recursion" transformation. Fails at 9.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    if (number%2 == 0) {
      return [2]+getPrimeFactors(number.intdiv(2))
    }
    return [number]
  }


Step 4. "unconditional -> if" transformation. Fails at 25.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    if (number%2 == 0) {
      return [2]+getPrimeFactors(number.intdiv(2))
    }
    if (number%3 == 0) {
      return [3]+getPrimeFactors(number.intdiv(3))
    }
    return [number]
  }


Step 5. Refactor to remove duplication. No change to behaviour.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    for (int divisor=2; divisor<=3; divisor++) {
      if (number % divisor == 0) {
        return [divisor] + getPrimeFactors(number.intdiv(divisor))
      }
    }
    return [number]
  }


Step 6. "constant -> scalar" transformation. Success for all cases.
  static def getPrimeFactors(int number) {
    if (number == 1) return []
    for (int divisor=2; divisor*divisor<=number; divisor++) {
      if (number % divisor == 0) {
        return [divisor] + getPrimeFactors(number.intdiv(divisor))
      }
    }
    return [number]
  }

One slightly undesirable characteristic is that this method attempts to use composite divisors, although these can never succeed. Some may consider it better to construct a list of primes as potential divisors. I'll look into this another time, but I suspect that the additional work to construct the list of primes will add complexity, and not improve performance too much (unless the primes are calculated just once, and there are many calls to getPrimeFactors).

Two nice wines and Baguette

I don't have a very good way to record memorable wines. So let me note a couple here from the last couple of months before I forget them: 2008 Langmeil Orphan Bank and 2004 Reschke Bos.

And while I'm typing epicuriously, let me also note that two work friends and I had a great meal at Baguette this week. All three of us noted the complementary flavours of our dishes - the duck, the rabbit, and my pork belly, boudin noir, lentils de puy, and garlic snails. Can't wait to go back there again.

Playing with Batik

Helen's group at school has been asked to put together an "interactive trade-show booth" on the topic of Indonesia. I decided that this was a good excuse to try to develop some kind of quiz to help. It seemed that a geographical one would be good.

The concept involved a map, and two types of questions: (a) the user is asked to click on a particular place (island, country, city), and (b) a particular place is highlighted visually on the map, and the user has to choose/guess the name of that place using a multiple choice answer.

I didn't want to build a full scale GIS, so SVG was the obvious answer. I found a great map of the world at wikimedia commons. Using Inkscape, I could crop it and choose an appropriate part of SE Asia - essentially a square covering Burma to Tasmania. The really nice thing about that map is that all the paths are neatly grouped into countries, and they're even labelled inside the SVG source. (The Inkscape XML Editor was invaluable for understanding the structure and locating different paths).

Although groovy is my new language of choice at work, I decided to use plain java/swing for this project, with Apache Batik for the SVG interface. It was easy to highlight a particular country by using the DOM interface to find elements with the appropriate id, and setting the style attribute. I found later that it was critical to do this in the correct thread, and also that you must wait at startup until the map has been rendered.

I found it somewhat frustrating though, to try to detect mouse clicks on the relevant countries. I found that there were two steps necessary: I had to register as a listener for the "click" event on the main layer element - that was obvious enough. But I also had to set an onclick attribute on that element. I don't really know why, but if I don't do it, it doesn't work.

  Element e = getElementById("layer1");
  EventTarget t = (EventTarget) e;
  t.addEventListener("click", this, false);
  e.setAttribute("onclick", "var x = 1;"); // not sure why

I was quite diligent with my test-driven development of the main controller, and used Mockito extensively. It really was fun, and (I'm convinced) much faster than trying to do a big-up-front-design.

The final stage was to package it all up as a single jar. I'd used Simon Tuffs' clever one-jar before at work, and knew that it would be the best thing to use. All of those 15 or so Batik jars, plus the ever-so-useful miglayout - I didn't want to have all those files floating around, having to manually set up a classpath.

There was one extremely frustrating part, though. I kept getting an "Invalid CSS Document" error, with this stacktrace:

Invalid CSS document.
mapquiz.jar (The system cannot find the file specified)
   at org.apache.batik.css.engine.CSSEngine.parseStyleSheet(CSSEngine.java:1149)
   at org.apache.batik.dom.svg.SVGDOMImplementation.createCSSEngine(SVGDOMImplementation.java:117)
   at org.apache.batik.dom.ExtensibleDOMImplementation.createCSSEngine(ExtensibleDOMImplementation.java:212)
   at org.apache.batik.bridge.BridgeContext.initializeDocument(BridgeContext.java:378)
   at org.apache.batik.bridge.GVTBuilder.build(GVTBuilder.java:55)
   at org.apache.batik.swing.svg.GVTTreeBuilder.run(GVTTreeBuilder.java:96)

I didn't even think that I needed or was using any CSS. After googling unproductively, I looked at the source, and discovered line 113 in SVGDOMImplementation.createCSSEngine()

 URL url = getClass().getResource("resources/UserAgentStyleSheet.css");

I tried simply making a resources directory and CSS file inside my one-jar, but that didn't work. After sorting out the differences between Class.getResource() and ClassLoader.getResource(), I decided that it was going to be looking for /org/apache/batik/dom/svg/resources/UserAgentStyleSheet.css. So I created that nest of directories, jar'd up the one-jar, and it worked. There's no way that I'm a classloader expert, but that area of java is now a little less mysterious than it once was.

Just need to see what grade I get now...

Favourite songs

Ken, a work colleague, said that there had been a recent discussion about peoples' top ten favourite songs. I thought that sounded like an interesting challenge. So at the risk of saving a few fairly fickle favourites forever, here is what I came up with (in roughly chronological order, rather than order of preference).

1. Pange Lingua (Gregorian)
2. Beatus Vir (Monteverdi)
3. Schafen können sicher weiden (Bach)
4. Nacht und Träume (Schubert)
5. Im Abendrot (Richard Strauss)
6. Come away, death (Quilter)
7. I've got you under my skin (as sung by Diana Krall)
8. Somewhere (Berstein)
9. Baby Grand (Joel)
10. Chili con carne (Real Group)

Perhaps these would be my "desert island discs", rather than absolute favourites. I do like some variety, even if I do listen to those Four Last Songs (Strauss) enough to bore my family.

Testing request- and session-scoped spring beans with JUnit

I recently updated my web app to use a UserDetails bean that has request scope. Then I found out that my integration tests didn't work.

java.lang.IllegalStateException: No Scope registered for scope 'request'
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:321)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:189)
 at org.springframework.aop.target.SimpleBeanTargetSource.getTarget(SimpleBeanTargetSource.java:33)
 at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.getTarget(Cglib2AopProxy.java:657)
 at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:608)
...

The Spring Documentation made it quite clear:

The request, session, and global session scopes are only available if you use a web-aware Spring ApplicationContext implementation (such as XmlWebApplicationContext). If you use these scopes with regular Spring IoC containers such as the ClassPathXmlApplicationContext, you get an IllegalStateException complaining about an unknown bean scope.

I looked for solutions, and found Thomas Webner's and Andreas Höhmann's, but they weren't really what I wanted. After a bit of messing around, I found that it was easiest to continue using the existing ApplicationContext (which was actually a GenericApplicationContext, not web-aware), and to add a "request" scope to it. This may not be sufficient once I do more complex tests, but it seems ok for now.

   @Before
   public void setupRequestScope() {
      if (applicationContext instanceof GenericApplicationContext) {
         GenericApplicationContext context = (GenericApplicationContext) applicationContext;
         ConfigurableListableBeanFactory beanFactory = context.getBeanFactory();
         Scope requestScope = new SimpleThreadScope();
         beanFactory.registerScope("request", requestScope);
      }
   }

GWT com.google.gwt.core.client.JavaScriptException: (null): null

I recently discovered this problem when doing GWT 2.0 tests in develoment mode. (Other reports of the problem.)
I've seen it arise in calls to constructors to GWT UiObjects/Widgets, such as FlexTables, when I restrict GWT to using only a particular browser, with a line in the module definition such as

<set-property name="user.agent" value="ie6">

Just remove that line, because the headless browser the htmltest framework uses is incompatible with IE6.

Glenn has more details.