It’s been a busy several weeks for me, mostly in my work-related universe.  I haven’t been completely disregarding the public sphere, though, and managed to get a CBR / CBZ utility script working, to wit cbfix.  Currently it only does one thing, but having the scaffolding around to open up CBR / CBZ files, mess with them, and replace them presents a lot of possibilities. In the meantime I still have a bunch of CoffeeScript and Scala stuff simmering on the back burner.

Well, that took a while

It’s been a minute since my last update, during which time I’ve learnt a great deal about node.js as well as the whole AMD / CommonJS mess mentioned in my last post. Without drowning in detail, here are a few things I’ve been thinking about.

I’ve made the switch to CoffeeScript for a good deal of my Javascript code, mostly because I was worried about wearing out the keys on my keyboard which comprise the phrase function(){}. I rather enjoy CoffeeScript’s cleaned-up syntax, at least for now, and I like how it hews to Javascript’s essential character while smoothing out its rougher bits. I’ve still got plenty of reservations about it, and it sometimes errs a bit too far on the loosey-goosey side of the syntax fence for my taste, but anything that will save me from the ongoing insults to programming-language aesthetics coming out of the Javascript world has my sincere gratitude. (The latest of these I’ve had to endure is the comma-first style, which is just one of the awful emergent results from the most-used data transport format of our day, the JSON spec, needing to fit into three quarters of a page of EBNF for some reason. Why complicate it with an optional comma at the end, or optional quotes around key values? But that’s a rant for another day.)

Meanwhile I also went down a fairly long rabbit hole in learning about electronics – this started with getting a Raspberry Pi and then got a little more low-level with some Ardiuno stuff, and by now I’ve got a bunch of nascent electronics projects in the offing, mostly concerning various robots. The Pi in particular has a robust Python infrastructure, and I have a pretty good imaging-related project in mind, but in the course of writing the software for it I decided that this would be a good opportunity to experiment with node.js development, and so I’ve since ported the stuff I’d written from Python to CoffeeScript and node.js.

More is surely to come about all that, and as usual I ought to have some code available on github before too long. In the course of all this I’ve also learned a great deal more about the server-side javascript ecosystem, and I think I should actually be ready to get the weighted-probability list library into much better shape.

Mysterious 404 responses with foursquare OAuth2 + google-app-engine

I’ve done a lot of work on my aforementioned foursquare / last.fm mashup project, which I’m still not exactly ready to start crowing about, but whose source is, as ever, available.

I’m currently hitting a bunch of problems in getting my Python code to work while it’s deployed out to Google’s appspot.com production environments.  I’ve been able to authenticate successfully against foursquare from my local environment, following their straightforward instructions, but every time I try to run the same code out in Google’s cloud, once I hit the point where I retrieve an access code from foursquare and make a request for a legit session token, this is what I wind up with:

    HTTP blarg, feh
    Traceback (most recent call last):
    File "/base/python27_runtime/python27_lib/versions/third_party/webapp2-2.3/webapp2.py", line 545, in dispatch
      return method(*args, **kwargs)
    File "/base/data/home/apps/s~how-you-been/0-2-4.357616581640073459/whence.py", line 97, in get
      accessCode = self.getFoursquareAccessToken(code)
    File "/base/data/home/apps/s~how-you-been/0-2-4.357616581640073459/howyoubeen/Foursquare.py", line 49, in getFoursquareAccessToken
      httpResponse = urllib2.urlopen(url)
    File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 126, in urlopen
      return _opener.open(url, data, timeout)
    File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 400, in open
      response = meth(req, response)
    File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 513, in http_response
      'http', request, response, code, msg, hdrs)
    File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 438, in error
      return self._call_chain(*args)
    File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 372, in _call_chain
      result = func(*args)
    File "/base/python27_runtime/python27_dist/lib/python2.7/urllib2.py", line 521, in http_error_default
      raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

Dang it!  My initial instinct is that I’m hitting a timeout in Google’s cloud environment, but are they honestly passing a 404 back to my stack at that point?  (I suspect, as an alternate explanation, something about my error-handling handlers are deficient.)

While looking into this I’ve stumbled upon a few Python OAuth2 libraries, about which OAuth2 isn’t exactly rocket science, but if the libraries have a decent interface and can overcome this issue, maybe I’ll give them a shake.  (On the js side of the world I’ve been having a few similar issues, where I’d like to move to an open-source library but the ones I’ve seen make odd assumptions about the execution environment.)

Another possibility I’ve considered is that there’s some sort of SSL wildcard issue at play, as suggested by this stack overflow answer which foursquare has bravely elected as their replacement for tech support.  But for now, it is a mystery.

Using pyjade with webapp2 on Google App Engine

And now for something completely different. I got a little distracted again by an idea for a simple social networking mashup I’ve had and decided that the best place to deploy it would probably on Google App Engine.  App Engine supports Scala and I’ve seen one or two positive reports about using Scalatra on it, so I thought it wouldn’t be too hard to spin something up quickly using my existing code.

It turned out not to be as easy as that, unfortunately; I may try to elaborate on the reasons in a later post, but it seems to come down to Scalate always checking the filesystem for its template routines.  In any event, I found that I was spending a lot of time trying to glue weird bits of infrastructure to one another rather than working on my idea.

With that in mind, I decided to start over in Python, and of course as my first task I spent a long time glueing random bits of infrastructure together.  In this case I’ve been successful, though, so here’s a brief writeup on getting Jade templates to work with Google App Engine and Google’s webapp2 in python.

1. Add dependencies to include webapp2’s jinja2 module in your project.

This is as simple as adding this bit of code to app.yaml:

- name: jinja2
  version: latest

2. Install pyjade in your environment.

I’ve been using a layout similar to the one mentioned in this Stack Overflow answer.  It’s puzzling that Google doesn’t really mention anything about a standard way to get libraries and stuff from the wider Python ecosystem installed in a development environment, but essentially I set up a virtualenv, ran pip install pyjade, and then symlinked the pyjade directory from the virtualenv lib directory into the project’s src directory.

3. Add a custom Jinja2 factory.

This is where the magic happens.  We write a factory function, jinja2_factory, whose purpose is to add pyjade’s included Jinja2 extension into the Jinja2 instance’s Environment.

  def jade_factory(app):
    j = jinja2.Jinja2(app)
    return j

Then, in our handler’s jinja2 method, we pass along the factory method to it:

  def jinja2(self):
    return jinja2.get_jinja2(app=self.app, factory=jade_factory)

(This is based on moraes’s Stack Overflow answer here.)

4. Use the factory and get a template.

Here’s the full source code for a simple “hello world” app. I’m following the examples on Google’s jinja2 page and creating a JadeHandler subclass of RequestHandler.

import os
import webapp2
from webapp2_extras import jinja2

class JadeHandler(webapp2.RequestHandler):
  # Per http://stackoverflow.com/a/7081653/87990
  def jade_factory(app):
    j = jinja2.Jinja2(app)
    return j
  def jinja2(self):
    return jinja2.get_jinja2(app=self.app, factory=JadeHandler.jade_factory)

  def render_response(self, _template, **context):
    # Renders a template and writes the result to the response.
    rv = self.jinja2.render_template(_template, **context)

class MainPage(JadeHandler):
  def get(self):
    context = {'message': 'Hello, world!'}
    self.render_response('index.jade', **context)

app = webapp2.WSGIApplication([('/', MainPage)], debug=True)


I’m by no means an expert in Google App Engine, and I’m a little worried that there’s something badly inefficient or similarly wrong about this approach. In particular, I’m hoping the returned templates are cached, but I’m not certain, and I’ve seen reports that Jinja2 Environment creation is an expensive operation on GAE. More to the point, pyjade seems to work as a preprocessor for Jinja2, so it would obviously be better not to need to run it for every request. In any event, this does at least find jade templates and render them properly as HTML.

Edit: as usual, the current code is available on GitHub.