Using pyjade with webapp2 on Google App Engine

And now for something completely different. I got a little distracted again by an idea for a simple social networking mashup I’ve had and decided that the best place to deploy it would probably on Google App Engine.  App Engine supports Scala and I’ve seen one or two positive reports about using Scalatra on it, so I thought it wouldn’t be too hard to spin something up quickly using my existing code.

It turned out not to be as easy as that, unfortunately; I may try to elaborate on the reasons in a later post, but it seems to come down to Scalate always checking the filesystem for its template routines.  In any event, I found that I was spending a lot of time trying to glue weird bits of infrastructure to one another rather than working on my idea.

With that in mind, I decided to start over in Python, and of course as my first task I spent a long time glueing random bits of infrastructure together.  In this case I’ve been successful, though, so here’s a brief writeup on getting Jade templates to work with Google App Engine and Google’s webapp2 in python.

1. Add dependencies to include webapp2’s jinja2 module in your project.

This is as simple as adding this bit of code to app.yaml:

libraries:
- name: jinja2
  version: latest

2. Install pyjade in your environment.

I’ve been using a layout similar to the one mentioned in this Stack Overflow answer.  It’s puzzling that Google doesn’t really mention anything about a standard way to get libraries and stuff from the wider Python ecosystem installed in a development environment, but essentially I set up a virtualenv, ran pip install pyjade, and then symlinked the pyjade directory from the virtualenv lib directory into the project’s src directory.

3. Add a custom Jinja2 factory.

This is where the magic happens.  We write a factory function, jinja2_factory, whose purpose is to add pyjade’s included Jinja2 extension into the Jinja2 instance’s Environment.

  def jade_factory(app):
    j = jinja2.Jinja2(app)
    j.environment.add_extension('pyjade.ext.jinja.PyJadeExtension')
    return j

Then, in our handler’s jinja2 method, we pass along the factory method to it:

  @webapp2.cached_property
  def jinja2(self):
    return jinja2.get_jinja2(app=self.app, factory=jade_factory)

(This is based on moraes’s Stack Overflow answer here.)

4. Use the factory and get a template.

Here’s the full source code for a simple “hello world” app. I’m following the examples on Google’s jinja2 page and creating a JadeHandler subclass of RequestHandler.

import os
import webapp2
from webapp2_extras import jinja2

class JadeHandler(webapp2.RequestHandler):
  # Per http://stackoverflow.com/a/7081653/87990
  @staticmethod
  def jade_factory(app):
    j = jinja2.Jinja2(app)
    j.environment.add_extension('pyjade.ext.jinja.PyJadeExtension')
    return j
  
  @webapp2.cached_property
  def jinja2(self):
    return jinja2.get_jinja2(app=self.app, factory=JadeHandler.jade_factory)

  def render_response(self, _template, **context):
    # Renders a template and writes the result to the response.
    rv = self.jinja2.render_template(_template, **context)
    self.response.write(rv)

class MainPage(JadeHandler):
  def get(self):
    context = {'message': 'Hello, world!'}
    self.render_response('index.jade', **context)

app = webapp2.WSGIApplication([('/', MainPage)], debug=True)

Caveat

I’m by no means an expert in Google App Engine, and I’m a little worried that there’s something badly inefficient or similarly wrong about this approach. In particular, I’m hoping the returned templates are cached, but I’m not certain, and I’ve seen reports that Jinja2 Environment creation is an expensive operation on GAE. More to the point, pyjade seems to work as a preprocessor for Jinja2, so it would obviously be better not to need to run it for every request. In any event, this does at least find jade templates and render them properly as HTML.

Edit: as usual, the current code is available on GitHub.

The cake falls

(I’m running out of snappy cake-related puns here, but I’m saving “the cake is a lie” for a special occasion.)

So after thinking about my interface / implementation dilemma for a while (recap: I’m aiming to separate out the implementations of hexmap from their interface), I decided to approach it from a Java vantage point and see how I’d implement the same thing in Java.  The immediately obvious thing that jumped out at me was that the specific problem I was dealing with was sort of a classic application of plain old vanilla inheritance from an abstract class.  Having drunk deeply from the well of Spring over my last several years of Java experience, plain old inheritance had begun to seem in some ways like a quaint relic of a bygone era (having been replaced in many cases by service composition).

At any rate, a quick bit of refactoring to get rid of traits altogether got me back to a fairly sane codebase, where I’ve got this:

abstract class HexMap (h: Int, w: Int) {
  val height = h
  val width = w
  // This is where the magic happens
  def data(addr: Address): HexData
}
abstract class HexData(a: Address) {
  def toJson(): JObject = JObject(List()) // default: empty JSON object
}

case class ColorData(a: Address, color: String) extends HexData (a) {
  override def toJson(): JObject = {
    ("color" -> color)  // json {"color": "foo"} via JsonDSL
  }
}
class RandomColorMap(h: Int, w: Int) extends HexMap(h, w) {
  private val store: Map[Address, ColorData] = createRandomMap()
  def data(addr: Address): ColorData = {
    store(addr)
  }
  // Note, we're able to access width and height here
  private def createRandomMap(): Map[Address, ColorData] = {
    val pairs = for (x <- 0 until width; y <- 0 until height) yield {
      val addr = Address(x, y)
      addr -> ColorData(addr, randomColor())
    }
    pairs toMap
  }
}

I will come back and revisit this later; I’m certain there’s a way traits could be useful in this code, but for now I’m leaving them aside and pressing on with more actual program functionality.

As far as that is concerned, my next worry is persisting the hexmap the user sees in their Scalatra session, so that when the user asks for a random path from the server, he or she gets one within the actual bounds of the actual map he or she is seeing. The Scalatra session docs make this look fairly straightforward; we’ll see.

The jadening

Bolting Scalatra on to the existing hexmap project turned out to be fairly easy, though I had the advantage of having an existing minimal project already (the nascent random-album-cover project).  My idea for this project is to have nearly all the server transactions handled through Ajax rather than form posts and server-side templating, but while I was getting the directory structure up and running I took the opportunity to translate the existing HTML that I developed for the hexmap into Jade.

My initial impulse, actually, was to use Mustache, and avoid rewriting the HTML altogether.  However, I found the layout documentation for scalate Mustache to be basically incomprehensible, and if I’m understanding it correctly, it requires you to insert extra stuff into your layout file and still not remove anything from your content file, which doesn’t exactly seem like it’s heading in the right direction as far as templating system design.  Besides, using Jade lets me pretend that I live in a universe where SGML-derived syntaxes never became something that almost every working programmer would have to deal with almost every day.

The downside of this is that I’m no longer able to monkey with the client-side code without a running jetty instance, but I think that should be OK for the near future.  The upside is that the code for index.html has shrank down to a svelte 36 lines.

At any rate, I’ve got the client code retrieving and painting a map from the server now.  My server-side map database currently looks like this:

  get("/map/new.json") {
    contentType = "application/json"
    """{ "map" : [["blue", "brown", "yellow"],
                  ["lightgreen", "yellow", "blue"],
                  ["grey", "grey", "darkgreen"]] }"""
  }

It’s a start. I’ve also pulled in the Lift-JSON libraries, which look to be useful. The next step is to implement a few different map stores, hook them up to the client, and test the output. After that, I need to figure out a way to populate actually interesting map data in a persistent store (presumably Mongo). The idea which keeps nagging at the back of my mind is that the existing code could probably be used as the start of a dynamic map editor, but I’m not positive I want to head down that route.