Yet another language

In the interest in keeping this blog somewhat regularly filled with content, I should note that I’ve been working on an interesting new project at work in my spare time. It’s a little game and I’m writing the back-end in Clojure, using http-kit for its websockets capability. I’m trying to figure out the best way to shoehorn core.async into it – it seems like a natural fit for websockets.

I also gave a little introduction to Clojure and ClojureScript to a technical user group at my job; the slides are up on github (well, really the source to the slides in AsciiDoc, but there are build instructions on the site, or you can just view the deck as markdown on github).

It’s interesting to me how different the programming approaches of Clojure and Scala are, despite them sharing a great deal of aesthetic ideals (in particular, immutability and good interoperability with the JVM platform).

Three quick Scala plugs

I’m still thinking about the dependency-injection and Akka stuff and will have at least one more longish post on the subject, but I’ve recently become distracted by refactoring some of my old JavaScript code into ClojureScript and figuring out Clojure’s new core.async library (ultimately I hope to form all of these distractions into a ring instead of a straight line, at which time I will be the acme of productivity, but that’s another story). But before I get too deep into Clojure land I wanted to plug a few Scala-related things I’ve come across recently.

Firstly, Derek Wyatt’s book Akka Concurrency: Building reliable software in a multi-core world is excellent, covering a great many Akka topics in an enjoyable style. In particular, his chapter on testing seems profoundly relevant to the mock injection topics I’ve been thinking about, but I haven’t quite absorbed it yet. The whole thing is refreshingly up to date, too (which makes sense since it was only published a month or two ago), and it doesn’t pad out its length with a lot of “learn Scala in 30 days” remedial material. Anyways, I recommend it thoroughly.

Secondly, John Sullivan’s long post about the cake pattern is by far the best treatment I’ve seen of it online, and is required reading for anyone interested in dependency injection in Scala. (It’s been up for several months now but somehow I missed it until recently.) John has written a dependency-injection framework, congeal, which makes instant intuitive sense for me as someone coming from a Java / Spring background; unfortunately it depends on some macro stuff which won’t make it into Scala’s mainline, so it isn’t ready for prime time and will need to be rewritten down the road once Scala’s macros reach their next stable state. There’s a video from ScalaDays 2013 describing the framework.

And finally, following up on the subject of the cake pattern, Daniel Spiewak’s keynote from NEScala, “The Bakery from the Black Lagoon,” is an excellent talk which made me think about the cake pattern in a new way (as more of a compiler-enforced module system than as a form of dependency injection). His implementation of the cake pattern is also interestingly different from most examples I’ve seen online – in particular, he mostly eschews self-types, with the exception of needing one for a virtual class.

More on Akka and dependency injection

Just as a quick follow-up to my previous post, I thought I’d note that the official Akka blog has published a post regarding Akka and dependency injection (kind of weirdly expressed as a mini white paper, as though the internet at large had issued an RFP for an actor-based concurrency system with dependency injection).

While, as I mentioned before, I should emphasize that I’m by no means an expert in Akka or actor best practices, I’m not convinced that this document addresses my particular concerns with the intersection of Akka and dependency injection. I’m still thinking about a larger post breaking down the approaches to this topic that I’ve seen online, but this one sort of falls into the “mock stuff outside of Akka” camp.

The document has two main points: firstly, if you have an existing dependency-injected service, you can pass along a factory which knows where to find it to the Props constructor of an actor, and there’s a way to attach a DI application context to an ActorSystem to support this, in what seems like a pretty convenient way. Secondly, if you need to expose an interface from actors to an existing system based on a DI framework, you can include an ActorSystem singleton in your DI object graph, and then expose a sort of regular-object facade over it which finds specific actors and returns either ActorRefs or futures resulting from sending ask messages to them.

That’s all well and good, but it seems more like a way to integrate between Akka and an existing synchronous DI-based system than anything that makes dependency injection useful or usable inside a purely Akka-based system. (In particular, the document’s unfortunate final section seems to be aimed squarely at recalcitrant middle managers who need to be convinced that a move to Akka will not result in a whole bunch of now-legacy code needing to be tossed out.) While I’m not incredibly interested in this topic myself, I thought Akka already had a talking point for this integration problem in the form of “typed actors“.

The bit that I still haven’t seen addressed is that if Akka likes actors to explicitly manage the lifecycles of other actors they supervise, there doesn’t seem to be any room for the inversion of control that is the hallmark of dependency injection frameworks in the first place. To put it in more concrete terms, if I’m running a partial integration test of my simple notification service and I want it to have a real database actor and mock REST web service actors, how can I tell the actor that supervises all the HTTP worker actors to create mock actors instead of live ones?

I have a half-formed idea of how this could work that involves having a sort of service locator / factory actor which is responsible for actor instantiation, but the idea in my head doesn’t particularly jibe with Akka’s supervision hierarchy, which as far as I can tell is coupled very tightly to actor instantiation.

Testing Akka: actors, dependency injection and mocks

I’ve been digging into what the expected way is to test my small Akka system, as described in my previous post on the subject. I think my problem partially arises from being unclear as to the proper mode of dependency injection in Akka. In other words, I don’t know what the proper way is for my Root actor to obtain a reference to its Database and HTTP sub-actors. Does it create them itself? Look them up from a service locator? And what if I need to inject mock actors into the system in some parts in order to test it?

Various bits of Akka documentation suggest different approaches to wiring actors together; for instance, this page in the official docs suggests either passing dependent actors as constructor arguments, creating them in a preStart() method, or passing them to their referring actors in a message and using become() to switch between initialization states. This example from the testkit docs takes the latter approach, but I can’t say I like the result:

class MyDoubleEcho extends Actor {
  var dest1: ActorRef = _
  var dest2: ActorRef = _
  def receive = {
    case (d1: ActorRef, d2: ActorRef) =>
      dest1 = d1
      dest2 = d2
    case x =>
      dest1 ! x
      dest2 ! x
  }
}
/* ... */

val probe1 = TestProbe()
val probe2 = TestProbe()
val actor = system.actorOf(Props[MyDoubleEcho])
actor ! (probe1.ref, probe2.ref)
actor ! "hello"
probe1.expectMsg(500 millis, "hello")
probe2.expectMsg(500 millis, "hello")

This does seem to work, but it seems to me that it pollutes the actor with a bunch of test-related code that probably doesn’t belong in production (by which I mean the receive pattern which takes the two dest parameters).

I have found an interesting take on this question in this presentation by Roland Kuhn, introducing akka-testkit from Scala Days 2012—the entire presentation is worth watching, but the part I’m interested in starts at around 22:05 or so.  After a not terribly helpful note about how if you have difficulty injecting mocks into your code, then there is probably something wrong with your design (there may be something to that, but it’s not all that helpful to hear when you’re looking for a solution for injection), Mr. Kuhn mentions a third option for users of the (then-new) Akka 2.0: actors can use actor path naming to look up their dependent actors; the test ActorSystem can then supplant the real implementations with mocks at the same locations.

Of course, all of this sort of assumes that you have a way of separating out actor creation and lifecycle control from dependency injection itself. A lot of the other Akka literature I’ve read seems to posit the integrated lifecycle management bits of Akka as a feature, right down to the “Let It Crash” maxim on the public Akka blog, and all of these features seem to be in direct opposition to the inversion of control notions that most dependency injection systems are founded on. In the last part of Mr. Kuhn’s talk above, he suggests breaking up actor models into somewhat discrete trees, which then use service locators or similar things to find one another; this might be something I can look into.

There was also a talk at this year’s Scala Days about integrating Spring and Akka, which might have some merit for this purpose, and I recently ran across this promising post which describes an approach to autowiring actors with Spring and Akka 2.2 (in Java). Overall, though, this doesn’t seem to be a problem with a clear solution.

Adventures in Akka

My current technical interest, mercurial as ever, is in Akka. My present employer is mostly a java shop, but they are open-minded and I have a notion to prototype out a rewrite of a simple system there into Akka and Scala.  The system is probably one of the simpler ones we have, known as the “notification service.” It periodically checks for new rows in a particular database table.  If it finds any, it fires off a JSON-formatted request to a REST web service, the “delivery service”; if it gets a successful response from this service it will mark the message as delivered in the database. There are a few wrinkles related to locking, and there are actually a few different web services involved, but that’s pretty much the basics. Something possessed me to make a diagram of the existing flow: 

Image

 

The purpose of this system is to deliver notifications to particular users, with the idea being that any subsystem which needs to send a notification to someone can put the right data in the database, where this system will pick up up and hand it off to an existing REST service which winds up doing most of the heavy lifting.  The existing service is implemented in Java and Spring, using Quartz as a cron job to kick off a polling method once every 30 minutes or so (we don’t need this service to run particularly swiftly).

It’s not really hard to see how this would translate into a message-based actor model in Scala.  You’d probably have one root actor coordinating things.  You’d have an actor talking to the database, maybe with a supervisor to restart it as needed, and you’d have another actor to handle the HTTP client calls.  Most likely the client actor would spawn off a new actor per individual row of data, and have each of these worker actors make a single HTTP request. On a success, the worker would send a message back to the database actor to update the database row as “completed”; on a failure the worker might just log an error and die.

A rough sketch of that might look like this (pardon my sub-par OmniGraffle skills):

Image

Note that the single-line arrows here represent the actor supervision hierarchy, not message-passing.  I’m also not positive that the “DB Worker” actor needs to exist, versus just having the “Database” actor do the work, but it simplifies things to do it this way and I suppose there might be more than one of those (more on this later).

I’m been struggling a bit to come up with a good way to represent message passing in a diagram, but I think I’ve got the gist of the design in this one:

Image

Everything is started by Tick messages which are sent to the root object every 5 seconds via Akka’s scheduler interface (this would be more like 15-30 minutes in production).  This causes the root actor to pass a PollDatabase message to the Database actor; the message includes a reference to the HTTP Client actor.  For each notification row the Database actor finds in the database, it sends a Notify message to the HttpClient actor.  This actor composes a MakeRequest message to one of a pool of Worker actors, including the data from the database and a reference to the Database actor.  The Worker performs the HTTP request; if it is successful it sends a RequestSucceeded message to the Database actor, which will ask a DB Worker thread to update the database to mark the relevant row as successfully delivered.  If the Worker gets an error, it sends a RequestFailed message to the HTTP Client actor, which at this point will just log the error and continue on.

I will have more to say about this, but this post is already decently long, so maybe I’ll leave this here so I can refer back to it later. There are a few things I’m struggling with:

  • Despite having read a lot of articles and blog posts on the subject, it’s not obvious to me what the correct way to instantiate and connect these actors is (constructor arguments, preStart() methods, dependency injection, etc).
  • Related to the above, it’s not clear to me how to test this system without mixing up test code and business logic.  In particular I’d like to replace the nodes in yellow above with mock objects and verify that the system still works properly.
  • I would like to have a reasonable interface to Oracle, without needing to include Spring or something in the project. The Typesafe, Inc solution is to use Slick, but I don’t have a burning desire to sell my co-workers on closed-source, commercial software in addition to a new language and framework. 

I’ll have more to say about all this in the days to come.

cbfix

It’s been a busy several weeks for me, mostly in my work-related universe.  I haven’t been completely disregarding the public sphere, though, and managed to get a CBR / CBZ utility script working, to wit cbfix.  Currently it only does one thing, but having the scaffolding around to open up CBR / CBZ files, mess with them, and replace them presents a lot of possibilities. In the meantime I still have a bunch of CoffeeScript and Scala stuff simmering on the back burner.

Well, that took a while

It’s been a minute since my last update, during which time I’ve learnt a great deal about node.js as well as the whole AMD / CommonJS mess mentioned in my last post. Without drowning in detail, here are a few things I’ve been thinking about.

I’ve made the switch to CoffeeScript for a good deal of my Javascript code, mostly because I was worried about wearing out the keys on my keyboard which comprise the phrase function(){}. I rather enjoy CoffeeScript’s cleaned-up syntax, at least for now, and I like how it hews to Javascript’s essential character while smoothing out its rougher bits. I’ve still got plenty of reservations about it, and it sometimes errs a bit too far on the loosey-goosey side of the syntax fence for my taste, but anything that will save me from the ongoing insults to programming-language aesthetics coming out of the Javascript world has my sincere gratitude. (The latest of these I’ve had to endure is the comma-first style, which is just one of the awful emergent results from the most-used data transport format of our day, the JSON spec, needing to fit into three quarters of a page of EBNF for some reason. Why complicate it with an optional comma at the end, or optional quotes around key values? But that’s a rant for another day.)

Meanwhile I also went down a fairly long rabbit hole in learning about electronics – this started with getting a Raspberry Pi and then got a little more low-level with some Ardiuno stuff, and by now I’ve got a bunch of nascent electronics projects in the offing, mostly concerning various robots. The Pi in particular has a robust Python infrastructure, and I have a pretty good imaging-related project in mind, but in the course of writing the software for it I decided that this would be a good opportunity to experiment with node.js development, and so I’ve since ported the stuff I’d written from Python to CoffeeScript and node.js.

More is surely to come about all that, and as usual I ought to have some code available on github before too long. In the course of all this I’ve also learned a great deal more about the server-side javascript ecosystem, and I think I should actually be ready to get the weighted-probability list library into much better shape.

Initial Jasmine impressions

Ok, I must confess that midway into my half-baked project of comparing Javascript testing frameworks, I’m getting a little bored with the entire project. Largely this is due to the fact that of the three major BDD-oriented frameworks I’ve seen so far, the syntax and semantics are roughly the same. Really the only major difference I’ve seen so far is that Mocha warns me about global namespace pollution, and the other frameworks do not.

As a point of reference, I was able to take the nascent set of tests I’d come up with for BusterJS and port them over virtually unchanged to Jasmine, with a few simple syntax changes relating to the setup / teardown methods. I’ll probably finish out the tests in one framework or the other and then move on to more interesting things, like figuring out the whole module mess that seems to be the current state of the art.

BusterJS vs Chai/Mocha

BusterJS, a node.js-oriented test framework, is notable for having a lot of ways to run tests, including one where it fires up an HTTP server that your browser connects to, as well as a way, seemingly, to attach several browsers to a single running test harness. Having opted against exploring the whole node.js / browser module dichotomy up until now, I skipped all of that stuff and went straight to the instructions for just running the thing in a browser. (I should note in passing that the test-runner Mocha, covered earlier, also has a lot of non-browser-focussed running modes which I didn’t cover, among which the most intriguing is the Nyan Runner).

Anyways, the browser stuff was quite easy to set up, and I returned my focus to writing tests in the BusterJS idiom.  As it turned out, there wasn’t all that much different about it.  BusterJS comes with a BDD-oriented test mode which functions almost exactly like Mocha’s does, and it also sports a BDD-flavored assertion style which is very similar, if a bit less tricksy in its syntax.

To illustrate, here’s a snippet of the Mocha/Chai tests:

describe('WeightedList', function() {

  describe('Constructor', function() {

    it('should succeed with no arguments', function() {
      expect(new WeightedList()).to.be.ok;
    });

    it('should throw errors on bad inputs', function() {
      var badConstructor = function() {
        return new WeightedList( {'wrong': 'field names'} );
      };
      expect(badConstructor).to.Throw(Error);
    });
  });
});

And here is their BusterJS equivalent:

var spec = describe("Weighted Lists", function() {

  describe('Constructor', function() {

    it('should succeed with no arguments', function() {
      expect(new WeightedList()).toBeDefined();
    });

    it('should throw errors on bad inputs', function() {
      var badConstructor = function() {
        return new WeightedList( {'wrong': 'field names'} );
      };
      expect(badConstructor).toThrow();
    });

  });
});

As you can see, the broad strokes are virtually identical. The main difference I’ve noticed is that Chai uses a little more trickery to get its assertions into a quasi-DSL style, with statements like result.should.have.length(1); and new WeightedList().should.be.ok; as opposed to BusterJS’s more conventional expect(result.length).toEqual(1); and expect(new WeightedList()).toBeDefined();.  While I appreciate the vigor with which the Chai folks were able to bend Javascript’s syntax to their will, I actually find the more conventional BusterJS syntax to be a little easier to read, since it looks basically like I expect Javascript to look.

Overall, these assertion frameworks both seem adequate for testing Javascript code. I’m really not thrilled about the way one tests code for exceptions in either language, where it’s necessary to pass in a function which is expected to throw an error, but I can’t really think of a better way to do it in Javascript. I’m going to continue porting the tests over to BusterJS, but I don’t know that I’ll have all that much more that’s interesting to say about it. If you’re interested in the nitty-gritty, the details can be found by contrasting the Chai and BusterJS assertion documentation.

I should probably also note in passing that the particular uses I’m putting the testing frameworks through are a little unusual. Most people writing tests won’t be testing a pure-javascript library with no user-interface whatsoever, and a lot of the appeal of the various frameworks seems to be the integration they offer with various other pieces of the Javascript ecosystem.

With that in mind, my next task is to finally figure out what the whole Javascript module system is all about, and likewise to figure out how I can package the library I’ve got into something usable in a node.js context. Meanwhile, based on a preliminary reading of the AMD vs CommonJS pedagogy in the blogosphere, my idea of just including the library code on a web page via a <script type="text/javascript" src="js-weighted-list.js"/> tag is laughably naive, and I’ll need to add a compilation step in there, because what is Javascript if not an Ada-style B&D language which benefits from the intense scrutiny of static program analysis?

In any event, adding a compilation step will probably give me a ready excuse to jump over to CoffeeScript.

Forging on with javascript testing frameworks

I’ve got what I consider to be a decent set of unit tests in Chai, though I’m certainly a good deal away from a complete set. (As an aside, the inherently non-deterministic nature of js-weighted-list presents something of a testing challenge in itself, totally apart from the particular framework in use; I’ll have more to say about this in future.)  My impressions of it aren’t radically different from how I felt about it last time – I’m still not crazy about the syntax, but it has offered me one or two pleasant surprises when it found scope bugs I had missed.

There’s another  aspect of the framework I’ve enjoyed as well. As some background, I’m mostly investigating BDD frameworks, but I have to confess that I don’t really see how BDD is all that different from unit-test based testing, apart from the terminology in use; frankly the difference between a test suite and a spec doesn’t seem all that large to me, and I’m dubious about the vague claims of readability to non-programmers that seem to haunt the rhetoric of some BDD sites I’ve seen. But anyways, according to the classic BDD (or TDD) approach, I’m going about things backwards: I’m writing a suite of tests to cover code which already exists instead of starting with the tests and then writing code to satisfy them.

Certainly I’m not the first programmer in the history of the world to approach testing this way, but I did enjoy adding in some undefined specifications for a few missing features for the library and seeing them show up as elements in the list:

    it('should increase the length of the list');
    it('should validate its weights');

Mocha testrunner with pending specs

Seeing the tasks come up with little blue dots in the test-runner output, and then change to red and then green as I implemented first the test code and then the implementing code in the library was a pretty satisfying process, and I can definitely see its appeal. This is a whole aspect of BDD I’ll probably miss out on in this comparison, since I’m not planning on reimplementing the library itself over and over again. In recognition of this, though, I’m hoping to think up some features and hold them in abeyance so I can try more specification-first type development.

At this point I have a decent feel for Chai, and I will move on to the next framework; I’m thinking I’ll try out BusterJS, since from a cursory examination of Jasmine it looks to be very similar to Chai. Another thing which is bubbling up towards the top of my priority list is trying out the coveraje library, which measures test coverage for javascript unit tests. However, as a node.js only project, this will require me to figure out how to get the same test suite and module system running on the command-line and the browser, and I’d just as soon keep my focus on the browser for the moment.

Follow

Get every new post delivered to your Inbox.