I got a question from a Play user about implementing call/response WebSockets in Play Framework. This is not something that comes up that often, since it means using WebSockets to do basically what AJAX does for you, so what's the point? But here are some use cases that I've thought of:

• You have some transformation of a stream that can only be done on the server side. For example, perhaps the transformation requires some heavy database work, or is too computationally expensive for a mobile client, or perhaps you want to encrypt the stream with a key that is private to the server.
• You are already processing a stream of events from the server using WebSockets, and the responses to the calls are just more events in this stream, so you'd like to share the same transport mechanism for these events.
• Your application is particularly chatty, and you don't want the overhead of the HTTP protocol on each call/response.

There are possibly more use cases - WebSockets is quite a new technology and as an industry we haven't really settled on what it's best use cases are.

A simple echo implementation

A Play WebSocket is implemented by providing an iteratee that consumes messages from the client, and an enumerator that produces messages for the client. If we simply wanted to echo every message that the client sent us, then we would want to return an iteratee whose input becomes the output of the enumerator that we return. Play doesn't come with anything out of the box to do this, but we will probably add something out of the box that does this in a future release. For now, I'm going to write a method called joined, that returns a joined iteratee/enumerator pair:

/**
 * Create a joined iteratee enumerator pair.
 *
 * When the enumerator is applied to an iteratee, the iteratee subsequently consumes whatever the iteratee in the pair
 * is applied to.  Consequently the enumerator is "one shot", applying it to subsequent iteratees will throw an
 * exception.
 */
def joined[A]: (Iteratee[A, Unit], Enumerator[A]) = {
  val promisedIteratee = Promise[Iteratee[A, Unit]]()
  val enumerator = new Enumerator[A] {
    def apply[B](i: Iteratee[A, B]) = {
      val doneIteratee = Promise[Iteratee[A, B]]()

      // Equivalent to map, but allows us to handle failures
      def wrap(delegate: Iteratee[A, B]): Iteratee[A, B] = new Iteratee[A, B] {
        def fold[C](folder: (Step[A, B]) => Future[C]) = {
          val toReturn = delegate.fold {
            case done @ Step.Done(a, in) => {
              doneIteratee.success(done.it)
              folder(done)
            }
            case Step.Cont(k) => {
              folder(Step.Cont(k.andThen(wrap)))
            }
            case err => folder(err)
          }
          toReturn.onFailure {
            case e => doneIteratee.failure(e)
          }
          toReturn
        }
      }

      if (promisedIteratee.trySuccess(wrap(i).map(_ => ()))) {
        doneIteratee.future
      } else {
        throw new IllegalStateException("Joined enumerator may only be applied once")
      }
    }
  }
  (Iteratee.flatten(promisedIteratee.future), enumerator)
}

This code might be a little scary if you don't understand iteratees, but as I said we will probably add this to Play itself in future. The rest of the code in this blog post will be simple.

Now that we have our joined iteratee/enumerator, let's implement an echo WebSocket. For the rest of this post we'll be assuming that all our WebSockets are sending/receiving JSON messages.

def echo = WebSocket.using[JsValue] { req =>
  joined[JsValue]
}

So now we have an echo call/response WebSocket. But this is not very useful, we want to do something with the incoming messages, and producing new outgoing messages as responses.

Processing messages

So now that we've expressed our call/response in terms of a joined iteratee/enumerator, how can we transform the call messages to be different response messages? The answer is enumeratees. Enumeratees can be used to transform iteratees and enumerators. We return both an enumerator and an iteratee, so which one do we transform? The answer is it doesn't matter, I'm going to use it to transform the iteratee. The enumeratee that we're going to use is the map enumeratee:

def process = WebSocket.using[JsValue] { req =>
  val (iter, enum) = joined[JsValue]

  (Enumeratee.map[JsValue] { json =>
    Json.obj(
      "status" -> "received",
      "msg" -> json
    )
  } &> iter, enum)
}

Enumeratees are one of the most powerful features of iteratees for end users. You could use any enumeratee here, but let's look at some examples of other common use cases.

What if we don't want to return a response to every message? There are numerous ways to do this, but the simplest is to use the collect enumeratee, which takes a partial function:

def process = WebSocket.using[JsValue] { req =>
  val (iter, enum) = joined[JsValue]

  (Enumeratee.collect[JsValue] { 
    case json if (json \ "foo").asOpt[JsValue].isDefined =>
      Json.obj(
        "status" -> "received",
        "msg" -> json
      )
  } &> iter, enum)
}

Perhaps we want to produce many responses for a single input. The mapConcat enumeratee can be used in this case, with our map function returning a sequence of JsValue messages to return:

def process = WebSocket.using[JsValue] { req =>
  val (iter, enum) = joined[JsValue]

  (Enumeratee.mapConcat[JsValue] { json =>
    Seq(
      Json.obj(
        "status" -> "received",
        "msg" -> json
      ),
      Json.obj("foo" -> "bar")
    )
  } &> iter, enum)
}

What if we want to do some blocking operations? In Play 2.2, this will be able to be done simply by providing an execution context suitable for blocking calls to whichever enumeratee you decide to use, but Play 2.1 does not yet support this, so we have to dispatch the callback to another execution context ourselves. This can be done using the mapM enumeratee:

val ec: ExecutionContext = ...

def process = WebSocket.using[JsValue] { req =>
  val (iter, enum) = joined[JsValue]

  (Enumeratee.mapM[JsValue] { json =>
    Future {
      // Some expensive computation, eg a database call, that returns JsValue
    }(ec)
  } &> iter, enum)
}

Pushing from an external enumerator

You may want to combine your call/response messages with messages from some other enumerator that spontaneously pushes messages to the client, for example a broadcasting enumerator for all clients. This can be done by interleaving your joined enumerator with the external enumerator:

val globalEvents: Enumerator[JsValue] = ...

def process = WebSocket.using[JsValue] { req =>
  val (iter, enum) = joined[JsValue]

  (Enumeratee.map[JsValue] { json =>
    ...
  } &> iter, Enumerator.interleave(enum, globalEvents))
}

Conclusion

Using WebSockets in a call response style may be something that your application needs. If so, using enumeratees to map the stream of messages coming in to messages going out is the most natural and idiomatic way of doing this in Play. It allows you to call on the large number of composable enumeratees that Play provides out of the box, and makes your code simple and easy to reason about.

Nothing is wrong with Monads. What is wrong is how people communicate them.

There are people out there that learn abstract concepts first, and then work out how to apply them to solving their every day problems. Then there are people that learn how to solve their every day problems, and then they can understand how these solutions can be generalised into abstract concepts for solving problems. I am firmly in the second camp, explain an abstract concept to me that I've never come across before, and it'll go in one ear and out the other. Explain a solution to a problem that I have at hand, and I'll understand. Tell me that the solution can be generalised into an abstract concept, and you won't need to tell me anything more about that abstract concept, I'll understand it immediately.

I envy the people that can understand the abstract concepts first before applying them to real problems. I really wish I had that ability. But I don't. And I don't think I'm alone. I suspect many, maybe even most developers learn the same way that I do.

So this is the problem with monads, the problem is too abstract. I tried to read up about monads quite a number of times, and I always got stuck. I even read about "burritos" and "semicolons", these simplifications just made me more confused. None of the stuff I was reading made any sense to me.

Then one day I started using Play 2. Play 2 is an asynchronous web framework. It heavily uses futures as a means to implement asynchronous code. I had been writing some asynchronous code for a while before I had started using Play 2, and I knew how painful it could be. When I saw futures in Play, it solved a problem that I had perfectly. The ability to return a result that could then be mapped/flatMapped was exactly what I needed.

I was so excited about this new discovery that I prepared and gave a presentation on Play's futures at the Berlin Play User Group. I put forward a real world problem, and explained how Future, map and flatMap could solve it. I got great feedback from everyone at the user group, it really helped a lot of people to understand asynchronous programming and futures. At that point, I still had no idea what a monad was, yet somehow I was teaching others about monads.

A while later I was at the Berlin Play User Group again, and a conversation about monads came up. I admitted to everyone there that I didn't know what a monad was. Someone looked at me funny, and said "it's basically just flatMap". Suddenly it all made sense to me. I went back and read the same papers and blog posts that I had tried to read before, and they all made perfect sense, I now had no problems understanding monads.

So why didn't I understand monads before that? What was missing was an explanation that was given within the context of a concrete problem that I could relate to. Why was it missing? Because monads are simply too abstract. Each specific different monad solves a different concrete problem. The option monad solves a problem involving values that might not exist. The future monad solves a problem involving values that don't exist yet. The sequence monad solves a problem involving a sequence of values. And so on. To explain the general concept of monads, you need to abstract it away from any of the specific concrete problems that each monad solves, into a general abstract problem. And so if, to explain monads, you start with monads, you are at an abstraction level that is twice removed from real world problems. That means if you try to explain them starting with the abstract concepts to someone such as myself, at best you'll just make my head explode.

This is why I say, if you want to explain monads to a developer who is anything like me, don't start with monads. Don't even use the word monad, the moment you do that you are throwing down a concrete wall in front of them 30 foot high that they will not be able to climb. Start with a real world problem - asynchronous programming with futures is an excellent one, and talk about how to solve that. Then maybe later talk about monads. But by that stage, does it matter if they know what a monad is? They've learnt the concepts already, probably well enough to give a presentation at a user group about them.

I'm an IntelliJ user, so I have the SBT idea plugin permanently in my global SBT plugins. This is fine, until I start working across versions of SBT, and particularly with milestone builds of unreleased SBT versions, because in that case, the version of the idea plugin that I usually use with other versions of SBT may not be available. Hence I need to have per SBT version global plugins. I think this might be easier to do in future, but for now, here's a way to do it:

libraryDependencies <++= (sbtBinaryVersion in update, scalaBinaryVersion in update) { (sbtV, scalaV) =>
  sbtV match {
    case sbt013 if sbt013.startsWith("0.13.") => Seq(
      Defaults.sbtPluginExtra("com.github.mpeltonen" % "sbt-idea" % "1.5.0-SNAPSHOT", sbtV, scalaV)
    )
    case _ => Seq(
      Defaults.sbtPluginExtra("com.github.mpeltonen" % "sbt-idea" % "1.4.0", sbtV, scalaV)
    )
  }
}

We frequently get questions about how to meet all sorts of different routing needs in Play Framework. While the built in router is enough for most users, sometimes you may encounter use cases where it's not enough. Or, maybe you want a more convenient way to implement some routing pattern. Whatever it is, Play will allow you to do pretty much anything. This blog post is going to describe some common use cases.

Hooking into Plays routing mechanism

If for some reason you don't like Plays router, or if you want to use a modified router, then Play allows you to do this easily. Global.onRouteRequest is the method that is invoked to do routing. By default, this delegates to the Play router, but you can override it to do whatever you want. For example:

override def onRouteRequest(req: RequestHeader): Option[Handler] = {
  (req.method, req.path) match {
    case ("GET", "/") => Some(controllers.Application.index)
    case ("POST", "/submit") => Some(controllers.Application.submit)
    case _ => None
  }
}

As you can see, I've practically implemented my own little routing DSL here. I could also delegate back to the default router by invoking super.onRouteRequest(req).

An interesting thing that could also be done is to delegate to different routers based on something in the request. A play router compiles to an instance of Router.Routes, and it will be an object called Routes itself. By default, any file with the .routes extension in the conf directory will by compiled, and will go in the package with the same name as the filename, minus the .routes. So if I had two routers, foo.routes and bar.routes, I could implemented a crude form of virtual hosting like so:

override def onRouteRequest(req: RequestHeader): Option[Handler] = {
  if (req.host == "foo.example.com") {
    foo.Routes.routes.lift(req)
  } else if (req.host == "bar.example.com") {
    bar.Routes.routes.lift(req)
  } else {
    super.onRouteRequest(req)
  }
}

So here are some use cases that overriding onRouteRequest may be useful for:

• Modifying the request in some way before routing is done
• Plugging in a completely different router (eg, jaxrs)
• Delegating to different routes files based on some aspect of the request

Implementing a custom router

We saw in the previous example how to use Plays Router.Routes interface, another option is to implement it. Now, there's no real reason to implement it if you're just going to delegate to it directly from onRouteRequest. However, by implementing this interface, you can include it in another routes file, using the sub routes include syntax, which in case you haven't come across this before, typically looks like this:

->    /foo         foo.Routes

Now something that people often criticise Play for is that it doesn't support rails style resource routing, where a convention is used to route commonly needed REST endpoints to particular methods on a controller. Although Play comes with nothing out of the box that does this, it is not hard to implement this today for your project, Play 2.1 has everything you need to support it, by using the routes includes syntax, and implementing your own router. And I have some good news too, we will be introducing a feature like this into Play soon. But until then, and also if you have your own custom conventions that you want to implement, you will probably find these instructions very helpful.

So let's start off with an interface that our controllers can implement:

trait ResourceController[T] extends Controller {
  def index: EssentialAction
  def newScreen: EssentialAction
  def create: EssentialAction
  def show(id: T): EssentialAction
  def edit(id: T): EssentialAction
  def update(id: T): EssentialAction
  def destroy(id: T): EssentialAction
}

I could provide default implementations that return not implemented, but then implementing it would require using override keywords. I think it's a matter of preference here.

Now I'm going to write a router. The router interface looks like this:

trait Routes {
  def routes: PartialFunction[RequestHeader, Handler]
  def documentation: Seq[(String, String, String)]
  def setPrefix(prefix: String)
  def prefix: String
}

The routes method is pretty self explanatory, it is the function that looks up the handler for a request. documentation is used to document the router, it is not mandatory, but it used by at least one REST API documenting tool to discover what routes are available and what they look like. For brevity in this post, we won't worry about implementing it. The prefix and setPrefix methods are used by Play to inject the path of the router. In the routes includes syntax that I showed above, you could see that we declared the router to be on the path /foo. This path is injected using this mechanism.

So we'll write an abstract class that implements the routes interface and the ResourceController interface:

abstract class ResourceRouter[T](implicit idBindable: PathBindable[T]) 
    extends Router.Routes with ResourceController[T] {
  private var path: String = ""
  def setPrefix(prefix: String) {
    path = prefix
  }
  def prefix = path
  def documentation = Nil
  def routes = ...
}

I've given it a PathBindable, this is so that we have a way to convert the id from a String extracted from the path to the type accepted by the methods. PathBindable is the same interface that's used under the covers when in a normal routes file to convert types.

Now for the implementation of routes. First I'm going to create some regular expressions for matching the different paths:

  private val MaybeSlash = "/?".r
  private val NewScreen = "/new/?".r
  private val Id = "/([^/]+)/?".r
  private val Edit = "/([^/]+)/edit/?".r

I'm also going to create a helper function for the routes that require the id to be bound:

def withId(id: String, action: T => EssentialAction) = 
  idBindable.bind("id", id).fold(badRequest, action)

badRequest is actually a method on Router.Routes that takes the error message and turns it into an action that returns that as a result. Now I'm ready to implement the partial function:

def routes = new AbstractPartialFunction[RequestHeader, Handler] {
  override def applyOrElse[A <: RequestHeader, B >: Handler](rh: A, default: A => B) = {
    if (rh.path.startsWith(path)) {
      (rh.method, rh.path.drop(path.length)) match {
        case ("GET", MaybeSlash()) => index
        case ("GET", NewScreen()) => newScreen
        case ("POST", MaybeSlash()) => create
        case ("GET", Id(id)) => withId(id, show)
        case ("GET", Edit(id)) => withId(id, edit)
        case ("PUT", Id(id)) => withId(id, update)
        case ("DELETE", Id(id)) => withId(id, destroy)
        case _ => default(rh)
      }
    } else {
      default(rh)
    }
  }

  def isDefinedAt(rh: RequestHeader) = ...
}

I've implemented AbstractPartialFunction, and the main method to implement then is applyOrElse. The match statement doesn't look much unlike the mini DSL I showed in the first code sample. I'm using regular expressions as extractor objects to extract the ids out of the path. Note that I haven't shown the implementation of isDefinedAt. Play actually won't call this, but it's good to implement it anyway, it's basically the same implementation as applyOrElse, except instead of invoking the corresponding methods, it returns true, or for when nothing matches, it returns false.

And now we're done. So what does using this look like? My controller looks like this:

package controllers

object MyResource extends ResourceRouter[Long] {
  def index = Action {...}
  def create(id: Long) = Action {...}
  ...
  def custom(id: Long) = Action {...}
}

And in my routes file I have this:

->     /myresource              controllers.MyResource
POST   /myresource/:id/custom   controllers.MyResource.custom(id: Long)

You can see I've also shown an example of adding a custom action to the controller, obviously the standard crud actions are not going to be enough, and the nice thing about this is that you can add as many arbitrary routes as you want.

But what if we want to have a managed controller, that is, one whose instantiation is managed by a DI framework? Well let's created another router that does this:

class ManagedResourceRouter[T, R >: ResourceController[T]]
    (implicit idBindable: PathBindable[T], ct: ClassTag[R]) 
    extends ResourceRouter[T] {

  private def invoke(action: R => EssentialAction) = {
    Play.maybeApplication.map { app =>
      action(app.global.getControllerInstance(ct.runtimeClass.asInstanceOf[Class[R]]))
    } getOrElse {
      Action(Results.InternalServerError("No application"))
    }
  }

  def index = invoke(_.index)
  def newScreen = invoke(_.newScreen)
  def create = invoke(_.create)
  def show(id: T) = invoke(_.show(id))
  def edit(id: T) = invoke(_.edit(id))
  def update(id: T) = invoke(_.update(id))
  def destroy(id: T) = invoke(_.destroy(id))
}

This uses the same Global.getControllerInstance method that managed controllers in the regular router use. Now to use this is very simple:

package controllers

class MyResource(dbService: DbService) extends ResourceController[Long] {
  def index = Action {...}
  def create(id: Long) = Action {...}
  ...
  def custom(id: Long) = Action {...}
}
object MyResource extends ManagedResourceRouter[Long, MyResource]

And in the routes file:

->     /myresource              controllers.MyResource
POST   /myresource/:id/custom   @controllers.MyResource.custom(id: Long)

The final thing we need to consider is reverse routing and the Javascript router. Again this is very simple, but I'm not going to go into any details here. Instead, you can check out the final product, which has a few more features, here.

I learnt functional programming, but then I decided I liked imperative programming better so I switched back.

— Nobody, ever

Moving from imperative programming to functional programming is a very common thing to do today. Blog posts on the internet abound with testimonies about it. Everything I've read and every person I've talked to, including myself, has the same story. Once they started functional programming, there was no looking back. They loved it, and in the early days, even the small amount they learnt gave them a thirst to learn more.

To me it seems clear, going from imperative programming to functional programming is a one way street under heavy traffic. It's a diode with a million volts sitting across it. It's a check valve on a mains water pipe. Not only can you not go back, but it comes with an irresistible desire to explore and learn more that pushes you further into functional programming.

Java 8 Lambdas

With Java 8 lambdas on the way, this poses an interesting turning point for one of the largest groups of developers on the planet. Lambdas in themselves don't necessarily equate to functional programming. But they do enable it. And as a developer here starts dabbling in functional programming, a library maintainer there, we'll start to see some new things in Java source code. Methods that previously might have returned null will start returning Optional. Libraries that do IO, for example HTTP client libraries, will start returning CompletableFuture. More and more functional concepts will start creeping into Java interfaces, there will be methods called fold, map, reduce, collect. And so will start the one way street of the Java masses moving from imperative programming to functional programming.

But will Java satisfy their thirst? Looking at the Lambda spec, I suspect not. I see an essence of genius in the Lambda spec, it makes many, many existing libraries instantly usable, with no changes, with Lambdas. The reason for this is that a Lambda is just syntactic sugar for implementing a single-abstract-method (SAM) interface. In Java you'll find SAM's everywhere, from Runnable and Callable in the concurrency package, to ActionListener in Swing, to Function and Supplier in Guava, and the list goes on. All of these libraries are Lambda ready today.

However, this also presents a problem. Functional programming gets interesting when you start composing things. The ability to pass functions around and compose them together gives a lot of power - but the Java 8 Lambdas are not composable. Java 8 does provide a Future SAM, but so does Guava, and many other libraries. To compose these together, you'd need all permutations of composition methods. Two SAM's of the same type aren't even very simple to compose, at least, not in a traditional Java way, since you can't add any methods to the SAM, such as a map or transform method, to do the composing.

So, without the ability to do one of the most basic functional concepts, composing functions, can Java ever become a functional language? Maybe there are some creative ways to solve this that I haven't thought of. And maybe it doesn't need to, I don't think the designers of Java 8 Lambdas had any intention of making Java a functional language, so you can't call this a fault of the Lambda spec. But the problem is the developers, having got a taste for functional programming, as I pointed out earlier, are going to want more, and going to want more fast. Even if Java could become a functional language, I don't think it will keep up with the movement of Java developers to functional programming.

So I'm going to make a prediction. Java 8 Lambdas will be eagerly adopted. So eagerly that Java itself will be left behind, and the majority of Java developers will move on to a language that meets their needs as eager budding new functional programmers.

Which language?

Before I speculate on which language Java developers will move to, let me first just qualify that I am both biased and ignorant. I work for Typesafe, and so am obviously biased towards Scala. And apart from playing with Haskell and ML at university, I have never used any other functional language in anger. So take my words with a grain of salt, and if you disagree, write your own blog post about it.

Scala as a transitionary language

So firstly, I think Scala makes a great transitionary language for imperative programmers to switch to functional programming. Having got a taste for functional programming with Java 8 Lambdas, a Java developer will find themselves very comfortable in Scala. They can still do everything in the same way they used to, they have vars and mutable collections, they have all the standard Java libraries at their finger tips. And of course, they can start deepening their knowledge in functional programming. So Scala provides a smooth transition from imperative programming to functional programming, you can adopt functional programming as quickly or as slowly as you like.

Scala as the destination language

Having transitioned to functional programming, will developers stay at Scala, or will they move on, like they moved on from Java, in search of a more pure language? My opinion here is no. Broadly speaking, I see two camps in the functional programming community. The first camp sees functional programming as a set of laws that must be followed. For this camp, Scala has a lot of things that are not necessary and/or dangerous, and they would probably not see Scala as the end destination.

The second camp sees functional programming as a powerful tool that should be widely exploited, but not a set of laws that must be followed. This is where I stand, and Scala fills the needs of this camp very well. Functional programming has first class support in Scala, but you can always fall back to imperative when it makes sense. I suspect that the majority of the Java community would be inclined to join this camp, otherwise, they would already be shunning Java and writing Haskell.

So I think Java 8 Lambdas are going to be very good for Scala, in that they give Java developers a taste for what Scala will do for them, and thus funnel the masses into Scala development.