Simple webhooks with Clojure and Ring

When your data moves into cloud applications and collaboration is the rule, web hooks provide a way to extend what your cloud provider gives you.

Webhooks have been touted as the basis for what Anil Dash calls the "Pushbutton Web" which enables large-scale real-time collaboration between applications and humans. The concept behind webhooks is simple (as explained in detail by Jeff Lindsay in this video) and boils down to: it's easy to build a lightweight HTTP server nowadays, so why not let cloud-based applications interact with simple RESTful HTTP requests containing a payload of information of interest.

In this article, I will show a simple web hook processor written in Clojure using the Mark McGranaghan's Ring, a lightweight HTTP server framework based on the ideas in Rack. This web hook was written as part of the system for maintaining the clojure-contrib documentation
website.

Clojure-contrib is a collaborative project in which contributors check in new code on their own schedule. I built the "autodoc robot" to build a corresponding documentation site that reflects the latest state of clojure-contrib. The result of that process is seen here.

Using Webhooks to react to Commits on GitHub

Github provides commit hooks which allow me to be notified via http when someone updates clojure-contrib. So I wrote a simple processor in Clojure to receive these notifications and to update the documentation web site in response.

The issues:

1. Not all calls to our webhook are legitimate. Whenever you open an HTTP port, bad guys will come sniffing around. We need to filter those out.
2. In some cases, we want to ignore even legitimate requests. For example, the documentation for clojure-contrib is in the same project (on a different branch) as the source code. If we built documentation in response to all updates, we'd find ourselves in an infinite loop.
3. Processing must be serialized. It take about 3 minutes to build the documentation on my server. In that time, more updates can come in. If we processed them immediately, the two processes would stomp on each other since they work on the same file spaces.

To address these issues (especially issue 3), I used an architecture that separates the HTTP request/response handling from the actual hook processing. In between, there is a queue that enforces single threading:



We can combine ring with this queuing model using the fill-queue function from clojure.contrib.seq-utils. fill-queue allows us to take this kind of multi-threaded data passing behavior and recast it as operations on a clojure seq, greatly simplifying the look of our code.

The main loop of our application looks like this:

(defn hook-server
  "Build a simple webhook server on the specified port. Invokes ring to fill a blocking queue,
   whose elements are processed by handle-payload."
  [port]
  (doseq [payload (fill-queue (fn [fill]
                                  (ring.jetty/run {:port port} (partial app fill)))
                            :queue-size 10)]
  (handle-payload payload)))

So the main loop of our program is simply a doseq over the sequence generated by incoming requests. fill-queue will create a new thread on which to run the producer function. This is important to prevent the request handler from blocking while we are doing processing.

Unlike a normal doseq, this doseq will block when the jetty server is waiting for a new request.

The first argument to the fill-queue function is the function that will produce the data. It, in turn, is supplied with another function to call when it has data (the argument fill to the lambda). We'll discuss that more below.

As requests come in, they are passed to the handle-payload function for processing.

Handling incoming requests

Ring wants to pass all requests to an app function that handles the request and returns the appropriate HTTP response. All the data is formatted as Clojure maps and vectors at this point.

(defn app
"The function invoked by ring to process a single request, req. It does a check to make
sure that it's really a webhook request (post to the right address) and, if so, calls fill
with the parsed javascript parameters (this will queue up the request for later processing.
Then it returns the appropriate status and header info to be sent back to the client."
[fill req]
(print-date)
(pprint req myout)
(cl-format myout "~%")
(if (and (= (:scheme req) :http),
 (= (:request-method req) :post),
 (= (:query-string req) nil),
 (= (:content-type req) "application/x-www-form-urlencoded"),
 (= (:uri req) "/github-post"))
;; TODO: respond correctly to the client when an exception is thrown
(do (fill (json/decode-from-str (:payload (parse-params (slurp* (:body req)) #"&"))))
{:status  200
:headers {"Content-Type" "text/html"}})
{:status  404
:headers {"Content-Type" "text/html"}}))

Here we see that after some debug logging, we simply check that various fields of the request have the "correct" type to be a real request for our webhook. In practice this (especially the "secret" URI) filters out all the attack attempts.

A GitHub webhook call is sent as an HTTP POST message with a parameter called "payload" which contains a JSON-encoded description of the update that occurred (the project, branch, files, etc.).

Ring passes the body of the request wrapped in an InputStream. We process that with slurp* from clojure.contrib.duck-streams to get all the data, then parse the parameters to get the payload parameter. Finally, we use Dan Larkin's clojure-json JSON decoder to decode the JSON into Clojure data structures.

Ring lacks much of what would be standard in a web framework designed for building full web sites. Mostly, this is a good thing for applications like this one. One thing that I need here that Ring doesn't have is the ability to parse out the POST parameters. For this, I stole a parse-params function from the Compojure framework.

Once I have the payload, I pass it to the fill function I was given by fill-queue. Note how I can use partial to slide that parameter through Ring and into the app function without Ring having to know about it at all. This can be a powerful technique for easily combining unrelated libraries without doing handstands.

Processing the payloads

Incoming payloads are processed sequentially as they get to the head of the queue created by fill-queue. The processing function, handle-payload, is quite simple:

(defn handle-payload
"Called when a request is dequeued with the parsed json payload. Sees if the
request matches anything in the action-table and, if so, executes the associated shell
command."
[payload]
(pprint payload myout)
(when-let [params (match-table payload)]
(cl-format myout "~a~%" (apply sh (concat  (:cmd params) [:dir (:dir params)])))))

We look at the payload using match-table and, if match-table returns information, we use that information to execute a shell command. In the case of clojure-contrib, this command runs the documentation builder that will launch the ant job that builds the documentation and pushes it back to GitHub. The code that does this is online at contrib-autodoc.

The model for matching the payload structure is very simple:

(def action-table
 [[[:repository :url] "http://github.com/richhickey/clojure-contrib"
   [:ref] "refs/heads/master"
   {:cmd ["ant"] :dir "/home/tom/src/clj/contrib-autodoc"}]
  [[:repository :url] "http://github.com/tomfaulhaber/hook-test"
   [:ref] "refs/heads/master"
   {:cmd ["echo" "got here"] :dir "/home/tom/src/clj/contrib-autodoc"}]])


(defn match-elem
"Determine whether a given request, m, matches the action table element, elem."
[m elem]
(loop [elem elem]
(let [ks (first elem)
      rem (next elem)]
  (if (nil? rem)
    ks
    (when (= (apply get-in m ks) (first rem))
      (recur (next rem)))))))

(defn match-table
"Match a request, m, against the action-table"
[m]
(some #(match-elem m %) action-table))

Each payload is parsed from JSON into hierarchical maps. The matcher takes a sequence in which each element is a pair of a vector of keys and value. The vector is used as hierchical key into the map using get-in. This key is used on the payload structure and compared to the value supplied. If the retrieved value equals the value provided, then that element matches. If all the elements of the sequence match, the the pattern matches. This is a simplistic type of matching, but it is sufficient for our purposes.

Conclusion

Webhooks provide a powerful way to interconnect and extend cloud-based applications. Building webhook-based services in Clojure can be done with a simple and elegant composition of work that already exists.

One important thing to remember when using webhooks (from any language) is that the webhook mechanism provides no real security. Therefore the webhook request should be used as a hint about what work should be done rather than a reliable source of information. Hopefully, future work on Web infrastructure will address this shortcoming in an otherwise powerful model.

The source code presented here lives on GitHub at github-hook

Correction: Ring is based on Rack, not Sinatra. I have corrected the text above.

Coming to Clojure

I tried to avoid it. I really did. I had bookmarked Rich Hickey's video, "Clojure for Lisp Programmers" and it sat there on my delicious list like a fresh packet of heroin on a junkie's dresser.

I didn't want to get immersed in a new language. About a year ago, I'd gone through a bad Haskell addiction and I had nothing to show for it but a stack of monad tutorials and a bunch of puzzle code. And I was busy: I had a job, after all, and a family and a billion activities. Plus I was busy working on a side project using some real technologies: Ruby on Rails and Facebook. Technologies that let you build deployable applications without building big teams.

But one fateful day in November, I opened that video. An hour later, it was pretty much all over. From watching Rich present Clojure, it was clear that he got it. He got both why people love Lisp and why people who love Lisp end up using Python, Haskell, Ruby, or even Java.

When Rich talked about Clojure, he talked about a elegant and coherent set of design decisions. I didn't really believe it, but I had to try it. Once I tried it, I was hooked. There'd be a lot more of those little packets on my dresser.

What Clojure is and why to use it has been widely discussed. And no one presents it better than Rich himself. But I feel compelled to write down those things that make it the language I want to use.

Clojure is a Lisp, first of all. It is very different from Common Lisp or Scheme, but it is clearly part of the family. But where Common Lisp feels like a classic 70's muscle car, powerful perhaps but filled with compromises from a bygone era, Clojure feels like what we expect from a high-end sport sedan in the 21st century: all fit and finish, every button in the right place, and zip when you need it.

Clojure has abandoned the primacy of the cons cell, which has dominated Lisps from the beginning, and built a model that gets leverage from sequences, but supports multiple first-class data types (lists, vectors, and maps) that can be entered as primitive data directly (as you'd expect in, say, modern languages like Python or Ruby).

Clojure makes functional natural. Lisp has always made functional programming seem like the right strategy, with first-order functions, map, apply and so on. Clojure takes this an important step forward with persistent data structures built right into the language. With these structures, the programmer doesn't feel like he needs to break the functional paradigm for efficiency reasons when, for example, updating a vector. The cost of creating a copy of a vector with a new value is a slot is so low as to seem free. (The functional theory is that the incremental cost of this copy is offset by the reduced bookkeeping, etc., that's required.)

Clojure is built on the JVM. In truth, this part has it's pros and cons. Being on the JVM means that Clojure is able to leverage wealth of libraries, gets excellent compiler technologies for free, and is instantly portable to most platforms. However, it also means programmers need to work with libraries based on the OO paradigm rather than the functional paradigm, some platforms end up being off-limits (can you say iPhone?), and the dreaded Java startup environment which comes with classpath hell and other features that mean it's hard to use Clojure as a scripting language the same way you'd use Python. The one factor that definitely puts JVM hosting into the plus column is the design of the Java interop functionality. This makes leveraging Java libraries incredibly easy and natural.

Clojure has community. One of the things that everyone who comes to Clojure notices is the feeling of community that exists among the Clojure programmers. Interaction in the various forums is helpful and respectful. People genuinely seem to appreciate each other. This is not an accident. Rich has been very careful is his own statements not to be competitive or incendiary (or to position Clojure above everything else) and he has followed that up with some mentoring (and occasional moderation) of the community. For me, this community spirit drives my desire to contribute to Clojure. I know this is a huge factor for other contributors as well.

Clojure has many other features (software transactional memory, agents, ...) which are nicely implemented and give you a lot of power, but these were the ones I wanted to highlight about my decision to make Clojure my first choice when I can choose a programming language.

I know Clojure is not the right thing for everybody (Lisp may not be your thing, the JVM may be a deal breaker, or whatever). We live during a period in which it's easy to choose from among many great options when it comes to programming environments.

But for myself and many others who have found Clojure, we feel like we've found a special sweet spot in the programming world.

Welcome to Tom's Blog

For a long time, I've wanted a place to record random thoughts and observations that I have run across during my explorations of the world of technology.

For anyone reading this who doesn't know me, my background is systems programming (operating systems, network stacks, etc.), management, and consulting. Until recently, I was happily living in that particular bubble.

Over the last two years, my eyes have been opened in a number of ways:

• I have become reacquainted with the world of Lisp and the power that comes from a tight, expressive language.
• I found Anarchaia and started thinking theoretical thoughts again.
• These two led me inexorably to Haskell and the power and beauty of pure functional programming.
• I became interested in web-based platforms, particularly the Facebook platform, as a way to quickly build and deploy applications in the cloud.
• I started to use Rails as a rapid prototyping tool to quickly develop some application concepts.

The initial idea behind this blog (and I'm sure it will change) is that I'll write about what I learn while doing this, mostly at the micro-level concerning development and about the things I'm working on or even rolling out.

I hope you find something interesting here!