Helder Ribeiro

Queue Size

Number of jobs that haven't been run yet or that failed and are going to be retried

Graphs make people happy. No denying that. So here’s a treat for you. Assuming you’re using Munin to monitor all sorts of stuff in your servers (Munin’s web interface is ugly but it’s really cool!), and using delayed_job as a queue, take this file:
http://github.com/helder/delayed_job/blob/423dc31b9ed6f2bee6e4c780e85d7d9a546e0a60/contrib/delayed_jobs_queue_size.rb

just drop it (or link to it) inside your /etc/munin/plugins/ on the servers that run jobs and that’s it! It’ll break!

Not what you were expecting huh? Ok, you have to do a little more work. First, you have to tell the plugin how to connect to your database. The easiest way, if you’re running Rails, is to set the environment variable DATABASE_YML to the path where that file is. Make sure the user who runs the plugin has access to that file. By default it’s the user munin, but you can specify a different one.

If you have some different setup, you can edit the plugin and pass a hash with username, password, etc. to Grapher.new. Don’t be scared! The code is really simple, take a look!

Also, if you’re running vanilla delayed_job (that is, from tobi’s repo), this plugin uses a table column that you don’t have, finished_at (we’ll come back to this column later). Just remove that extra AND finished_at IS NULL. Now it works! That’s how big your queue is right there, on that graph!

So why would you want that extra column again? Here’s something to wet your appetite:

Average Run Time

How long it's taking your jobs to run

Now, you can’t get that with vanilla delayed_job. Why not? Because it deletes finished jobs, so there’s no way of telling how long they took to run*.

How do you solve that?

Step 1: Run this migration:

Step 2: Use my branch of delayed_job (hey, gotta sell my fish). It adds the appropriate behavior related to those extra columns.

Step 3: Add the line Delayed::Job.destroy_successful_jobs = false to your environment.rb or, better yet, to some initializer.

Now your jobs stay there after they complete successfully so you can poke at them and gather statistics. Specifically, they keep the time when they first started, when they last started (these are different if there was more than 1 attempt) and when they finished successfully.

With that in hands, you can now add this one other plugin:
http://github.com/helder/delayed_job/blob/423dc31b9ed6f2bee6e4c780e85d7d9a546e0a60/contrib/delayed_jobs_run_time.rb

Follow the same instructions as with the first one, and you have the graph above with each point representing the average running time of jobs that got finished in the last 5 minutes. Sweet huh?

There’s more!

It’s a graphing spree! Know what else you can do with those statistics you’re gathering? From the README:

“This is useful for gathering statistics like how long a job took:
* to be picked up by the first worker: first_started_at – created_at
* in one successful run (the last one that didn’t fail): finished_at – last_started_at
* in failed retries: last_started_at – first_started_at”

Now these you’ll have to do yourself. I haven’t written them. But if you look at my plugins, they’re really simple, and it should be very easy for you to change them to output these other graphs. If you get around to doing that, please be nice and put them in that contrib folder. There’s certainly people who’ll find use for them!

Wait, so I’ll have these finished jobs lying around FOREVER?

If you don’t do anything, yep. But you can clean up with these handy rake tasks:
jobs:clear:all
jobs:clear:finished
jobs:clear:failed
Put them in a cron job somewhere and you’ll be fine.

* This is not entirely true. Delayed_job logs how long each job took on its last run. So you could parse the log and graph it. I don’t like parsing logs, and if you have more than one worker you’ll have to aggregate the different logs somehow, it’s just a big mess. Also, it only gives you how long it took to run, doesn’t give you when it started and finished, so you can’t graph that other stuff I mentioned at the end. My plugin, for example, has slightly different semantics, and tells you how long the job took IN TOTAL to run, counting retries. You can’t do that by parsing the log. I could’ve added the extra data to the log itself but, as I said, I hate parsing logs, and delayed_job already let you keep failed jobs around, so I thought it wouldn’t be too inconsistent to leave successful ones there too.

So I jumped on the bandwagon and joined all the cool kids in scorning relational databases and playing with CouchDB (for a sort of long and excellent wrap up on this versus that, read this). I installed it, read through a lot of docs, thought I understood it pretty well and immediately started searching the tubes for what Ruby framework/library/gizmo would best allow me to get kickstarted with using it on a new project.

Turns out that was a bit premature, as my brain couldn’t really handle trying to model the domain of my intended application onto this totally new way of thinking so abruptly. It’s like when you’re a native Portuguese speaker and you’re drunk, trying to make yourself pass as speaking Spanish and you keep spilling out the German you’ve been learning for the past two years. I’m sure everyone can relate to that.

Time to take a step back.

The best way I found to get more familiar with this new type of database was to get rid of all the mental cruft I had around it. So I forgot about my app, its data model and the web framework and went on to play with the database alone.

I had seen this blog post about how to store hierarchical data in CouchDB and decided to play with the example data and views the guy provided (big thanks to Paul Bonser!). [Note: If you can follow that, you don't need to read this, as it's basically an expanded version of one of his examples.]

This is the data:

{
  "docs": [
    {"_id":"Food", "path":["Food"]},
    {"_id":"Fruit", "path":["Food","Fruit"]},
    {"_id":"Red", "path":["Food","Fruit","Red"]},
    {"_id":"Cherry", "path":["Food","Fruit","Red","Cherry"]},
    {"_id":"Tomato", "path":["Food","Fruit","Red","Tomato"]},
    {"_id":"Yellow", "path":["Food","Fruit","Yellow"]},
    {"_id":"Banana", "path":["Food","Fruit","Yellow","Banana"]},
    {"_id":"Meat", "path":["Food","Meat"]},
    {"_id":"Beef", "path":["Food","Meat","Beef"]},
    {"_id":"Pork", "path":["Food","Meat","Pork"]}
  ]
}

Which corresponds to this tree:

To create a database and import this data, save that snippet as a file somewhere (I’ll use /tmp/data.json). CouchDB talks to the world in HTTP. We’re gonna use curl for that so you really see what’s going on. Web browsers are for wimps.

Usually CouchDB runs locally on 127.0.0.1, port 5984. To create a new database all you need to do is PUT to that address with the name you want your DB to have in the URL and no payload. We’ll save that URL in a variable because we’re gonna use it a lot.

DB="http://127.0.0.1:5984/hierarchical_data"
curl -v -X PUT $DB

Here -v means verbose, and -X lets you choose the HTTP method.

We have our database, now we import the data using the bulk document API. We specify the payload (data) with -d and feed it as a string from the file by prefixing its path with a @ (that’s one bash trick I didn’t know about!):

curl -v -d @/tmp/data.json -X POST $DB/_bulk_docs

And this is the view I was interested in (it lists all descendants of a node, including itself):

{
  "language": "javascript",
  "views": {
    "descendants": {
      "map": "
        function(doc) {
            for (var i in doc.path) {
                emit(doc.path[i], doc)
            }
        }"
    }
  }
}

This thing about views going through all objects in your database took a little time to sink in with me. Initially I thought the query took place in the view, that I would somehow pass the node from which I wanted the descendants as the doc argument to that function. That’s not how it works. The query actually takes place in the view parameters, and the view function itself only flattens everything out into a convenient array so you can query it better.

This view I just mentioned, for example, doesn’t actually give you the elements in a sub-tree. It goes through each object (document) in the database and adds it to the array of results once for each of its ancestors.

To see if for yourself, save it in a file somewhere (/tmp/view.json in my case) and add it to the database. We do that by creating a special design document:

curl -v -d @/tmp/view.json -X PUT $DB/_design/tree

Now, to run it, just execute:

curl -v -X GET $DB/_design/tree/_view/descendants

Or see it in the browser: http://localhost:5984/hierarchical_data/_design/tree/_view/descendants

This is what you get:

{"total_rows":29,"offset":0,"rows":[
{"id":"Banana","key":"Banana","value":""},
{"id":"Beef","key":"Beef","value":""},
{"id":"Cherry","key":"Cherry","value":""},
{"id":"Banana","key":"Food","value":""},
{"id":"Beef","key":"Food","value":""},
{"id":"Cherry","key":"Food","value":""},
{"id":"Food","key":"Food","value":""},
{"id":"Fruit","key":"Food","value":""},
{"id":"Meat","key":"Food","value":""},
{"id":"Pork","key":"Food","value":""},
{"id":"Red","key":"Food","value":""},
{"id":"Tomato","key":"Food","value":""},
{"id":"Yellow","key":"Food","value":""},
{"id":"Banana","key":"Fruit","value":""},
{"id":"Cherry","key":"Fruit","value":""},
{"id":"Fruit","key":"Fruit","value":""},
{"id":"Red","key":"Fruit","value":""},
{"id":"Tomato","key":"Fruit","value":""},
{"id":"Yellow","key":"Fruit","value":""},
{"id":"Beef","key":"Meat","value":""},
{"id":"Meat","key":"Meat","value":""},
{"id":"Pork","key":"Meat","value":""},
{"id":"Pork","key":"Pork","value":""},
{"id":"Cherry","key":"Red","value":""},
{"id":"Red","key":"Red","value":""},
{"id":"Tomato","key":"Red","value":""},
{"id":"Tomato","key":"Tomato","value":""},
{"id":"Banana","key":"Yellow","value":""},
{"id":"Yellow","key":"Yellow","value":""}
]}

As you can see, there was no view parameter in that call, and this looks nothing like a list of descendants. Each emit call is responsible for one line of the output, which contains the id of the doc object, a key and a value. [I replaced doc as the value in the original emit call with '' to make it more readable.]. Note that lines do not appear in the order they were emitted (otherwise you’d see lines with the same id grouped together). CouchDB automatically sorts them by key. Another thing you’ll notice is that all the lines whose keys have the same element are also a descendant of that element. Convenient, huh?

Now, to get the descendants of one particular node, just query the view with that node’s name in the key:

curl -v -X GET 'http://localhost:5984/hierarchical_data/_design/tree/_view/descendants?key="Fruit"'

Or, again, use the browser: http://localhost:5984/hierarchical_data/_design/tree/_view/descendants?key=%22Fruit%22

And bingo!

{"total_rows":29,"offset":13,"rows":[
{"id":"Banana","key":"Fruit","value":""},
{"id":"Cherry","key":"Fruit","value":""},
{"id":"Fruit","key":"Fruit","value":""},
{"id":"Red","key":"Fruit","value":""},
{"id":"Tomato","key":"Fruit","value":""},
{"id":"Yellow","key":"Fruit","value":""}
]}

The ‘loophole’

Lots of people write frameworks and plugins and libraries intended for use on the Web. A lot of that is FOSS. But, as everybody knows, free software licenses like the GPL lose their “virality” when used for the web, since you’re not giving away any software (which would then force you to make the code available), you’re just providing a service. As long as you don’t give away the software, you can keep your modifications nice and cozy inside your server, without anybody knowing about it, and it’s still all legal.

Less Sharing

Occasionally I’ve made small contributions to such software projects and I might eventually release some of my own. But you know, I like Linus’ tit-for-tat attitude, and I’d like to get back improvements from people who are using my web-related stuff.

Sure, a lot of people contribute back, as you can see on GitHub, even if they don’t have to. Some folks are just that nice. But, and I’m taking a wild guess here, there’s a large hidden part of users who make modifications and don’t give them back simply because it’s extra work making it look nice, maybe conforming to the project’s coding standards, removing private, company-specific stuff, etc., and the employer only cares that it works, not that it’s shared back. So a lot of potential hacking freedom mojo is lost due to time/budget constraints or just pure laziness.

A solution?

Apparently I’m not alone in thinking this, as the FSF wrote the Afferos General Public License (AGPL) specifically to cover this area, which was previously overlooked. Now you have to give back modifications on AGPLed code, whether you give away the code or just provide a service with it over a network. Great! Now I can make a Rails plugin and get back some contributions!

Small problem though: like the GPL, the AGPL is viral, and so the installation of a mere plugin would require the whole app to be open-sourced as well. Fat chance.

The same problem existed in the desktop world, so they came up with the LGPL, which lets you use the library in proprietary software, but forces you to give back changes you made to the library itself. Sadly, there’s no “ALGPL” equivalent for us web folks. So what now? Are we doomed to have our FOSS mojo drained forever into the gutters of licensing loopholes?!

No!

The solution!

While not as good as making an ALGPL, the clever FSF guys put a nice little linking exception in both GPLv3 and AGPLv3, allowing you to use software licensed under any of them together. What does that mean?

It means that now you, application developer, can take my AGPL plugin/library and use it on your app, which is GPL, and give me back only changes you make to my stuff.

Wait a minute! So you’re saying I have to license my code as GPL?! Kind of, yes. But since you don’t give out your code anyway, to the outside world it’s completely unknowable if it is indeed GPL or not. You just stick a LICENSE file in there with the GPL text and that’s it. Nobody has to know about it, and you don’t have to do a thing (and I won’t even check if the file is actually there). For the web, GPL == proprietary.

Sounds convoluted? Nah. It’s actually quite simple. Summarizing:

1. Put the GPL text in a hidden file in your web app;
2. Install my plugin/library;
3. Change it around, make it better;
4. Give me the changes.

That’s it. Since other people will also be giving me their improvements, you will also benefit, all of that while keeping your application nice and proprietary. Good, eh?

I’m posting here the full version of my proposal that was accepted for the Google Summer of Code this year. I’ll be blogging mostly about this for the next few months, so it’s good to give some context. I hope it can also be useful to other students when writing their application next year.

Abstract

ReviewBoard’s increasing adoption across many diverse development environments has led to a need for integration with other tools that has outgrown its development capacity and focus, driving a growing need for extensibility. There is already an HTTP API and support for third-party extensions is on the way, but we still need a standard way to support event-based integration with other apps. This project aims to fill that gap by implementing support for WebHooks [1].
Content:

Background

My name is Helder dos Santos Ribeiro, born and living in Brazil. In 2007 I was mentored by Aaron Patterson in the Google Summer of Code, where I successfully built TestGen4Ruby [2], a TestGen4Web-to-FireWatir bridge [3][4], allowing the recording of user actions on a browser and programatic replaying of them on an open Firefox instance. I’ve spoken about it at RubyConf2007 [5].

I’m a Ruby programmer with a lot of experience in web development, being familiar with its main concepts and technologies. I’ve done a lot of freelance work for Europe’s leading local reviews web startup, which was built using Ruby on Rails.

Other work includes professionally writing C applications for embedded Linux platforms using the Enlightenment Foundation Libraries; heavy data processing and visualization using Java for scientific/governmental projects; and a popular Twitter bot for notifying students of the lunch menu at the cafeteria :)

I’m currently studying Computer Engineering at Unicamp (State University of Campinas). Unicamp is considered to have one of the best
Computer Engineering courses in Latin America.

Motivation

ReviewBoard is one very important tool in the development process, helping keep code quality, share knowledge, build team interaction and train new hires. As important as it is, it sits in a much bigger ecosystem in the development process, composed of version control systems, continuous integration/build/staging servers, bug trackers, mailing lists, IRC channels, etc. Integration among these is crucial to allow for increased automation in the process, reducing its overhead so people can focus on the product. Given the myriad of different applications used for each of those roles, it is unrealistic to expect ReviewBoard to provide integration with every single one of them.

Reviewboard’s HTTP API has helped bridge that gap, allowing other applications to interact with it by requesting or adding content. One thing that is missing, though, is a callback mechanism, allowing ReviewBoard to notify other apps based on events that happen within itself. One emerging way of doing this is WebHooks [1]. It uses HTTP to POST JSON payloads to callback URLs, conveying information about the event which can then be handled by the receiving application however it likes. The use of HTTP and JSON are key to its effectiveness, as these are standards understood by most applications and for which many tools and libraries are available.

The goal of this project is to abstract ReviewBoard’s current email notification mechanism into a generic, signals-based event callback API that will be made available to its to-be-released extensions framework; refactor email notifications to plug into that API; and then build support for WebHooks as an extension.

I’m working in a small team of Ruby on Rails developers and have been responsible for organizing and tuning our development process. We’ve been using ReviewBoard daily for the past few months and have grown very fond of it. The idea for this project grew out of our own need for better extensibility and notification. Its completion will have a direct impact on our productivity, so I’m very commited to it.

Deliverables

* Build a common infrastructure for events using signals:
** make it very easy to register new event triggers and listeners;
** allow listeners to listen only to events by a user, a group or all.

* Refactor email notification code (currently hard-coded) using signals;

* Write support for WebHooks as an extension. This includes:
** Admin interface:
*** specify hook URL(s) to POST all events on the site to;
*** control whether user/group-level hooks are allowed;
*** (optional) specify which types of event will be sent.
** User/Group interface:
*** specify hook URL(s) to POST user/group-scoped events to;
*** (optional) specify which types of event will be sent.
** Backend:
*** WebHooks table with proper join tables for users/groups;
*** Specification of JSON format for serialized events;
*** Listen to signals and issue POST requests asynchronously and with retries:
**** Implement using django_webhooks [6], Hookah [7] or custom solution

And, of course, docs, tutorials and whatnot.

Availability

Full disclosure: I live in the Southern Hemisphere, so I don’t have a big summer break to work on this. I’m not taking too many classes this semester, so I’ll start working right after the results come out (including during the community bonding period), which gives me almost 4 months to finish everything. I also have a 1-month winter break in July, so it’s perfectly doable.

Timeline

Weeks 1, 2, 3, 4 (April 20th – May 17th):

* Learn Python and Django (easy with my Ruby/Rails background);
* Study the ReviewBoard codebase and get familiar with its development process;
* Start work on signals infrastructure;

Weeks 5, 6, 7, 8 (May 18th – June 14th):

* Finish signals infrastructure;
* Refactor email notification code to use signals;
* Study which of django_webhooks, hookah or custom solution are the best choice;

Weeks 9, 10, 11, 12 (June 15th – July 12nd):

* Build admin/user/group interfaces and tables to record options;
* Tie options/URLs to listeners with the appropriate scopes without POSTing (just printing the URL);

Weeks 13, 14, 15, 16 (July 13th – August 10th):

* Tie POST issuing code to signal handlers;
* Test, fix, test, fix, test;
* Write documentation, tutorials, blog posts, etc.

References

[1] WebHooks: http://webhooks.org/
[2] TestGen4Ruby: http://code.google.com/p/tg4rb/
[3] TestGen4Web: https://addons.mozilla.org/firefox/addon/1385
[4] FireWatir: http://code.google.com/p/firewatir/
[5] RubyConf Presentation: http://rubyconf2007.confreaks.com/d3t3p1_summer_of_code_firewatir.html
[6] django_webhooks: http://github.com/johnboxall/django_webhooks/tree/master
[7] Hookah: http://blog.webhooks.org/2009/03/02/announcing-hookah-the-web-hook-event-dispatcher/