Olivier Scherrer

I was so happy the first time I used socket.io, seeing how fast, reliable and easy to use it is, that when I started doing XHRs I felt that something was wrong. Current State of the Art implies using AJAX calls coupled with JSONP for getting data from a 3rd party server. To new applications we’re also adding WebSockets that allow to push data from the server to the client. Three different techniques for carrying data, can’t we unify them?

Actually, it turns out that we can. Let’s imagine something like this: data is fetched by the node.js server. By data I mean any data, whether it comes from a database, a web service, the file system, actually anything that node.js can have access to. Each data source is accessed by what would be called a request handler. And when the data is available on the node.js server, it can be accessed from within the application and be streamed to the client using socket.io.

It would resemble something like this:

How would it work from the client side?

A component, let’s call it Transport, would be responsible to query the requests handlers. For instance, verifying a browserID assertion would look like:

Transport.request("BrowserID", assertion, callback);

Fetching data from CouchDB would look like:

Transport.request("CouchDB", { path: "/db/_all_docs" }, callback);

And opening a keep-alive socket that would trigger the callback for every new chunk:

Transport.listen("CouchDB", { path: "/db/_changes" }, callback);

And the exact same syntax would work on the server side with a Transport directly connected to the requests handlers.

How to trick socket.io to transport the requests params and push the result back to the client?

Think of it like JsonP. You load a script on a different website which pads the result with a function name that you’ve previously defined. When you get this result, the callback is executed.

So what I do is, I send the requests params to the server and I ask it to reply on a previously defined channel (a unique one). The server executes the request through the handler, and streams the result back to the client. If transport.request is called, the channel is closed right away, otherwise a function responsible for closing it is returned.

The solution has 5 major benefits

• One unique transport
• Multi-purpose transport that can stream binary data too
• Based on well known technologies
• Easy to use
• No cross-domain issues

Want to see it in action?

The solution was successfully implemented in Olives JS Framework, which Transport module can be found here, and requests handlers are explained here.

There’s a simple application currently being developed that also uses this Transport module and another use case

Warning

A discussion is going on on socket.io’s google groups. I’m exposing a database to the browser’s application to showcase this solution but this has security issues. I’m looking for a way to secure this and will post about it, until then, just be aware of the implications.

I’m really happy to announce that Emily and Olives have been officially released today.

It’s hard to find the words to express what I’m feeling now. These two frameworks have consumed almost all my spare time since last September, and they’re the achievement of 2 years of trial and error, research and development, joy and fear. To be honest, had I known how hard it is to come up with what I’ve got today, I think I’d have given up on day 1.

My first Pirates JS Framework attempt has failed, it was in 2010. Emily was born in July 2011 from the ashes of Pirates. Emily is runtime agnostic, it provides tools for manipulating data and common design patterns such as states and observable. Olives is more about creating UIs, connecting views to models and fetching data from the server.

Running both frameworks allows to create incredible realtime applications on node.js with no effort. The API was designed to solve the developers’ problems, not mine.

But allow me to talk about their features in forthcoming articles.

For now, numbers:

• ~1300 hours work since last september
• 15 AMD/commonJS modules
• 100% code coverage, 100% TDD
• 408 tests and many, many more assertions
• Emily: 1815 lines of code, 3499 lines of tests
• Olives: 1301 lines of code, 2258 lines of tests
• Emily is only 12,858b and Olives 8,264b
• That’s 2.3 lines of working code per hour
• And 4.4 lines of tests per hour
• 0 regression. Never saw one.
• And as of today, 3 happy developers using them and making feedback for improvements.

There’s a working demo of Olives based on Addy Osmani’s todo app. There’s a suggestion application too that’s being developed, showing the full potential of Olives.

And more to come!

Also, I wanted to thank Olivier Wietrich @owietrich for his tremendous help. He’s brought lots of good ideas such as Olives plugins, he’s tested the frameworks, contributed to the code and was the first ever to actually develop on them. Many many thanks!