Sections

Browser wars and JavaScript performance
The rehabilitation of JavaScript
Node.js solves a real problem
Evented I/O
Sharing code between the browser and server
Critical mass for Node.js
Node is not the "next" anything

In the short time since its initial release in late 2009, Node.js has captured the interest of thousands of experienced developers, grown a package manager and a corpus of interesting modules and applications, and even spawned a number of startups.

What is it about this technology that makes it interesting to developers? And why has it succeeded while other server-side JavaScript implementations linger in obscurity or fail altogether?

The key factors are performance, timing, and focusing on a real problem that wasn't easily solved with other server-side dynamic languages.

Browser wars and JavaScript performance

In the early 2000s, AJAX web development was coming into its own and placing increasing demands on browsers' JavaScript engines. New JavaScript libraries such as YUI, Dojo and jQuery were allowing developers to do much more with web user interface (UI), creating a user experience for web applications that mimicked the behavior of desktop applications.


As JavaScript libraries and websites became more complex and users started to notice poor performance in their browsers, browser developers started to focus seriously on their JavaScript engines.

The race for faster JavaScript engines heated up in September 2008 when Google released Chrome and the Chromium source code. The engine behind it was V8 and it outperformed all others. This helped spur the developers of Firefox, Safari, Opera and Internet Explorer to improve JavaScript performance in their browsers and it opened a new front in the browser wars.

Technically speaking, V8 takes a slightly novel approach to improving performance. Certain JavaScript objects are dynamically compiled directly into native machine code before execution based on a predictive analysis of the code.


This, along with a new approach to property access and a more efficient garbage collection system enabled Chrome to initially post significantly faster benchmarks than other browsers.

Save 50% - JavaScript Ebooks and Videos

JavaScript is everywhere: servers, rich web client libraries, HTML5, databases, even JavaScript-based languages. If you've avoided JavaScript, this is the year to learn it. And if you don't, you risk being left behind. Whatever your level, we have you covered:

Introductory  /  Intermediate  /  Advanced

One week only—offer expires 14 June. Use discount code HALFD in the shopping cart. Buy now and SAVE.

The other browsers responded with improved or completely rewritten JavaScript engines that matched or exceeded V8's benchmarks. These optimizations are still going on, and Google's V8 is benefiting from the healthy, often technically brilliant, competition. Compared to the interpreters for server-side dynamic languages like Ruby, Python, PHP and Perl, JavaScript now has several efficient and incredibly fast runtimes.

Ryan Dahl, creator of Node.js, chose the V8 engine for Node. This has an additional benefit for a server-side implementation.


The predictive optimization of JavaScript works fairly well in the Chrome browser, but it is much more effective for server applications where the same chunks of code tend to be run multiple times. V8 is able to refine its optimizations and soon ends up with very efficient cached machine code.

Node has an additional performance advantage (a big one) that is not directly tied to V8, but we'll get to that in a bit.

The rehabilitation of JavaScript

JavaScript was once widely regarded as an awful hack of a language. Many programmers still feel this way, but the prejudice is starting to fade, mostly because there is a growing body of good code that shows off the language.


One person who has done much to pinpoint JavaScript's technical weak points is Douglas Crockford. Fortunately, instead of stopping there, he has also created JSLint and written "JavaScript: The Good Parts" to help developers write better code while avoiding most of the "bad parts" of the language. In his presentations and posts, one of his core assertions is that:

... despite JavaScript's astonishing shortcomings, deep down, in its
core, it got something very right. When you peel away the cruft, there
is an expressive and powerful programming language there. That
language is being used well in many Ajax libraries to manage and
augment the DOM, producing an application platform for interactive
applications delivered as web pages. Ajax has become popular because
JavaScript works. It works surprisingly well.


Without getting into the details of which parts are good or bad, we have seen in the past few years that professional developers have come to realize that JavaScript is not going away. Many of developers have gotten on with the task of building complex, well-designed applications and libraries. There are still problems with JavaScript and with its specification, but programmers are now much less likely to dismiss it out of hand.

Previous server-side JavaScript frameworks had a much harder time overcoming the negative mindset about the language. By the time Node arrived, JavaScript had overcome the most of its image problem.

Node.js solves a real problem

Wikipedia has a fairly comprehensive "Comparison of server-side JavaScript solutions". Node is in there, but most of the others listed are not nearly so well known. The use of the term "solutions" is interesting, as most of these projects are solutions to problems that have already been solved by other languages.

Python, Java, Ruby, PHP, Perl and others are all still extremely good choices for most types of dynamic web applications. They talk to databases, crunch numbers, validate data, and parse templates. They are high-level languages, and there are several MVC frameworks for each of them for quick web app creation. Node is sometimes touted as the next Ruby-on-Rails, but this a bad comparison and misses the point of what Node is for.

Node is not trying to solve the same problems as Rails, and it's not competing head-on with any of the other languages or frameworks in the areas where they do well. It was made for, and is most successful at, solving a special set of problems with modern web applications. What can it do that these other languages cannot?

It turns out that what JavaScript can do is the flip side of something it can't do: blocking I/O.

Evented I/O

JavaScript itself can't actually read or write to the filesystem. This ability was omitted from the language because it wasn't necessary for its job in the browser, so Node was able to start from scratch with an I/O system based on event loops.

Node is all about "evented I/O," but what does that actually mean?

To those of us who are either not programmers or are not familiar with event loops, an analogy might help.

You're in a grocery store with a list of items to buy. You wheel your cart around the store, pick up one item at a time, put it in your cart, then take the cart through the checkout. You can optimize this slightly by fetching the items in a sane order, but you can't go get the milk while you're waiting at the deli counter.

If you're in a hurry, you might start thinking of crazy ways to speed up the process. You could enlist a number of other shoppers with shopping carts and send each out to buy a single item. This would create bottlenecks in narrow isles and a huge traffic jam at the checkout

This is clearly an insane way to solve the issue because it throws more shopping carts and cash registers at the problem than needed.

Programming languages that block on I/O often try to solve similar problems by spawning additional threads or processes (c.f. Apache, sendmail). This can be expensive in terms of memory usage, and an analysis of Python's Global Interpreter Lock shows just how expensive the traffic jam can be in terms of CPU utilization.
JavaScript and Node use event loops and callbacks to approach the problem differently.

Returning to the shopping example: If you had a group of kids along with you on your shopping trip, you could send each off to get a single item and return them to the cart. When they've all returned, you can proceed through the checkout. The time taken in fetching items would be the maximum time for retrieving a single item (the kid who had to wait at the deli counter), rather than the sum (picking up the items in sequence). Using runners for the small, simple task of fetching items is a more efficient way of parallelizing the problem than sending out full-fledged shoppers and carts.

It's not a perfect analogy by any means, but more succinct and accurate descriptions involve code or pseudo-code. Ryan Dahl's initial presentation at JSConf 2009 used the following example:

var result = db.query("select..");
// use result

Here the database query blocks the program from doing anything else until the query is returned, whereas in an event loop:

db.query("select..", function (result) {
// use result
});

... the program can continue doing things while waiting for the function to call provide its callback.

Node provides non-blocking libraries for database, file and network access. Since I/O is not a fundamental part of JavaScript, nothing had to be taken away to add them. Python's Twisted and Ruby's Event Machine have to work around some basic language components in order to get similar evented behavior.

So, in addition to the performance wins Node gets "for free" by using the V8 JavaScript engine, the event loop model itself allows Node servers to handle massive concurrencies in network connections very efficiently. It often approaches the benchmarks achieved by high-performance reverse proxies like Nginx (which is also based on an event loop).

Sh[…]

Sublime Text 2, also known as TextMate 2 in my office, is becoming increasingly useful to me. The only trouble thus far is the lack of community involvement to the extent of TextMate. That’s to be expected, it’s an alpha that went cross platform two seconds ago.

All that aside though, it’s an impressive alpha. I seriously love it’s potential and I’m really hoping to make it my own over the next few months. Naturally, the biggest hangup is the lack of tools I’ve become so accustomed to in TextMate. The great thing there is that the snippets and color schemes can be carbon copied into Sublime Text 2, the only outstanding items are Commands. Those aren’t directly portable from TextMate to Sublime Text 2.

I work in HTML, CSS, JavaScript, and PHP. In TextMate I make heavy use of the validators because it’s not an IDE and doesn’t have inline validation as you would find in an integrated environment. Sublime Text 2 doesn’t come with any such validators out of the box, but there are some forum posts. I found some resources on getting JavaScript validation working in Sublime Text 2:

Sublime Text 2 (Alpha) Google Closure JavaScript Linter Build package
(V2 only) JavaScript Google Closure Linter Build

I went through the process of getting Google Closure Linter installed and running fine via Terminal, but got [Errno 2] File not found errors in Sublime Text 2. Additional Googling to find out why got me nowhere so I went with JSLint for the time being and thought I’d share how.

Adding JSLint validation to Sublime Text 2
To implement JSLint we’re going to first need JSLint. There are a number of resources for this, but I went with an OS X executable provided courtesy of javascriptlint.com. I made a new folder in my Applications folder and called it a day. JSLint is prepped and ready to go.

The other half of this puzzle is integration with Sublime Text 2. To do that, you’ll need to create a new Build System:

Upon invoking that menu, Sublime will give you an empty file to work with, outlining the structure of Build Systems. Sweet, but what to do with that? We want to send the current file we’re working with to JSLint for validation. To do that, our Build System will look like this:

{
"cmd": ["/Applications/jsl/jsl", "-process", "$file"],
"selector": "source.js"
}
Note the path to Applications/jsl/jsl — that’s where I store my JSLint install, but you will need to modify that to the path on your system. The next parameter is the -process flag that JSLint uses to define the file to validate, and the final parameter is the $file flag that will be replaced with the file path to the file you’re currently working on.

The selector will help Sublime Text 2 automatically use this JSLint Build System when you’re working within a JavaScript file.

After modifying for your system, save the file to the default path Sublime Text 2 brings up in the save dialog and restart Sublime. Open up a .js file and hit CMD+B, or use Tools > Build. An inline prompt will appear giving you the JSLint results:

I’ll continue fiddling around with getting Closure Linter to work as I’d like to have the option, but if anyone’s got a tip regarding the error I was getting I’d love to hear some feedback.

Everybody is Spamming Everybody Else on MTurk -- one researcher found >40% of HITs are spammy, but this author posted a Mechanical Turk HIT to supply recommendations for visitors to a non-existent French city and got responses from people expecting that every response would be paid regardless of quality.
Javascript Garden -- a growing collection of documentation about the most quirky parts of the JavaScript programming language. It gives advice to avoid common mistakes, subtle bugs, as well as performance issues and bad practices that non-expert JavaScript programmers may encounter on their endeavours into the depths of the language.
A 5 Minute Framework for Fostering Better Conversations in Comments Sections (Poytner) -- Whether online or offline, people act out the most when they don’t see anyone in charge. Next time you see dreck being slung in the bowels of a news story comment thread, see if you can detect whether anyone from the news organization is jumping in and setting the tone. As West put it, news organizations typically create a disconnect between the people who provide content and the people who discuss that content. This inhibits quality conversation.
Full Text RSS Feed -- builds full-text feeds for sites that only offer extracts in their RSS feeds. (via Jason Ryan)

RSS is Dying and You Should Be Worried -- If RSS dies, we lose the ability to read in private.
What Could Have Been Entering The Public Domain on January 1, 2011? -- a list of the works that won't be entering the public domain in the US because the copyright term was extended in 1976. Think of the movies from 1954 that would have become available this year. You could have showed clips from them. You could have showed all of them. You could have spliced and remixed and made documentaries about them. (You could have been a contender!) Instead, here are a few of the movies that we won’t see in the public domain for another 39 years .... This list will be viewed two different ways by different groups, reinforcing instead of changing their views: copyright minimalists will say "what a tragedy" but copyright maximalists will say "look at these great works we protected, they're still earning money for their creators therefore they're still valuable and thus worth protecting". (via Bill Bennett on Twitter)
ProxClone -- cloner for proximity cards, cost of parts around $30. (via Hacker News)
2011 Is The Year of Server-Side Javascript -- explanation of why the author will be doing back-end coding in Javascript this year. Good to see an honest assessment that it's still early days for server-side Javascript: Most of the libraries out there are young, buggy and incomplete. I got Node.js to segfault a few times. There’s no killer framework on the same caliber as Rails, nor anything that comes close to ActiveSupport and a decent standard runtime library (hmm … that gives me an idea). But then, it’s not much different than what Ruby was five years ago, or Java back in the late 90′s. We’ve all got to start somewhere.

Now we have something else to be thankful for. Nathan Smith of Sonspring has created a library that gives designers and developers “some measure of control over form elements, without changing them so drastically as to appear foreign in a user’s operating system.” Smith calls his new library Formalize CSS:

I’ve attempted to bridge the gap between various browsers and OS’s, taking the best ideas from each, and implementing what is possible across the board. For the most part, this means most textual form elements have a slight inset, and all buttons look consistent, including the button tag.

For more, including demos, options, screenshots, thanks, and the library itself, read Smith’s write-up at SonSpring | Formalize CSS. Hat tip and happy Thanksgiving to my good friend Aaron Gustafson for sharing this gem.

In 2005, Jesse James Garrett at Adaptive Path published the seminal blog "Ajax: A New Approach to Web Applications" and ushered in new age of web architecture. Ajax meant using the possibilities latent in JavaScript (specifically, the XMLHttpRequest object) so that a web page could contact the server asynchronously and request new data.

This was revolutionary; within months, we were seeing pages that were more dynamic and interactive. Ajax short-circuited the submit/response loop that dominated web applications up to that time. Instead of making an HTTP request, receiving an entire web page, and rendering that page as a replacement for the current page, the browser requested a chunk of data. It used that chunk of data to interact with the DOM and rewrite the page it was displaying on the fly.

Around the same time, the RESTful paradigm started taking hold. REST represented a much simpler, web-oriented way for servers to interact with their clients. As Roy Fielding pointed out in his dissertation, the basic operations of the HTTP protocol were capable of providing general access to data; stateful applications could be built upon stateless protocols; hypermedia could be used to maintain application state. Although Fielding's dissertation dates back to 2000, it took a few years of bad experience with SOAP and its heirs to realize its importance. With REST, it becomes easier for a website to see itself as a source of data for machines to process, rather than as a source of content for humans to read. Websites become data servers.

The HTML page you get from the new Twitter is largely a bunch of empty divs, with a big wad of JavaScript. The JavaScript is the entire application.

Important as Ajax and REST have been to the history of the web, each only represents half of a larger revolution. And in the past few months, we've seen some new sites that have taken the revolution to its logical conclusion. Specifically: take a look at the new Twitter. It's a nice web application, sure -- but look at the HTML. There's not much there. The HTML page you get from Twitter is largely a bunch of empty divs, with a big wad of JavaScript. What's happening? The JavaScript is the entire application; the divs exist only to provide tags so the JavaScript can rewrite the DOM at will. In turn, the JavaScript is constantly (and asynchronously) making requests from the Twitter site, which is just returning data from its API. In fact, the Twitter site is returning the same data for its web page that it would return for its mobile app, for TweetDeck, or for any of the apps in the Twitter ecosystem.

This design isn't particularly new; we've seen it ever since developers started reverse-engineering GMail and Google Maps to get ideas for their own projects. Those big Google apps may have been the first examples of this architectural trend. They were certainly among the first to use JavaScript as a full-fledge client programming language. But we're seeing many more sites built along these lines. Why now, what does this shift mean, and why is it important?

"Why now" is perhaps the easiest question to answer. A few short years ago, web developers only had one platform to support, and that was the "browser." Granted, there were a dozen or so browsers of significance, and the browser world was riddled with incompatibilities. We're in a different world now. Browser compatibilities have been ironed out, to some extent (though conscientious developers still support "legacy" browsers, all the way back to IE6 or even IE5). But it's no news that the most important new apps these days run on devices ranging from phones (iPhone, Android, BlackBerry, Windows Phone), tablets (iPad, Android/ChromeOS devices), and potentially ebook readers and other new devices. With these new devices on the table, browser incompatibilities pale in significance. It's another sign of the times that I can't conceive of an interesting application that doesn't access data across the network. A static application that never accesses remote data -- that's so 1990s.

So, while it's tempting to say that the new age is characterized by the browser as platform, and that applications running in the browser can do anything that native code can do, that's looking in the wrong place. HTML5 certainly ups the ante, as far as browser capabilities -- and is supported to some extent by all of the other devices we're concerned with. But the real meaning and importance of this architectural shift is on the server side, driven both by the need to support many heterogenous device and application types, and by the primacy of live data in modern applications.

Related books and videos

Ajax
-- Head First Ajax
-- Ajax, TDG

REST
-- Rest in Practice
-- RESTful Web Services
-- RESTful Web Service Cookbook

HTML5
-- HTML5: Up and Running
-- HTML5 Mobile Web Development (video)
In the browser-dominated world, static content and data were inevitably mixed. Yes, we had templating systems that let developers separate static content and design elements from data. But once the application server did its magic, what was delivered to the browser was HTML pages mixing data with other content. Browsers were similar enough that, with some browser detection hacks on both the server and client side, it was relatively easy (though a pain) to generate pages that would run anywhere. That doesn't work any more. It's naive to think that you can wrap some HTML around data and be done with the job; the chances are that you're leaving a huge chunk of your human audience behind, and making things more difficult for another audience -- machines that just want to consume your data. To build a modern application, developers must focus on the data: they must see themselves as data providers, they must develop documented and stable public APIs for accessing their data. Over the past few years, we've realized the importance of data. What's the value of Google without the data behind it? Or Facebook? Or, going back 15 or so years, GNN? It took a long time for us to understand the importance of data, as opposed to "content." But when you've gotten that lesson, your design goals change: designing and publishing a stable API to a data service becomes the highest priority.

That's the driving force behind this architectural shift. Front ends, user interfaces, clients, apps, whatever you decide to call them, don't disappear. But we have learned how important it is to keep the data interface separate from the user interface. Your next project will probably have multiple front ends, some delivered through HTML5, and some delivered through native code. Building them on a common data API is going to be much cleaner and simpler. In addition, third parties can build their own apps on top of your API. An important component of Twitter's success has been the ecosystem of applications that have built on their data: TweetDeck, TweetGrid, Tweetie, etc.; Twitdom, the Twitter applications directory, lists more than 1,800 apps. Until the "new Twitter" went live, the Twitter website was significantly less capable than most of the third-party apps.

Although it has been a long time coming, we're finally finishing the revolution that started with Ajax. Get data that users care about, make it available via an API, provide a data presence that's distinct from your HTML-based web presence, and build multiple front ends to serve your customers, on whatever platforms they care about. Your value is in the data.

Related:

What is data science?
The SMAQ stack for big data
Data as a service
Building data startups
Lies, damn lies, and visualizations

Do you feel that your front-end code isn’t quite up to enterprise standards? Want some good tips on how to take your HTML, CSS, and JavaScript to the next level? Go to Enterprise CSS, Enterprise HTML, and Enterprise JavaScript for loads of useful tips and best practices.

Note: To eliminate any risk of misunderstanding I think it’s best to add that this is a joke – most of the tips are actually anti-best practice. The sad thing is that the front-end code of many “enterprise-level” sites and content management systems really do look like many of these “tips” have been taken seriously.

Some of my favourite tips are these:

Store your application’s state as a hash in a single hidden input on the page, because page-weight is not my problem
Creating forms without the need for cumbersome label elements
Bullet-proof rounded corners that work all the way down to Netscape 4.79
Liberally using documentation to describe sections of content
Every single tag gets an id or class, because how else would you style it?
Read full post
Posted in (X)HTML, CSS, JavaScript.

Closure Compiler Service (via). A hosted version of the Google Closure Compiler (JavaScript minifier) running on App Engine. It has both a user interface and a REST API, which means you can use it as part of an automated build process without needing to set up a local copy of the software.

jQuery.queueFn. “Execute any jQuery method or arbitrary function in the animation queue”. I’m surprised this isn’t baked in to jQuery itself—the plugin is only a few lines of code.

SproutCore -- open-source HTML5 application framework (i.e., lots of Javascript goodness) that'll work with any backend. To code for this, you put most of the logic in the front-end and leave the back-end much simpler.
RDF for Intrepid Unix Hackers -- an interesting series, showing how to use common Unix tools to manipulate RDF data from the commandline. (via Edd Dumbill)
How to Thrive Among Pirates (Kevin Kelly) -- a look at how indigenous movie-makers make money in countries like China, India, and Nigeria where piracy is rampant. In short, they make cheap movies, sell near the price of inferior-quality knockoffs, and take advantage of unique experiences that movie theaters offer (e.g., air-conditioning).
On Complaints (PublicStrategist) -- a very good analysis of complaints departments and expectations of people who complain. But there is also a vital question of what the organisation thinks the purpose of a complaints process is. If it is a safety valve, a means of finding and correcting the most egregious failures or a means of channelling immediate anger and dissatisfaction into a swamp of unresponsiveness, then it can’t provide any broader value. That’s where the Patient Opinion model starts to look really attractive. It is deliberately and carefully constructed to elicit feedback, not just complaints. More than half the stories it gets told are positive, even some of the most harrowing, and it therefore creates a picture which is as clear about what is valued as it is about what is seen as in need of improvement.

flashblockdetector. Mark Pilgrim’s JavaScript library for detecting if the user has a Flash blocker enabled, such as FlashBlock for Firefox and Chrome or ClickToFlash for Safari. One good use of this would be to inform users that they need to opt-in to Flash for unobtrusive Flash enhancements (such as invisible audio players) to work on that page.

Smashing Magazine rounds up a few tricks for “CSS Layouts, Visual Effects and Forms”. Not everything in here looks useful to me, but I did notice a few gems while quickly browsing through. Bookmarked for later investigation. link

Plupload (via). Fantastic new open source project from the team behind TinyMCE. Plupload offers a cross-browser JavaScript File uploading API that handles multiple file uploads, client-side progress meters, type filtering and even client-side image resizing and drag-and-drop from the desktop. It achieves all of this by providing backends for Flash, Silverlight, Google Gears, HTML5 and Browserplus and picking the most capable available option.

Kindle Development Kit APIs -- Amazon will release a Kindle SDK. These are the API docs. (via obra on Twitter)
rePublish -- all-Javascript ebook reader. (via kellan on Twitter)
Peer Review: What's it Good For? (Cameron Neylon) -- harsh and honest review of peer review with some important questions for the future of science. But there is perhaps an even more important procedural issue around peer review. Whatever value it might have we largely throw away. Few journals make referee’s reports available, virtually none track the changes made in response to referee’s comments enabling a reader to make their own judgement as to whether a paper was improved or made worse. Referees get no public credit for good work, and no public opprobrium for poor or even malicious work. And in most cases a paper rejected from one journal starts completely afresh when submitted to a new journal, the work of the previous referees simply thrown out of the window. Some lessons in here for social software, too.
Analog IMDB -- The transition is moving slowly, but it’s moving. It’s a fascinating thing to watch. The technology is the dull part: what’s interesting is the shift in perception. You know how sometimes you turn off a certain section of your brain and force yourself to see a word not as a piece of language with meaning, but as a sequence of black shapes and white spaces? It’s like you’re seeing that image for the very first time and suddenly “bird” seems like a very odd thing. I’ve been buying all of my in-print books electronically for a couple of years. Physical books aren’t weird to me yet. But damn, that old copy of the Maltin guide was a freaky and bizarre object. It’s the first time I looked at a book and didn’t see a container for information. I saw dead wood.

Dojo: Still Twice As Fast When It Matters Most. Alex Russell shows how Dojo out-performs jQuery on the TaskSpeed benchmark, which attempts to represent common tasks in real-world applications and has had code that have been optimised by the development teams behind each of the libraries.

Four Possible Explanations for Google's Big China Move (Ethan Zuckerman) -- I'm staying out of the public commentary on this one, but Ethan's fourth point was wonderfully thought provoking: a Google-backed anticensorship system (perhaps operated in conjunction with some of the smart activists and engineers who’ve targeted censorship in Iran and China?) would be massively more powerful (and threatening!) than the systems we know about today. It's deliciously provocative to ask what the world's strongest tech company could do if it wanted to be actively good, rather than merely "not evil".
Gordon -- An open source Flash™ runtime written in pure JavaScript. (via Hacker News)
Pop Software -- great blog post about this new category of software. The people who are consuming software now are a vast superset of the people who used to do so. At one time, especially on the Mac, we’d see people chose software based upon how well it suited their requirements to get a job done. This new generation of software consumers isn’t like that - they’re less likely to shop around for something rather they shop around for anything. These are people who want to be entertained as much as they want to have their requirements met. [...] Apps are not Applications - they are their own things. They are smaller. They are more fun. Pop software has amazing scale, is hit-driven, is a very hard business for developers, and isn't going away. (via timo on Delicious)
Why Hasn't Scientific Publishing Been Disrupted? -- an analysis of the scientific publishing world: what roles it serves, how some of those roles can be better served by new technology, and which roles are still mired in traditions and performance plans anchored to the old models. As is often the case, people won't move to the new system when the amount they're paid is determined by the old system. (via timoreilly on Twitter)

Working at an agency I am involved more and more on projects in which client side code is developed internally then sent out to a separate team for implementation. You provide static HTML, CSS and JavaScript which then get placed into the CMS and brought to life as an actual website. As you can imagine this can sometimes lead to frustrations. However many safeguards you include, handing over your code to someone else is always a difficult thing to do effectively.

In this article I will show you how you can create a JavaScript implementation checker and that will give you more time for drink based activity as your web site and apps are launched quicker and with less unwanted drama!

An all too frequent occurrence

You’ve been working on a project for weeks, fixed all your bugs and send it to be implemented. You hear nothing and assume all is going well then a few days before it’s meant to launch you get an email from the implementation team informing you of bugs in your code that you need to urgently fix.

The 24ways website with a misspelt ID for the years menu

Being paranoid you trawl through the preview URL, check they have the latest files, check your code for errors then notice that a required HTML attribute has been omitted from the build and therefore CSS or JavaScript you’ve hooked onto that particular attribute isn’t being applied and that’s what is causing the “bug”.

It takes you seconds drafting an email informing them of this, it takes then seconds putting the required attribute in and low and behold the bug is fixed, everyone is happy but you’ve lost a good few hours of your life – this time could have been better spent in the pub.

I’m going to show you a way that these kind of errors can be alerted immediately during implementation of your code and ensure that when you are contacted you know that there actually is a bug to fix. You probably already know the things that could be omitted from a build and look like bugs so you’ll soon be creating tests to look for these and alert when they are not found on the rendered page. The error is reported directly to those who need to know about it and fix it. Less errant bug reports and less frantic emails ahoy!

A page with an implementation issue and instant feedback on the problem

JavaScript selector engines to the rescue

Whether you’re using a library or indeed tapping into the loveliness of the new JavaScript Selector APIs looking for particular HTML elements in JavaScript is fairly trivial now.

For instance this is how you look for a div element with the id attribute of year (the missing attribute from top image) using jQuery (the library I’ll be coding my examples in):


if ($(‘div#year’).length) {
alert(‘win’);
}

Using this logic you can probably imagine how you can write up a quick method to check for the existence of a particular element and alert when it’s not present — but assuming you have a complex page you’re going to be repeating yourself a fair bit and we don’t want to be doing that.

Test scripts

If you’ve got a lot of complex HTML patterns that need testing across a number of different pages it makes sense to keep your tests out of production code. Chances are you’ve already got a load of heavy JavaScript assets, and when it comes to file size saving every little helps.

I don’t think that tests should contain code inside of them so keep mine externally as JSON. This also means that you can use the one set of tests in multiple places. We already know that it’s a good idea to keep our CSS and JavaScript separate so lets continue along those lines here.

The test script for this example looks like this:


{
"title": "JS tabs implementation test",
"description": "Check that the correct HTML patterns has been used",
"author": "Ross Bruniges",
"created": "20th July 2009",
"tests": [
{
"name": "JS tabs elements",
"description": "Checking that correct HTML elements including class/IDs are used on the page for the JS to progressively enhance",
"selector": "div.tabbed_content",
"message": "We couldn't find VAR on the page - it's required for our JavaScript to function correctly",
"check_for": {
"contains": {
"elements": [
"div.tab_content", "h2"
],
"message": "We've noticed some missing HTML:</p><ul><li>VAR</li></ul><p>please refer to the examples sent for reference"
}
}
}
]
}

The first four lines are just a little bit of meta data so we remember what this test was all about when we look at it again in the future, or indeed if it ever breaks. The tests are the really cool parts and firstly you’ll notice that it’s an array – we’re only going to show one example test here but there is no reason why you can’t place in as many as you want. I’ll explain what each of the lines in the example test means:


name – short test name, I use this in pass/fail messaging later
description – meta data for future reference
selector – the root HTML element from which your HTML will be searched
message – what the app will alert if the initial selector isn’t found
check_for – a wrapper to hold inner tests – those run if the initial selector does match

contains – the type of check, we’re checking that the selector contains specified elements

elements – the HTML elements we are searching for
message – a message for when these don’t match (VAR is substituted when it’s appended to the page with the name of any elements that don’t exist)




It’s very important to pass the function valid JSON (JSONLint is a great tool for this) otherwise you might get a console showing no tests have even been run.

The JavaScript that makes this helpful

Again, this code should never hit a production server so I’ve kept it external. This also means that the only thing that’s needed to be done by the implementation team when they are ready to build is that they delete this code.


<script src="sleuth.js" type="text/javascript"></script>
<script type="text/javascript">
$(document).ready(function() {
sleuth.test_page.init(‘js_tabs_test.js');
});
</script>

“View the full JavaScript:/examples/self-testing-pages-with-javascript/js/tests/test_suite.js

The init function appends the test console to the page and inserts the CSS file required to style it (you don’t need to use pictures of me when tests pass and fail though I see no reason why you shouldn’t), goes and grabs the JSON file referenced and parses it. The methods to pass (tests_pass) and fail (haz_fail) the test I hope are pretty self-explanatory as is the one which creates the test summary once everything has been run (create_summary).

The two interesting functions are init_tests and confirm_html.

init_tests


init_tests:function(i,obj) {
var $master_elm = $(obj.selector);
sleuth.test_page.$logger.append("<div id='test_" + i + "' class='message'><p><em>" + obj.name + "</em></p></div>");
var $container = $('#test_' + i);
if (!$master_elm.length) {
var err_sum = obj.message.replace(/VAR/gi, obj.selector);
sleuth.test_page.haz_failed(err_sum, $container);
return;
}
if (obj.check_for) {
$.each(obj.check_for,function(key, value){
sleuth.test_page.assign_checks($master_elm, $container, key, value);
});
} else {
sleuth.test_page.tests_passed($container);
return;
}
}

The function gets sent the number of the current iteration (used to create a unique id for its test summary) and the current object that contains the data we’re testing against as parameters.

We grab a reference to the root element and this is used (pretty much in the example shown right at the start of this article) and its length is checked. If the length is positive we know we can continue to the inner tests (if they exist) but if not we fail the test and don’t go any further. We append the error to the test console for everyone to see.

If we pass the initial check we send the reference to the root element, message contains and the inner object to a function that in this example sends us on to confirm_html (if we had a more complex test suite it would do a lot more).

confirm_html


confirm_html:function(target_selector, error_elm, obj) {
var missing_elms = [];
$.each(obj.elements, function(i, val) {
if (!target_selector.find(val).length) {
missing_elms.push(val);
}
});
if (missing_elms.length) {
var file_list = missing_elms.join('</li><li>');
var err_sum = obj.message.replace(/VAR/gi, file_list);
sleuth.test_page.haz_failed(err_sum, error_elm);
return;
}
sleuth.test_page.tests_passed(error_elm);
return;
}

We’re again using an array to check for a passed or failed test and checking its length but this time we push in a reference to each missing element we find.

If the test does fail we’re providing even more useful feedback by informing what elements have been missed out. All the implementation team need do is look for them in the files we’ve sent and include them as expected.

No more silly implementation bugs!

Here is an example of a successful implementation.

Here are some examples of failed implementations – one which fails at finding the root node and one that has the correct root node but none of the inner HTML tests pass.

Is this all we can check for?

Certainly not!

JavaScript provides pretty easy ways to check for attributes, included files (if the files being checked for are being referenced correctly and not 404ing) and even applied CSS.

Want to check that those ARIA attributes are being implemented correctly or that all images contain an alt attribute well this simple test suite can be extended to include tests for this – the sky is pretty much up to your imagination.

Real time online activity monitor example with node.js and WebSocket. A neat exploration of Node.js—first hooking a “tail -f” process up to an HTTP push stream, then combining that with HTML 5 WebSockets to achieve reliable streaming.

flXHR. I was looking for something like this recently, glad to see it exists. flXHR is a drop-in replacement for regular XMLHttpRequest which uses an invisible Flash shim to allow cross-domain calls to be made, taking advantage of the Flash crossdomain.xml security model.

LABjs: new hotness for script loading. Created in collaboration with Steve Souders, LABjs is a JavaScript loading library which makes it easy to have scripts download in parallel while still ensuring that they execute sequentially where required to ensure dependencies are met. It’s unclear how you would decide to use this over concatenating all scripts together in to a single file.

I gave a talk on Friday at Full Frontal, a new one day JavaScript conference in my home town of Brighton. I ended up throwing away my intended topic (JSONP, APIs and cross-domain security) three days before the event in favour of a technology which first crossed my radar less than two weeks ago.

That technology is Ryan Dahl’s Node. It’s the most exciting new project I’ve come across in quite a while.

At first glance, Node looks like yet another take on the idea of server-side JavaScript, but it’s a lot more interesting than that. It builds on JavaScript’s excellent support for event-based programming and uses it to create something that truly plays to the strengths of the language.

Node describes itself as “evented I/O for V8 javascript”. It’s a toolkit for writing extremely high performance non-blocking event driven network servers in JavaScript. Think similar to Twisted or EventMachine but for JavaScript instead of Python or Ruby.

Evented I/O?

As I discussed in my talk, event driven servers are a powerful alternative to the threading / blocking mechanism used by most popular server-side programming frameworks. Typical frameworks can only handle a small number of requests simultaneously, dictated by the number of server threads or processes available. Long-running operations can tie up one of those threads—enough long running operations at once and the server runs out of available threads and becomes unresponsive. For large amounts of traffic, each request must be handled as quickly as possible to free the thread up to deal with the next in line.

This makes certain functionality extremely difficult to support. Examples include handling large file uploads, combining resources from multiple backend web APIs (which themselves can take an unpredictable amount of time to respond) or providing comet functionality by holding open the connection until a new event becomes available.

Event driven programming takes advantage of the fact that network servers spend most of their time waiting for I/O operations to complete. Operations against in-memory data are incredibly fast, but anything that involves talking to the filesystem or over a network inevitably involves waiting around for a response.

With Twisted, EventMachine and Node, the solution lies in specifying I/O operations in conjunction with callbacks. A single event loop rapidly switches between a list of tasks, firing off I/O operations and then moving on to service the next request. When the I/O returns, execution of that particular request is picked up again.

(In the talk, I attempted to illustrate this with a questionable metaphor involving hamsters, bunnies and a hyperactive squid).

What makes Node exciting?

If systems like this already exist, what’s so exciting about Node? Quite a few things:

JavaScript is extremely well suited to programming with callbacks. Its anonymous function syntax and closure support is perfect for defining inline callbacks, and client-side development in general uses event-based programming as a matter of course: run this function when the user clicks here / when the Ajax response returns / when the page loads. JavaScript programmers already understand how to build software in this way.
Node represents a clean slate. Twisted and EventMachine are hampered by the existence of a large number of blocking libraries for their respective languages. Part of the difficulty in learning those technologies is understanding which Python or Ruby libraries you can use and which ones you have to avoid. Node creator Ryan Dahl has a stated aim for Node to never provide a blocking API—even filesystem access and DNS lookups are catered for with non-blocking callback based APIs. This makes it much, much harder to screw things up.
Node is small. I read through the API documentation in around half an hour and felt like I had a pretty comprehensive idea of what Node does and how I would achieve things with it.
Node is fast. V8 is the fast and keeps getting faster. Node’s event loop uses Marc Lehmann’s highly regarded libev and libeio libraries. Ryan Dahl is himself something of a speed demon—he just replaced Node’s HTTP parser implementation (already pretty speedy due to it’s Ragel / Mongrel heritage) with a hand-tuned C implementation with some impressive characteristics.
Easy to get started. Node ships with all of its dependencies, and compiles cleanly on Snow Leopard out of the box.

With both my JavaScript and server-side hats on, Node just feels right. The APIs make sense, it fits a clear niche and despite its youth (the project started in February) everything feels solid and well constructed. The rapidly growing community is further indication that Ryan is on to something great here.

What does Node look like?

Here’s how to get Hello World running in Node in 7 easy steps:

git clone git://github.com/ry/node.git (or download and extract a tarball)
./configure
make (takes a while, it needs to compile V8 as well)
sudo make install
Save the below code as helloworld.js
node helloworld.js
Visit http://localhost:8080/ in your browser

Here’s helloworld.js:

var sys = require('sys'),
http = require('http');

http.createServer(function(req, res) {
res.sendHeader(200, {'Content-Type': 'text/html'});
res.sendBody('<h1>Hello World</h1>');
res.finish();
}).listen(8080);

sys.puts('Server running at http://127.0.0.1:8080/');

If you have Apache Bench installed, try running ab -n 1000 -c 100 ’http://127.0.0.1:8080/’ to test it with 1000 requests using 100 concurrent connections. On my MacBook Pro I get 3374 requests a second.

So Node is fast—but where it really shines is concurrency with long running requests. Alter the helloworld.js server definition to look like this:

http.createServer(function(req, res) {
setTimeout(function() {
res.sendHeader(200, {'Content-Type': 'text/html'});
res.sendBody('<h1>Hello World</h1>');
res.finish();
}, 2000);
}).listen(8080);

We’re using setTimeout to introduce an artificial two second delay to each request. Run the benchmark again—I get 49.68 requests a second, with every single request taking between 2012 and 2022 ms. With a two second delay, the best possible performance for 1000 requests 100 at a time is 1000 requests / (1000 / 100) * 2 seconds = 50 requests a second. Node hits it pretty much bang on the nose.

The most important line in the above examples is res.finish(). This is the mechanism Node provides for explicitly signalling that a request has been fully processed and should be returned to the browser. By making it explicit, Node makes it easy to implement comet patterns like long polling and streaming responses—stuff that is decidedly non trivial in most server-side frameworks.

djangode

Node’s core APIs are pretty low level—it has HTTP client and server libraries, DNS handling, asynchronous file I/O etc, but it doesn’t give you much in the way of high level web framework APIs. Unsurprisingly, this has lead to a cambrian explosion of lightweight web frameworks based on top of Node—the projects using node page lists a bunch of them. Rolling a framework is a great way of learning a low-level API, so I’ve thrown together my own—djangode—which brings Django’s regex-based URL handling to Node along with a few handy utility functions. Here’s a simple djangode application:

var dj = require('./djangode');

var app = dj.makeApp([
['^/$', function(req, res) {
dj.respond(res, 'Homepage');
}],
['^/other$', function(req, res) {
dj.respond(res, 'Other page');
}],
['^/page/(\\d+)$', function(req, res, page) {
dj.respond(res, 'Page ' + page);
}]
]);
dj.serve(app, 8008);

djangode is currently a throwaway prototype, but I’ll probably be extending it with extra functionality as I explore more Node related ideas.

nodecast

My main demo in the Full Frontal talk was nodecast, an extremely simple broadcast-oriented comet application. Broadcast is my favourite “hello world” example for comet because it’s both simpler than chat and more realistic—I’ve been involved in plenty of projects that could benefit from being able to broadcast events to their audience, but few that needed an interactive chat room.

The source code for the version I demoed can be found on GitHub in the no-redis branch. It’s a very simple application—the client-side JavaScript simply uses jQuery’s getJSON method to perform long-polling against a simple URL endpoint:

function fetchLatest() {
$.getJSON('/wait?id=' + last_seen, function(d) {
$.each(d, function() {
last_seen = parseInt(this.id, 10) + 1;
ul.prepend($('<li></li>').text(this.text));
});
fetchLatest();
});
}

Doing this recursively is probably a bad idea since it will eventually blow the browser’s JavaScript stack, but it works OK for the demo.

The more interesting part is the server-side /wait URL which is being polled. Here’s the relevant Node/djangode code:

var message_queue = new process.EventEmitter();

var app = dj.makeApp([
// ...
['^/wait$', function(req, res) {
var id = req.uri.params.id || 0;
var messages = getMessagesSince(id);
if (messages.length) {
dj.respond(res, JSON.stringify(messages), 'text/plain');
} else {
// Wait for the next message
var listener = message_queue.addListener('message', function() {
dj.respond(res,
JSON.stringify(getMessagesSince(id)), 'text/plain'
);
message_queue.removeListener('message', listener);
clearTimeout(timeout);
});
var timeout = setTimeout(function() {
message_queue.removeListener('message', listener);
dj.respond(res, JSON.stringify([]), 'text/plain');
}, 10000);
}
}]
// ...
]);

The wait endpoint checks for new messages and, if any exist, returns immediately. If there are no new messages it does two thing[…]

In Issue No. 290 of A List Apart, for people who make websites…

The Case for Content Strategy—Motown Style

by MARGOT BLOOMSTEIN

Over the past year, the content strategy chatter has been building. Jeffrey MacIntyre gave us its raison d’être. Kristina Halvorson wrote the call to arms. Panels at SXSW, presentations at An Event Apart, and regional meetups continue to build the drum roll. But how do you start humming the content strategy tune to your own team and to your prospective clients? Listen up and heed Aretha Franklin. No, really.

JavaScript MVC

by JONATHAN SNOOK

As JavaScript takes center stage in our web applications, we need to produce ever more modular code. MVC (Model-View-Controller) may hold the key. MVC is a design pattern that breaks an application into three parts: the data (Model), the presentation of that data to the user (View), and the actions taken on any user interaction (Controller). Discover how MVC can make the JavaScript that powers your web applications more reusable and easier to maintain.