iOS: Previously mentioned collaboration tool Asana made waves when it launched, but at the time it was missing support for mobile devices. Now, anyone with an iOS device can log in to their Asana account on the go, check in on their projects and activities, comment on tasks, and more. More »

There's no shortage of to-do managers and organizers on the web, but Sandglaz, a new webapp that combines the best things from a day planner and a calendar together into an easy-to-use webapp, really stands out. Adding to-dos, reordering them, and browsing your next actions are all easy operations that don't take a long tutorial to figure out. Sandglaz manages to be powerful and customizable without being difficult to use. More »

This guest post comes from Amber Case, co-founder of Geoloqi, a private, real-time platform for location sharing. She also speaks frequently on Cyborg Anthropology, the study of humans and computers.

In this post I’ll describe how we planned, built and tested a truly real-time location-based game with Socket.io, Redis, Node.js, and what we learned along the way. Over the past few months, we’ve spent the majority our free time building a real-time game as a test for our location platform, Geoloqi. We call the game MapAttack! due to its map-based nature. Two teams compete to capture the most points on the gameboard. The gameboard, in this case, is the city streets of the neighborhood the players are in.

We set each geofence up with a point value that would give players points for entering geofences. The idea was that a virtual map would be set up on top of the real world, and players on red and blue teams would try to capture all of the geofence points in the game before the other team. To capture a point, the phone would have to detect when the player entered the fence, determine the point value of the fence, notify the player that he received the point, turn the geofence the color of the team, and then add the point to the player score and the overall team score.

Why Build a Real-Time Geofencing Game?
We wanted to create a game that allowed people to physically interact with the real world instead of a computer console like a first person shooter or a real-time strategy game. We were inspired by playing a real-life version of Pac-Man called Pacmanhattan, invented by graduate students at the Interactive Telecommunications Program at NYU in 2004. We played it at Portland’s WhereCamp conference in 2008, and we wanted to see if we could make a GPS version of the game, as Pacmanhattan relied entirely on phone calls and physical maps. We also needed a good demo of Geoloqi’s streaming API.

Technical Challenges
Here is an overview of the problems we had to focus on in order to build the game.

Handling the detection of users entering and leaving 200+ geofences concurrently.
Handling the volume of location-updates from all the phones in a given game (20 or more users per game).
Allowing each phone and web browser watching the game to be able to see the movements of players and the geofences changing color in real time. Every phone in the game sends its location to the server, which broadcasts that data to every other phone and browser watching the game.
Handling errors and differences in GPS technology on different smart phone models in order to ensure a fair gameplay experience.

Differences in GPS Hardware
GPS signals are known for reflecting off of tall buildings in urban settings. This causes inaccuracy and inconsistency in location data. It is less-pronounced in newer phones, but it greatly shows in older ones.

Before our Streaming API
Before we finished the Geoloqi streaming API and before we started using Node.js and Socket.io, everything was based on polling for new updates. Phones reported their location at 5 second intervals and the browsers would update the game board in 5 second intervals.

Using Socket.io, Node.js, Redis, and Sinatra Synchrony

Socket.io

Socket.io is a cross-browser web socket implementation allowing us to do real-time data updates on the browser and also supports older browsers. We can use the latest technology without requiring all of our users to update to the newest browsers, thanks to Socket.io falling back to older technologies in older browsers. This allow us to do instant updates across browsers and the phones in the game.

Node.js

Node.js is Evented I/O for V8 Google’s Javascript implementation for Chrome, implemented with a reactor pattern, that enables for large amounts of asynchronous data traffic.

We use a Node.js server to stream the location data from the phones to the Redis pub/sub channel. It publishes to Redis, and another Node server subscribes to that redis channel. Our Node.js server receives updates from the phones using a custom protocol similar to Google’s Protocol Buffers, which is essentially a very compact binary JSON.

When a browser wants to start streaming data, it connects to the Socket.io server and that server then subscribes to the Redis pub/sub channel. The Socket.io server sends that data via Websockets to the browser, falling back to Flash or long-polling if Websockets is not available.

In essence, Socket.io allows us to use Websockets, which are completely new, but also allows this to work on older browsers thanks to the fallback tricks.

Redis

Redis is an open source, advanced key-value store that has support for message queues using something called publish/subscribe, or pub/sub (not to be confused with PubSubHubbub).

From the higher level what this lets us do is handle the difficulty of sending data to all of the phones in the game and the browser in real-time. Every phone in the game sends its location to the server, which broadcasts that data to every other phone and browser watching the game.

One of the interesting things about the publish/subscribe system is that with a traditional system you have to maintain connections and iterate through each in order to pass data through them. The alternative would be that if you had 10,000 users you’d have to iterate through an array of 10,000 connections, which would be very slow and prone to locking up on socket problems.

Using Redis pub/sub is like starting a radio station. Once it is turned on, people (in this case, browsers) can just listen in. This allows us to do real-time data updates to clients (browsers and phones) at a massive scale.

Sinatra Synchrony

Sinatra::Synchrony is a small extension for Sinatra that dramatically improves the concurrency of Sinatra web applications. Powered by EventMachine and EM-Synchrony, it increases the number of clients your application can serve per process when you have a lot of traffic and slow IO calls (like HTTP calls to external APIs). Because it uses Fibers internally to handle blocking IO, no callback gymnastics are required! This means we can just develop as if we were writing a normal Sinatra web application.

Sinatra::Synchrony allows us to do asynchronous programming (ala Node.js), except that it wraps the callbacks in Fibers (which are basically co-routines in Ruby). This allows you to do synchronous programming while taking advantage of asynchronous code. Aside from being easier to program this way, it also allows us to switch to a different concurrency/parallelism strategy if we need to. Kyle Drake developed Sinatra Synchrony specifically for MapAttack. Drake’s work became popular after he made a presentation on Sinatra::Synchrony at PDX Ruby.

The MapAttack Game Server
Finally, there is the MapAttack Game Server. In this case the Game Server is a simple database that takes care of storing the player point data that is displayed on the map and on the phones as players grab points in real-time.

Source Code
We made the source code for MapAttack available for download. You can download or fork the source code for the MapAttack website, iPhone application and Android application. If you build anything interesting with it, please let us know.

Upcoming Games
We’ll be bringing MapAttack! to WhereCamp Portland on October 7-9, 2011. We’ll give an overview of the technology there as well. If you plan to be in the area, please join us.

Sponsored by

Related ProgrammableWeb Resources Geoloqi API Profile, 2 mashups

"If Engineering at Etsy has a religion, it's the Church of Graphs," Ian Malpass writes on the Etsy Code as Craft blog. And how is Etsy fulfilling its religious obligations? With a Node.js daemon called StatsD, which the company has made available on GitHub. StatsD listens for messages on a UDP port, conducts its counting or timing, and then sends the info off to graphite for graphing.

It's based on a Perl daemon of the same name created by Cal Henderson for Flickr.

Sponsor

Malpass ellaborates:

We like graphite for a number of reasons: it's very easy to use, and has very powerful graphing and data manipulation capabilities. We can combine data from StatsD with data from our other metrics-gathering systems. Most importantly for StatsD, you can create new metrics in graphite just by sending it data for that metric. That means there's no management overhead for engineers to start tracking something new: simply tell StatsD you want to track "grue.dinners" and it'll automagically appear in graphite. (By the way, because we flush data to graphite every 10 seconds, our StatsD metrics are near-realtime.)
Discuss

Love Vim? Use Gmail? Then check out Vmail, a Ruby-based Gmail client for Vim written by Daniel Choi. Vmail gives you a full command line interface for Gmail from within Vim, including reading, composing, starring, deleting, archiving and marking spam.

Of course, Emacs users have been able to check e-mail from within Emacs for years. "A common criticism that text editor (vim) fans throw in the face of operating system fans (emacs) is that a text editor should simply edit text," writes Steve Klabnik at The Changelog. "I'm slightly torn, but it's still pretty freaking cool." Don't worry Vim purists: it's just a plugin, not a change to the Vim core.

Sponsor

Screenshot via The Changelog

Vmail requires:

a Gmail account
a relatively recent version of Vim (vmail is developed against Vim 7.3)
Ruby with SSL support compiled in (vmail is developed using Ruby 1.9.2)
RubyGems (if Ruby version is older than 1.9)
the lynx text-only-mode web browser is required to view HTML mail parts in vmail

The current version of Vmail requires a Unix-like environment, but Choi says he hopes to support Windows in the future.

Discuss

J Chris Anderson, CFO of CouchOne, has been hosting a curious CouchApp on his personal site: Twebz. Twebz is a "decentralized Twitter client." We were curious about what that meant, so I did an interview with Anderson about the project. He says it's a just a demonstration of what CouchApps are capable of. But it also hints at what CouchDB is really trying to accomplish: a radical re-architecting of the Web into a more decentralized system. Read on for the full interview.

Sponsor

Klint Finley: Let's start at the top: what exactly is Twebz? It's described as a "decentralized Twitter client." What exactly does that mean?

J Chris Anderson: The aim is to allow you to interact with Twitter when Twitter is up and you are online. But if Twitter is down for maintenance or you are in the middle of nowhere, you can still tweet. And when you can reach Twitter again, it will go through.

If lots of folks are using it, then they can see each other's tweets come in even when Twitter is down.

Mostly the goal was to show the way on how to integrate CouchDB with web services and APIs.

It seems like it could be really useful for people in situations where Internet access is spotty, though.

Definitely. It's not production software yet, lots of rough edges. But it could be cleaned up and packaged like a normal desktop twitter client easily.

Do you plan on releasing it?

Not currently - I did it when I was around the house right after my daughter was born. So it's really just to show other developers how to build these things.

So I guess the idea is that you'd run a local copy and it would connect with other instances of Twebz running on other people's machines and everyone's client would try to connect to Twitter when possible?

Yes. The hard part is ensuring that someone else can't trick your client into tweeting something as you that you didn't write. So I had to do some JSON cryptography stuff to protect against that.

I was going to ask about that - so your Twitter credentials are passing through other people's machines?

No, your credentials are private to your machine. Potentially someone could send you a tweet that looked like it's supposed to be from/by you. The crypto makes sure that the Twitter-posting code can't be fooled by that.

So if you did release this, and people started using it, and then one day Twitter decided "We're done. We're going to go raise pigs in the Ozarks," Twebz would actually still be up and running fine basically forever and everyone could keep reading each other's Tweets.

Yep. And as a side effect you have a complete personal Twitter archive of the folks you follow.

There's even a feature to pull in the complete history of a user, so you can get the back fill of your closest friends if you want.

The full history is what is used to power these types of word clouds for a user. Aside from the autocomplete, I think the word clouds are the best feature. It subtracts the global frequency of each term from the per-user frequency of a term. So for each user you see the terms that are distinctive to them.

How does the autocomplete work?

Autocomplete uses CouchDB map reduce to build a prefix index. Then CouchDB sorts the matches by popularity when each time you type a letter into the search box. So it may start out: aardvark apple alligator (as things that start with "a"). But CouchDB will sort it so you get apple first, as that's the most popular. It was inspired by Google Instant.

If people want to try it, the code and instructions are here.

It says on GitHub that it doesn't work yet.

That's the same as on the site - it certainly isn't ready for the faint of heart. But someone comfortable hacking node.js and CouchApp would be able to dig in and run it.

I wish I had time to clean it up for release, but this week will continue to be busy with all that actual business stuff I've got to do.

Could CouchDB and Node be used in conjunction to create some sort of decentralized darknet? Something along the lines of Freenet?

Node is a good fit for CouchDB because Couch encourages asynchronous background processes, but people also use Ruby / Python / Java for the same purposes. But yes, eventually the plan is that CouchDB will make web applications a lot more robust because they will no longer depend on a centralized point of failure. E.g., even if Twitter goes out of business, people can continue to share messages.

The turnover of Web 2.0 startups is so fast that I think users get discouraged from signing up for services. Why bother with a new photo share if there's a chance it won't be around in a year? But when those are CouchApps, users can continue to use them even if no one is maintaining them, which makes it more rational to invest time in using them. Imagine if Pownce or Dodgeball were still being run by fans.

Have you seen CouchAppspora?(Disclosure: CouchAppspora contributor Tyler Gilles is a ReadWriteWeb employee.)

Yeah, I like that project because the social aspects of it get to the core of the stuff that "comes for free" with the CouchApp platform.

For another example of how CouchDB is useful in low-connectivity settings, check out this case study on how Better Health Outcomes through Mentoring and Assessments is using CouchDB in rural Zambia.

Discuss