How do you manage configuration files across a wide array of games, played by millions of users and served out of multiple data centers?

Over the years Zynga has found that the right solution to this problem varies for each unique service and product. I’d like to present one of these solutions today: Apache ZooKeeper.

Apache ZooKeeper allows Zynga to update thousands of configuration files in under a second. Game servers at Zynga need to interact with various levels of persistent and volatile storage. To provide increased performance, game nodes talk directly to these storage nodes via IP addresses and domain names stored in a local configuration file.

ZooKeeper allows distributed processes to coordinate with each other through a shared hierarchical name space of data registers (we call these registers znodes), much like a file system. Unlike normal file systems ZooKeeper provides its clients with high throughput, low latency, highly available, strictly ordered access to the znodes. The performance aspects of ZooKeeper allows it to be used in large distributed systems. The reliability aspects prevent it from becoming the single point of failure in big systems. Its strict ordering allows sophisticated synchronization primitives to be implemented at the client.

The name space provided by ZooKeeper is much like that of a standard file system. A name is a sequence of path elements separated by a slash ("/"). Every znode in ZooKeeper's name space is identified by a path. And every znode has a parent whose path is a prefix of the znode with one less element; the exception to this rule is root ("/") which has no parent. Also, exactly like standard file systems, a znode cannot be deleted if it has any children.

The main differences between ZooKeeper and standard file systems are that every znode can have data associated with it (every file can also be a directory and vice-versa) and znodes are limited to the amount of data that they can have. ZooKeeper was designed to store coordination data: status information, configuration, location information, etc. This kind of meta-information is usually measured in kilobytes, if not bytes. ZooKeeper has a built-in sanity check of 1M, to prevent it from being used as a large data store, but in general it is used to store much smaller pieces of data.

This is the first part in a series of posts going over Noah

As you may have heard (from my own mouth no less), I’ve got a smallish side project I’ve been working on called Noah.

It’s a project I’ve been wanting to work on for a long time now and earlier this year I got off my ass and started hacking. The response has been nothing short of overwhelming. I’ve heard from so many people how they’re excited for it and nothing could drive me harder to work on it than that feedback. To everyone who doesn’t run away when I talk your ear off about it, thank you so much.

Since I never really wrote an “official” post about it, I thought this would be a good opportunity to talk about what it is, what my ideas are and where I’d like to see it go in the future.

"look at this effing rainbow I just made for you"

Noah is an application registry inspired by Apache ZooKeeper

What does that mean? From the ZooKeeper Home Page:

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications.

Essentially Noah is a port of parts of that functionality into a stateless RESTish application.

ZooKeeper: A Distributed Coordination Service for Distributed Applications

ZooKeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives that distributed applications can build upon to implement higher level services for synchronization, configuration maintenance, and groups and naming. It is designed to be easy to program to, and uses a data model styled after the familiar directory tree structure of file systems. It runs in Java and has bindings for both Java and C.

Coordination services are notoriously hard to get right. They are especially prone to errors such as race conditions and deadlock. The motivation behind ZooKeeper is to relieve distributed applications the responsibility of implementing coordination services from scratch.