Things to consider when optimizing at the 99th percentile. Variance is key and is one of the biggest reasons EBS blows.

Counterintuitively, he suggests synchronizing variances across a cluster (for example, a Puppet run). Even though for that moment in time all requests are slow, it's better to do that than let every request potentially be slow because it fanned-out to a temporarily slow node.

Stand back, I'm going to try science.

Design documentation of LinkedIn's Kafka queue broker. It's distributed but relies on Zookeeper for orchestration. The section on the log file's implementation is particularly interesting.

OCaml, but a distributed POSIX- and NFS-compliant filesystem.

(Dumping GitHub watches into Pinboard.)

Distributed fault-testing tools.

Vector clocks!

Gossip-based eventually-consistent (but for a small cluster likely always consistent) configuration storage engine like Noah.

Git by another name?

Distributed computing project from Berkeley. "...build systems that are OOM bigger than are building today, with OOM less effort than traditional programming methodologies."

MOAR NOSQL

Jonhnny Weslley just added support for this to BashReduce. Need to read to see if it's worth merging.

This intrigues me: "unlike most key/value stores, you can query for ordered ranges of keys." In my messing around with Cassandra, it's Thrift Interface seemed to actively hinder doing anything like this.

I could bolt libketama on the front of Net::Tyrant, or I could translate this to PHP.

Another POSIX-compliant cluster filesystem.

Worth a second look

DHT in library form? From Neils Provos so very much not a joke.

Leonard landed on TC with his own hashing/distribution. Someday we'll have eventual consistency.

"Again, open source runs on its communities. Collections of people using the benefits of multiple brains to solve, either deliberately or by random walk, complex problems. "

LPUSH "Atomic appends!"

Well, Digg uses it, so it must be at least worth reading.

Back at it, this time with a beast of a server, a dataset that won't fit in memory (joy of joys!) and (inexplicably) more OCaml.

This is so meta.

Via Schachter

Using Hadoop's distributed filesystem with Amazon S3. Perhaps a better solution than SimpleDB for real data storage?