python minority game simulation

how to profile pypy performance. not completely obvious.

Library claiming orders of magnitude speed improvement over hadoop and more flexibility than mapreduce. Already has amazon ec2 images ready to fire up. c++ is the native language; POJO claim to be also a first class citizen, and thus other JVM langauges. Hm.

hadoop-alike. Major attractions - uses scala, optimised for RAM-happy operations, fast native iteration, and supports a Google Pregel clone called Bagel.

it is not enough to have a language for specifying your prior structural beliefs—instead we must have a language for such which results in computationally tractable solutions.

mapreduce alternative thingy for distributed nodes

how google does it.

handy educational tutes on various concurrency models; a bit of a hadoop emphasis

Amazon's recommended GPU thingy for python  - sounds like a GPU accelerated SciPy.

Not only is the arrival of Tesla GPUs in the cloud very welcome, the intro blog is a great introduction to resources for using them.

faul tolerant distributed computing by yahoo

"As Narayan has eloquently argued many times, it no longer makes sense for most institutions to run their own HPC, if you take into account the true costs of power, AC, and hardware. The only reason it looks like HPCs work well is because of the way institutions play games with funny money (a.k.a. "overhead charges"), channeling it to HPC behind the scenes - often with much politicking involved. If, as a scientist, your compute is "free" or even heavily subsidized, you tend not to think much about it. But now that we have to scale those clusters 10s or 100s or 1000s of X, to deal with data 100s or 1e6s of times as big, institutions will no longer be able to afford to build their own clusters with funny money. And they'll have to charge scientists for the true computational cost of their work -- or scientists will have to use the cloud."

I always fall down on the step about not trying to be clever.

Nerdiest blog post ever. 1557 words about how to save 1 byte in your transparent PNGs.

"PyCUDA lets you access Nvidia‘s CUDA parallel computation API from Python. Several wrappers of the CUDA API already exist–so what's so special about PyCUDA?

Object cleanup tied to lifetime of objects. This idiom, often called RAII in C++, makes it much easier to write correct, leak- and crash-free code. PyCUDA knows about dependencies, too, so (for example) it won’t detach from a context before all memory allocated in it is also freed.
Convenience. Abstractions like pycuda.driver.SourceModule and pycuda.gpuarray.GPUArray make CUDA programming even more convenient than with Nvidia’s C-based runtime.
Completeness. PyCUDA puts the full power of CUDA’s driver API at your disposal, if you wish.
Automatic Error Checking. All CUDA errors are automatically translated into Python exceptions.
Speed. PyCUDA’s base layer is written in C++, so all the niceties above are virtually free.
Helpful Documentation."

google analytics considered as inferior to mongodb for pageview tracking. (query API-wise, i agree. client-side, not so sure.)

max/msp style flow control programming for python, with concurrency and weirdness. Seems to be an open-source yahoo pipes re-implementation.

nicer wire protocol than redis (http/json!), very scale-y

how a massive site like reddit scales is pretty instructive. spoiler: cache everything. memoise functions

parallelizing cellular automata using clojure

Green Unicorn (gunicorn) is an HTTP/WSGI Server for UNIX designed to serve fast clients or sleepy applications.

This is a port of Unicorn in Python.

uWSGI is a fast (pure C), self-healing, developer-friendly WSGI server, aimed for professional python webapps deployment and development. Over time it has evolved in a complete stack for networked/clustered python applications, implementing message/object passing and process management. It uses the uwsgi (all lowercase) protocol for all the networking/interprocess communications. From the 0.9.5 release it includes a plugin loading technology that can be used to add support for other languages or platform. A Lua wsapi adaptor, a PSGI handler and an Erlang message exchanger are already available.

hi performance minimal queu protocol

intriguing approach to distributed high performance computing, where continuations are passed about as needed to scale to the computation

ubuntu instances excute their user-data in bash!

"import cloud; cloud.call(my_function, arguments)" - now your functions has been serialised and executed in a cloud cluster.

a performance-oriented, DVCS API-compatible drupal fork

event IO and system forks makes you potent

writing your first twitter/telnet apps with node.js

Fascinating expose on how kiva crowdsources volunteer management to be the sleekest NGO ever

redis-backed queue for reasonably heavyweight jobs with a simple ruby interface.

awesome music geekery by the guy behind echonest

how do you evolve the bloody things in silico

A pregel-like graph computation framework for hadoop. apparently working

intersting run around various non-sql data stores and which work for last.fm

"There’s another problem with Unix programming in Ruby that I’ll just touch on briefly: Java people and Windows people. They’re going to tell you that fork(2) is bad because they don’t have it on their platform, or it sucks on their platform, or whatever, but it’s cool, you know, because they have native threads, and threads are like, way better anyways.

Fuck that.

Don’t ever let anyone tell you that fork(2) is bad. Thirty years from now, there will still be a fork(2) and a pipe(2) and a exec(2) and smart people will still be using them to solve hard problems reliably and predictably, just like they were thirty years ago."

generator based pyton keep-the-connection-alive thingy. rumoured to have poor test coverage, though.

non-blocking masively parallel python webserver, as seen on friendfeed

a postres single master replication app.

thingy that caches map tiles in a map-tile aware way making it, well, fast. IIRC this is used by Open Street Maps.

fascintating hybrid of schemaless and schema'd high performance datastore at friendfeed. - stashing indexed free(ish)-form serialised python objects in the fields.

go, floguy. TRadeoffs for some trendy non-relations DBMSs

an alternative process manager that doesn't require you to write daemonising startup scripts, and manages your unforked processes itself.

how to make them etags behave nice-like. an save CPU time to boot.

Even the w3c endorses this idea: that you should make your data as easy to understand as possible.

managing rapdily mutating cloud server setups

pluses and minuses of mongodb vs MySQL

Jacob does the whole deal - scalability, deployment, replication etc

nice little python blog

"django-denorm provides a declarative way of denormalizing models in Django based applications while maintaining data consistency." - another solution to db scalability problems - use JOINS for canonical refs, but load denormalised copies. also allows some nice queries not natively supported by the django ORM, which this framework makes transparent.

Cassandra is the new hotness. V. distributed, non-relational data store with flexible indexing and data models. open-sourced by facebook. Overkill for everything I do, (i will never have that many db writes that i want to live with its warts) but nice to know it's there.

facebook's own google bigtable workalike. Except it's better and stuff. and crashy.

python mapreduce framework, for all those massive corups analysis tasks. written in erlang, btw.

the sexiest REST db in the game.

There are two ways to have a fast learning algorithm: (a) start with a slow algorithm and speed it up, or (b) build an intrinsically fast learning algorithm. This project is about approach (b), and it's reached a state where it may be useful to others as a platform for research and experimentation.

There are two algorithms, one implementing specialist gradient descent (GD) on squared loss and the other implementing specialist exponentiated gradient descent (SEG) on squared loss.

crazy - a distributed, high security language for computation over crappy, slow, compromised or untrusted networks.

google making moves to open their large scale graph analysis infrastructure. Or at least discuss the implementation.

"MongoDB is a high-performance, open source, schema-free document-oriented database. MongoDB is written in C++ and offers the following features:

Collection oriented storage - easy storage of object-style data
Dynamic queries
Full index support, including on inner objects
Query profiling
Replication and fail-over support
Efficient storage of binary data including large objects (e.g. videos)
Auto-sharding for cloud-level scalability (Q209)
A key goal of MongoDB is to bridge the gap between key/value stores (which are fast and highly scalable) and traditional RDBMS systems (which are deep in functionality)."

an indexed store of JSON objects that get deserialised into whatever client language you connect using. handy. looks faster and easier than couch

wunderschön! a decorator that allows you to write unit tests for django to detect threading collisions under load

essay in progress in cloud hosting futures

AMQP ("queueing") HOWTO for python, based around RabbitMQ, the erlang AQMP flagship.

how does simon willison get across al these handy things so rapidly? Anyway, amazon will deploy your mapreduce problem into hadoop for you, if you don't have the spare brainpower to both write your algorithm and support the infrastructure. One-off incredibly massive data munging job? NP.

consdiering scaling problems of democracy. Network governance and all that.

massively parallel python clusters. why not? Incredibly simple API.

a kind of hybrid between memcached and a db, offering some datastore intelligence and persistence, but speed and sloppiness too.

more on the python-thread shennanigans. Better than most, but like many, seems to ignore GUI programming. Did webdev eat *everything* else?

turns out the naïve pythonic way of computing pagerank is not far from the sophisticated matrix approach. go forth and analyse network weights on your laptop!

google's stab at tile-caching fro web apps.

eke some semblance of asynchronous IO out of python using coroutine magic. this i the very light event-y python networking thing that is built upon greenlet, if you were askin', as opposed to the crazy world of twisted or what-have-you.

why queues are trendy

do your slow geocoding etc ion the background.

write your bulk processing jobs in python (well, jython) and then run them on your cloud via mapreduce. Perfect for those bulk content analysis jobs. or for running freebase.com, which is apparently what they do with it.

an hour log video from that flickr bloke about what django needs to scale. included bonus kittens.

Wow - couchdb seems to work with django, really easily. Perhaps I should have read this before I rolled my own custom versioning backend for my blog. I didn't, though, and I'm never getting that weekend back.

neat hacks for handling bulk XML in smart ways using python, elementtree and/or stackless

message queues for stateless fun