Stoyan and I were discussing this and wondered if there was a way to have longer cache times and update resources when necessary. We came up with a solution. It’s simple and reliable. Adopting this pattern will reduce unnecessary HTTP requests resulting in faster pages and happier users, as well as better Page Speed and YSlow scores.

Using http headers for more efficient RSS serving.

Asynchronous HTTP calls in Java via @cowtowncoder

I hope above shows that while doing incremental, "streaming" processing is bit more work, it is not super difficult to do.

Not even when you have bit of pipelining to do, like uncompressing (or compressing) data on the fly.

"According to the HTTP Archive, which gathers stats on the top million sites in the world (as ranked by Alexa), the average web page has surpassed the 1 MB mark..."

good to see an API for constructing, parsing, and resolving URIs , this should be part of any web-friendly environment.

proxy_support = urllib2.ProxyHandler({})
>>> opener = urllib2.build_opener(proxy_support)