At the end of this discourse, to borrow a phrase from my hero, Edward R. Murrow, a few people may accuse this reporter of fouling his own comfortable nest. But if you've seen this nest recently, you know that if it was fouled to any considerable degree, it might not look all that different anyway.

At one of Microsoft's sessions on HTML5 and CSS3 a few weeks ago, the lead program manager for Internet Explorer 10, John Hrvatin, was introducing Web developers to the basic concepts of layout. These were folks who held up their hands to show they've built Web sites for a decade or more. And for many of them, this was the first experience they ever had in considering the following elements: Column flow. White space. Gutter adjustment. Pagination. Visibility at a distance. Symmetry.

Sponsor

The reason HTML5 (in all its many manifestations) is such a foreign object to so many companies - especially publishers - is because we have gone so long without acknowledging these basic elements of style that they have ceased to be part of our diet. Blogs, which house a majority of the Web's daily content, are the fast food of today's publishing society. They give publishers a convenient means for manufacturing roughly equivalent, bite-size chunks of content in a long, single column. And in turn, they give aggregators such as Google News and now, even more prominently, Facebook an easy means for repackaging and broadcasting that content to whatever passes today for a mass audience.

The blog format relieves publishers from the tiresome duty of producing covers and front pages and things to make their content more attractive and make readers want it. In some cases, it enables publishers to surrender any responsibility for making content attractive in the first place.

There is a prophetic scene in the magnificent movie "Wall-E" where, after having floated in space for centuries in a self-contained shopping mall, the remains of the human race return to Earth. There, upon realizing that food once grew on trees and that trees must be cared for, the people ponder for the first time in their lives just how the pizza and ice cream sprouted forth from these stem-like thingies.

This is what sitting in that room reminded me of. Here, Hrvatin was preaching principles I've believed my entire career: that an online publication should become functional, like software, and that software should become informative and balanced, like a magazine. HTML5 restores to the public conscience all those prospects and possibilities we publishers lost sight of when we became bloggers:

1. Not all articles should be created equal. Blogs are singular conveyor belts of nuggets of text. But a major news story, a feature on how to build a private cloud in your office, an interview with a mobile app developer, and some guy ranting about the stupidity of the blog format, are different beasts with varying life spans. Longer-living articles should be allowed to live longer, rather than being hurled off the conveyor belt into the void of invisibility when more replacements come along. HTML5 offers the possibility of componentized, two-dimensional layout where the Table of Contents can live and breathe again.

2. Publications are packages. The great thing about the magazine was that the reader tended to consume most or all of it. Readers appreciated it as a whole. Blogs are designed to be entered through the middle, into the meat of an article, and then exited through-and-through like a bullet. Even folks who consider themselves regular readers of a particular brand find themselves entering and exiting, letting their aggregator of choice lead the way. By reviving an old concept of scaling components to suit flexible-sized containers, HTML5 enables Tables of Contents to zoom in and out of unused spaces, so that readers entering through the middle are treated to the entire package.

3. An article does not have to be an "article." In a blog, anything functional - a video, a podcast, an interactive chart - must be embedded in the text, and the frame and context for the functional element must fit within whatever the text allows. This is, and has always been, backwards. HTML5 recognizes, and to at least some extent addresses, the fact that functional elements do not belong in static containers. In other words, apps are not text. So if a publisher wants to at least try to produce a piece of useful software as a feature of its publication, it need not be concerned with the syntax of the embed tag or the requirements of the plug-in.

HTML5 offers a glimpse of the freedom to define the Web publication the way it should have been defined fifteen years ago: as a more functional descendant of the magazine and the broadcast. But it can only do so to the extent that the content management systems upon which publishers have entrusted their livelihoods embrace, incorporate, and advance that vision. This is not a certainty; in fact, I fear it's not even a likelihood. The CMS of today is a machine that produces blogs - that shapes and forms content to fit in equal-sized nuggets dropped onto a conveyor belt, the length of which constitutes the breadth of their natural lives. It's about as prepared to embrace the full prospect of HTML5 as government is prepared to embrace "change." As long as Web publishers continue to rely on today's CMS to define the nature of their content, they'll remain stuck in deep space aboard the Starship Buy 'n' Large.

Discuss

For the past three years, I have been doing custom-made video screencasts for private consulting clients. These are moving captures of the images on a PC screen with my own voice-over narrations about IT-related products (you can see the entire collection here). And lately, more vendors have stepped up their own efforts to produce their videos as a way to explain what their products do, or as Mike Lee has said, what they might eventually do. There is also a growing awareness that these screencasts can be used as way of product documentation and support.

Let's talk about what tools you need, some best practices that I have gleaned, and some other places to learn more about this craft.

Sponsor

Tool time
Here are the tools that I use. While certainly there are lots of similar ones, they are pretty much the minimum set that you'll need.

PiratePad for collaborative editing. This is a free tool, and if you are working on a script with others around a conference table or around the world, it is very easy to use. While Google Docs and other services have real-time editing tools, this one is dirt simple to use and get started. It makes producing the script a lot easier.
Audacity for audio editing. It comes in Windows, Mac, and Linux versions. There is a great book here that will help you get started with using it. You'll also need a good microphone, I use an old Plantronics USB headset but there are many things you can use. Just don't use the built-in one on your computer.

Camtasia Studio for video editing and screen recording. (You can see their video editor above.)Their Windows version is superior to their Mac version. There are other tools out there for video screen recording, such as IShowU. Find one that you are comfortable with and learn its secrets.
Wistia.com for video embedding and tracking tools. We covered them in an article earlier this summer here. I like Wistia because you can track exactly who watched your video and at what point they either stopped watching or rewound it to see it again. There are other hosting services that offer embedding but not the level of tracking detail that you get here.
TubeMogul for posting videos to multiple sites at one click. You have to be careful with this because you can propagate something that you didn't want very quickly too.
WordPress or any other blogging software. If you are going to use WordPress, make sure to host it yourself (meaning on a computer you own or a virtual server): the hosted WordPress.com site isn't as good for putting up video content. If you use other software tools, make sure they can handle the embedded video player that you choose. Some blogs fall down miserably in this area.

Next page: Some tips and best practices

Some tips and best practices

Okay, so you have your toolkit. Now I'll share some of my tips that I have learned down through the years.

A word about your audio: I write the script and record it first, with an eye and an ear towards what I am going to end up displaying on the screen. Some people do it the other way around. I have found that it is easier to edit with audio first, particularly to get the synchronization and timings right. Speak clearly, crisply, and engagingly. Use your audio editor to remote any umms, you-knows, and other verbal pauses or mistakes. Vary your tone: you are reading a script, but don't want to sound like you are. Think about when you read books to your young kids, that is the kind of voice I am talking about. Also, talk faster than you would normally in conversation: a viewer can always hit the pause or rewind buttons if they want to hear or see something again.

Keep the video simple. Camtasia and other products have all sorts of special effects, to zoom in and out and cross fade and so forth. Resist the temptation to remake Star Wars here and just deliver the goods. While you should zoom in if the area that is of interest is small, we don't need a lot of fancy tricks because ultimately they can distract. I use 800x600 resolution, which seems to be just the right compromise between bandwidth and clarity of the presentation. YouTube and others now offer HD resolution and wide-screen formats too: but not everyone else does yet, so 4:3 formats are probably best for now.

What about overall length? Three minutes is ideal. Longer and you will lose viewers. A three-minute video will be about 500 words of script, give or take. Use the time wisely, and combine your narration with the visual element to deliver your message. Cut out the dialogue that doesn't describe any action. Here is an example of the kind of drop off of one of my videos (from Wistia's tracking service), where you can see less than half of the people that started watching this three minute screencast were there at the end:

Pick where you are going to distribute and name your channels consistently. Certainly YouTube is a good place to start, but there are other how-to sites that are important too. My favorites are 5Min.com, Vimeo.com and WonderHowTo.com. When looking for a site to host your content, see how much tech-related content they already have, and how easy is it to brand your content if you want to post more than one or two videos?

It isn't about going viral. While certainly it would be nice to have one of your videos hit the stratosphere, you are looking at building an audience and keeping them coming back for more. The only way to do that is by word of mouth, and for people to gradually find you. So slow growth is better, and that also means regular infusions of new content, just like the best magazines (or TV series for that matter). Come up with a regular production schedule that will guarantee video content getting posted regularly.

Don't forget about text descriptions around your video frame. Some of the better video hosting sites allow you to post up to 1,000 words of descriptive content around the video. Some don't allow for much beyond a title and a few tags or keywords. This text is very important for organic search traffic, and also as a way to set expectations for your viewers once they land on your video page. Create a great title that conveys what you are trying to do. Write your text with a compelling call to action. Put in a URL where viewers can go for more information. Include things like pricing, what requirements are needed to use your product or service, and other details here too.

Next page: Want to learn more?

Want to learn more?
There are a number of places that will also teach you how to do these screencasts yourself using a video, naturally. The folks at CodeSchool have a wonderful hour-long video that goes into tremendous detail of their style of screencasting. While I don't agree with everything they recommend, it is a good solid tutorial on how to get started here. A far shorter tutorial can be found here from the folks at Attachments.me here (the background music is a no-no IMHO):

Finally, if you want some suggestions of places to look for great screencasts (other than my own humble examples above), a good place to start is this article from our archives.. It is a bit outdated has plenty of links to suggested videos. Good luck with preparing your own screencast!

Discuss

Analytics appliance vendor, Netezza (which was acquired by IBM last year posted a series of videos dedicated to Thomas Bayes, a Presbyterian minister and mathematician who lived from 1702 to 1761. Bayes formulated Bayes' Theorem, the foundation for probability theory and predictive analytics. The theorem wasn't published until after Bayes' death, when his friend Richard Price presented it to the Royal Society.

Sponsor

Here's the first video:

If you'd prefer to read, rather than listen, you can check out the introduction to Bayes on the Netezza blog by Mike Kearney. Kearney also points to this post at Less Wrong for an illustrated explanation of Bayes Theorem.

It's a great story that rises above the hype and promise of analytics to remind us of the humble origins of the theories behind the tech.

Discuss

It's been 11 days since the first batch of U.S. diplomatic cables was released by the online organization WikiLeaks.

A lot has happened in those 11 days. Below is a recounting of the key events, issues and debates that have arisen. There's enough going on that it's almost sure to include some details you weren't familiar with, dear reader, but we also invite you to correct the history we're working on documenting as it unfolds.

Sponsor

Cables Begin to Be Released

It's been 11 days since the first batch of U.S diplomatic cables was released by the online organization WikiLeaks. WikiLeaks is described on Wikipedia, the online encyclopedia dedicated to a neutral point of view in all things, as "an international new media non-profit organization that publishes submissions of otherwise unavailable documents from anonymous news sources and leaks."

Launched in 2006, WikiLeaks has won numerous awards, from organizations like Amnesty International and The Economist magazine, for exposing human rights violations in other countries around the world.

The newest documents, named Cablegate by WikiLeaks, are now stored on thousands of peoples' home computers around the world and are being downloaded onto more from torrents every hour of the day. The first batch was indexed by the torrent search engine The Pirate Bay within hours of their release and can now be downloaded by anyone in the world in less than five minutes.

The Complicated Arm of the Law

Julian Assange, the intensely controversial leader of WikiLeaks is in a U.K. jail, held without bail for alleged sex crimes but possibly safer there than he would be free. Release of new documents continues daily; WikiLeaks claims to have released so far less than 1% of the cables in its possession.

Of the more than 1,000 documents that have been released so far, 53% were unclassified by the U.S. government, with no security restrictions on viewing them; 40% were classified as Confidential, a security clearance held by more than 3 million people in the U.S. The remaining 7% of documents are classified as Secret, the second-highest security clearance in the U.S. government, behind Top Secret.

The man allegedly responsible for passing these documents to WikiLeaks is in U.S. custody, but it's not clear that WikiLeaks has committed any crimes by publishing the stolen documents.

Doing Business With the Enemy, Or Not, Whoever They Are

Businesses around the Web have stopped doing business with WikiLeaks, from hosting its website to processing online donations for the organization.

Thousands of anonymous people around the world have downloaded software to attack the websites of those businesses, in concert and under the direction of self-appointed and anonymous leaders.

PayPal was taken down through brute force, Visa was down and is now back up, Mastercard is still down after nearly a full day, Amazon withstood the group's attacks and now PayPal has become a target again.

The group is fully permeable, recruited from social networking sites and trained in using the attack software in a dedicated chat room. Law enforcement, security researchers and journalists are known to have infiltrated the Anonymous online chat, but its members are practiced enough at remaining anonymous that they don't seem to care.

This morning a 16-year-old in the Netherlands was arrested for participating in the attacks. Apparently he wasn't as good at being anonymous as the others have become and has reportedly confessed.

The Anonymous communication infrastructure is hosted on a Russian server that claims to specialize in being impermeable to DDOS attacks itself, but the co-ordinators' chat has been taken down several times throughout the past few days by anti-Anonymous counter attackers as well.

Twitter and Facebook have shut down Anonymous accounts, but new accounts are created within minutes and publicized through the group's chat room.

One Big Distraction?

Meanwhile, some have raised the concern that the crude, anonymous battle between Anonymous and businesses that have refused to service WikiLeaks threatens to overshadow the revelations made public in the diplomatic cables themselves.

Those revelations include that U.S. diplomats have spoken candidly among themselves about:
spying on international allies (after promising they weren't)lying to the international press when asked if the U.S. military was involved in various conflicts around the world by saying that it was notbribing countries with things like presidential visits to convince them to take released Guantanamo Bay prisonerscovering up homosexual child sex trafficking by contractors in the huge corps of private military contractors augmenting the U.S. military around the worldand using blackmail-style leverage on parties in climate change discussions who seek more drastic action against global warming than U.S. negotiators support, among other things.

Those items aren't really news, many of the particular matters had been reported on months or years ago (i.e.). That the U.S. government is willing to engage in almost endless nefarious actions in defense of its empire is not news, either. Many of its own citizens support that. The news is simply that behind closed doors, U.S. officials explicitly plan the things that critics have long accused them of doing and now the documents are available to eyes around the world.

The anonymous brute force attacks by 3,000 to 5,000 thousand volunteer botnet participants threaten to become the biggest part of the story because that's what's still unknown. Who will they attack next? Can they be stopped? Will their cloak of anonymity ever be lifted?

What will the political consequences of these multiple layers of distributed identity and anonymity be (WikiLeaks, torrented files, DDOS botnets, Twitter, Facebook)? Will the end result be a more repressive global Internet regime? An Internet kill switch? Online anonymity outlawed? Licensing for journalists?

Or will the global political order be remade? It could become more agile, more transparent, more accountable, more distributed, more contemporary, more egalitarian.

No one knows for sure - but that seems to be where we stand right now.



Discuss

Heroku is a platform that offers an effective join of the best parts of scaling cloud infrastructure with simple but great tools for immediately provisioning Ruby applications. Last week, at the Under the Radar event, where Heroku is a alumni, the company announced that they are nearly at 60,000 applications - marking a growth rate of over 1,000 new applications hosted weekly.

In this quick analysis, we'll review Heroku and New Relic as two pieces of cloud infrastructure that helps web sites perform to service level agreements even the developer can love.

Sponsor

$ sudo gem install heroku - or - Getting Started is Easy

Feel like impressing the boss?

Tell 'em you can transform that whiteboard sketch into an working web application in two weeks.

That is what some inspired Ruby developers are doing. Some significant enterprises are giving it a shot. We found this list of enterprises that are known to have a Ruby application in production.

So, the boss says "do it"

What do you do next? If the answer needs to be "now", Heroku can fit in nicely as a place to launch your application without having to bring new technology or skills into the organization.

Heroku's ruby platform lives on top of Amazon Web Services. The company sells a unit of computing called a Dyno, and bundles packages like the Ronin that are comprised of compute plus storage packages. All of Heroku's offerings come with infrastructure curation build ton top of EC2, S3 and a host of Amazon Web Services.

Shown here is a snapshot of the Heroku Add-ons, partners the company offers to developers.

It offers simplicity to the developer in the way the platform is bundled into Ruby. It has simple documentation that almost makes it fun to flip through architecture diagrams. And, it uses a model for add-ons that both promote the partner and make it easy to on-board. For example, when buying the popular Ruby application performance tool, New Relic for use in Heroku, the billing comes directly through Heroku's console and process.

Recently, Heroku teamed up with NorthScale to introduce a memcache implementation to Heroku customers. Now, memcache is a command away, provisioned in your Amazon infrastructure cloud, all tuned and orchestrated by Heroku. All of the sudden, the cloud looks even smarter for developers scaling Ruby.

New Relic Saves the SLA

So, your app is ready, it looks exactly like your team wants it to. Is it ready for production launch? This can be an important time, and more and more often, developers are turning to tools like New Relic to test for application bottlenecks as part of the acceptance process.

Sometimes, however, something is missed, and an application starts getting reports of "slowness", perceived or real. New Relic is ready to offer help, where you can tune your application, or do a quick two-minute install and troubleshoot.

Here is a demonstration application company hosts with a sample application.

New Relic has become a dominant application performance management tool. Its services provide a way to tune Ruby (and now Java) applications and report on a number of factors such as application performance satisfaction.

The company has chosen to guide users towards simplifying the way SLAs are defined by implementing Apdex (Application Performance Index) which buckets application SLAs into three buckets, "satisfying" "tolerable", and "frustrating".

By taking this approach to judging performance, the company moves users to the true experience of the web application instead of the raw metrics. What this boils down to is business owners being able to pinpoint where they need to be satisfied with the overall application performance.

Ruby hosting in the cloud is catching on. With cloud offerings for real-time performance tuning and scaling up in the cloud a whole new door for growth with the language and adoption for the enterprise.

Platforms like Heroku and tools like New Relic are bending the time-honored boundaries of Information Technology. The old joke "quality, time, cost - pick any two" is about meeting reality face-to-face. Yet, we wonder if Ruby in the cloud will offer the opportunity to break the rules of reality and let developers have it all.

With commands such as "heroku scale memcache" directly near our fingertips, it may be time to claim a future where quality, time, and cost are joined as one.

Discuss

Facebook blew peoples' minds today at its F8 developer conference but one sentiment that keeps coming up is: this is scary. The company unveiled simple, powerful plans to offer instant personalization on sites all over the web, it kicked off meaningful adoption of the Semantic Web with the snap of the fingers, it revolutionized the relationship between the cookie and the log-in, it probably knocked a whole class of recommendation technology startups that don't offer built-in distribution to 400 million people right out of the market. It popularized social bookmarking and made subscribing to feeds around the web easier than ever before. And it may have created the biggest disruption to web traffic analytics in years: demographically verified visitor stats tied to peoples' real identities. There was so much big news that the analytics part didn't even come up in the keynote.

This is so much new technology and it's tied in so closely with one very powerful company that there is big reason to stop and consider the possible implications. There are reasons to be scared. The bargain Facebook offers is very, very compelling - but it's not a clear win for the web.

Sponsor

We won't go into all the details in this post. You can read our blow-by-blow in our live blog, other coverage on Techmeme and discussion of particular developments here on ReadWriteWeb throughout the day. I just want to talk about one overriding concern.

This is why Facebook did a 180 degree shift on privacy last December: because it wanted to use that formerly private user data to make the web social. Privacy remains a major concern in the new scenario, but it also got a couple of nods in the use of iFrames on 3rd party sites and the big support for the OAuth password-free log-in system.

Semantic web developers are liable to be concerned that decades of their work is being ignored, but Facebook's Open Graph Protocol sure seems intended to be respectful of prior art. Shelley Powers calls it "a bit of a rough start, but it's a start."

Centralization is a dangerous thing and Facebook is a young company that's proven willing to break its contract with users in the past.Data portability advocates will find it difficult to argue with the fact that Facebook users can now export their data to an off-site developer's cache and thus potentially to another social network. That said, Gnip's Eric Marcoullier has begun the conversation with Tweets today like "By 'Open,' Zuck means he is open to taking all your data and not giving anything back."

At first blush, it's hard from a user's perspective to find anything to criticize Facebook for in today's announcements. Those criticisms will no doubt start to form once people wrap their heads around all the particulars. On principal, though, there's going to be so much more Facebook around the internet that it feels like a real cause for concern. Centralization is a dangerous thing and Facebook is a young company that's proven willing to break its contract with users in the past (see Facebook's Privacy Move Violates Contract With Users).

For hundreds of millions of people, Facebook already was the internet. That's liable to be even more true in the future, thanks to the changes announced today. For all intents and purposes, when it comes to social networks, there is no other option for most people. That's a very vulnerable place for the web to be.

Discuss

I love to sit on the beach.  One of the coolest things about the beach is the number of layers of visual depth.  Look at the sand and it's beautiful, but zoom your eyes in closer and you'll see a whole layer of life running around on the sand that you didn't see before.  Look even closer and you can see individual grains of sand, water and light dancing between them.  Look closer still and you see that each grain of sand is a unique object with its own texture.  If your eyes are strong enough, or you have a machine to help you, you can see even more layers by looking closer still.

That's what Twitter is going to be like with the launch of Twitter Annotations this Summer. It's a beautiful vision, with huge potential, but there's another way to look at this analogy: you don't build on the beach sand because it shifts too much. Will Annotations live up to its incredible promise?

Sponsor

What Annotations Are

Last week Twitter announced a forthcoming feature called Twitter Annotations: it's a system for almost any metadata to be connected to any Twitter message when it's published. Inside every Tweet is now a space where you could put or find anything, including links out to further instructions or larger bodies of information. That's always been the case with the 140 characters of content - but now we're talking about systematic metadata intended for machines, to augment the content. The idea is dripping with potential, but also some risk.

Isn't much of life's meaning found in the play between limits and the infinite?Twitter has been considering adding Annotations for at least two years, according to Platform Team member Raffi Krikorian. That's a relatively large portion of the company's young life. Every time a new bit of metadata was added to Tweets, like geolocation information was last Fall, the company would ask itself "should we be doing this, or should we just open up the platform for and and all metadata?"

Now the company has decided to do just that. Twitter publishing tools can now add a description to any tweet their users publish, not as a part of the 140 character message, but as a small machine-readable metadata field that travels along with the content.

What might this look like? We could see Annotations fields like:
Link to a media file, like podcast enclosures, photos linked to, etc.
Context about the Tweet like where was the author when it was published, maybe what the weather was like there at the time.
Your Twitter publishing interface could offer you a special option to write reviews of movies, books, or links you're sharing. The ISBN of the book, a link to a preview of the movie and the number of stars in your rating could be included in the Tweet Annotations.
Any way you can classify, describe, append or otherwise enrich a Tweet with words or numbers can be included in Annotations.

You Tweet, you (or more likely your Twitter app) attach a characteristic or quality, you define the characteristic and then you provide a value of how or what that Tweet did relative to the quality being referenced. Twitter clients like Seesmic, Tweetdeck and more will make it easy for users to add these annotations. Yes, this is meaningful in large part because of the 140 character limit on Twitter messages themselves, but isn't much of life's meaning found in the play between limits and the infinite?

From Annotations Come Analysis

Annotating a single Tweet is uninteresting, it's when you hit the Twitter databases and gather together all the Tweets that share a characteristic that things get exciting. When those selected Tweets can then be cross-referenced with other sets of data from outside Twitter - that's when the word fecund starts feeling inadequate.

Show me all the Tweets from my friends that have links to music and play me those songs. Twitter clients like Seesmic, Tweetdeck and others are going to make viewing that kind of data a whole lot easier.

Tweetmeme's Nick Halstead believes that Annotations will be used most extensively to communicate webhooks, links to instructions for a Twitter client to follow. He thinks it will enable game play and help Twitter start acquiring more users again.
"Because of the size of the data you can put in the annotations, I think people will come up with links to offsite resources. Seesmic is building their own platform for Windows to support plug-ins, but this reaches much further, but this lets Twitter clients augment a tweet with other services. Sf you were Stocktweets, you could attach a link in the namespace that's in stocktweets, Seesmic could follow that link back to Stocktweets and ask it how to render it. So you could put a chart and any other associated information. It's like FBML [Facebook Markup Language], the ability to embed applications inside the Twitter clients. Maybe threaded conversations. A game of Scrabble where the link points at a currently rendered scrabble board, so other people could look at the board and join in playing it. Annotations and webhooks would allow gaming to start happening on Twitter."

Halstead believes an Alpha version of Annotations could be made available to developers in a month.

How about showing me all the Tweets from anyone that are referencing the President of the United States (subject: POTUS?), analyze the sentiment in the messages, show me where those Twitter users were located and tell me how those local sentiments change over time. Send me an alert when one of those starts to shift radically.

Show me all the Tweets by people in their 20's and in their 50's (imagine an author age tag in Annotations, why not?), living near the site of a disastrous event. How do those discussions differ?

There are all kinds of interesting questions that could be tackled when the developer world's imagination runs wild on the terms of description applied to our messages.

Of course it will be tempting to draw all kinds of conclusions from this rich data. We'll surely be able to draw a whole lot of value from it. "You can learn something from almost anything," Big Data cruncher and 80Legs CEO Shion Deysarkar says. "Just give me enough data, I'll figure out something."

But let's keep in mind the words of social network scientist danah boyd, who wrote the following on her blog this morning:
Time and time again, I see computational scientists mistake behavioral traces for cultural logic...Big Data creates tremendous opportunities for those who know how to assess the context of the data and ask the right questions into it. But mucking with Big Data alone is not research. And seeing patterns in Big Data is not the same as hypothesis testing. Patterns invite more questions than they answer.

Tweet Power Politics

Twitter's Krikorian says the site will probably list "trending annotations" just like it lists trending topics today. There will probably be a wiki where anyone can find out what namespaces are being used for what purposes.

Really though, the classification system is going to be determined by the market. That's something that worries a lot of people. "People who believe in building standards are conerned about our blase attitude about how we want to run annotations," Krikorian says. He believes that the developer community will work things out for itself, just as it has in the past. "There has been a lot of emergent behavior around how to relate to tweets anyway, without our imposing much structure around it. The Twitter platform is continuously evolving - the developers will figure it out. Twitter developers iterate in public."

That's likely to be cold comfort for people focused on the power of structured data standards. Many people are calling for Twitter to embrace the well-built efforts of the Semantic Web community. Krikorian says that 90% of Twitter developers don't know what the Semantic Web is but that there's certainly room for standards lovers to work within the Annotations scheme.

It's not just about standards, either. "We need serious consideration from folks who know their stuff before we create a convention," says Teresa Boze, who suggested the American Society of Indexers in particular. It's hard to think that creating a giant living library without consulting some librarians is a good idea.

The absence of standard terminology could really be a problem. Annotations can't be changed retroactively, either. Krikorian says that major players will dominate the obvious use cases for Annotations and the company will monitor and highlight really innovative Annotations developed by people on the margins.

We'll see how well that will work.

Imagination will make the sky the limit for this publishing platform used easily by more than 100 million people around the world. But a shortage of forethought, planning and agreed-upon standards may bring that platform's aspirations back down to earth quickly in the future. Time will tell.

Discuss

The long-closed nature of Apple's iPhone OS ecosystem is coming to a head with the addition of major new restrictions on developers. If there ever was a time when the Android world had a chance to out-innovate Apple, this could be it.

Each day this week, developers have pointed out another indignity Apple's legal framework subjects them to. Could this be the pressure that gets resolved by the rise of a compelling Android offering? It seems like a long shot.

Sponsor

People creating applications on the iPhone and iPad platform are apparently no longer allowed to build in development environments abstracted from the preferred form of code, 3rd party analytics services are believed to be no longer allowed to track use of apps, Apple has baked in its own advertising platform and the essential requirement of winning Apple's permission to deploy apps on its platform is feeling more onerous every day.  

At the same time, no one else has come close to building a User Experience that can rival the iPhone and iPad.  If someone could, a grand battle could emerge.  Instead, right now it's looking ugly. On the positive side, the number of Android applications is growing faster and faster.

The Anguish

Prominant iPhone developer Dan Grigsby articulated today what could become an increasingly common sentiment in a goodbye post announcing the closure of his popular iPhone development blog Mobile Orchard:

Ask permission environments crush creativity and innovation. In healthy environments, when would-be innovators/creators identify opportunities the only thing that stands between the idea and its realization is work. In the iPhone OS environment when you see an opportunity, you put in work first, ask Apple's permission and then, only after gaining their approval, your idea can be realized.

I've always worked at the edge; it's where the interesting opportunities live. None of the startup I've created would have been possible in an ask permission environment.... I won't work in this ask-permission environment any longer.

As Google's Chris Messina put it well in some poignant speculation this afternoon, "It occurs to me that Apple is crossing a chasm. To where, I don't know. But its early proponents seem to be being left behind."

Another Perspective: Despite Its Problems, Apple's Ecosystem Remains the Best

Raven Zachary, President of leading iPhone development shop Small Society, offers another perspective.
Android needs a better OS before we'd even begin to see iPhone developers leave. I didn't fall in love with iPhone OS due to the elegance of Apple's legal terms. It's the platform that I fell in love with. It's the best mobile platform out there, and while I appreciate the analysis by the community and the hard questions being asked, I remain committed to the iPhone platform.

Of course the most probable outcome of all this is that most developers will stay where the users, the money and the best user experience are. Some will be unhappy and some will leave - but probably not enough for consumers to notice.

If only someone could build an Android device that rivaled Apple's hardware, and if the issues with different versions of Android across devices could be fixed, if the Android OS was just betteer - then there would be an incredible opportunity to lure away developers and finally get more users drawn to their applications. The iPad is really incredible though and there are a whole lot of very big "ifs" in play.

An effective challenge by Android sure feels like a long-shot right now, doesn't it?

Discuss

As computer scientists and technologists, we're used to dealing with large numbers in the abstract. But expressed in human terms, the mind-boggling numbers of the Haiti earthquake -- 250,000 dead, 300,000 injured and more than 1 million people left homeless -- are hard to comprehend.

The recovery from a disaster of this magnitude presents some important tasks for information technology: coordination of effort, triaging those most in need, and getting good data into the hands of decision makers and aid workers.

Here's a partial list of aid, relief, and rescue organizations currently in Haiti, gleaned from Wikipedia:

An Argentine military field hospital.
The Red Cross/Crescent, in various forms.
The U.S. military.
Multiple U.N. agencies.
Remnants of the Haitian government.
The French navy.
Sri Lankan relief workers.
At least 2,000 rescuers from 43 different groups (along with 161 search dogs).

A wealth of collaborators like this presents unique challenges around information fusion. Unlike business competitors or opposing sides of a war, the different groups want to share as much information as possible to achieve their common goal.

Each organization has a produced a fairly detailed picture of the parts of Haiti they are interacting with. Each organization also wants to consume every other organization's detailed knowledge of the situation. To act effectively, they need to integrate that knowledge into a common operating picture that accurately models the situation on the ground yesterday, today, and tomorrow.

Better coordination through data

Our reaction to the earthquake was to try to help in the best way we knew how. We set up a publicly available instance of our Palantir Government product, already loaded with relevant data, for use by aid workers and organizations working in Haiti. Using relevant, open-source data, we started modeling a picture of what's going in Haiti.

Our first cut was to include the locations and names of collapsed buildings, internally displaced people (IDP) camps, and Misson 4636 SMS messages, among others. We also added in map layers that let us see what administrative zone any point on the map was located in.

Having mapped the data into this model, users have access to it through a suite of visualization, analysis, querying, and collaboration tools that allow them to get useful answers to practical questions. Here are some examples:

Which administrative sectors have had the most SMS requests for food in the past 24 hours?

What collapsed buildings are suspected to contain hazardous materials?

Are any IDP camps close enough to hazmat sites to warrant special precautions? Should residents be moved?

Next: Stay ahead of Haiti's rainy season

With the infrastructure of the country destroyed, Haiti's rain and hurricane season will be more dangerous than usual. Not only are the normal structures that protect citizens from the waters gone, but people have moved out of the ruins of Port-au-Prince to hastily constructed IDP camps, some of which are sitting in the flood plains of Haiti's waterways.

The essential question facing relief workers is: Which of the approximately 2,500 IDP camps are most at risk from flooding?

In a place like the United States, an earthquake response and recovery team could engage the services and expertise of the U.S. Geological Survey (USGS), which maintains the National Water Information System. No such luck in Haiti, where the closest thing to the USGS is the Centre National de l'Information Géo-Spatiale. A quick look at the organization's website shows it didn't really make it through the earthquake.

We decided to help out. Since we're starting from square one, we put together data from the Army Geospatial Center, the U.N., NOAA, Haiti-based NGOs, a number of academic papers, and even geo-tagged photos from Flickr. The time it took to integrate this data? About six hours. Time it took to do the analysis? About seven minutes. Amount of that work that is reusable? All of it.

The best way to improve this analysis is to add detailed information about flooding, gathered from the field. We're looking to get new conduits of information into the Haiti instance as the rains really pick up.

If you'd like to help us, we're accepting new data sources, analyses, and contact with
relief organizations.

Volunteers, supplies, and goodwill are only the raw ingredients to recovery. It's the efficient and timely application of those resources to Haiti's most pressing problems that will make recovery a reality.

Related:

How crowdsourcing helped Haiti's relief efforts