Most venture capitalists obsess on the latest shiny object for the 18-34 demographic. That’s remarkably shortsighted. The aging U.S. population is a potential gold mine for entrepreneurs who can build technologies to help this huge demographic remain active and stay in their homes as long as possible.

The cohort of Americans over age 55 is massive. “This is a huge market —  $3 trillion in annual disposable income,” said Jody Holtzman, SVP of thought leadership for the AARP.

Holtzman was one of a panel of experts who spoke this week on The Rise of the Grey Market an event hosted by Redstar, a Cambridge, Mass. company that’s somewhat like an incubator. Redstar focuses on the older demographic and a few other underserved markets. So why is Redstar keen on retirees? Here’s what the panel had to say:

1. Technologies and services to foster “aging in place” will be huge
The number-one desire of most aging Americans is to stay in their homes. That age-in-place trend represents a $10 billion to $20 billion market opportunity in and of itself, Holtzman said.

Still, it’s a thorny problem.  Most senior citizens aren’t wild about having surveillance cameras or other intrusive technologies tracking them at home. And yet many of them live alone (42 percent of women over the age of 65 in the U.S. live solo), and have distant children who worry about their well-being. So research must be done to make technologies that are acceptable to them and let family members know whether they’re okay or not. The AARP estimates the unpaid value of care provided to family members by children or siblings is $450 billion annually.

“If we can blend monitoring and managing with social media tools and sharing or connecting with other people, that might be more attractive,” said Joe Coughlin, director of MIT’s AgeLab, which is working on the e-Home project, which uses wireless sensors to signal potential problems, like a stove being left on for hours.

2. When tech products hit big with older Americans, it’s often by mistake
The Wii and Apple’s iPad are big hits with aging Americans, but that was a happy accident. “Wiis are huge in nursing homes, unbeknownst to Nintendo,” said panelist Joe Coughlin, director of MIT’s AgeLab.

Smart companies will design products for older people while not condescending to them. For example, car companies should focus on personalized dashboards that will help older people drive safely longer. Similarly, car doors should be designed to allow easy entry and exit, and seats should be crafted with stronger lumbar support.

“Connected health” efforts, like Vitality’s smart pill bottle cap, could have a huge impact. That product alerts pharmacies when a refill is needed and reminds a person to take her pills on time. ”The challenge for these connected health initiative si to embed things in everyday objects,” said David Rose, CEO of Vitality, another panelist.

Also, those who design things need to make them look good. Many elderly people don’t wear their emergency alert devices because they’re ugly, panelists said.

3. Target market: baby boomer care givers
Baby boomers — people between 47 to 65 years of age — face a dilemma. Many are still caring for children while also assuming more care for aging parents. A huge percentage of that group is both willing and able to pay for services and products that will keep their parents safe and happy — and give the caregiver peace of mind.

“As parents age, life is more complex. When you turn 50 to 55, that’s the in-between stage, and your life gets exponentially more complex. How do you take people in my situation and make their lives simpler?” asked Frank Moss, director of new media medicine at MIT’s Media Lab.

4. Think services, not just products
SilverRide, a Bay Area company that started up to help retirees get to doctors appointments, quickly found its users wanted more. “They found demand pull from people was not just for rides but to introduce customers to other customers,” said Coughlin.  ”People want to connect — they want social interaction and activities. SilverRide now promotes and hosts group events.

Home delivery and check-in services, and add-ons around them, could be very big.

5. Research, research, research
The findings may surprise you — and force you to rethink your game plan.  Coughlin mentioned one research project that wired up people’s medicine cabinets to monitor their use of prescriptions. “What they found was very few people keep their meds in a medicine cabinet. It’s all about bags, baskets and boxes,” Coughlin said.

Panelists recommended talking to social scientists and anthropologists as well as senior citizens themselves to gauge behaviors and preferences. AARP makes a lot of its information public as does the MetLife Foundation, Holtzman said.

But the biggest takeaway from the event remained that this “huge bulge of people getting older is a huge opportunity for entrepreneurs and investors,” said RedStar co-founder Jeet Singh.

Some rights reserved by Marcel Oosterwijk

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Connected world: the consumer technology revolutionConnected Consumer Q2: Digital music meets the cloud; e-book growth explodesDisruptapalooza 2011: how Amazon’s Kindle is changing the portable media game

The iPad has been a success for Apple in business, apparently in spite of Apple’s lackadaisical approach to promoting its products directly to enterprise customers. But there’s a specific vertical market where the company is clearly making a concerted effort to promote professional adoption of the iPad: medicine.

Apple has a medical market manager, Afshad Mistri, who was profiled by Wired in a feature on Monday. Mistri is rare because he has a specific type of business to sell to: health care. Mistri is behind the dedicated iTunes store section for professional health care apps, has organized conferences on how to use the iPad in medicine, and is known to make house calls for medical professionals hoping to set up their organizations with iPads for use in treatment and patient care.

We have talked in the past about how iPads can help hospitals and doctors modernize their record-keeping systems. A program instituted in July offers doctors incentives for using electronic medical record (EMR) software on the iPad, and during our recent RoadMap conference, MIT Media Lab’s director of new media medicine, Frank Moss, said that “everyone’s got an iPad” at the nation’s leading medical schools these days.

Much of the iPad’s use in medical settings so far has been in the form of pilots and trials, but it’s getting ready to take off in a much bigger way. The Veteran’s Administration in the U.S. is looking at rolling out as many as 100,000 tablets across 152 hospitals, says Wired, based on the success of the 1,500 trial iPads it currently has in use. Over 80 percent of U.S. hospitals have similar trials in place, according to recent comments made by Apple CEO Tim Cook, which means that many more could soon take the plunge, resulting in a huge uptick of orders from medical organizations for the generally consumer-oriented device.

IPads can help on both sides of the stethoscope. For patients, they can act as a source of entertainment, providing a way for those who are bed-bound to escape their situation and just browse the web, play games or watch a movie privately and in comfort. Doctors can use them to consult more easily while out of office, and they increase the likelihood of uptake for EMR programs, since they make such records convenient and accessible, instead of a chore tied to a stationary desktop.

Apple’s iPad is a hit with consumers, that much is certain. But its success in health care, which, due to the slower nature of institutional adoption is only now beginning to become significant, might be the key to its remaining the king of the tablet heap. Apple can offer apps, security and a uniformity of experience both within and between medical organizations that Android devices can’t match; if the iPad becomes a tool young doctors just can’t live without, as it already appears to have become in many ways, it could be the go-to slate for healing hands for decades to come.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Connected world: the consumer technology revolutionForecast: Tablet App Sales To Hit $8B by 2015Connected Consumer Q3: Netflix fumbles; Kindle Fire shines

Thanks to the efficiency boost provided by cloud computing’s debut about five years ago, web applications can now launch on almost a shoestring budget. That’s why there are so many new web companies that deal in things such as photo sharing, daily deals websites, travel planning and the like. The truth is, it’s easier than ever to put together a web or mobile app and call yourself a startup.

But there are a few recent developments that, taken together, are creating an even more powerful efficiency boost: one that puts resources that were once limited to well-funded corporations and research universities within the reach of a new generation of startup founders. Perhaps it’s time entrepreneurs took advantage of this new environment to solve larger problems, instead of building yet another lifestyle app.

The way I see it, the big components at play here are:

New money is investing in big ideas
Even though the larger economy is rocky, there are a lot of people just itching to pour money into the next big technological thing — hence pre-launch photo sharing startups that net $41 million in funding. While many tech investors are focused on funding sure short-term bets (i.e. the tried and true realm of web and mobile apps), there’s a budding sect aggressively looking to invest in larger, long-term innovations.

Peter Thiel’s Breakout Labs is one of the most explicit examples of this. As we reported at the program’s launch last month, Breakout Labs will aim to fund nascent research proposals: opportunities too early stage or radical to attract dollars from VCs or government grants. Basically, Thiel, who recently told the New Yorker he doesn’t consider the iPhone to be a major technological breakthrough, is saying: Enough with the toys and games. It’s time for us to make something big.

Supercomputers are going mainstream
The next “something big” in tech might not require all that much money to make.

If you want to build something really complex — think aeronautics, new pharmaceutical drugs, medical devices, jet engines, and the like — you need high-performance computing (HPC). HPC solves advanced computational and scientific problems by using a massive amount of computing power to solve very complicated problems that involve a lot of moving parts.

HPC facility at Argonne National Labs (attribution below)

It has historically been so prohibitively expensive to do HPC that only entities such as governments, militaries, well-funded universities, or huge corporations have the kind of access to the machines needed for computational fluid dynamics problems and the like.

But last year, Amazon started offering HPC as a service with “Cluster Compute,” making high-performance computing available in the same way that EC2 made regular servers available in the cloud. Earlier this month, Amazon souped up its Cluster Compute offering significantly — now, Amazon’s HPC-as-a-Service offering provides access to one of the world’s top 500 supercomputers for around $1,000 per hour. Meanwhile, tools such as CUDA and OpenCL give programmers the ability to harness massive numbers of compute cores without having to learn a special programming language.

This takes HPC out of the realm of scientists and makes programming for massively multicore HPC systems accessible to software engineers. What Amazon’s EC2 did for democratizing the ability to develop scalable web apps, HPC-as-a-Service can do for democratizing the ability to solve computationally heavy engineering problems or build gigantic predictive models.

3-D printing is becoming a reality
Printed engine prototype by Mcor Technologies

Once challenging technology problems have been mastered with the help of HPC, some of the solutions will need to be prototyped and put into physical production. This is still a very labor- and cost-intensive process, which is a big reason why many startups prefer to stay in the virtual realm. But the emergence of viable 3-D printing technology is on the cusp of changing that, making it cheaper and easier than ever before to make a physical prototype of a new design.

How much of a reality is 3-D printing today? It’s now available at the consumer level with a startup called MakerBot, which makes a 3-D printer called a Thing-O-Matic. The Thing-O-Matic costs $1,200 and makes relatively simple items such as small toys and gadgets like bottle openers on demand. Three-dimensional printers from companies such as Mcor Technologies are aimed at making more complex prototypes for enterprise-level applications.

Of course, startups have the option of skipping the prototype step and selling simply the IP of their HPC-developed designs to a larger company. But if a startup wants to have more control over the production of what it has made, 3-D printing brings that much more within small companies’ reach.

What will be the hot startup of the next era?
If everything works out as it should, the smart, early stage entrepreneurs of the near future won’t be thinking about how to build the perfect restaurant recommendation app. Instead, they’ll devote their energy to designing a more efficient airplane wing to conserve jet fuel, or a tiny device that can perform real-time monitoring of kidney enzyme levels, or an even more awesome landing gear apparatus for the next Mars Rover. Starting the next SpaceX or Virgin Galactic won’t need the kind of funding that only an Elon Musk or Richard Branson can provide.

Today, the lion’s share of companies that emerge from incubators such as Y-Combinator and 500 Startups deal in consumer-focused web apps. Here’s hoping that in the near future, incubators will look for startup founders who are taking real advantage of their new-found access to serious tech tools to build bigger and bolder products. It seems to me that driving toward that kind of world is where the attention of the tech industry — and the media that covers it — should focus.

“What’s Next?” image courtesy of Flickr user Crysti

Image of the HPC facility at the Center for Nanoscale Materials at the Advanced Photon Source courtesy of Flickr user Brian Howard on behalf of the Argonne National Laboratory.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Flash analysis: lessons from Solyndra’s fallCleantech Financing Trends: 2010 and BeyondConnected world: the consumer technology revolution

EBay has announced it is buying New York startup Hunch, a recommendation engine created by Chris Dixon and Caterina Fake, to help improve its recommendation services. The purchase price hasn’t been announced, but venture capital investor and former TechCrunch editor Michael Arrington has said it was about $80 million.

EBay said it will use Hunch’s “taste graph” technology to provide its users with non-obvious recommendations for items based on their unique tastes. The company said it will also apply Hunch’s technology to other areas such as search, advertising and marketing, in order to better surface product information based on its customers’ tastes.

“We are engaging consumers in innovative ways and attracting top technologists to shape the future of commerce,” said Mark Carges, Chief Technology Officer and Senior Vice President, Global Products, Marketplaces. “With Hunch, we’re adding new capabilities to personalizing the shopping experience on eBay to the individual relevant tastes and interests of our customers. We expect Hunch’s technologies to benefit eBay shoppers as they browse and buy, and to bring sellers on eBay new ways to connect the right products with the right customers.”

Hunch will remain in New York with co-founders Chris Dixon, Tom Pinckney and Matt Gattis staying on board with the company. Fake, who also co-founded Flickr, stepped away from the company earlier this year.

Dixon said in a blog post that he and the team decided to sell because of the opportunity to apply Hunch’s Taste Graph to one of the top e-commerce leaders. He said Hunch will continue to operate a standalone site and will be providing eBay with predictive merchandising, interpreting unstructured data and creating merchant insights. Hunch had reportedly turned down previous offers for as much as $60 million from suitors including Google .
I think the purchase could improve eBay in a number of ways. For one thing, it could help the retailer better compete with Amazon, and it keeps Hunch out of the hands of its rival, which has been well-regarded for its ability to recommend products to users based on their purchase history and preferences. EBay launched its own recommendations last year to help suggest items based on past searches, but it clearly has a long way to go in matching Amazon, which moved ahead of eBay in sales last year. Earlier this year, Om wrote that Amazon should buy Hunch because of its ability to create an interest graph that can be tied into social commerce. And as retailers like Amazon and eBay build out almost limitless inventories, matching recommendations to these products becomes even more important. Here’s what Om wrote in April:

“Amazon should buy Hunch. It could use the decision engine to help customers sift through the ever-expanding array of offerings and make purchasing decisions. That little kernel of an idea still looms large in my thinking, especially as I wonder what the future of media and e-commerce looks like….Interest graph, for me, is the underpinning of a new kind of e-commerce experience. Think of it as a new kind of social commerce experience that goes beyond the notion of group shopping (Gilt Groupe, Groupon), shopping communities and recommendation engines,”

Hunch could allow eBay to create some very smart recommendations based on more than just searches. I think past search or purchase history is limiting, and sometimes an inaccurate predictor for recommendations because a lot of gifting happens on these sites, which can skew the suggestions. Hunch uses what it knows about a person’s likes and interests along with answers to personal questions to help understand a user’s tastes, then makes very intelligent recommendations based on those answers and their correlation to other data. Hunch originally began a consumer-facing service but changed course last year to license its technology to retailers who participated in its Hunch Partner Platform.

I think this could also help with eBay’s self-described goal to become a technology-driven company. The company is in the midst of a big push to become much more than just a seller of goods. You could see that at eBay’s X.commerce Innovate conference, where it showed off its X.commerce platform for retailers and developers. EBay has been on a buying binge recently, buying a number of companies that have helped it aspire to be a sort of plumbing or infrastructure for e-commerce. It has bought GSI, Magento, WHERE, Milo and RedLaser, much of which is being put into the new X.commerce platform and also its bid to enable payments in-store. Hunch is a very intelligent company that uses a lot of machine learning and data mining to create recommendations. And it has a smart team led by the well-known Dixon, who can also help build out eBay’s presence in New York

Adding Hunch could be another tool for X.commerce customers, who could plug smart recommendations into their business operations. EBay is really trying to build a commerce operating system for businesses, and it’s putting all these new assets together into one big resource to help enable sales for retailers and developers. Increasingly, retailers need better ways to engage consumers and personalize service to the tastes of their users. By providing a tool like Hunch to X.commerce customers, it can make the platform that much more attractive.

I think the move makes sense for eBay and follows through on CEO John Donahoe’s attempts to remake the online seller into a technology company. Donahoe said at the X.commerce conference that eBay is not done buying up companies and it looks like it will continue to add more pieces to its growing collection of e-commerce assets.

Here are a couple of video interviews we did earlier with Dixon:



Watch this video for free on GigaOM





Watch this video for free on GigaOM



Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Connected world: the consumer technology revolutionThe future of mobile: a segment analysis by GigaOM ProFlash analysis: Collaborative consumption – a first look at the new web-sharing economy

Although it appears significant venture capital investment in Platform-as-a-Service startups is drawing to a close, Boulder, Colo.-based Standing Cloud was able to raise another $3 million, it announced on Thursday. The money comes from existing investors Foundry Group and Avalon Ventures, bringing Standing Cloud’s total investment to $8 million. The company likely can thank its unique spin on PaaS for keeping it in investors’ good graces.

Unlike more-popular PaaS options such as Heroku or Microsoft Windows Azure, Standing Cloud isn’t so much about automation and scalability as it is about choice. Developers choose the infrastructure cloud on which they want to deploy (you name it; Standing Cloud supports it); build their application stack (i.e., web servers, database, programming language, etc.); port their code; then launch their server. Standing Cloud also hosts a variety of open-source applications that customers can choose to launch, as well, which takes the platform into SaaS territory. It’s not as sexy as what some other PaaS providers are doing, but it’s very functional and probably very approachable for a large number of small businesses.

At this point, though, PaaS is becoming passé, at least with regard to general-purpose platforms for hosting web applications. It’s not that PaaS adoption has peaked — far from it, actually — but that the market seems to be getting saturated with regard to how many platforms it can support. Heroku, Google App Engine, Windows Azure, Amazon Elastic BeanStalk and Red Hat’s OpenShift are all backed by the deep pockets of public companies. In the startup realm, DotCloud, AppFog and CloudBees are all operating with double-digit-millions in funding and are regularly expanding their capabilities. Then there’s Standing Cloud, the similarly application-focused BitNami and veteran Engine Yard.

Assuming the world doesn’t need dozens of PaaS offerings — there are only a handful of legacy application platforms that PaaS providers are trying to displace, after all — it’s hard to imagine what investors would have to see to find a new PaaS investment worthwhile. Maybe it’s a focus on a new breed of social and mobile developers, or a business model that blends SaaS and PaaS in such a manner as to make those labels irrelevant. Thankfully for anyone trying to build those next-generation platforms, projects such as VMware’s Cloud Foundry help eliminate the need to invest in foundational PaaS capabilities so companies can just focus on differentiation.

But when it comes to broad platforms targeting applications generally, it seems investors have picked their horses and are backing them to the finish line. Hopefully, that means a wave of innovation to push the cloud platform discussion to the next level.

Image courtesy of Standing Cloud.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Connected world: the consumer technology revolutionInfrastructure Q3: OpenStack and flash step into the spotlightNewNet Q2: Google closes the quarter with a bang

It’s time those front-facing cameras on smartphones were used for more than just video chatting or taking low-resolution pictures: Pantech is adding software in its phones that will allow users to control the handset with gestures. The concept is similar to Microsoft’s Xbox Kinect but on a smaller scale.

Pantech’s new Vega LTE smartphones will be the first products to gain the technology, which is powered by eyeSight, an Israeli startup we highlighted in June of 2010. Using the eyeSight solution, handset owners will be able to use a gesture to answer a call, start music playback, control games and more. The idea is that the gesture interaction is useful when the smartphone is in a hands-free mode such as driving, cooking or when the phone is sitting in a dock.

Last year, when we first heard of eyeSight, the company’s founder and CEO, Itay Katz, said this in the news release: “Users are looking for ways to ease, improve and enjoy their day-to-day interaction with their mobile phone, ideally aiming to gain effortless control of the device’s applications and functions, which is where eyeSight’s solution comes to place.”

While I’m not sold that the general public will take to gestures on smartphones like it has with the Xbox Kinect, I agree with Katz’s sentiment about users looking for improved device interaction. For those who prefer a smartphone in a hands-free use case, eyeSight’s solution should bring that improvement. But the smartphone is a device that is generally meant to be held. As such, I think there’s a limited audience for gesture controls on a handset.

Still, the news that eyeSight’s solution getting picked up by a hardware manufacturer — Pantech is one of Korea’s top three handset makers — shows that some are willing to think outside the box when it comes to user interfaces on mobile devices. And it’s not the only solution either: Take a look (or a listen?) at Siri on Apple’s iPhone 4S and you’ll see another example of a user interaction improvement.

Touch is still a key user interface on mobile devices, but with all the sensors in our smartphones, expect to see more of these forward-thinking ideas as devices begin to adopt what I call the “invisible interface.”

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Siri: Say hello to the coming “invisible interface”What the history of handset platforms can teach the future of mobileSocial media reactions to the iPhone 4S

Video editing startup WeVideo is launching a new product that will enable organizations to use its collaborative editing products in the cloud. The WeVideo Commercial product is aimed at bloggers, journalists, marketers and other video creators who wish to easily edit, manage and publish videos from a single online platform.

By putting editing in the cloud, WeVideo is taking advantage of cloud-based processing power to eliminate the need for expensive editing hardware and software. It’s also betting on collaborative editing, which allows multiple editors to make changes and to approve videos before they go live.

The platform offers a suite of tools that match most prosumer video editing software packages, and it comes with a set of royalty-free audio clips, transitions and graphics. Since all editing is done in a web browser, the platform is agnostic to the device it’s being edited on, whether it’s Mac or PC — and it can even be used to edit on smartphones and tablets when connected to the Internet. WeVideo is also agnostic as to the source file or publishing destination.

Once completed, videos can be published on a corporate website, as well as popular destination sites such as YouTube or Vimeo. WeVideo also incorporates social sharing tools to let users publish or embed videos on social networks like Facebook or Twitter.

WeVideo operates on a freemium model, and it already offers a free, consumer-based product, which it launched at the Demo conference in September. It’s also available to YouTube users as a way to edit videos directly on that site. With the launch of WeVideo Commercial, it’s giving enterprise users a much greater amount of storage (50 GB), as well as up to 1080p video resolution and 24-hour support for $79 a month.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
NewNet Q3: Facebook remakes headlines in social mediaPlayers and Strategies for Real-Time In-Stream AdvertisingManaging infinite choice: the new era of TV user interfaces

The mobile web version of Nokia Maps now looks and behaves more like a standard native application on Google Android and Apple iOS devices, thanks to HTML5: The navigation service now provides offline downloading of maps. This ability can reduce mobile broadband data charges or allow map usage in areas that have limited or no wireless data service.

Enthusiast site Android Community noted the updates on Monday by way of the HandHeld Blog. In addition to the downloadable maps, the service — found at http://m.maps.nokia.com — also adds public transit directions to supplement the existing walking and driving navigation as well as points of interest (POI) and guides to the local area.

Nokia’s mapping service is arguably one of the best software products to come from the Finland-based handset maker, and this update makes it even better. Why else would Microsoft decide to integrate Nokia Maps in the Windows Phone platform going forward? I used the web version of Nokia Maps earlier on Monday, finding it to be so full-featured that it was almost difficult to believe it to be a web application.

 LoadingNextPreviousPicture 1 of 6 nokia-maps-1-save-local

The offline mapping mode is welcome, especially when many smartphone owners pay for set amounts of wireless data. Google, too, recently introduced downloadable maps, partially for this reason. Nokia’s implementation is somewhat limiting, though, at least in my short tests. The initial geographic area I wanted to map was too large, so Nokia Maps wouldn’t save it. I had to keep zooming and cropping before saving.

The end result was a reasonable size — about 15 square blocks of Philadelphia — and I had to boost the storage limits allocated to the service to get the 19 MB area map downloaded. Nokia calls these “neighborhood maps,” so if you’re planning to visit several areas, each neighborhood will have to be downloaded separately. That differs from Google’s solution, where I was able to grab a map of 10 square miles. Once you have a local map from Nokia stored on the device, you don’t have access to the guides and POIs, but you can zoom in for greater detail, just like Google’s version.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
App Developers: Are You Ready for HTML5 and Metered Data?Mobile Q3: the fight for OS domination continuesThe future of mobile advertising, 2011 – 2016

When an investor or entrepreneur makes high-risk bets on early stage companies, failure is always part of the equation. For Kleiner Perkins Caufield & Byers Partner Bill Joy, he says more of the early-stage greentech startups he invests in with fail early, because he takes on riskier startups. But if those startups make it past the early stage, they can turn into something more meaningful and have a higher-impact on the industry and society.

“I try to have a first  [investment] round that has a 50/50 mortality rate,” said Joy at MIT’s Emerging Technologies Conference on Wednesday. So essentially the early stage startups he likes to back have a 50-percent chance of not working. “Now a 90-percent mortality rate is not a good use of my time. That’s something for the government to back,” added Joy.

Joy has 10 to 12 startups he works with at anyone time, including companies like Siluria, which converts natural gas into ethylene (a key ingredient in plastics) and Renmatix, which makes sugar from biomass that can be turned into biofuel and biochemicals. Joy joined Kleiner six years ago after starting Sun Microsystems in the early ’80s.

The switch from IT to greentech has been “a learning experience,” and Joy says he’s learned enough about physics and chemistry “to not be too dangerous to our [firm's] financial future.” Joy explained: “I felt like I’d done enough with computer industry. So I was happy to be allowed to pioneer for us in this area.”

Another key way to work with failure, is to fail cheaply. Joy says Kleiner will spend a couple million dollars backing an early stage risky company to see if they can help de-risk the investment and add value. But if the company ends up taking more like $20 million to fail, then that’s not good economics for them. Entrepreneur Bill Gross has explained the same thing before.

To find his investments, Joy said the Kleiner team thought about grand challenges that it wanted to solve, like desalinating water for 10 cents per cubic meter — today desalinating water costs closer to $1 per cubic liter, said Joy. Joy would then think about the best way to get to that goal (in the desalination problem, he points to using thermal energy instead of electricity) and then would start looking for scientists that are working on those innovations. Another way is to use the new tools that have been developed in the last decade — carbon nano tubes, robotics, ionic liquids — and apply those to older problems.

When looking at competing technologies, Joy says he uses the “secretary algorithm.” When you hire a new secretary, you can’t just pick the first one you interview because she seems pretty good — you have to get a sense of the quality of the population and then pick the one that stands out, said Joy.

At the end of the day, though, one of the problems with tackling high-risk problems and backing high-risk companies, is that some of these problems and innovations just aren’t scientifically possible. So it’s one of Joy’s jobs to determine at an early point if the company can’t be taken any farther with Kleiner’s funds. That’s the fail early part.

Image courtesy of hans.gerwitz.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Green IT’s Q4 Winners: Wind Power, Solar Power, Smart EnergyFlash analysis: lessons from Solyndra’s fallSmart Grid Apps: Six Trends That Will Shape Grid Evolution

It might sound crazy, but I think there’s an argument that Facebook could become a leading enterprise software vendor for the webscale world if the social-networking kingpin is so inclined. Over the past couple years, it has released details on a number of its internal efforts to automate and simplify the management of its massive infrastructure. As we continue to consume web applications, and cloud services and webscale data centers become more common, Facebook’s tools and expertise could be a cash cow.

During a fireside chat at Structure 2011, NetApp Founder Dave Hitz joked to me that perhaps Facebook would be the next big enterprise vendor nobody saw coming. As I was putting together the third-quarter wrapup report for GigaOM Pro, analyzing the summer’s news highlights and picking out key trends, it occurred to me that Hitz’s offhand comment might actually be right on the mark. In July, Facebook explained how it moved its 30-petabyte Hadoop cluster without taking it offline. In September, it talked about a system called FBAR that helps automate the resolution of system errors to the point that two administrators can manage half of Facebook’s massive infrastructure.

But that’s just in the last two months. In May, Facebook detailed how it moved operations into a new data center thanks in part to a homegrown configuration, provisioning and testing tool called Kobold. Over the past few years, it has blessed data types with a plethora of entirely new products and techniques, from the Cassandra NoSQL database to geographically distributed Hadoop clusters. And say what you will about its reliance on MySQL, but Facebook has undeniably done masterful work to make an old database work at a scale for which it was never designed. It also knows a thing or two about data center design.

Other companies likely would, and certainly should, be willing to pay large sums of money for Facebook’s webscale expertise. Twitter, Reddit and — just a few days into its life as a cloud provider — Apple have already established reputations for shoddy uptime. Other growing companies such as Zynga and LinkedIn, and even the next generation of webscale companies, are also going to run into the same problems that Facebook has. Why recreate the wheel trying to solve problems Facebook has already solved?

It’s already happening elsewhere. Google has converted its deep expertise in running a webscale search engine into its wide array of enterprise services that includes Google Apps and App Engine. Yahoo spun off Hortonworks to capitalize on its extensive Hadoop knowledge. These companies had developed internal skill sets in next-generation technologies, and when markets emerged for those skills, they productized them.

Systems management software and support is a huge market, but few, if any, legacy vendors have products and knowledge that easily translate into webscale environments. Facebook could stand to make a lot of money by consulting with customers on how to build their data centers and architect their applications, and then selling them the software tools to keep those apps up and running. I’m not saying it’s going to happen, but it wouldn’t surprise me in the least if it did.

Image courtesy of Facebook.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Infrastructure Q3: OpenStack and flash step into the spotlightNewNet Q2: Google closes the quarter with a bangInfrastructure Q1: IaaS Comes Down to Earth; Big Data Takes Flight

It’s no big surprise that Facebook wants to make it as easy as possible for users to share with their friends what they’re reading, listening to and watching. That was the key message at the social network’s f8 developers conference. But when it comes to video, a little-known law from the ’80s could hold back those ambitions, at least for users who want to seamlessly share what they’re watching on Netflix or Hulu.

The Video Privacy Protection Act (VPPA), which was enacted in 1987 in the wake of the Robert Bork’s Supreme Court nomination hearings (after Bork’s video rental records were released to a newspaper by his local video store), prohibits companies from sharing viewer records publicly. The act was also at the center of a class-action lawsuit aimed at Blockbuster after the rental store began sharing its records with Facebook back during the Facebook Beacon debacle of 2007.

As a result, some video companies — most notably Netflix — have shied away from integrating with Facebook’s Open Graph, at least in the U.S. While Netflix will allow users to share viewing history with Facebook friends in Canada and in Latin America, it doesn’t plan to roll out the same functionality in the U.S. until Congress removes some uncertainty around the VPPA.

The House Judiciary Committee is considering a bill that would amend the VPPA, but concerns from certain Congressmen could require users to explicitly opt-in each time they want to share what they’re watching. The bill, HR 2471, would clarify what is now ambiguous language about when and how users share their viewing information. In particular, it seeks to amend the type of disclosure necessary for companies like Facebook, Hulu and others to make such information available to others:

“to any person with the informed, written consent (including through an electronic means using the Internet) of the consumer given at one or both of the following times:

(i) The time the disclosure is sought.
(ii) In advance for a set period of time or until consent is withdrawn by such consumer.”

In other words, the amendment as it stands would now let users opt-in once to sharing information with friends and social networks. That is, until they withdraw such consent, or decide they don’t want to share anymore.

That would be a boon for Hulu, Netflix and other video distributors which seek to connect with Facebook’s Open Graph. But some Congressman have expressed concern about the lack of privacy oversight that comes from enacting such an amendment. In particular, Congressman Bobby Scott of Virginia and Congressman Mel Watts from North Carolina had voiced some concerns about the bill during the mark up hearing.

According to Tech of the Hub:

“Congressman Watts is concerned the current bill trades convenience at the expense of privacy. He offered an amendment to require a consumer’s permission explicitly each time video viewing data is shared as opposed to the current bill’s up-front one-time consent… Watts stated the bill ‘does not adequately address the realities of privacy in this age of instant and wide-spread information distribution and consumption.’”

Those concerns are due to the “dynamic nature of friend lists on social networks as well as the lack of safeguards for children.” Under Netflix’s current proposed implementation of Facebook Connect, for instance, only the Netflix account holder would be able to share his or her viewing history on the social network. But many Netflix accounts are shared within a household, which means that the history of multiple users could be broadcast under a single Facebook account — and as a result, a child’s viewing history can be shared without any controls placed on it.

It’s not clear which, if any, version of the bill will be put forward or if it will be passed. But it’s important to note the concerns about the effect that the sort of frictionless sharing would have on social networks, particularly as Netflix, Hulu and others haven’t implemented a satisfactory way to enable multiple Facebook-authenticated users to share the same account.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Connected Consumer Q3: Netflix fumbles; Kindle Fire shinesConnected Consumer Q2: Digital music meets the cloud; e-book growth explodesA 2011 Connected Consumer Forecast

Cloud-based DNA-sequencing specialist DNAnexus has closed a $15 million second round led by Google Ventures and TPG Biotech. Elsewhere, we learned Wednesday that agribusiness giant Monsanto has deployed Cloudant’s NoSQL database as the underpinning of the company’s genomics system. Big data technologies, it seems, can now include biotech — and genomics, specifically — among their many killer apps.

Innovation blowing past Moore’s Law
In a recent interview, DNAnexus Co-Founder and CEO Andreas Sundquist explained the opportunity he sees for his company’s services, which include cloud-based storage and processing of DNA-sequencing data. The problem and the opportunity genomics researchers face is that innovations in the field are “outpacing Moore’s Law,” he said, which has resulted in the cost of a DNA profile being pretty much on par with that of any other standard medical test. Soon, everyone will have DNA profiles as part of their medical records.

This “will change the way medicine is done” and could grow into a hundred-billion-dollar market, he explained, but it also will result in lots of data generation: hundreds of gigabytes per person. Whereas the high-end research facilities might have access to high-performance computing and storage necessary to perform DNA sequencing, the hospitals that will now be doing those analyses on a regular basis certainly will not.

And that’s where Sundquist thinks a service such as DNAnexus becomes indispensable. Leveraging the storage capacity and processing power of Amazon Web Services and Google Cloud Storage, his company provides sequencing facilities and researchers with the infrastructure and the software to run the analyses and display the results. Because of its investment relationship with Google and the sheer scale of its operations on AWS, Sundquist said DNAnexus actually works very closely with both companies.

However, unlike some areas where analytics are primarily focused on algorithms because access to storage is just a matter of buying more commodity gear, capacity is still a big issue in genomics. By using the cloud, Sundquist said DNAnexus lets customers share and collaborate on data without actually transferring hundreds to thousands of gigabytes.

It also helps with DNAnexus’ latest undertaking: hosting the Short/Sequence Read Archive. The comprehensive set of sequencing data was hosted by the National Center for Biotechnology Information but was slated for sunsetting because of budget cuts.

That database is currently at about 400TB, but Sundquist says that’s just the tip of the iceberg. He said DNAnexus has actually “blown past” analyzing data at the Hadoop/MapReduce scale and is now focusing on parallelizing computation across 100,000 nodes and scaling its storage infrastructure into the exabyte range.

Genomics aren’t just for humans
Say what you will about Monsanto’s business in genetically engineering food, but it does involve high science on par with DNA sequencing in humans and presents many of the same data problems. That’s why Monsanto deployed Cloudant’s BigCouch database as the focal point of its massively distributed genomics system.

According to Mike Miller, co-founder and chief scientist of Boston-based Cloudant, NoSQL offerings such as his company’s CouchDB-based product are actually ideal for the genomics space because they allow for cheap, horizontal scalability and high throughput. Cloudant is particularly well-suited, he explained, because of the incremental MapReduce engine built into BigCouch.

Miller compares it to Google Percolator, the data-processing framework Google recently deployed to replace its legacy MapReduce system. Whereas traditional MapReduce implementations such as that found in Hadoop are designed for batch processing, Percolator and Cloudant’s MapReduce implementation enable near-real-time analysis because they let users process data as it enters the system and update the dataset accordingly.

This is important for Monsanto because BigCouch isn’t just an analytics system, but an operational database serving a wide variety of users. Some users who aren’t data scientists, but consumers of the data, need up-to-date information and must rely on the system to provide it.

Ultimately, Miller paints a picture of DNA sequencing very similar to what DNAnexus’ Sundquist does. Innovation is rampant, but data growth is outpacing the ability to analyze it, making faster, cheaper and more scalable data systems integral to leveling the playing field. If they can help bridge the gap between the data and the algorithms to analyze it, Miller says, “We’re going to see things in the space beyond our wildest dreams.”

Feature image courtesy of Flickr user micahb37.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes FlightDefining Hadoop: the Players, Technologies and Challenges of 2011Putting Big Data to Work: Opportunities for Enterprises

Thanks to smartphones and wireless networking and SMS and Twitter, we are all swimming in an ocean of real-time information — an ocean that can often seem overwhelming. Are new technologies going to help, or are they going to increase the problem? A presentation at the recent Society for News Design conference included a video called “The Storm Collection,” whose creators imagined a near future in which real-time updates about a news event would be shown in virtual heads-up displays on picture frames, car windshields and even eyeglasses. But would this kind of technology make our information-overload problem better or worse?

In the video (embedded below), the two creators — Matt Thompson, editorial product manager at National Public Radio, and writer/blogger Robin Sloan, who is also part of the media partnerships team at Twitter — talk about some of the technologies we already have for getting live updates about something like a tornado hitting a small town: SMS for text messaging people, mobile media websites with official information, Instagram for sharing photos of damage, and Twitter and Facebook for talking about the event and posting status updates, pictures, videos and so on.

Want to watch a video from an unmanned drone?
What might things look like a few years from now? Among other things, Sloan and Thompson imagine photo frames that include news-related and location-based updates about the person who appears in the picture (“Sarah checked in at Tornado Safety Zone”), televisions and video terminals that show a real-time video feed from unmanned drones flying over the damaged areas, and heads-up displays embedded in car windshields and eyeglasses that give live updates about damage near the user of the device (“Downed power lines in area, please use caution”).

I have no doubt that the things that Sloan and Thompson are describing are close to becoming reality, if they aren’t already. And the benefits of having a photo frame that could update you about the location of the loved one in the picture in case of emergency — by using GPS or a location-based service such as Foursquare, presumably — seem pretty obvious. And if anyone likes a constant stream of real-time news and information about the world, it’s me; I am connected to Twitter almost all the time (as my family will tell you), and I use Instagram and Facebook and plenty of other social services to keep track of what friends and family are doing.

But I wonder whether a world like the one portrayed in “The Storm Collection” would be a positive thing for many people, and particularly those who already feel overwhelmed by the waves of information that they are already subjected to — whether it’s 24-hour TV news programs, or Twitter and Facebook, or all the other real-time sources we are all bombarded by throughout the average day.

The biggest issue is one that media analyst and journalism professor Clay Shirky has described, when he said the problem with the digital age isn’t so much information overload as it is “filter failure.” As Robin Sloan is no doubt aware, Twitter is a great example of this phenomenon in action: it allows you to follow the comments of hundreds of even thousands of people, as I do, but it doesn’t really provide all that many great ways of filtering that content so that it is manageable. Trending topics is one way, and lists are another, but neither of these provides a great solution.

How do we deal with “filter failure?”
Twitter isn’t alone in this; Facebook has implemented “smart lists” and now allows people to “subscribe” to others in the same way Twitter does, but in some ways this actually exaggerates the problem instead of solving it. Google+ has Circles, which allow users to create mini-groups that they can either follow or post their comments to, but many people don’t use them (just as many people don’t use Facebook or Twitter lists, which have been around for a while now). As a result, it’s still easy to become overwhelmed by the “activity stream,” whether it’s Twitter or Facebook or Google.

To come back to “The Storm Collection” example, it looks as though Sloan and Thompson see most of the real-time news updates and unmanned-drone videos and so on coming from media outlets (in this case the Fresno Bee, owned by McClatchy). But the filtering problem exists for media companies as well; in fact, in some ways, it’s worse, because they are taking in so many other sources of content in addition to Twitter and SMS and photos. This is complicated by another phenomenon Sloan and Thompson dealt with in a 2004 video project called EPIC 2014: the explosion of “user-generated content” or citizen journalism that blogging and other tools allow.

As a post from a BBC “user-generated content” editor described earlier this year, filtering and making sense of this never-ending stream of information isn’t an easy task: The UGC desk at the British broadcaster has a large staff that try to verify and “curate” news reports from Twitter and Flickr and YouTube about events such as the revolutions in Egypt. Andy Carvin of NPR has turned his personal Twitter stream into a one-man newswire of curated and verified “citizen journalism,” but even his output is so massive in some cases that it needs a second set of curators using tools like Storify to filter what he has already filtered.

In the presentation to the Society of News Design, Sloan said: “If the world is suddenly this new terrain full of all these new screens and all these new ways to get stories out there, [journalists] should be in the business of identifying rich new territory, sending out scouts, and seizing it.” In the end, the problem may not be identifying or seizing these new territories, but coming up with ways to prevent them from becoming just another piece of flotsam in a never-ending sea of real-time content.

Post and thumbnail photos courtesy of Flickr user Ed Kohler

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Players and Strategies for Real-Time In-Stream AdvertisingWhy Google Should Fear the Social WebFacebook and the future of our online lives

Members of the OpenStack community are, for the most part, pleased by news that RackSpace will relinquish control of the OpenStack effort but want to see how the foundation is set up and staffed before endorsing it wholeheartedly.

Any foundation must replicate and build upon successful past open-source group efforts such as the Apache Software Foundation and the Linux Foundation, said attendees of the OpenStack Conference in Boston. They said lingering concern over Rackspace’s power of the OpenStack effort to date drove the move.

“The model is there for this. The Eclipse Foundation did an awesome job–but IBM had to get out of the way first,” said a software developer for one of the Linux distributions who was at the conference.

Whatever the foundation’s makeup, the goal is to keep the resulting platform open and evolving, said Chris Kemp, CEO of Nebula and former CTO of NASA. Rackspace said the transfer of the OpenStack intellectual property and trademarks should be done by 2012.

The challenge is huge as OpenStack partisans position their effort as the open-platform equivalent to Amazon Web Services and VMware efforts.

“If this platform does not emerge as the Linux of the open data center, then we’ve done the wrong thing,” Kemp told attendees Thursday morning.

“I will bluntly state when I see a company like Sun acquired by Oracle, whose  value is a completely integrated platform, that’s scary. It locks everybody else out of that ecosystem, and Exadata becomes ‘Exadollar’ to lock out innovation, interoperatiblity and portability,” Kemp said. Exadata was the first of a handful of hardware-software data center appliances launched by Oracle in the past two years.

The anti-Oracle meme surfaced several times both in keynotes and in attendee discussions. Lew Moorman, president of Rackspace’s cloud effort, flashed a photo of Oracle’s glossy headquarters building and referred to it as “the high temple of lock-in.”

“There’s a lot of work to be done and there’s already been a lot of politicking behind the scenes,” said an executive with a hardware company that backs OpenStack who would not speak for attribution. “It depends on who the leadership is and the devils are in the details,” he added.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes FlightInfrastructure Q2: Big data and PaaS gain more momentumA field guide to cloud computing: current trends, future opportunities

Karmasphere, a startup focused on helping analysts write better big data applications, raised $6 million in a Series B round from Presidio Ventures, Hummer Winblad and US Venture Partners. That brings the Cupertino, Calif.-based company’s total funding to $11 million, and the announced VC investment in Hadoop to more than $30 million in the last 30 days.

As the old saying goes, you’d have to have been living in a cave for the past year to have not heard about Hadoop, the open-source, data-processing engine. Once a darling among large web sites such as Yahoo and Facebook, Hadoop has spread like wildfire into web companies of all sizes and, thanks in part to Cloudera’s presence, into banking, intelligence and other traditional industries. So promising is the technology that IBM, EMC and now Oracle are getting into fray as well.

Karmasphere’s Analyst product sits above the core Hadoop distribution platform layer where companies such as Cloudera, EMC and MapR play, instead helping companies write applications, or workflows, that run on Hadoop. As I explained last week in covering the latest version of Karmasphere Analyst:

Karmasphere is trying to woo data analysts who are well versed in working within data warehouses but still need some guidance to translate that knowledge into a Hadoop environment. To that end, the company has devised a workflow for accessing, assembling, analyzing and acting upon big data.

Every VC I’ve spoken with in the past several months has touted big data as an area of particular interest, so I have to assume the Hadoop-funding train is just getting started. Hadoop is just a portion of the greater big data puzzle, but happens to be the focal point right now and a great core around which to launch a company and get some cash.

Feature image courtesy of Flickr user zzzack.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes FlightDefining Hadoop: the Players, Technologies and Challenges of 2011Putting Big Data to Work: Opportunities for Enterprises

Salesforce.com GM of Platforms (and former Heroku CEO) Byron Sebastian

Last week, Facebook and Heroku announced a partnership through which Facebook developers could easily launch applications on Heroku’s cloud Platform-as a Service via the Facebook development portal. That appears to have been a smart partnership for Heroku, which reports it saw more than 33,800 Facebook applications launched on its service since the social network giant unveiled new features at yesterday’s f8 conference.

On the official Heroku blog, Adam Seligman notes “that’s more than 20 [applications] a minute. Facebook has again innovated and captured the excitement of the developer community.”

However, in the comments to both Heroku’s post and on Hacker News, there’s some debate over whether these are “fake apps” launched to get access to the new Timeline feature. It’s difficult to tell, especially because developers don’t need to launch on Heroku to access those features, some commenters claim.

Assuming at least a good portion are actual applications, though, such a large number is also a ringing endorsement for PaaS, in general, which increasingly appears to ideal for developers wanting to build and launch lightweight applications. For individual developers, PaaS is a way to host an application without getting caught up in systems management or other low-level concerns. Enterprise developers get the same benefits, even if they only utilize right now for non-mission-critical Facebook or mobile applications.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes FlightInfrastructure Overview, Q2 2010Infrastructure Q2: Big data and PaaS gain more momentum

We learned a lot about two people’s lives during Facebook’s f8 keynote on Thursday: Mark Zuckerberg and Nicholas Felton. Zuck showed off photos of his dog and his childhood to demonstrate the company’s new Timeline feature, which made sense. But why was Facebook VP of Product Management Chris Cox talking so much about Felton’s life, elaborating some 10 minutes about how Felton really loved to collect data that would bore most other people? And who the heck is Nicholas Felton?

In short, he’s the god of data visualization. Felton’s work has been featured in Wired, the New York Times  and the Wall Street Journal. He’s received props for turning unwieldy things like all the things that have been written about on Wikipedia or abstract concepts like the economy of ideas into compelling infographics. But he’s probably best known for the mundane aspects of his everyday life. Like the music he listens to and the kind of booze he drinks.

The kind of music Nicholas Felton listened to in 2008 to, as visualized by Nicholas Felton.

The reason for that is that Felton has been turning this kind of data into annual reports about his own life and the life of people close to him ever since 2005. These annual reports contain beautiful data visualizations, which is why Cox called them works of art on stage on Thursday.

However, the reports also provide interesting clues on how to utilize and get value out of data that others would simply discard, which is exactly what Facebook’s new Timeline feature is about. When working on this, Facebook had to decide how to make sense of a person’s life, how to summarize some behavior and feature other events. No wonder they turned to Felton for advice. The data visualization expert was hired by Facebook in April — and on Thursday, it looked like this key hire more than paid off.

Image courtesy of (CC-BY-SA) Flickr user poptech.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Flash analysis: the tech startup investment environment, Q3 2011Millennials in the enterprise, part 2: benchmarking IT’s readiness for the new digital workforceMillennials in the enterprise, part 1: strategies for supporting the new digital workforce

With the growing number of social networks out there, users have plenty of ways to share the minutiae of their daily lives. But few are positioned to help them collect and share rich family memories. That could change with the launch of Storytree, which aims to provide a platform for users to share stories with their family and friends.

Storytree, which is launching at the 500 Startups demo day Tuesday, was borne out of founder Matt Sullivan’s time at the Stanford d.school. The project emerged out of his Master’s thesis, which was centered on finding ways for the elderly to share their wealth of rich family stories. In a phone interview, Sullivan said he spent a lot of time at retirement homes, and found there were few tools available that made it easy for them to do so.

Originally the team, which is made up of Sullivan and co-founder Zach Weiner, were thinking about a system that would have family members mailing cheap video cameras to each other and to “hubs” where the videos would be uploaded, put together and made into DVDs that could be shared. But the web and mobile apps offered a cheaper and quicker way for family members to communicate.

With Storytree, users can create private pages, then tell their own stories through pictures, audio and video, which they can then share with friends or family. Or they can ask other members to contribute their own stories by emailing them. The whole system works entirely in the web browser, letting a user record a video memory with a webcam or record audio from his or her computer’s built-in mic. Soon, Storytree will also have an iPhone app, allowing users to record and add to their stories from the mobile phone. The hope is to get users to share stories based on “trigger events,” like a wedding or birth of a child, to share stories.

While the company is still in beta, it’s exploring different revenue models. Sullivan said he was considering a freemium model whereby subscription users would get additional features, like the ability to export to DVD. The company is also looking to make its platform available for larger organizations to aggregate video memories of regular users.

Storytree has received some funding as part of the 500 Startups accelerator program, as well as the program’s Design Fund. The startup has two full-time employees — Sullivan and Weiner — but is looking to raise more money and expand its team.

Storytree: Remember the Time from StoryTree on Vimeo.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
3-D Untethered: A Look at Mobile 3-D TechnologyFlash analysis: Collaborative consumption – a first look at the new web-sharing economyMobile payments: forecasts, technologies and opportunities

Step right up to test your app!

No, this isn’t some kind of carnival pitch. It’s a new product offering from Mu Dynamics called blitz.io that allows developers building in the cloud to load test their apps for as little as a buck. With blitz.io Mu Dynamics, which has a product for ISPs that shows them how 1 million people downloading Netflix or playing FarmVille might affect their networks, is branching out into the cloud sector.

Kowsik Guruswamy, the CTO of Mu Dynamics, said today that the company realized there was an opportunity to take what it calls its blitz infrastructure used to test how apps will affect ISPs’ networks and offer load testing for developers building in the cloud. For a detailed look at how Mu built the blitz platform, check out Guruswamy’s post here. Friday, Mu said it could offer developers an hour of testing time on the Heroku platform for just $1. (It has other pricing plans as well.)

So now, developers using the Heroku or Red Hat Platforms-as-a-Service can deploy their apps and use Blitz.io to terrorize them with hits to see how well they scale. Developers get back results that tell them how many hits their app can handle. And because this is a simple and cheap test to run, they can do it as often as they want, trying it out after they upload new code or just running it over and over again as the develop tries to optimize an aspect of the application.

Calling the current way of building apps for the web or mobile devices one of “continuous deployment, Guruswamy says the iterative (and cheap) approach offered by the Blitz platform works with the way people are building out apps. It used to take six months to build an app and a month to test it, but that’s not as relevant to an entirely new crop of developers that are building on various clouds. It’s a point emphasized by the emergence of several startups such as Parse or Kinvey that are offering a variety of back-end or development services to speed up the process of building out new mobile apps or web services.

And for those thinking how it might be awesome to prank your fellow Y Combinator buddies by spending a buck to test out their app on the sly, Guruswamy says the company makes you prove the app you want to load test belongs to you. He expects to launch service for new platforms on an ongoing basis, so if your PaaS isn’t there yet, it may be soon.

Image courtesy of Flickr user Loozrboy.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes FlightInfrastructure Overview, Q2 2010Infrastructure Q2: Big data and PaaS gain more momentum

DreamIt Ventures, a Philadelphia-based accelerator program, graduated its first New York class yesterday, sending 14 new start-ups into the wild. I enjoyed the demo day and the investor pitches and I find it always inspiring to see entrepreneurs making a play for the big time.

DreamIt’s class tackled education, e-commerce and app publishing and a few used video conferencing in interesting ways.

Here are three that stood out to me:

Take the Interview CEO and co-founder Danielle Weinblatt

Take the Interview. Hiring manager and recruiters often lose time winnowing down the field of applicants to a handful they can bring in for real interviews. Often, there’s a lot of time devoted to scheduling and conducting phone screening interviews. What Boston-based Take the Interview does is provide a video interviewing platform that allows companies to solicit up to five video-recorded answers from applicants that they can review.

It allows HR teams to screen candidates more quickly and get a better feel for people using a medium that many younger applicants are already used to: video. The videos can be organized by question and can be shared and commented on within an organization. Recruiters can also pull up an applicant’s LinkedIn profile while watching a video. Take the Interview launched this summer and has about 80 companies participating in its beta, including Deloitte and Bausch + Lomb. Monster and Craigslist have also started embedding a Take the Interview button on their sites.

I like Take the Interview, because it seems like a simple way to get gain more valuable information from candidates and use video to really see if there’s a potential fit with an applicant. And as video conferencing and webcams become a common way for people to communicate, it makes sense to leverage that more for hiring.

Re-Vinyl. This Los Angeles-based start-up is taking on the demise of albums with an application that allows labels and artists to create their own digital albums complete with cover art, streaming music, lyrics and liner notes. There’s also an opportunity for high quality advertising within the app including promotions for the artists themselves.

There’s also a chance to include more interactivity with fans using a Remix feature, in which an artist can ask questions of their audience and fans can respond by dragging and dropping media into the app. Right now, Re-Vinyl is working with artists to create apps, which can take about a week, but the company is poised to release a self-serve tool to allow artists to make these apps on their own.

That, I think, will be cool because it will allow serious and also budding musicians to create next-generation digital albums. We’ve seen some of the promise of this kind of interactive multimedia apps from Push Pop Press and 955 Dreams. But this could be, if it catches on, it a way to help musical albums make the transition to the app world. Re-Vinyls plans to make money by selling Vinyls or subscriptions, taking a percentage of advertising revenue or through sponsored content.

AppAddictive CEO and founder Michael Onghai

AppAddictive. While brands move to social media platforms to advertise and engage their users, it’s not always easy or very effective, especially as they try to branch out to multiple networks. AppAddictive, based in New York City, is working to solve that with a marketing platform that allows companies to manage their content, ads and analytics on multiple networks. It comes down to AppAddictive’s ability to engage with users using mini-apps like quizzes, contests, coupon offers, video galleries.

That makes a brand’s page more dynamic and interesting and allows them to gather more information from users on what they like and are interested in. And by leveraging social networks, it can bring in new customers through viral channels for less money.  Founder Michael Onghai, one of the first employees at Geocities, left his hedge fund job to give AppAddictive a try. He said already 5,500 companies are testing the apps and 50 million people have tried one of the apps. The company plans on making money through performance-based advertising.

I think as advertisers look to tap the power of social media, having a tool like AppAddictive makes some sense. They can make their brand pages smarter and more engaging and they can gain valuable information on their users while lowering the cost of user acquisition.

I also enjoyed Hoot.me, an interactive learning platform built on top of Facebook that allows users to study with friends, connect via multi-person video chat and bring in tutors to aid in learning. KeepIdeas is also interesting, allowing people to organize recipes and food content in the cloud. You can view a longer list of all the graduates here.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Mobile payments: forecasts, technologies and opportunitiesCleantech Financing Trends: 2010 and BeyondWhy Startup Visas Don’t Go Far Enough

It looks like we can add security to the growing list of killer apps for Hadoop. The open-source, data-processing tool is already popular for search engines, social-media analysis, targeted marketing and other applications that can benefit from clusters of machines churning through unstructured data — now it’s turning its attention to security data.

At a high level, using Hadoop — or any big data tool — to sniff out security problems makes sense for the exact same reason to use Hadoop to do anything: Organizations have lots of data, and they’re trying to glean as many insights as possible from it.

I think targeted applications for specific industries and uses will help drive Hadoop and other big data tools into mainstream businesses, and security is a great starting point. Security concerns are universal, and anything that can bring a proven and white-hot technology like Hadoop to bear on them should garner serious attention.

Chris Hoff, senior director and security architect at Juniper Networks, recently wrote on his Rational Survivability blog about the problems and promise of big data as it relates to security. Essentially, he argues, even though it has been possible for quite a while to capture and analyze security data, the effectiveness has always been limited because it was difficult to draw insights outside the realm of the security applications from which the data was coming.

“Even when we do start to be able to integrate and correlate event, configuration, vulnerability or logging data, it’s very IT-centric,” Hoff explains. “It’s very INFRASTRUCTURE-centric. It doesn’t really include much value about the actual information in use/transit or the implication of how it’s being consumed or related to.”

Hadoop addresses this problem because it serves as a central repository for all of an organization’s unstructured data. Security-specific data can be analyzed against other data sets to create a clearer picture of what’s really going on, and how it might be affecting other parts of the business. However, leveraging new technologies such as Hadoop isn’t necessarily an easy prospect if organizations don’t already have in-house data analytics expertise.

Unless, of course, someone were to create an application that’s tuned to address a specific problem using a new technology, but that buries its complexities within the business logic.

That’s exactly what’ starting to happen with security. Tuesday, for example, big data startup Zettaset announced its Security Data Warehouse, which is designed specifically to mine security data using Hadoop as the storage and processing engine. As part of Zettaset’s greater Data Analytics Platform, the new product ties into an organization’s entire data warehouse and carries on the legacy of making it easier for them to process, visualize and consume that data. Hadoop is at the core of Zettaset’s products, but users interact with a specialized API and management interface designed to be more intuitive than using Apache Hadoop alone.

A few months ago, I covered another startup, called ipTrust, that uses Hadoop to process web-traffic data and assign reputation scores to IP addresses. When deployed as a service or within a security vendor’s firewall product, ipTrust makes it easier to determine how risky it will be to allow network access to traffic from certain points. But security administrators interacting with the product have no interaction at all with the Hadoop cluster that powers it.

And that’s the trick. Hadoop is helpful by itself, but it’s a lot more helpful if someone else takes out the guesswork of managing the cluster and figuring out how to put it to use for a specific task. Whether pointed at security or some other workload, applications that bury Hadoop under specialized interfaces and algorithms, or that deliver its results as a service, are the future of big data for most businesses.

Feature image courtesy of Flickr user Paul Keller.

Related research and analysis from GigaOM Pro:Subscriber content. Sign up for a free trial.
Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes FlightPutting Big Data to Work: Opportunities for EnterprisesWill Hadoop Vendors Profit from Banks’ Big Data Woes?

The size of Hadoop deployments appears to have tripled since October, according to statistics from Cloudera. If accurate, they help prove assumptions that Hadoop usage grows quickly once organizations wrap their heads around how to use it. This seems especially true in industries where better customer insights are tied directly to more revenue.

In his DBMS2 blog this morning, database expert Curt Monash quotes Cloudera VP of Customer Solutions Omer Trajman as stating that his employer counts 22 Hadoop clusters (not counting non-Cloudera users Facebook and Yahoo) in production that are managing more than a petabyte of data apiece. Additionally, Trajman said the average cluster size has grown to more than 200 nodes, roughly tripling since the company conducted a survey of attendees at its Hadoop World conference in October 2010.

I relied in part on that Hadoop World survey in writing a GigaOM Pro report in March (subscription required). What’s interesting, if Trajman’s stats are accurate, is how fast the high end of Hadoop clusters has grown. At that point, the average cluster size was about 67 nodes, but it was less than 40 nodes once you excluded three outliers claiming 999 or 1,000 nodes. Trajman notes that most clusters are still less than 30 nodes, but the increasing number of large clusters is boosting the average size.

Presumably, there are also a greater number of those starter 30-node clusters, which eventually will grow closer to the 200-node average.

This aligns with the findings of a Karmasphere survey conducted contemporaneously with the Hadoop World survey. In the Karmasphere survey, 32 percent of respondents expected to be managing a cluster between 10 and 49 nodes in a year, compared with 36 percent claiming they presently were managing a cluster in that range. However, 61 percent expected to manage clusters between 60 and 1,000-plus nodes in a year, compared with only 32 percent managing that size cluster at the time of the survey.

Also noteworthy is Trajman’s observation that the biggest population of large clusters is in the advertising space. This shouldn’t be surprising: Using analytics to improve targeted advertising directly results in more money. We have highlighted this trend before, specifically as a part of video-advertising strategies, and, earlier this week, as a component of social-media-based ad targeting.

Of course, Facebook, Yahoo and Google (which uses its MapReduce-plus-Big-Table system, on which Hadoop is based) also rely heavily on Hadoop to serve the advertising programs that drive their revenues. And they all have huge deployments spanning multiple clusters around the world. Yahoo now manages more than 42,000 Hadoop nodes.

The amazing thing about this purported growth is that we’re just getting started. Many see a market that is over a billion dollars for Hadoop products, but it’s nowhere near that right now. As Hadoop best practices and technologies improve along with the overall understanding of big data across industries, who knows how large deployments will become. Perhaps today’s SMB will be tomorrow’s Yahoo.

Feature image courtesy of Flickr user miheco

Related content from GigaOM Pro (subscription req’d):
Defining Hadoop: the Players, Technologies and Challenges of 2011Infrastructure Q1: IaaS Comes Down to Earth; Big Data Takes FlightPutting Big Data to Work: Opportunities for Enterprises

Over a year ago on this blog, I posed the question, Can Networking Be Made Cool Again? A lot has changed since then. The pendulum has swung back, and now networking is hot. In the past year Cisco hit a major speed bump, the OpenFlow networking protocol has burst into the popular consciousness and the first new wave of startups has emerged to meet the changing requirements of the large-scale, highly virtualized data center network.

Many have been waiting for it, but this year, the cracks in Cisco’s dam finally burst open for all to see. The networking giant has lost $60 billion in market cap in the past year, hammered by Wall Street for its flirtations with consumer, collaboration and other noncore businesses. Simultaneously, Cisco flubbed the emergence of the web-scale data center customer. Its UCS servers and Nexus switches are out of touch with these customers’ performance requirements — laden with features they will never use yet must pay handsomely for.

The emergence of OpenFlow
In the same time period, OpenFlow has emerged from the Stanford labs and exploded on the scene. OpenFlow is a protocol that defines how external controllers communicate with switches to program their forwarding tables. The reaction of the networking industry reminds me a lot of what happens when I put a shiny new toy in front of my six-month-old daughter. First she stares at the toy intently, then bats at it a few times, ultimately succeeding in grabbing hold. Then she jams it straight into her mouth. We are currently still in the frantically swatting stage. You’ll know the industry has jammed OpenFlow into its mouth when you see Cisco supporting third-party controllers programming their switches.

Despite the purported revolutionary nature of OpenFlow, the concept of separating the control plane from forwarding is nothing new. One example of this architectural practice is the use of a parallel signaling network for call setup and teardown in the plain old telephone system. An OpenFlow control serves a conceptually similar function for packet switches. It’s ironic to see this decades-old voice concept now wreaking havoc in the data world.

But just because the concept isn’t new doesn’t mean this isn’t an important development. However, OpenFlow itself is just an interface protocol. As Nicira CTO Martin Casado put it, “OpenFlow is about as exciting as USB.” What matters is the value you can deliver in the controllers leveraging it and the potential change in industry structure enabled by the standardization of this interface. Standardization and abstractions allow the emergence of third-party controller vendors who don’t make switches. This is the truly exciting development and the one that scares the incumbent switch vendors, no matter how much lip service they pay to embracing OpenFlow.

Don’t look for the VMware of networking
In most of the discussion around OpenFlow the market seems infatuated with trying to identify the VMware of networking. This makes for splashy headlines but is a flawed analogy. What is really needed before that concept can take hold is the Linux of networking. I am sure that that won’t be the network operating systems from the major vendors, including Cisco’s NX-OS, Juniper’s JUNOS, or Arista’s EOS.

All of this virtualized software is great, but let’s show some respect to the physical layer. Web-scale data centers still need switches — big, dense switches to connect servers. Interop was the coming-out party for OpenFlow, and the number of vendors that showed up for the OpenFlow Lab at Interop this year was impressive. Cisco and Arista were conspicuously absent. Not surprising, since the whole concept behind OpenFlow — taking the control plane and routing computation physically out of the router — is a scary thought for them.

I suspect the next shoe to drop is that the networking industry gets Open Computed. It is inevitable that an open-source hardware architecture for the large chassis switch gets released, likely driven by a consortium of large customers. When combined with the external software control enabled by OpenFlow, this will really shake things up. Once this happens, it will be akin to the shift that took place in the server and database layers in the transition from Web 1.0 to Web 2.0. Early large-scale websites were built with Sun servers and Oracle databases, while today commodity servers and open-source databases are the norm in many environments.

Hurry up and wait
While the disruptions coming to the networking industry are real, it is important not to overhype the speed at which they will take place. These transitions take time, lots of time; Cisco and Juniper are amazing companies, full of world-class technologists and with massive installed bases of customers. They are not going out of business tomorrow because of OpenFlow, nor are they going to sit idly on the sidelines. This isn’t the first time a new technology emerged that purported to be a major threat to Cisco’s dominance. MPLS is a good example of a technology that was supposed to do serious damage to Cisco but that it ultimately embraced and came to lead.

It has been exciting to see networking back in the spotlight again, and I suspect the next year will be even more exciting and dynamic. Two promising startups, Big Switch and Vcider, launched this week at Structure 2011, and a number of others are still percolating in stealth mode.

Perhaps the most exciting promise of the software-defined networking movement is simply that it will increase the velocity of innovation in the industry. As the promise of these new technologies is delivered in the next 12–24 months, they will find use cases that are not restricted to the largest web properties but reach into other applications, including network security and potentially all the way down into the enterprise wiring closet. It has the potential to disrupt and reshape the industry. This means networking is cool again.

Alex Benik is a principal at Battery Ventures.

Related content from GigaOM Pro (subscription req’d):
The Structure 50: The Top 50 Cloud InnovatorsInfrastructure Overview, Q2 2010How Much Integration Is Too Much in the Cloud?

iPad users aren’t stingy with their devices, according to a new usability report by the Nielsen Norman Group focusing on Apple’s tablet. In fact, iPad owners who lived with one or more individuals reported that they shared their iPads freely, unlike the iPhone. The report also illuminated many things we like and don’t like about the apps we use on our iPads.

For example, the study found that users aren’t crazy about using their iPad devices to deal with complicated forms that require lots of user input, especially if those forms are found in non-optimized websites, rather than housed in an app. Users would skip registrations processes rather than deal with inputting information in many cases. The solution to such a problem would be to make forms simpler, requiring less information, and reduce the need for repeat entry of information (so apps that offer to remember login details are better, for example).

iPad users also aren’t as able to decipher non-obvious control systems as some developers might think. In cases where it wasn’t made clear what tapping an item that wasn’t obviously a button (i.e., a logo) would do, users often missed the functionality. Examples cited in the report include the logo in the top left of The Daily app, which returns users to the app’s home screen. USA Today  originally used a similar mechanism, but changed their logo to include a “Sections” label to tell users that it was in fact designed to be tapped and tied to a function.

Likewise, gestures in apps can sometimes cause trouble when there are no visual cues to provide information about how they work. Don’t think that placing an instructional video or graphic at the beginning of the app will solve the problem, either. Many users don’t read instructions, though visual instructions that are incredibly obvious, like those used by Bing for iPad, tested well with those participating in the study, since users couldn’t avoid grasping their meaning even when they quickly dismissed them. Nielsen Norman Group advises developers that they’re much better off including visual markers throughout, indicating that swipes and other gestures can be used. For example, magazine apps like Wired include arrows that show the direction a user should swipe to unveil more content.

Another alternative is to provide explicit tips in the form of dialog boxes, like Adobe Photoshop Express does. The iPad Photoshop app uses gestures to control effects like “soft focus,” and pops up notifications to alert users of what to do. Tips can be hidden at any time, so they won’t become annoying.

What users find very annoying according to the report are splash or loading screens. No matter how clever, or how easy on the eye, splash screens and animations become annoying very quickly. Startup sounds, in particular, are singled out as especially bad, because of the potential they have for unpleasantly surprising people who open apps in surroundings where noise might not be appreciated.

Also, almost universally, apps will benefit from having back buttons on nearly every page, and should aim for a simple homepage-like table of contents over more complicated navigation schemes. Users prefer a home base from which to operate without having to hunt through carousels or wade through long columns of thumbnails, and they always want the option to go one step back from their current position, because of accidental taps or to refer back to something they just saw.

As mentioned above, iPads tend to be communal devices, at least within the household. But the report also highlighted some other interesting points regarding how we use the Apple tablet. Generally, we use it for gaming, checking email and communicating via social networking, watching videos/movies and reading news. We also tend to shop, but the participants in the study generally preferred shopping on their desktops, and some even perceived iPad shopping to be more risky from a security perspective. iPads also tend to be carried around by many users, or at least taken along for the ride when long waits or trips are expected.

Now that the iPad is more than a year old, it’s interesting to see how people are using it, and what is and isn’t working when it comes to app usability design. No doubt there’s still plenty of innovation left in iPad app interface design, but this report illustrates that some things never go out of style when it comes to user experience.

How does your experience with the iPad either agree or disagree with the findings described above?

Related content from GigaOM Pro (subscription req’d):
Mobile Q1: All Eyes on Tablets, T-Mobile and AT&TConnected Consumer Q1: The Over-the-Top vs. Pay TV Battle Heats UpA Media Tablet Forecast, 2011 – 2015

Zynga CEO Mark Pincus got lobbed some softballs this morning at the TechCrunch Disrupt conference by his investors at Kleiner Perkins: Bing Gordon (who is on the Zynga board) and John Doerr. However, he did give up some specific metrics about how his very metrics-driven company runs, and he talked about the larger themes at stake.

Pincus said three-and-a-half-year-old Zynga has 215 million monthly active users, 33 million of which played its games yesterday. The company has a staff of 1,200, half of them engineers, with 13 games studios in eight locations. It has 20,000 servers not including its heavy use of Amazon Web Services, with 1,000 added per week (more detail on that here).

Pincus projected that the rest of the web will come to look a lot more like Zynga, with advertising only be a small portion of revenue. “As big as ads are, I think what I call the user-pay economy will be much bigger than the advertising economy.” The web should continue to become more app-like, with categories like travel, shopping and health starting to look a lot more like gaming does today.

Zynga is known for making decisions about its product based on metrics, but that also extends to the way Pincus runs the company. He said that every employee at Zynga is treated as a CEO, with goals and metrics they are graded on using the concept of OKRs: one objective and three key results. The company measures its success using its Net Promoter Score (the measure of how likely users say they are to recommend a service) and aggregating daily survey data of customers taken after they go through support.

“We are data junkies,” said Pincus. “We measure everything. We’ve invested incredibly in data warehousing. We capture and track every click in every game every day.”

Related content from GigaOM Pro (subscription req’d):

How to Measure Social Media Advertising
The Real Impact of Facebook’s New Approach to Social Gaming
Multiple Models for Social Media Businesses