(This article was first published on All Things R, and kindly contributed to R-bloggers)

Mash-up Airlines Performance Data with Historical Weather Data to Pinpoint Weather Related DelaysFor this exercise, I combined following four separate blogs that I did on BigData, R and SAP HANA.  Historical airlines and weather data were used for the underlying analysis. The aggregated output of this analysis was outputted in JSON which was visualized in HTML5, D3 and Google Maps.  The previous blogs on this series are:Big Data, R and SAP HANA: Analyze 200 Million Data Points and Later Visualize in HTML5 Using D3 - Part IIBig Data, R and HANA: Analyze 200 Million Data Points and Later Visualize Using Google MapsGetting Historical Weather Data in R and SAP HANA Tracking SFO Airport's Performance Using R, HANA and D3In this blog, I wanted to mash-up disparate data sources in R and HANA by combining airlines data with weather data to understand the reasons behind the airport/airlines delay.  Why weather - because weather is one of the commonly cited reasons in the airlines industry for flight delays.  Fortunately, the airlines data breaks up the delay by weather, security, late aircraft etc., so weather related delays can be isolated and then the actual weather data can be mashed-up to validate the airlines' claims.  However, I will not be doing this here, I will just be displaying the mashed-up data.I have intentionally focused on the three bay-area airports and have used last 4 years of historical data to visualize the airport's performance using a HTML5 calendar built from scratch using D3.js.  One can use all 20 years of data and for all the airports to extend this example.  I had downloaded historical weather data for the same 2005-2008 period for SFO and SJC airports as shown in my previous blog (For some strange reasons, there is no weather data for OAK, huh?).  Here is how the final result will look like in HTML5:Click here to interact with the live example.  Hover over any cell in the live example and a tool tip with comprehensive analytics will show the break down of the performance delay for the selected cell including weather data and correct icons* - result of a mash-up.  Choose a different airport from the drop-down to change the performance calendar. * Weather icons are properties of Weather Underground.As anticipated, SFO airport had more red on the calendar than SJC and OAK.  SJC definitely is the best performing airport in the bay-area.  Contrary to my expectation, weather didn't cause as much havoc on SFO as one would expect, strange?Creating a mash-up in R for these two data-sets was super easy and a CSV output was produced to work with HTML5/D3.  Here is the R code and if it not clear from all my previous blogs: I just love data.table package.###########################################################################################  # Percent delayed flights from three bay area airports, a break up of the flights delay by various reasons, mash-up with weather data###########################################################################################  baa.hp.daily.flights <- baa.hp[,list( TotalFlights=length(DepDelay), CancelledFlights=sum(Cancelled, na.rm=TRUE)),                              by=list(Year, Month, DayofMonth, Origin)]setkey(baa.hp.daily.flights,Year, Month, DayofMonth, Origin)baa.hp.daily.flights.delayed <- baa.hp[DepDelay>15,                                     list(DelayedFlights=length(DepDelay),                                       WeatherDelayed=length(WeatherDelay[WeatherDelay>0]),                                      AvgDelayMins=round(sum(DepDelay, na.rm=TRUE)/length(DepDelay), digits=2),                                      CarrierCaused=round(sum(CarrierDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2),                                      WeatherCaused=round(sum(WeatherDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2),                                      NASCaused=round(sum(NASDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2),                                      SecurityCaused=round(sum(SecurityDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2),                                      LateAircraftCaused=round(sum(LateAircraftDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2)), by=list(Year, Month, DayofMonth, Origin)]setkey(baa.hp.daily.flights.delayed, Year, Month, DayofMonth, Origin)# Merge two data-tablesbaa.hp.daily.flights.summary <- baa.hp.daily.flights.delayed[baa.hp.daily.flights,list(Airport=Origin,                           TotalFlights, CancelledFlights, DelayedFlights, WeatherDelayed,                            PercentDelayedFlights=round(DelayedFlights/(TotalFlights-CancelledFlights), digits=2),                           AvgDelayMins, CarrierCaused, WeatherCaused, NASCaused, SecurityCaused, LateAircraftCaused)]setkey(baa.hp.daily.flights.summary, Year, Month, DayofMonth, Airport)# Merge with weather databaa.hp.daily.flights.summary.weather <-baa.weather[baa.hp.daily.flights.summary]baa.hp.daily.flights.summary.weather$Date <- as.Date(paste(baa.hp.daily.flights.summary.weather$Year,                                                            baa.hp.daily.flights.summary.weather$Month,                                                            baa.hp.daily.flights.summary.weather$DayofMonth,                                                            sep="-"),"%Y-%m-%d")# remove few columnsbaa.hp.daily.flights.summary.weather <- baa.hp.daily.flights.summary.weather[,             which(!(colnames(baa.hp.daily.flights.summary.weather) %in% c("Year", "Month", "DayofMonth", "Origin"))), with=FALSE]#Write the output in both JSON and CSV file formatsobjs <- baa.hp.daily.flights.summary.weather[, getRowWiseJson(.SD), by=list(Airport)]# You have now (Airportcode, JSONString), Once again, you need to attach them together.row.json <- apply(objs, 1, function(x) paste('{\"AirportCode\":"', x[1], '","Data\":', x[2], '}', sep=""))json.st <- paste('[', paste(row.json, collapse=', '), ']')writeLines(json.st, "baa-2005-2008.summary.json")                 write.csv(baa.hp.daily.flights.summary.weather, "baa-2005-2008.summary.csv", row.names=FALSE)Happy Coding!

To leave a comment for the author, please follow the link and comment on his blog: All Things R.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series,ecdf, trading) and more...

Healthcare providers are increasingly required to do more with less. Regulations, HIPAA, Meaningful Use, recovery audit contractor (RAC) audits and decreasing revenues are motivating providers to consider cloud computing as a solution to potentially help them cut costs, maintain quality, meet regulations, and increase productivity.

Some electronic health record (EHR) vendors are offering solutions as a cloud-based offering. This offers an approach intended to help providers better manage the IT investments that need to be made to support EHR implementations. And just as we've seen in other industries, there is an ongoing debate within healthcare as to the viability of cloud-based solutions given the care needed for patient privacy and sensitive personal information.

Providers' trust in the public cloud is still relatively weak, but increasing numbers are considering using private clouds. However, EHR applications hosted in the cloud do seem to be gaining traction.

One example of a cloud-based EHR offering is CareCloud. My fellow Radar blogger Andy Oram wrote about them two years ago at HIMSS, and they have made significant progress since then. CareCloud creates apps that help medical professionals run their businesses. Those apps include a community collaboration and communication platform to securely share patient information, a medical practice management system for billing and scheduling, and a revenue cycle management service. CareCloud also provides electronic health records. It's built with Ruby on Rails, a highly abstracted programming language quite well suited for rapid development of web applications. CareCloud was a co-winner of the IBM Global Entrepreneur Silicon Valley SmartCamp competition in 2010 (see video below).

I ran into the folks at CareCloud at the HIMSS 2012 conference and was impressed with both their use of open source and their strategy on leveraging the cloud in healthcare. Mike Cuesta, CareCloud's director of marketing and user experience, defined CareCloud's strategy as one of future survival.

"Being able to deliver the product across platforms is crucial," Cuesta said. "In healthcare there is a glaring lack of modern web apps. What we wanted to do was create an elegant and user-friendly application that is accessible anywhere. Companies have to be able to deliver a desktop-class experience that works across platforms."

CareCloud relies on open source. "I had my eyes opened to open source about eight years ago when I was looking for a project management system," said CareCloud CTO Tom Packert. "I discovered I could use something like dotproject, which is a GPL-licensed PHP-MySQL web-based project management application. It only took us a day to put it up on SUSE Linux and we didn't need SQL seat licenses. Open source allows you to scale horizontally. It's not as scary as a lot of people think it is."

Another EHR in the cloud is athenahealth. Athenahealth's co-founders Todd Park, the new U.S. chief technology officer (CTO), and Jonathan Bush, purchased a birthing practice in 1997. Soon, like most medical practices, they were buried in paper and spent most of their resources trying to get paid. Searching for innovative solutions led them to create their own software. Enlisting the help of Todd's younger brother Ed, a software developer, they created an EHR and financial revenue cycle system with a rules engine of dynamic billing rules data. I met Ed Park at HIMSS when I remarked that he looked a lot like Todd Park, and Jonathan introduced him to me as Todd's "younger, smarter, and much better looking brother." Apparently his programming skills are paying off ...

This year, athenahealth was named to the TR50, Technology Review's third annual list of the world's most innovative technology companies. At this year's HIMSS conference, athenahealth showed the company's plans for an iPhone app that will gives its EHR users access to certain features of its athenaClinicals cloud-based platform. An iPad version of the web-based athenahealth EHR app is also currently under development and set to launch in 2013.

Being based on cloud technology makes athenahealth much more nimble in launching mobile products in services. In the video below, I discuss with Jonathan Bush how athenahealth is using the cloud in their EHR.

(Thanks to Nate DeNiro and Open Affairs Television for their assistance with this video.)

Related:

AI will eventually drive healthcare, but not anytime soon
Medical imaging in the cloud: a conversation about eMix
Health gets personal in the cloud

Terrier IR -- open source (Mozilla) text search engine, now with Hadoop support.
s3ql -- open source (GPLv3) Linux filesystem which stores its data on Google Storage, Amazon S3, or OpenStack. (via Adam Shand)
Esprima -- open source (BSD) fast Javascript parser in Javascript. (via Javascript Weekly)
Hogan.js -- open source (Apache) Javascript templating engine from Twitter. If it proves anywhere near as good as Bootstrap, it'll be heavily used.

Terrier IR -- open source (Mozilla) text search engine, now with Hadoop support.
s3ql -- open source (GPLv3) Linux filesystem which stores its data on Google Storage, Amazon S3, or OpenStack. (via Adam Shand)
Esprima -- open source (BSD) fast Javascript parser in Javascript. (via Javascript Weekly)
Hogan.js -- open source (Apache) Javascript templating engine from Twitter. If it proves anywhere near as good as Bootstrap, it'll be heavily used.

The question of how to print from wireless devices has been thrust once again into the limelight recently thanks to the printing-anemic iPad. Longtime notebook and mobile device users are quite familiar with the printing conundrum—cables, drivers and all.

Google has announced that it's looking to address this problem in the form of Cloud Print. Part of the Chromium and Chromium OS projects, Cloud Print aims to allow any type of application to print to any printer. This includes Web, desktop, and mobile apps from any kind of device—potentially, this could be used on a BlackBerry, Windows machines, Macs, or even the iPad. (That is in addition to Google's own offerings: "Google Chrome OS will use Google Cloud Print for all printing. There is no print stack and there are no printer drivers on Google Chrome OS!" says the company.)





Read the comments on this post

Crawling webpages isn’t something most of us are set up to do. That’s why 80legs turned it into a service, spidering two billion web pages per day. It launched with only Java support. Now the company has added an API Kit for Python programmer, responding to its users most popular request (our 80legs profile).

The web crawling service starts with a basic account that is free for limited use. There are also subscription options, in addition to paying by the million pages and CPU hour. The fees, of course, are small in comparison to creating your own farm of web crawling servers.

It’s interesting–and potentially convenient–that the two languages the service uses are the same two supported by App Engine, Google’s cloud-based application hosting service (our Google App Engine API profile). However, CEO Shion Deysarkar said in an email that not many 80legs developers are using App Engine. “We did Java first because we write all of our stuff in Java.  Python came second because it was the most requested language,” said Deysarker.

Last fall we wrote about the 80legs contest, which launched in preparation for its 80apps store. Developers can sell apps, keeping 100% of the revenue–just paying 80legs for the page crawling and CPU-hours.

Related ProgrammableWeb Resources 80legs API Profile

Apple urgently needs to improve its strategy on the cloud. The iPad and the iPhone are perfect smart terminals for cloud computing. At some level Apple knows this, as it was pushing a MobileMe discount with iPads this weekend. But when you get your hands on an iPad, you realize that Apple missed a real opportunity for deep integration with its cloud offerings.

I've been a MobileMe user for a little while, since the transition from .Mac, and I like how it is integrated with OS X setup. On the iPhone, I love the over-the-air syncing of my bookmarks, contacts and calendar. I had expectations that the iPad would take this a step further.

However, the iPad is no more advanced than the iPhone in its cloud integration. I would have loved to have switched on the iPad, keyed in my MobileMe login, and automatically had my email, browser bookmarks, calendar and contacts set up for me, as well as the ability to load in ebooks through my iDisk, and have my photo galleries available.

Instead I was forced through the painfully overloaded iTunes application, and had to tether my device via USB to get all of my content on it. Setting things up was a crazy dance involving configuration in both iTunes and in the iPad's settings panel. To make matters worse, the iPad doesn't want to charge over USB. This means I need to plug it in twice: once to the charger, and then somewhere else to sync. Decent cloud access would have mitigated this a little.

I was genuinely surprised that the iWork and Photo applications for the iPad don't have built-in support for MobileMe. Email appears the be the only generally universal way of getting things out of the iPad.

Both OS X and Ubuntu offer me a much more pleasant out-of-box setup experience for connecting and synchronizing with cloud services. I suspect that because the iPad is divided up into little silos for each application, and iPhone OS doesn't offer any general notion of cloud services, it can only be this way for now.

I am hoping for a future where all I need to supply a device with is my identity, and everything else falls into place. This doesn't even have to be me trusting in a third-party cloud: there's no reason similar mechanisms couldn't be used privately in a home network setting.

I think the iPad is an amazing piece of hardware, and the most pleasant web browsing experience available. It is still very much a 1.0 device though, and its best days certainly lie ahead of it. I hope part of that improvement is a simple story for synchronization and cloud access.

Somewhat to my surprise, I'm equally as excited about the upcoming Ubuntu 10.04 (Lucid) release for netbooks as I am by the iPad. The iPad is not yet a netbook-killer.

Cloud computing is big right now, but the sheer number of options, and the lack of interoperability, can be an issue for developers. Vendor lock-in reduces the options available to those working on cloud computing initiatives, while the variability between APIs inevitably leads to many hours spent pouring over documentation when comparing cloud service providers.

There are a number of projects that can reduce, or even eliminate, some of these problems by exposing the functionality of a number of cloud service providers through a consistent interface. Here is a list of 7 such projects.

DeltaCloud is a RehHat initiative that provides a REST API and user tools to manage EC2, RHEV-M, RackSpace and RimuHosting, with VMWare ESX and more coming soon.

libcloud is a pure python client library for interacting with many of the popular cloud server providers. The libcloud web pages lists support for 11 cloud services provides.
The Simple Cloud API brings cloud technologies to PHP and the PHPilosophy to the cloud, starting with common interfaces for File Storage Services, Document Storage Services and Simple Queue Services. The API page shows support for Amazon S3, Windows Azure Blob Storage and Nirvanix, with more to come.
jclouds is an open source framework that helps you get started in the cloud and reuse your java development skills. Jclouds support many clouds including Amazon, VMWare, Azure, and Rackspace.
The Dasein Cloud API provides an abstraction for applications that wish to be written independent of the clouds they are controlling.
OpenNebula is an open and flexible tool that fits into existing data center environments to build any type of Cloud deployment. OpenNebula 1.4 supports Xen, KVM and VMware virtualization platforms and on-demand access to Amazon EC2 and ElasticHosts Cloud providers. It also features new local and Cloud interfaces, such as libvirt, EC2 Query API and OGC OCCI API.

Cloudloop is a new open source Java API and command-line management tool for cloud storage. Unfortunately the cloudloop web site was hacked recently, but you can find more information with this post on TheServerSide.

Just as with the growth of these cloud APIs and platforms, there are more libraries and abstraction initiatives all the time. Feel free to share any others we missed in the comments.