rahuldave + d3   5

Editorial Radar: Functional languages
Functional Languages are driving a broader set of choices for programmers. O'Reilly editors Mike Loukides and Mike Hendrickson sat down recently to talk about the advantages of functional programming languages and how functional language techniques can be deployed with almost any language. (The full conversation is embedded below.)

Andy Hunt and Dave Thomas have long recommend learning a new language each year, especially those languages that teach new concepts [discussed at the 02:02 mark]. Functional languages have made that easier. They behave in a different way than the languages many of us grew up on — procedural like C or languages derived from C. Plus, the polyglot programming movement has driven the interest in functional languages as one of the languages you might want to learn.

Programmers need to understanding the advantages of using a functional language, such as productivity, power of expressiveness, reliability, stateful objects, concurrency, natural concurrency, modularity, and composability [05:37]. Though a search still exists for a magic bullet [06:29] to make it easier for programers to better solve the problem of concurrency. CPU speeds have been stuck at roughly the same level for the last four to five years. Programmers have been given is more transistors on a chip, hence more CPUs and more cores to work with making concurrency one of the most difficult issues facing computer scientists today. Enter functional programming with improved debugging and the ability to write more reliable code in a concurrent environment.

Additional highlights from this conversation include:

Print book sales of functional languages are growing, especially books on R programming. And while Loukides doesn't consider R to be a functional language, some debate exists about its classification. Though it's clear the data science movement has driven the use of R because it's well designed for statistics and dealing with data. [Discussed at the 00:29 mark]

We'll see F# grow in the Microsoft development environment while Scala and Clojure are dominating the open source space. Erlang will also be around for a long time for building highly reliable concurrent systems. [Discussed at the 03:01 mark]

Since the publication of Doug Crockford's JavaScript: The Good Parts, coders have discovered the functional language abilities of JavaScript and Java. Google's release of Maps and Gmail revolutionized how JavaScript is used. Some of today's best examples include Node for high-performance websites and D3 for creating exotic and beautiful data visualizations. [Discussed at the 08:15 mark]

While JavaScript isn't a functional language, it's designed loosely, so it's easy to use as a functional language. You might also be interested in how functional programming techniques can be used in C++ — a blog post written by John Carmack. [Discussed at the 10:36 mark]

Java isn't intended as a functional language. Though Dean Wampler's Functional Programming for Java Developers provides an approachable introduction to functional programming for anyone using an object-oriented language. [Discussed at the 11:41 mark]

The use of a functional language or functional language techniques can make your code more robust and easier to debug. [Discussed at the 12:09 mark]

You can view the entire conversation in the following video:

Tune in next month for a discussion of NoSQL and web databases.

Fluent Conference: JavaScript & Beyond — Explore the changing worlds of JavaScript & HTML5 at the O'Reilly Fluent Conference (May 29 - 31 in San Francisco, Calif.).

Save 20% on registration with the code RADAR20

Related:

Subscribe to the free Code podcast through iTunes
See more Code podcasts
Editorial Radar: Machine learning, 3D printing, devices and JavaScript
Clojure: Lisp meets Java, with a side of Erlang
A rough guide to JVM languages
Programming  clojure  codepodcast  concurrency  d3  f  functionalprogramming  java  javascript  node  rprogramming  scala  from google
28 days ago by rahuldave
Big Data, R and SAP HANA: Analyze 200 Million Data Points and Later Visualize in HTML5 Using D3 – Part III
(This article was first published on All Things R, and kindly contributed to R-bloggers)

Mash-up Airlines Performance Data with Historical Weather Data to Pinpoint Weather Related DelaysFor this exercise, I combined following four separate blogs that I did on BigData, R and SAP HANA.  Historical airlines and weather data were used for the underlying analysis. The aggregated output of this analysis was outputted in JSON which was visualized in HTML5, D3 and Google Maps.  The previous blogs on this series are:Big Data, R and SAP HANA: Analyze 200 Million Data Points and Later Visualize in HTML5 Using D3 - Part IIBig Data, R and HANA: Analyze 200 Million Data Points and Later Visualize Using Google MapsGetting Historical Weather Data in R and SAP HANA Tracking SFO Airport's Performance Using R, HANA and D3In this blog, I wanted to mash-up disparate data sources in R and HANA by combining airlines data with weather data to understand the reasons behind the airport/airlines delay.  Why weather - because weather is one of the commonly cited reasons in the airlines industry for flight delays.  Fortunately, the airlines data breaks up the delay by weather, security, late aircraft etc., so weather related delays can be isolated and then the actual weather data can be mashed-up to validate the airlines' claims.  However, I will not be doing this here, I will just be displaying the mashed-up data.I have intentionally focused on the three bay-area airports and have used last 4 years of historical data to visualize the airport's performance using a HTML5 calendar built from scratch using D3.js.  One can use all 20 years of data and for all the airports to extend this example.  I had downloaded historical weather data for the same 2005-2008 period for SFO and SJC airports as shown in my previous blog (For some strange reasons, there is no weather data for OAK, huh?).  Here is how the final result will look like in HTML5:Click here to interact with the live example.  Hover over any cell in the live example and a tool tip with comprehensive analytics will show the break down of the performance delay for the selected cell including weather data and correct icons* - result of a mash-up.  Choose a different airport from the drop-down to change the performance calendar. * Weather icons are properties of Weather Underground.As anticipated, SFO airport had more red on the calendar than SJC and OAK.  SJC definitely is the best performing airport in the bay-area.  Contrary to my expectation, weather didn't cause as much havoc on SFO as one would expect, strange?Creating a mash-up in R for these two data-sets was super easy and a CSV output was produced to work with HTML5/D3.  Here is the R code and if it not clear from all my previous blogs: I just love data.table package.###########################################################################################  # Percent delayed flights from three bay area airports, a break up of the flights delay by various reasons, mash-up with weather data###########################################################################################  baa.hp.daily.flights <- baa.hp[,list( TotalFlights=length(DepDelay), CancelledFlights=sum(Cancelled, na.rm=TRUE)),                              by=list(Year, Month, DayofMonth, Origin)]setkey(baa.hp.daily.flights,Year, Month, DayofMonth, Origin)baa.hp.daily.flights.delayed <- baa.hp[DepDelay>15,                                     list(DelayedFlights=length(DepDelay),                                       WeatherDelayed=length(WeatherDelay[WeatherDelay>0]),                                      AvgDelayMins=round(sum(DepDelay, na.rm=TRUE)/length(DepDelay), digits=2),                                      CarrierCaused=round(sum(CarrierDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2),                                      WeatherCaused=round(sum(WeatherDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2),                                      NASCaused=round(sum(NASDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2),                                      SecurityCaused=round(sum(SecurityDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2),                                      LateAircraftCaused=round(sum(LateAircraftDelay, na.rm=TRUE)/sum(DepDelay, na.rm=TRUE), digits=2)), by=list(Year, Month, DayofMonth, Origin)]setkey(baa.hp.daily.flights.delayed, Year, Month, DayofMonth, Origin)# Merge two data-tablesbaa.hp.daily.flights.summary <- baa.hp.daily.flights.delayed[baa.hp.daily.flights,list(Airport=Origin,                           TotalFlights, CancelledFlights, DelayedFlights, WeatherDelayed,                            PercentDelayedFlights=round(DelayedFlights/(TotalFlights-CancelledFlights), digits=2),                           AvgDelayMins, CarrierCaused, WeatherCaused, NASCaused, SecurityCaused, LateAircraftCaused)]setkey(baa.hp.daily.flights.summary, Year, Month, DayofMonth, Airport)# Merge with weather databaa.hp.daily.flights.summary.weather <-baa.weather[baa.hp.daily.flights.summary]baa.hp.daily.flights.summary.weather$Date <- as.Date(paste(baa.hp.daily.flights.summary.weather$Year,                                                            baa.hp.daily.flights.summary.weather$Month,                                                            baa.hp.daily.flights.summary.weather$DayofMonth,                                                            sep="-"),"%Y-%m-%d")# remove few columnsbaa.hp.daily.flights.summary.weather <- baa.hp.daily.flights.summary.weather[,             which(!(colnames(baa.hp.daily.flights.summary.weather) %in% c("Year", "Month", "DayofMonth", "Origin"))), with=FALSE]#Write the output in both JSON and CSV file formatsobjs <- baa.hp.daily.flights.summary.weather[, getRowWiseJson(.SD), by=list(Airport)]# You have now (Airportcode, JSONString), Once again, you need to attach them together.row.json <- apply(objs, 1, function(x) paste('{\"AirportCode\":"', x[1], '","Data\":', x[2], '}', sep=""))json.st <- paste('[', paste(row.json, collapse=', '), ']')writeLines(json.st, "baa-2005-2008.summary.json")                 write.csv(baa.hp.daily.flights.summary.weather, "baa-2005-2008.summary.csv", row.names=FALSE)Happy Coding!

To leave a comment for the author, please follow the link and comment on his blog: All Things R.

R-bloggers.com offers daily e-mail updates about R news and tutorials on topics such as: visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series,ecdf, trading) and more...
R_bloggers  business_analytics  business_intelligence  cloud  D3  D3.js  Google  google_maps  HANA  HTML5  predictive  R  r-project  RApache  SAP  SFO  SJC  TechEd  twitter  from google
29 days ago by rahuldave

Copy this bookmark:



description:


tags: