Revolutions

More data apps spawned by Sandy

As the clean-up continues on the eastern seaboard, I wanted to follow up on Monday's post on tracking Hurricane Sandy with Open Data with a couple of other R-based data applications spawned by the storm.

Josef Fruehwald created an R script to tap into local weather sensors to keep track of air pressure, wind speed and rainfall near his home and for other affected locations. Check out the plunge in barometric pressure as the centre of the storm passed near Atlantic City on Monday night, and then over Philadelphia a couple of hours later:

The greater scale of destruction in Atlantic City compared to Philadelphia is reflected in this comparison of wind speeds over the same period:

Joseph used R to download the data from Weather Underground station records (like this one) in XML format, and then used the ggplot2 package to create the charts. You can find the complete R script at RPubs: Analysis of a Philadelphia Weather Station Data during Hurricane Sandy.

Twitter's Ed Chen took a different look at the effects of Sandy: how the storm knocked out power around the NY region, and the race by the utulities to restore power. Where did he find such data? Tweets from Twitter users, of course! Geolocated tweets (from smartphones, presumably) along the lines of "I've just lost power!" , "Still no power here!" and "The power is back on!" gave Ed information he needed to use R to create an animation over time of the power going out and coming back on across NYC.

The above is just a snapshot; visit Ed Chen's blog for an animated version, with a slider to see the changes during the night of the storm.

R among TechCrunch's 5 Trendy Open-Source Techs for Big Data

Tim Gasper (Product Manager at Big Data platform Infochimps) has an informative article at TechCrunch that provides an overview of five open-source technologies trending now for Big Data applications. They are:

• Storm and Kafka (for processing stream data)
• Drill and Dremel (for ad-hoc queries of big data)
• R (for data science with big data)
• Gremlin and Giraph (for graph analysis, e.g. of social networks)
• SAP HANA (for in-memory analytics). HANA isn't an open-source tool though, so perhaps the fifth slot should really go to ...
• Honourable mention D3, for web-based visualization.

Regarding R, Gasper says that it is "incredibly powerful", "the new standard for statistics", and that "the R community is one of the most thrilling places to be in Big Data right now". He also mentions the RHadoop project ("R work very well with Hadoop") and the up-and-coming Julia project.

You can read Gasper's complete overview of R and the other trending big-data technologies at link below.

TechCrunch: Big Data Right Now: Five Trendy Open Source Technologies

Tracking Hurricane Sandy with Open Data and R

Hurricane Sandy is shaping up to be a major, and very dangerous, meteorological event for the US's East coast. Naturally, everyone is looking for the latest information and forecasts. Fortunately, the wealth of public meteorological data available on the open web, combined with real-time on-the-ground updates via social media, means that an ecosystem of on-line apps is now available providing all the up-to-date information you need. O'Reilly's Strata site has a good overview of the open-data Sandy-tracking apps available.

For example, R user Bob Rudis has written an R script to grab up-to-date hurricane tracking data from Unisys Weather, and combine it with the forecast cone from Google's crisis map, to create the following combined picture of where Sandy's been, and where it's going:

The forecast track shows Sandy heading north into Canada. It differers from the version on Bob's blog post because I ran the R script just now (at 16:38 PST) — that's the power of using live data!

If you're anywhere near Sandy, stay informed and above all, stay safe. You can find information on how to keep safe at Ready.gov.

Update Oct 30: See also this Analysis of a Philadelphia Weather Station Data during Hurricane Sandy by Josef Fruehwald, featuring charts of barametric pressure, wind speed and rose charts of wind direction created from live data using R.

rud.is: Watch Sandy in “R” (Including Forecast Cone)

Because it's Friday: Gangnam Halloween

It's almost All Hallow's Eve, and in the it's tradition in many places here in the States to decorate houses with "spooky" decorations. Some homeowners take the process to extremes — to both delight and chagrin of neighbours — such as with this light show in Leesburg, Virginia set to the Korean pop hit Gangnam Style:

 

The display includes more than 8,500 lights, and required 250 separate lighting channels to support the computer-generated animation. Enjoy the Hallowe'en festivities in your own region this weekend!

R 2.15.2 now available

As promised, the source distribution for R 2.15.2 is now available for download from the master CRAN repository. (Binary distributions for Windows, MacOS and Linux will be available from the CRAN mirror network in the coming days.) This latest point-update — codenamed "Trick or Treat" — improves the performance of the R engine and adds a few minor but useful features. Detailed changes can be found in the NEWS file, but highlights of the improvements include: 

• New statistical analysis method: Multistratum MANOVA
• Hyman's method of constructing monotonic interpolation splines is now available.
• Improved support for Polish language users
• Functions in the parallel package (such as parLapply) will make use of a default cluster if one is specified.
• Improved performance and reduced memory usage for some commonly-used functions including array, rep, tabulate and hist
• Increased memory available for data on 64-bit systems (increased to 32Gb from 16Gb)
• Several minor bugfixes

There is likely to be at least one further update to the 2.15.x series: a round-up of any further changes will probable be released as 2.15.3 shortly before R 2.16.0 is released, most likely around March 2013.

r-announce mailing list: R 2.15.2 is released

 

Allstate compares SAS, Hadoop and R for Big-Data Insurance Models

At the Strata conference in New York today, Steve Yun (Principal Predictive Modeler at Allstate's Research and Planning Center) described the various ways he tackled the problem of fitting a generalized linear model to 150M records of insurance data. He evaluated several approaches:

1. Proc GENMOD in SAS
2. Installing a Hadoop cluster
3. Using open-source R (both on the full data set, and on using sampling)
4. Running the data through Revolution R Enterprise 

Steve described each of the approaches as follows.

Approach 1 is current practice. (Allstate is a big user of SAS, but there's a growing contingent of R users.) Proc GENMOD takes around 5 hours to return results for a Poisson model with 150 million observations and 70 degrees of freedom. "It's difficult to be productive on a tight schedule if it takes over 5 hours to fit one candidate models!", Steve said.

Approach 2: It was hoped that installing a Hadoop cluster and running the model there would improve performace. According to Steve, "a lot of plumbing was required: this involved coding the matrix equations for iteratively-reweighted least squares as a map-reduce task and manually coding the factor variables as indicator columns in the design matrix. Unfortunately, each iteration took abour 1.5 hours, with 5-10 iterations required to convergence. (Even then, there were problems with singularites in the design matrix.)

Approach 3: Perhaps installing R on a server with lots of RAM would help. (Because open-source R runs in-memory, you need RAM in the order of several times the size of the data to make it work.) Alas, not even a 250Gb server was sufficient: even after waiting three days, the data couldn't even be loaded. Sampling the data down into 10 partitions was more successful, and allowed for the use of the glmnet package and L1 regularization to automate the variable selection process. But each glmnet fit on a partition still took over 30 minutes, and Steve said it would be difficult for managers to accept a process that involved sampling.

Approach 4: Steve turned to Revolution Analytics' Joe Rickert to evaluate how long the same model would take using the big-data RevoScaleR package in Revolution R Enterprise. Joe loaded the data onto a 5-node cluster (20 cores total), and used the distributed rxGlm function, which was able to process the data in 5.7 minutes. Joe demonstrated this process live during the session.

So in summary, here's how the four approaches fared:

Approach	Platform	Time to fit
1: SAS	16-core  Sun Server	5 hours
2: rmr / map-reduce	10-node (8 cores / node) Hadoop cluster	> 10 hours
3: Open source R	250 GB Server	Impossible (> 3 days)
4: RevoScaleR	5-node (4 cores / node) LSF cluster	5.7 minutes

That's quite a difference! So what have we learned:

• SAS works, but is slow.
• It's possible to program the model in Hadoop, but it's even slower.
• The data is too big for open-source R, even on a very large server.
• Revolution R Enterprise gets the same results as SAS, but about 50x faster.

Steve and Joe's slides and video of the presentation, Start Small Before Going Big, will be available on the Strata website in due course.

Quick notes from Strata NYC 2012

The O'Reilly Strata conferences are always great fun to attend, and this latest installment in New York City is no exception. This one is super-busy though; the conference has been sold out for weeks -- and not just marketing-sold-out, it's fire-department-sold out. It's non-stop conversations and presentations, and it's tough to move through the hallways in between.

Nonetheless, I thought I'd pause for a couple of minutes and share some of the highlights for me so far.

• Ed Kohlwey and Stephanie Beben gave a three-hour tutorial on the RHadoop project, showing the packed room how to crunch big data. They shared how consulting firm Booz Allen Hamilton uses R and Hadoop for data exploration; to run many tasks in parallel; and to sort, sample and join data. They've also create a very handy VirtualBox VM including R, Hadoop, RHadoop and RStudio (along with demonstration script files) which I hope to be able to post a download link for soon.
• Stan Humphries from Zillow gave a presentation on how data and statistical analysis drives Zillow's home valuation service. One fascinating tidbit: while Zillow has long used R to fit their valuation model, until recently they recoded the model scoring algorithm in C++ for use on the production site. The process of re-implementing a new version of the model, validating it, and deploying it used to take 9 months. But now that they run R in production via the Amazon cloud, without the need to recode the model in another language, the deployment time for new valuation models is just four weeks.
• Mike Driscoll from Metamarkets shared the technology behind their data stack: node.js and D3 for visualization; R and Scala for analytics; Druid as the data store; and Hadoop and Kafka for ETL. Druid is MetaMarket's home-grown high-performance, which they announced today is now available as open source software.
• In a similar vein, Cloudera announced the release of Impala, an open-source project two years in the making to bring high-performance real-time analytics to Hadoop.
• And there were even more announcements: Kaggle launched a partnership with EMC to give Greenplum users direct access to the roster of Kaggle data scientists competitors.

It's been a great conference so far, and this is only day one! Looking forward to more great talks and conversations tomorrow.

Two Talks on Data Science, Big Data and R

On Thursday next week (November 1), I'll be giving a new webinar on the topic of Big Data, Data Science and R. Titled "The Rise of Data Science in the Age of Big Data Analytics: Why Data Distillation and Machine Learning Aren’t Enough", this is a provocative look at why data scientists cannot be replaced by technology, and why R is the ideal environment for building data science applications. Here's the abstract:

The reason why Big Data is important is because we want to use it to make sense of our world. It’s tempting to think there’s some “magic bullet” for analyzing big data, but simple “data distillation” often isn’t enough, and unsupervised machine-learning systems can be dangerous. (Like, bringing-down-the-entire-financial-system dangerous.) Data Science is the key to unlocking insight from Big Data: by combining computer science skills with statistical analysis and a deep understanding of the data and problem we can not only make better predictions, but also fill in gaps in our knowledge, and even find answers to questions we hadn’t even thought of yet.

In this talk, David will:

• Introduce the concept of Data Science, and give examples of where Data Science succeeds with Big Data … and where automated systems have failed.
• Describe the Data Scientists’ Toolkit: the systems and technology components Data Scientists need to explore, analyze and create data apps from Big Data.
• Share some thoughts about the future of Big Data Analytics, and the diverging use cases for computing grids, data appliances, and Hadoop clusters
• Discuss the skills needed to succeed
• Talk about the technology stack that a data scientist needs to be effective with Big Data, and describe emerging trends in the use of various data platforms for analytics: specifically, Hadoop for data storage and data “refinement”; data appliances for performance and production, and computing grids for data exploration and model development.

You can register for this free webinar at the Revolution Analytics website. (Update Nov 6: the slides and a replay of the talk are now available at that link.)

Also, if you're attending the Strata / Hadoop World conference in New York this week, be sure to check out Thursday's talk by Steve Yun from Allstate Insurance and Joe Rickert from Revolution Analytics, which will include some real-world benchmarks of doing big-data predictive modeling with Hadoop and Revolution R Enterprise.

Start Small Before Going Big

The availability of Hadoop and other big data technologies has made it possible to build models with more data than statisticians of even a decade ago would have thought possible. However, the best practices for effectively using massive amounts of data in the construction and evaluation of statistical models are still being invented. As is the case with most difficult complex problems: “If you’re not failing, you’re not trying hard enough”. The majority of ideas tried do not work. Best practices should include keeping failures small and inexpensive, quickly eliminating approaches that are not likely to work out, and keeping track these failures so they won’t be repeated. Every development environment should encourage trying multiple approaches to problem solving.

This talk presents a case study of statistical modeling in the insurance industry and examines the trade-offs between working with all of the data in a Hadoop cluster, dealing with complex programming, significant set-up times and a batch-like programming mentality, versus rapidly iterating through models on smaller data sets in a dynamic R environment at the possible expense of model accuracy. We will examine the benefits and shortcomings of both approaches and include model accuracy, job execution time and overall project time among the performance measures. Technologies examined will include programming a Hadoop cluster from R using the RHadoop interface and the RevoScaleR package from Revolution Analytics.

You can find more details about this talk and the Hadoop World conference (of which Revolution Analytics is a proud sponsor) here.

Eight new R User Groups worldwide

There are new local R user groups in eight (!) countries to announce this month:

• Sweden is host to the first R user group in Scandinavia. StockholmR has been holding meetings since September, and their next meeting on November 28 will be on Teaching R and Data visualization using R.
• In Taiwan, the Taipei-based Taiwan useR Group holds regular Monday meetings on machine learning and data mining with R. Their next meeting is on October 29.
• The first R user group in Thailand, Chang-Mai R has brings together R users in Chiang Mai province. You can also connect with other Thai R users via their Facebook page.
• The Marburg R Local User Group boosts a strong R presence in Germany, bringing the total number of R user groups in the country to eight.
• Luxembourg also boasts a new local R user group. Join LuxR on MeetUp to help organize their first meeting. 
• The Poznań R Users is a new group for R users in Poland. Visit the Thinking in R blog for news of upcoming meetings and slides and reviews from previous meetups.
• The brand-new New Delhi UseR Group joins Bangalore as the second R user group in India. No meetings are scheduled yet, but you can join the group to help organize the first.
• Last but not least, Comunidad R Hispano is an on-line forum for R users across Spain. Local chapters in Madrid and Barcelona host meetings for local R users in those cities.

Several of these groups have taken advantage of sponsorship from Revolution Analytics to get their R user group up and running. If you're thinking about starting an R user group in your city, or help organize an existing group, applications for R user group sponsorship for 2013 are now open. If you'd like to join an R user group in your area, you can find a directory at the link below.

Revolutions: Local R User Group Directory

Because it's Friday: 7 billion-person 'continents'

The population of the world has been over 7 billion for about a year now. But those seven billion aren't distributed equally around the globe. 1.2 billion people — about  in India alone (despite it havingjust 2% of the world's land area). At the other end of the spectrum, the entire continent of Australia houses about 0.3% of Australia.

So what would the world look like if we divided it into continents not by political or geographic boundaries, but by population? For the sake of argument, what if the Earth were divided into seven continents of a billion inhabitents each? Armed with MS paint and Wikipedia's table of country populations, Redittor delugetheory drew this map:

Each color on the map represents 1 billion people. The continent of Africa, neatly, already has a total population of around a billion, as do North America and South America combined. (For some reason Greenland and Australia/NZ, with comparatively negligible populations, got thrown in Continent Green as well). For the rest of the world we need some judicious reassembling of the continents. 

While this map was assembled by hand, it would be interesting to see what map an algorithmic approach would create. It would be kind of like a 7-way knapsack problem, I guess. Unfortunately, you couldn't just use country populations as the basis, since China and India already have more than 1 billion inhabitants and need to be split apart. I guess you could arbitrarily break them up into 2 countries each, or perhaps work from the city population list (and distribute the remaining population in countries uniformly over their area). Any R programmers up to the task of automating this problem?