Revolutions

Using R in the Human Resources department

Ajay Ohri, author of R for Business Analytics, was recently interviewed about using R instead of Excel for human resources management. Human Resources is an increasingly analytics-driven field, with predictive modeling now used to prioritize applicants for interview and to organize workspaces, to give just two examples. As Ajay points out, R gets over the limitations of spreadsheets:

There is a lot of data out there and it’s stored in different formats. Spreadsheets have their uses but they’re limited in what they can do. The spreadsheet is bad when getting over 5000 or 10000 rows – it slows down. It’s just not designed for that. It was designed for much higher levels of interaction.

In the business world we really don’t need to know every row of data, we need to summarise it, we need to visualise it and put it into a powerpoint to show to colleagues or clients.

In a similar vein, the social network Facebook uses R for a variety of internal human-resources analytics applications. (The fact that Facebook employees use Facebook to collaborate internally provides a wealth of data to work with.) These applications include:

• Selecting peer reviewers during performance review season
• Setting up optimally constructed teams, and even optimizing seating charts to improve interactions and productivity
• Automatically filtering internal news feeds of employee content
• Suggesting new colleague interactions, and giving managers insight into employee interactions.

You can read more about how Facebook uses R for HR applications in Eric Sun's presentation (PDF), Criss-Crossing the Org Chart: Predicting Colleague Interactions with R.

Video: How John Deere uses R

Farming equipment manufacturer John Deere uses R, and in yesterday's webinar their manager of forecast analytics, Derek Hoffman, explained what they use it for:

 

In the presentation, Derek gave a spirited argument of why R is critical for John Deere's operations: from forecasting demand for equipment, to forecasting crop yields (they produce forecasts for more than half the world's food supply), and even optimizing the build order on the production line that produces the tractors. Derek, the longest-serving statistician at John Deere, showed how R is used for time series modeling and geospatial analysis in a reliable — and above all, reproducibe — way, and how the results are integrated with Excel and SAP. He'll wrap up with some thoughts on the "Big Data" hype cycle, and the impact that professional statisticians will have in the future.

You can download a video of the replay and the slides at the link below.

Revolution Analytics webinars: Order Fulfillment Forecasting at John Deere: How R Facilitates Creativity and Flexibility

More data apps spawned by Sandy

As the clean-up continues on the eastern seaboard, I wanted to follow up on Monday's post on tracking Hurricane Sandy with Open Data with a couple of other R-based data applications spawned by the storm.

Josef Fruehwald created an R script to tap into local weather sensors to keep track of air pressure, wind speed and rainfall near his home and for other affected locations. Check out the plunge in barometric pressure as the centre of the storm passed near Atlantic City on Monday night, and then over Philadelphia a couple of hours later:

The greater scale of destruction in Atlantic City compared to Philadelphia is reflected in this comparison of wind speeds over the same period:

Joseph used R to download the data from Weather Underground station records (like this one) in XML format, and then used the ggplot2 package to create the charts. You can find the complete R script at RPubs: Analysis of a Philadelphia Weather Station Data during Hurricane Sandy.

Twitter's Ed Chen took a different look at the effects of Sandy: how the storm knocked out power around the NY region, and the race by the utulities to restore power. Where did he find such data? Tweets from Twitter users, of course! Geolocated tweets (from smartphones, presumably) along the lines of "I've just lost power!" , "Still no power here!" and "The power is back on!" gave Ed information he needed to use R to create an animation over time of the power going out and coming back on across NYC.

The above is just a snapshot; visit Ed Chen's blog for an animated version, with a slider to see the changes during the night of the storm.

Quick notes from Strata NYC 2012

The O'Reilly Strata conferences are always great fun to attend, and this latest installment in New York City is no exception. This one is super-busy though; the conference has been sold out for weeks -- and not just marketing-sold-out, it's fire-department-sold out. It's non-stop conversations and presentations, and it's tough to move through the hallways in between.

Nonetheless, I thought I'd pause for a couple of minutes and share some of the highlights for me so far.

• Ed Kohlwey and Stephanie Beben gave a three-hour tutorial on the RHadoop project, showing the packed room how to crunch big data. They shared how consulting firm Booz Allen Hamilton uses R and Hadoop for data exploration; to run many tasks in parallel; and to sort, sample and join data. They've also create a very handy VirtualBox VM including R, Hadoop, RHadoop and RStudio (along with demonstration script files) which I hope to be able to post a download link for soon.
• Stan Humphries from Zillow gave a presentation on how data and statistical analysis drives Zillow's home valuation service. One fascinating tidbit: while Zillow has long used R to fit their valuation model, until recently they recoded the model scoring algorithm in C++ for use on the production site. The process of re-implementing a new version of the model, validating it, and deploying it used to take 9 months. But now that they run R in production via the Amazon cloud, without the need to recode the model in another language, the deployment time for new valuation models is just four weeks.
• Mike Driscoll from Metamarkets shared the technology behind their data stack: node.js and D3 for visualization; R and Scala for analytics; Druid as the data store; and Hadoop and Kafka for ETL. Druid is MetaMarket's home-grown high-performance, which they announced today is now available as open source software.
• In a similar vein, Cloudera announced the release of Impala, an open-source project two years in the making to bring high-performance real-time analytics to Hadoop.
• And there were even more announcements: Kaggle launched a partnership with EMC to give Greenplum users direct access to the roster of Kaggle data scientists competitors.

It's been a great conference so far, and this is only day one! Looking forward to more great talks and conversations tomorrow.

The rapidly increasing ideology of the US Republican Party

The chart below comes by way of the is.R blog and shows the average ideology of the members of the United State House of Representatives within the Republican (red) and Democratic (blue) parties. (Other parties are shown in green.) The chart is shown as a time series, from the first US congress in 1789, to the most recent full congress (the 111th, from 2010). The 80th congress first met in 1947.

Ideology here is measured according to the DW-NOMINATE scale, a score that ranges from -1 (extremely liberal) to +1 (extremely conservative) based on legislative votes, and is designed to be comparable across time. (The source data is available for download at voteview.com.) The width of each band represents the spectrum of all but the 10% most extreme legislators within each party; the dark dot shows the median. 

The graph shows that while both parties have always been distinct in their ideology, since about World War 1 there's at least been some slight overlap. All that changed in the early 70's though, as successive Republican congresses became increasingly more conservative in their voting records, while Democratic congresses remained much the same. Today, there's no ideological overlap between members of the two parties.

The chart above was created using the R language, and R programmers should take a look at the source code posted at the is.R blog. It's a great example of using the ddply function (from the plyr package) to aggregate the individual legislator ideology scores into medians and quantiles by party and congress. The ggplot function was used to create the chart itself, by overlaying the interquartile range and median points over a line chart of the 90% range. For more details, check out the full post at the link below.

(Update Nov 6: See another version of this chart based on code from Mike Lawrence in the comments below.)

is.R: The distribution of ideology in the U.S. House (with plyr)

Designing real-world 3-D objects with R

The Maker Movement has led to the production of open-source 3-D printers and other manufacturing machines that allow hobbyists to design, create and produce real-world objects affordably. Now R user Ian Walker, in a post at the Psychological Statistics blog, shows how to use the R language to transform 3-D surfaces into real-world physical objects with a 3-D printer. Ever had trouble visualizing exactly what a 3-D surface looks like when constrained by a boring old 2-D monitor screen? Now you can print it, and rotate and inspect it in realspace:

Walker's open-source R code works much like the standard R persp function, but instead of translating a grid of (x,y,z) points to a mesh surface on the screen, it creates a stereolithography file for use with 3-D printers, which in turn can be used to create an object like that shown above. Now, if only I had a 3-D printer to play with...

Psychological Statistics: Visualizing data using a 3D printer (via Dirk Eddelbuettel)

Using R in production: industry experts share their experiences

I had a great time yesterday moderating the "R in Action" panel discussion at the DataWeek conference in San Francisco. Each of the panelists represented a company that is actively using R and/or Revolution R Enterprise. Here (from memory, since I couldn't take notes) are some the things they shared:

• Jesse Bridgewater from eBay talked about how R is used within eBay's operations to analyze massive volumes of data. 
• Anthony Goldbloom from Kaggle confirmed that more Kaggle competition winners use R than any other software package. Random Forests is a particularly popular predictive modeling technique in the competiitions.
• Jim Porzak from Minted espoused the benefits of using R: the flexibility of the language, the data visualization capabilities, and the extensive user community and CRAN pacakages.
• Hsiu-Khuern Tang from Intuit described how R is used in the creation of the Intuit Small Business Index, and some of the advanced modeling packages (such as glmnet and the Rmetrics suite) used at Intuit.
• John Wallace from Upstream Software talked about dealing with Big Data in R, and using the modeling routines in Revolution R Enterprise to fit regression models for use in production.
• Damien Weldon from Corelogic contrasted using (previously) SAS and (now) R for financial data analysis.

Other topics of conversation included building data science teams to work in modern analytical environments, the strengths and weaknesses of R, and the technology stack and other applications working with R. All in all, it was a lively and entertaining discussion — my sincere thanks go to all of the panelists for sharing their insights and expertise.

A video map of the hottest songs in the US

Today's guest post comes from Revolution Analytics data scientist Luba Gloukhov — ed.

As a big fan of R’s ever-expanding geospatial plotting capabilities, I jumped at the chance to create a map using The Million Song Dataset (MSD).  For each of the 10,000 songs in the million song subset containing non-missing latitudes & longitudes and non-zero artists familiarity scores, I added a song ‘break-out’ score — the ratio of song hotttnesss to artist familiarity (similar that introduced by Echo Nest’s Hottt or Nottt blog post).  I then subset the data further to only include songs with a break-out score of greater than 1.

Exploring the map, I found myself YouTubing songs. Many of them I had never heard of before — a sign of either generally low artist familiarity scores or my own tanking hipppnesss.  I decided against exploring the “why?” fearing it might just prove the latter.  In an effort to simplify my process of exploration (and “what’s hottt”-self-education), I used R to embed YouTube videos in the map’s marker info windows. 

Explore the interactive map here.

For each song in my subset, I added a new variable containing code that embeds the first YouTube video search result of the artist name and song title. For plotting, I used Milan Kilibarda’s plotGoogleMaps package. You can access my code and data via github.

Oakland’s own Del tha Funkee Homosapien’s 1991 hit Mistadobalina came up as a recent break-out song (with a ratio of 1.04).  Since 1991, Del’s had numerous successes – as a member of Hieroglyphics, with the album Deltron 3030 and, perhaps most prominently, as a member of the Gorrilaz.  One would think that, by now, Del’s familiarity would surpass Mistadobalina’s hotttnesss, generating a ratio of less than 1.  How does this ratio compare to those of other Del songs? Is it really the case that Mistadobalina remains Del’s biggest break-out hit or do the hotttness scores of Del’s songs in general surpass his familiarity?  Perhaps more enlightening than momentary snapshots of these metrics would be an investigation of how the variables have changed over time.  

So much data exploration, so little time! I’ll have to save these topics for another R play date.

Luba Gloukhov is a Pre-Sales Engineer at Revolution Analytics. When not playing with R or helping others do the same, she can be found lifting heavy objects, thinking light thoughts or eating delicious food.

 

Designing Data Apps with R at Periscopic

Today's guest post comes to us from Andrew Winterman, Data Designer at data visualization company Persiscopic. He shares with us the process of using the R language and other tools to create an interactive data application for a client — ed. 

The Hewlett Foundation contacted us a few months ago because they were interested in exploring ways to visualize the distribution and impact of their grantmaking efforts over the last ten years. They hoped to make a tool with three functions: It would provide insight into where the Foundation has made the largest impact; provide grant seekers context for their applications; and help the Foundation’s officers make decisions about new grantmaking efforts, based on their existing portfolio. They had one request: No maps.

The data arrived, as it so often does, in the rough: An Excel document compiled quickly, by hand, with the primary goal of providing an overview, rather than complete accuracy. At this point in the process, we paint with broad brushes. We learn the data’s characteristics, determine which facets are interesting, and prototype visualization ideas.

At the beginning of a project, I always explore a few simple visualization techniques to get a feel for the data. For example, simple bar charts as shown in Figure 1, scatter plots, and choropleths, are great ways to get a visual sense of what the data is saying. 

 

Figure 1

My main tools for this process are d3.js, R — ggplot2 in particular — and Tableau. For this project I used ggplot2 (version 0.9 came out halfway through) and the CRAN package 'beanplot'.

 

Once we have a feel for the data, we start brainstorming, and trying out ideas. For example, an early idea led us to explore using concentric circles to represent the tree of geographic categories (Hemisphere, Continent, Country, Region, County, City), and then filling an arc of a circle with a scatter plot to show individual grants. You can see this idea sketched, with mostly fake data in Figure 2. We ultimately decided the technique didn’t use space effectively enough for what we needed to convey. 

Figure 2

Our next idea was to use modified beanplots [Figure 3] to succinctly describe the distribution of various quantities at the same time. These were made with the beanplot package available on CRAN. We denormalized them — meaning we hacked the beanplot function to make the total area of the beanplot proportional to volume. With traditional beanplots, the total area of each bean is always the same, since they represent probability distributions rather than counts. This is counter-intuitive if the viewer is unfamiliar with statistics. We actually went as far as developing a working tool using these modified bean plots. 

Figure 3 : Beanplot

The width of the bean at a given dollar amount shows the probability the next dollar falls at the given amount. After extensive user testing, this proved too high a cognitive hurdle for the casual viewer. Users liked the visual presentation, but were confused as to their meaning, even with a detailed page showing how to interpret the beanplot.

We decided to consider alternatives to the beanplot that still accomplish the same goals. We also wanted a very simple technique that could be explained in a phrase. After a few iterations, we agreed that interactive heatmaps [Figure 4] would be a good solution. You will be able to see them in action at Persiscopic.com when the final product launches at later this year.

Figure 4: Heatmap

R provides an ideal toolkit to explore methods to visualize data distributions. Between specialized packages and comprehensive toolkits like ggplot2, a wide range of techniques are available to the analyst. In particular, the transparent structure of most R functions make them easy to pull apart and put back together again, lending great flexibility to the patient programmer.

Andrew Winterman does Data Design for Periscopic. An inquisitive humanist, he is motivated by the promise of making ours a more rational society. He applies his skills to the problem of converting data into information, a process requiring scripting and research into the relevant fields of study. He holds a B.A. in Mathematics from Reed College, and patiently pursues a Masters of Science in Biostatistics at the Oregon Health and Sciences University. He greatly enjoys his daily bicycle commute, Portland’s artisanal culture, searing vegetables in cast iron, and thinking about epidemiology.

Kickstarter facilitates $50M in indie game funding

The social crowdfunding site Kickstarter announced today that it has enabled, via community contributions, $50M in funding in 2012 for new indie games. The second largest category of funding was for independent films. The blog post announcing the news includes analysis (using R, natch) of the breakdown in categories and funding sources. 

On a personal note, two Kickstarter projects I've contributed to recently are for video games: one to bring the Star Trek: The Next Generation pinball table to the iPad, and another to bring an entirely new pinball table from legendary designer Pat Lawlor to the digital realm. (Regarding the latter project, I had a small hand in the design of the Pro Pinball tables back in the '90's, so I'm psyched that they're getting remastered.) If you're a pinball fanatic like me, or just love great video games, you might want to check 'em out.

Kickstarter blog: Year of the Game