Revolutions

Big Analytics: Closing the "clue gap" with Big Data

There's been an growing discussion over the past couple of years on the topic of Big Data: how to deal with the situation when you have more data than can be conveniently managed and analyzed by traditional software tools. But Big Data has little intrinsic value in its own right: its value is only realized when you can deploy analytical expertise to draw actionable information from the data. Dion Hinchcliffe at ZDNet aptly calls this "closing the clue gap": 

As accumulated information has become a top-line asset in large companies, the ability to tap into it and release value from it is not growing to match in most traditional firms. Yet this information is currently growing exponentially and becoming a challenge in its own right. ... Smart organizations can apply Big Data methods to solve existing business problems, implement new business models, and drive growth in innovative new ways. That is, if they can find a way to move beyond their parochial ways.

Tim O'Reilly put it more succinctly, with this quote (which inspired Hinchcliffe's article):

Companies that have massive amounts of data without massive amounts of clue are going to be displaced by startups that have less data but more clue.

Our take is that the "clue" comes from the expertise of data scientists, who can apply the methods of statistical analysis and predictive analytics to create value from Big Data. We call this Big Analytics. In a new white paper, Revolution Analytics' CEO Norman Nie explores how statistical analysis techniques are critical to deriving value from Big Data, by allowing analysts to:

• Deal with a non-linear world, and move beyond the linear approximation with advanced software tools like the R language
• Create powerful models and predictions with data mining
• Detect and capitalize on rare events
• Extracting and analyze ‘low incidence populations’
• Address the conundrum of 'statistical significance' in Big Data 

Dr. Nie introduces the white paper as follows:

The enormous growth in the amount of data that the global economy now generates has been well documented, but the magnitude of its potential impact to drive competitive advantage has not. It is my hope that this briefing urges all stakeholders—executives who must fund analytics initiatives, IT teams that support them and data scientists, who uncover and communicate meaningful insight—to go boldly in the direction of “Big Analytics.” This opportunity is enormous and without precedent.

The white paper is available for download now (with free registration) from the link below.

Revolution Analytics: The Rise of Big Data Spurs a Revolution in Big Analytics

What language is R written in?

On of the nice things about R is that a lot if it is written in the R language. That means, as an R user, if you want to see how R calculates a certain statistic, or you want to modify an existing function for your own use, you can just look at the R code by typing the name of the functions. Sometimes, though, you'll see just a couple of lines of R code even for quite complicated functions -- in these cases, the work is mostly being done in a lower-level programming language like C or Fortran via a call to .Internal or .Call. (Of course, since R is open source, you can take a look at the C or Fortran code as well, if you like.)

So the natural question follows: how much of R is written in the R language? The librestats blog did an analysis of the latest R source code distribution, and found that 22% of the R's lines of code are in R. 

For comparison, about 50% of R is written in C, and just under 30% in Fortran -- for the current 2.13.1 release, at least. Of course, these ratios have changed over time with each release of R, as this analysis of the R codebase from Ohloh shows:

Note that R's source distribution includes the language/graphics engine and only the base packages. All distributions of R also include the recommended packages, and most users install third-party packages from CRAN as well. For packages in general, the ratios are a bit different: when considering the 12.4 million (!) lines of code that make up the 3249 packages on CRAN, just shy of 50% of that code is in R. (Thanks again to librestats for the analysis.)

So in conclusion: while R itself is mostly written in C (with hefty chunks in R and Fortran), R packages are mostly written in R (with hefty chunks written in C/C++).

librestats: How Much of R is Written in R Part 2: Contributed Packages

Sharing live R functions with OpenCPU

OpenCPU is a new initiative from R user Jeroen Ooms to make innovations in statistics, visualization and data-science more widely applicable. Based on open-source principles, it's a web-based service that lets you upload data visualizations and analyses as R scripts, and allow others to run them on demand. For example, you can upload a script to visualize a company's stock performance, which can then be embedded as a live chart, like this:

That chart was generated on-demand by the OpenCPU R server, and should display an up-to-date stock history for IBM. It was generated by a few lines of R code that Jeroen published to OpenCPU, and I called from this blog post using the OpenCPU API. You can publish your own code there as well, and share new data science techniques implemented in R in the same way. 

As an open platform, be aware that any everything you do and post on OpenCPU should be considered public, and so isn't suitable for anything that contains private or sensitive data. If you're looking for a secure platform for distributing R code, where access to sensitive or resource-intensive code needs to be restricted to authenticated users, then you should take a look at the RevoDeployR platform from Revolution Analytics. (Jeroen gave a presentation about the use of RevoDeployR to address such issues at the useR! 2010 conference.)

OpenCPU: Documentation

Because it's Friday: Parkour Plus

Since I missed doing a BIF post last week due to flight schedules, here's two videos for you this week. Both show some amazing skills as these athletes interact with urban environments. The first, more professionally produced (and featuring some outstanding cinematography) has Danny Macaskill doing some incredible things with a bicycle (thanks to reader IF for the tip):

The second comes from parkouriste (and presumably, aspirant stuntman) Damien Walters. He's produced slicker showreels in recent years, but this one from 2009 remains my favorite, if only for the flamboyant stunt at 0:56:

Enjoy the weekend, and for those in the path of Irene: stay safe.

Revolution R: 100% R and More - slides and replay

If you missed this week's webinar, the slides from my presentation Revolution R Enteprise: 100% R and More may be useful as an introduction to R and the additional capabilities of Revolution R Enterprise.

The slides themselves and the replay video are also available for download from the link below.

Revolution Analytics webinars: Revolution R Enterprise: 100% R and More

9 more ways to bring data into R

Here's a followup to yesterday's post on using the rdatamarket package to import data into R. Ajay Ohri at the DecisionStats blog offers nine additional methods for bringing data into R, from sources including InfoChimps, the Google Prediction API, the World Bank World Development Indicators, Bloomberg Market Data, and much more. See Ajay's post at the link below for the details.

DecisionStats: Using #Rstats for online data access

How to access 100M time series in R in under 60 seconds

DataMarket, a portal that provides access to more than 14,000 data sets from various public and private sector organizations, has more than 100 million time series available for download and analysis. (Check out this presentation for more info about DataMarket.) And now with the new package rdatamarket, it's trivially easy to import those time series into R for charting, analysis, or anything. Here's what you need to do:

1. Register an account on DataMarket.com (it's free)
2. Install the rdatamarket package in R with install.packages("rdatamarket")
3. Browse DataMarket.com for a time series of interest (I found this series on unemployment)
4. Copy the URL of the page you're on (the short URL works too, I used "http://data.is/qb61uf")
5. Use the dmseries function with the URL to extract the time series as a zoo object

Here's an example:

> library(rdatamarket)
> dminfo("http://data.is/qb61uf")
Title: "Persons Unemployed 15 weeks or longer, as a percent of the civilian labor force"
Provider: "Federal Reserve Bank of St. Louis" (citing "U.S. Department of Labor: Bureau of Labor Statistics")
Dimensions:
> unemp <- dmseries("http://data.is/qb61uf")
> plot(unemp)
> str(unemp)
‘zoo’ series from Jan 1948 to Jul 2011
  Data: num [1:763, 1] 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 0.5 ...
 - attr(*, "dimnames")=List of 2
  ..$ : chr [1:763] "1" "2" "3" "4" ...
  ..$ : chr "Persons.Unemployed.15.weeks.or.longer..as.a.percent.of.the.civilian.labor.force"
  Index: Class 'yearmon'  num [1:763] 1948 1948 1948 1948 1948 ...

Created by Pretty R at inside-R.org

With this package, you can go from finding interesting data on DataMarket to working with it in R in less than a minute. With such a wealth of data so easily available to the power of R, this will be a fantastic tool for all data scientists and data journalists.

CRAN: rdatamarket package

New R User Group at University of Utah

There's a new local R user group in Salt Lake City, based at the University of Utah. (There used to be another group in Salt Lake devoted to R/Weka/Processing, but it appears to now be defunct.) This new group has been meeting regularly for some time, and their next meeting, on September 9, will be devoted to short talks about R.

University of Utah - R Users Group: Home

 

This infographic goes to 11

I don't mean to rag on the folks at Microsoft -- they make some good products, and they're a partner of ours, and it's not technically even by them -- but I agree: this infographic is so bad, it's almost campy B-movie good.  

Click to enlarge: it's huge. I think my favourite part is the shining barchart-cum-sun rising over Cloud Mountain (is there any relationship between the numbers and the lengths of the rays?). The bar-balloons are equally wonderful, with the half-deflated nature adding to the pathos considerably. And the straw-thing penetrating the island is sublime in its ridiculousness. If I were to quibble, though (and I just may ...) I'm pretty sure most people understand cloud more than 3. At least 8, for sure.

Microsoft Asia: Cloud Computing Accelerating in Asia Pacific (via)

The problem with R? Too much new stuff!

In a tongue-in-cheek post at the Information Management blog, Steve Miller shares his "frustration" with R: package developers keep on releasing new functionality for R that makes his own work obsolete. For example, there's now pre-packaged functionality in R for enhanced dotplots, Economist-style graphics, additive regression models and more, which all obviate the need for Steve to implement such things himself. Steve reveals his agenda in the last paragraph: 

Of course, my tongue-in-cheek observations are in fact a celebration of R rather than a critique. I'm continually amazed at the ever-expanding bounty of contributions from some of the top quantitative analysts in the world. With this wealth of goodies, though, comes an admonition from the old buy versus build conundrum: do your research on what's already available before you invest a lot of time developing from scratch. Often as not, what you're looking to do is already out there – for free. I need to practice that lesson even as I preach it.

Check out Steve's full post at the link below.

Information Management (Steve Miller): Frustrations With R?