Revolutions

Vote in the KDnuggets poll on Analytics Software

The 14th annual KDnuggets poll measuring use of analytics software is open for voting. The poll asks, "What Predictive Analytics, Big Data, Data mining, Data Science software you used in the past 12 months for a real project?" and allows up to 20 choices from commercial software, open source software, and "big data" software. R was the leading choice in 2012, and currently leads the pack in the voting for 2013. 

You can enter your vote now at the link below.

KDnuggets: New Poll: Predictive Analytics, Big Data, Data Mining, Data Science Software Used

Big Data Republic's list of Top 20 Big Data Accounts on Twitter

I'm very proud to have been selected today for Big Data Republic's list of the Top 20 Big Data Twitter accounts. With such Big Data practitioners, analysts and thought leaders like Kirk Borne, Hilary Mason, Pete Skomoroch, Mike Olson and Doug Henshchen on the list, I'm humbled to be listed at #3. My co-presenters from the recent data science webinar, Carla Gentry and Gregory Piatetsky (#1) are also on the list.

Special thanks to judging panel member (and Teradata's Chief Analytics Officer) Bill Franks, for these kind words about my tweets:

He seems to come up with content on a more frequent basis than most. At the same time, his content is consistently good. That’s a hard combination to find.

If you'd like to follow all 20 of the accounts recognized by the judging panel, I created this Twitter list: BDR Top 20 #BigData. Check out at all of the members and the comments from the judging panel at the link below.

Big Data Republic: @kdnuggets Voted the Best Big Data Twitter Account!

Social Network Analysis at New Frontiers in Computing 2013

by Joseph Rickert

This past Saturday, the New Frontiers in Computing Conference (NFIC 2013), held at Stanford University, explored the theme: Social Network Analysis: It’s Who You Know. The speakers were a well-chosen, eclectic lot who covered a remarkable array of issues in less than a full day. Ian Hersey, former CTO of Attensity spoke on Lessons from Large-Scale Social Analytics. Michael Wu, Chief scientist of Lithium Technologies, provided an introduction to social network analysis and very gamely conducted a live experiment building a social network of attendee tweets during the conference. Rong Yan, the Engineering Manager for Ads Relevance and Quality at Facebook spoke about machine learning insights. Zahan Malkani, an engineer at Facebook, presented “Dog”, the yet to be released social media programming language. Shivakumar Vaithyanathan, Chief Scientist for Text Analytics at the IBM Almaden Research Center that is built around IBM’s Annotation Query Language (AQL). Laura Jacob, a Factset engineer and president of the IEEE’s Society on the Social Implications of Technology spoke about “Context Collapse”, a fundamental cause for the damaging “oversharing” trap that so many Facebook and Twitter users fall into. Finally, John Rehling, Senior Research Scientist at Reputation.com, “cleaned up” with an alarming discussion of the mind boggling hazards we all face in just using the Internet.

Although most of the talks were obviously enhanced versions of corporate presentations, there was nothing superficial about the day. Collectively, the presentations and panel discussions provided a comprehensive, multidimensional look at the technologies, issues and challenges associate with social networks. Most refreshingly, the day was mostly hype free — no beating the drum for big data or promoting unreasonable expectations for Hadoop.  The presenters all seemed to pretty much be in agreement about the current best practices in technology. Hadoop, for example, was characterized as being the place for massive amounts of persistent data, but not a suitable platform for ingesting social media data where low latency is of paramount importance. And, Rong Yan pointed out that although Facebook is a big Hadoop shop they do not use Map-Reduce for analyses that require status sharing among processors distributed across the cluster. R came up at various times during the discussions in a matter of fact way. Rong pointed out for example, that for data stored in Hadoop clusters Pig or Hive will typically be used to aggregate data at which point it is no longer big data. After that R, Matlab or SQL might be used for analysis. He indicated that most business questions can be answered with relatively small data sets. When it really is necessary to work with a large data set then the analysis is likely to be done in C++. At one point Shivakumar casually remarked that AQL syntax looks a lot like R.

A technical highlight of the day was Michael Wu’s introduction to social network analysis (SNA). With the help of an open source plug-in to Excel he was able to start from first principles and work up to explaining some fairly sophisticated performance metrics for social network graphs such as eigenvector centrality. Basically, this is the notion of giving high scores to nodes that are connected to nodes that are themselves central within the network. (For a very nice explanation of this idea and pointers to the source papers have a look at the Plos paper by Gabrielle Lohmann et al.)

Michael gave a remarkably clear presentation and although he did not use R he could have. For anyone with an interest in getting started with SNA I recommend the 2010 Social Network Analysis Labs in R written by McFarland, Messing and Nowak. The labs use functions from the igraph package and data from the NetData package to provide a challenging introductory SNA course. The first plot (from the 4th lab) shows a network graph of student interactions using the studentnets.S641 data set.

This next plot shows the Eigenvector centrality score for each student.

The most fascinating and distressing presentations and discussions happened in the section on Privacy Implications for SNA. Laura Jacob started things off here by providing some social theory background for the problem of inadvertently oversharing on social media sites. Frequently this sort of thing happens when the imagined audience for a tweet, message or photo turns out not to be the actual audience. This “context collapse” results from the tension between the individual’s attempt to establish some level of privacy and the social media site’s desire obtain information. Laura explained that social media sites know that if they put you a certain context you are more likely to share information that is appropriate for that context. However, unless you are really careful about the privacy settings the actual context might include a wider audience than intended. At some level, participating in social media is like continually reliving that part of your wedding day where you worked very hard to limit the conversation between your new in-laws at Table 1 and your Vegas party friends seated in Table 12. For more on the theory take a look at Laura’s suggested reading list of (Goffman 1959) and (Marwick 2010)

In the final presentation of the day, John Rehling took the attendees through the “Spectrum of Social Distance”: self < younger self < family < friend < acquaintance < enemy; recounted a number of cases where reputations were tarnished and irrevocable damage done by people closer than family and then pointed out that in the future we can expect to live in a world where individually innocuous bits of information will be assembled to form damaging information.

This very brief summary of the conference does not do justice to any of the presenters, but will end here with Ian Hersey’s list of ongoing challenges for SNA:

• The growth in the volume of data (10% increase per month)
• Data Quality Assurance
• Rich natural language processing in many languages across many domains
• The sparseness of geocoded data
• Veracity (There is lots of gaming going on in social media)
• Irony / sarcasm detection

Finally, I'm betting that not long after Dog we will have “RDog”.

SAS Big Data Analytics Benchmark (Part Two)

by Thomas Dinsmore

On April 26, SAS published on its website an undated Technical Paper entitled Big Data Analytics: Benchmarking SAS, R and Mahout.  In the paper, the authors (Allison J. Ames, Ralph Abbey and Wayne Thompson) describe a recent project to compare model quality, product completeness and ease of use for two SAS products together with open source R and Apache Mahout.

This is the second post in a two-part series.  Last week, I covered simple mistakes and errors. this week's post will cover the authors' methodology and findings.

I have contacted the authors and asked if they plan to release their data to the community.  To date, I have received no response. 

Methodology

The methodological problems in the SAS benchmark paper are too numerous to cite comprehensively, but I'll review the most glaring issues.

Choice of Products to Test:  Although the authors compiled the list of algorithms to be tested through a survey of SAS users, no single SAS product supports all three.  R supports all three, as does Mahout (with a workaround). 

The choice of products for comparison is strangely "apples and oranges".  Mahout is an early-stage analytics library; R is a mature analytic programming environment; SAS Rapid Predictive Modeler is a semi-automated ensemble modeling framework, and SAS High Performance Analytics Server is a high-end analytics platform designed for use in an MPP computing environment.  Comparing the two open source products with two high-end SAS applications on “ease of use” is a straw man comparison.  One would expect the two high-end SAS applications to be easier to use; that’s why SAS charges a premium for them.    

Oddly enough, SAS chose to omit its own product SAS/STAT from the benchmark. As an analytic programming language, SAS/STAT offers a more apt comparison to R, and it is far more widely used than either of the two high-end SAS products.

Analytic Workflow:  In the real world, data resides in external sources, and must be loaded into each of the respective platforms; analysts have choices when marshalling and loading data, and can do so in a way that best suits the application. The authors used SAS to prepare the data for use in all four products, starting with structured data sets loaded into Hadoop. This biases the "ease of use" assessment toward SAS.  

Structured data stored in Hadoop is not "native" to Hadoop but structured externally and loaded into the Hadoop file system; the authors did not assess the effort needed to do this, which biases the "ease-of-use" assessment towards SAS High Performance Analytics Server, which cannot operate directly on raw data. It also biases the assessment against Mahout, since the principal advantage of Mahout is its ability to work with data stored natively in Hadoop.  

The authors' decision to split the data into training and validation sets in SAS rather than using native splitting for each tool is puzzling. The stated reason for using this approach — that random seeds may differ among the tools — is fallacious; random samples of sufficient size are generalizable to one another even when produced with different random seeds.

Computing Environments: The computing environments used in this benchmark are vastly different. This is attributable in part to the widely different requirements of the products compared, but nevertheless there are some unexplained anomalies. There is no justification for using machines with very different memory profiles to host open source R and SAS Enterprise Miner; the two machines can and should be identical. By the same token, a five node Hadoop cluster is not equivalent to a sixty node MPP Greenplum appliance.  

Assumptions About User Skill: The authors tested the open source tools and the SAS high-end applications using a naïve "hands-off" approach that depends on "out of the box" functionality.  This is a trivial comparison that simply proves the obvious. When Deep Blue beats Kasparov at chess, it's news; when Deep Blue beats a three-year-old at chess, it's not news.  

Findings

The authors propose five main findings:

(1) The types and depth of classification models that could be run in SAS products and R outnumbered the options for Mahout. 

(2) The effort required by individual modelers to prepare a representative table of results was much greater in both R and Mahout than it was in SAS High-Performance Analytics Server and SAS Rapid Predictive Modeler for SAS Enterprise Miner.

(3) The object-oriented programming in R led to memory-management problems that did not occur in SAS products and Mahout; thus, the size of customer data that could be analyzed in R was considerably limited. 

(4) Overall accuracy was "comparable" across software in "most instances". 

(5) When the customer modeling scenarios were evaluated based on event precision and rank ordering, SAS HighPerformance Analytics Server models were more accurate. 

The first two points are true but trivial. Anyone with a smattering of knowledge understands that Mahout is an early stage project with limited functionality. The two SAS tools are high-end applications with a number of automated features. That's fine, there's nothing wrong with that for users willing to pay the license fees.  A reasonably competent R user can produce a comparable table of results with little effort; moreover, the table can be easily customized, which is not so easy to do with "out-of-the-box" tools.

The third point is silly. The authors attribute memory management issues encountered with R to its object-oriented nature. A more obvious explanation is that the authors deployed R on a machine with much less memory than the machine they used for SAS.

Points four and five relate to model quality, and deserve some analysis.

According to the authors, the overall misclassification rate (the inverse of overall accuracy) achieved by the four products is "comparable in most instances." They do not disclose the overall accuracy statistics, which leaves the reader in the dark about what they mean by "comparable" and which instances the overall accuracy was not comparable.

However, the SAS products produced models with higher precision and lift in the first decile. The authors choose to highlight this finding, reporting only the precision and lift statistics in the summary tables, while failing to report overall accuracy statistics.

If the models produced by SAS are more precise but not more accurate, it means that they were produced using techniques that minimize false positives at the expense of false negatives (since the overall error rate is the same). There are situations where this is a good thing — for example, in marketing campaigns where the goal is to optimize campaign ROI, the analytic goal is to target a list as precisely as possible. On the other hand, there are circumstances where the analyst must also be concerned about false negatives — for example, in a medical diagnostic situation where the cost of a false positive is "patient receives unnecessary dose" and the cost of a false negative is "patient dies."

While it is generally best to evaluate models using a range of measures and over the entire range of possible predictions (through a gains chart or ROC analysis), in the absence of information about the cost of errors, accuracy is the best single measure. And, as the authors admit, all of the products produced models of comparable accuracy.

But let's stipulate that increased precision is a good thing where models are equally accurate. How did the SAS tools produce the improved precision? As noted in the Discussion section on page 11, the SAS products accomplished this through use of well-known techniques:

• Oversampling the target event;
• Modification of model priors;
• Missing value imputation;
• Variable selection.

All of these methods are available in R; the authors simply chose not to use them. Which brings us back to a basic flaw in the authors' approach: it's news if SAS' automated tooling can beat a trained and reasonably competent analyst using R; it's not news if SAS' automated tooling can beat a passive and untrained analyst.

The reader should take note that SAS' automated tooling would also outperform a passive and untrained analyst using SAS/STAT. While all of the techniques detailed above can be implemented in SAS/STAT through the SAS Programming Language — a point recognized by the authors — they are not "automatic."

While there is a role for semi-automated model building tools in analytics, evidence from the marketplace suggests that working analysts in the real world prefer the flexibility and control offered by an analytic programming environment over "black-boxy" applications. This is true in the SAS user community as well as the broader community of analysts; users of SAS/STAT and the SAS Programming Language far outnumber users of the high end SAS applications included in this "benchmark."

Derek Norton, Andrie de Vries, Joseph Rickert, Bill Jacobs, Mario Inchiosa, Lee Edlefsen and David Smith all contributed to this post. 

Explaining real-time predictive analytics with big data (video)

In my presentation to the Strata Santa Clara 2013 conference earlier this year, my goal was to give a succinct (under 20 minutes!) explanation of three terms that are two often used as mere buzzwords: predictive analytics, real time, and big data.

 

You can download the slides for my presentation, Real-time Big Data Analytics: From Deployment to Production, from SlideShare.

In the talk I referenced the example UpStream Software's marketing attribution model. Last week I posted some details of how they used R to create and deploy the model for clients like Williams Sonoma, so follow that link if you're interested in the details.

I didn't have much time to get into why I believe that the R language is the ideal environment for creating such models, but I go into more depth in this longer version of the presentation.

Extending RevoScaleR for Mining Big Data - Naive Bayes

by Derek McCrae Norton, Senior Sales Engineer

In this third installment (following part 1 and part 2) of Extending RevoScaleR for Mining Big Data we look at how to use the building blocks provided by RevoScaleR to create a Naive Bayes model.

Motivation: Fit a Naive Bayes model to big data.

Naive Bayes is a simple probabilistic classifier based on applying Bayes' theorem with strong (naive) independence assumptions. This is often a good benchmark for other more complicated data mining models.

Really you are just calculating proportions for categorical variables (with possible Laplace correction), and probabilities based on a normal distribution for numeric variables. The proportions are easily calculated using rxCrossTabs, and the normal probabilities are easily calculated given a mean and standard deviation which we can get from rxSummary.

We can use existing e1071 code and replace the calculation of proportions and probabilities with big data versions. The results are not only not big data, but existing methods work on object!

You can test this out yourself with the function rxNaiveBayes at github.

Conclusions

It is pretty easy to extend RevoScaleR to do many tasks. These are only three example, but there are more on the github page. I also have a few more complicated examples that should be up eventually.

If you have an interest in helping to extend the functionality of RevoScaleR or just want to test some of the things I have created, please have a look at RevoEnhancements on github.

SAS Big Data Analytics Benchmark (Part One)

by Thomas Dinsmore

On April 26, SAS published on its website an undated Technical Paper entitled Big Data Analytics: Benchmarking SAS, R and Mahout.  In the paper, the authors (Allison J. Ames, Ralph Abbey and Wayne Thompson) describe a recent project to compare model quality, product completeness and ease of use for two SAS products together with open source R and Apache Mahout.

Today and next week, I will post a two-part review of the SAS paper. In today's post I will cover simple factual errors and disclosure issues; next week's post will cover the authors' methodology and findings.

Mistakes and Errors

This section covers simple mistakes by the authors.

(1) In Table 2, the authors claim to have tested Mahout "7.0".  I assume they mean Mahout 0.7, the most current release.

(2) In the "Overall Completeness" section, the authors write that "R uses the change in Akaike's Information Criteria (AIC) when it evaluates variable importance in stepwise logistic regression wheras SAS products use a change in R-squared as a default."  This statement is wrong.  SAS products use the R-squared to evaluate variable importance in stepwise linear models, but not for logistic regression (where the R-squared concept does not apply).  Review of SAS documentation confirms that SAS products use the Wald Chi-Square to evaluate variable importance in stepwise logistic.

(3) Table 3 in the paper states that R does not support ensemble models.  This is incorrect.  See, for example, these packages:

• gbm
• mboost
• party
• randomForest
• caret

(4) The "Overall Modeler Effort" section includes this statement: "Bewerunge (2011) found that R could not model a data set larger than 1.3 GB because of the object-oriented programming environment within R." The cited paper (linked here) makes no general statements about R, it simply notes the capacity of one small PC and does not demonstrate a link between R's object-oriented approach and its use of memory. The authors fail to state that in Bewerunge's tests R ran faster than SAS in every test where it was able to run, and that Bewerunge (a long-time SAS Alliance Partner) drew no conclusions about the relative merits of SAS and R.

Disclosure Issues

Benchmarking studies should provide sufficient information about how the testing was performed; this makes it possible for readers to make informed decisions about how well the results generalize to everyday experience. For tests of model quality, publishing the actual data used in the benchmark ensures that the results are replicable.

As we know from the debate over the Reinhart-Rogoff findings, even the best-trained and credentialed individuals can commit simple coding errors. We invite the authors to make the data used in the benchmark study available to the SAS and R communities.

In addition, we think that additional disclosures by the authors will help readers evaluate the methodology and interpret findings from the paper. These include:

(1) Additional detail about the testing environment.  I'll remark on the obvious differences in the hardware provisioning in next week's post, but for now I will simply note that the HPA environment described in the paper does not appear to match any existing Greenplum production appliances;

(2) Actual R packages used for the benchmark;

(3) Size of the data sets (in Gigabytes);

(4) Actual sample sizes for the training and validation sets for each method, together with more detail about the sampling methods used;

(5) Details of the model parameter settings used for each method and product;

(6) The value of "priors" used for each model run (which alone may explain the observed differences in event precision);

(7) In the results tables, detailed model quality statistics for each test, including sensitivity, specificity, precision and accuracy, the actual confusion matrices and method-specific diagnostics;

(8) Detailed model quality tables for the Marketing and Telecom problems, which are not disclosed in the paper;

We invite readers to review the paper and share your thoughts in the Comments section below.

Continue reading: SAS Big Data Analytics Benchmark (Part Two)

Derek Norton, Joseph Rickert, Bill Jacobs, Mario Inchiosa, Lee Edlefsen and David Smith all contributed to this post.

How UpStream uses R for Attribution Analysis

Major retailers like Williams Sonoma use UpStream Software for marketing analytics, including revenue attribution, targeting, and optimization. In the video below Tess Nesbitt (senior statistician at UpStream) describes how she uses Revolution R Enterprise and Hadoop to figure out the impact on various marketing channels (for example direct mail, email offers, and catalogs) on consumer retail sales.

 

(The slides for Tess's presentation, The Impact of Big Data on Marketing Analytics, are available for download.) Because Tess needs to massive amounts of consumer behaviour data to tease out the effects of the different marketing channels, she switched from SAS/WPS and now uses the big-data capabilities of Revolution R Enterprise to fit a survival model to more than 36 million records in less than 4 minutes. (You can find some of the details of the model and the R code used to fit it in Tess's recent presentation to the Bay Area R User Group, Statistical Marketing Analytics with Big Data (PPT).) By reducing the time it takes to fit the model from an overnight wait to the time it takes to make a cup of coffee, Tess has more opportunity to refine and improve the model, as she describes in a recent Channel Reseller News article:

"Revolution R Enterprise 6.2 enables UpStream to build highly efficient statistical models on extremely large data. Because their parallelized algorithms are so efficient, it enables us to take multiple passes at the data, build iterative models, and it provides everything we need to glean as much information and build the best models we can for our customers."

These more powerful models in turn mean that retailers can finally understand what marketing activities are most likely to lead consumers to make purchases, as Mohan Namboodiri (VP Customer Analytics, Williams Sonoma) describes in this GigaOM Structure 2013 panel discussion.

To learn more about how UpStream uses Revolution R Enterprise, take a look at this Revolution Analytics case study.

Revolution Analytics named to inaugural CRN Big Data 100

Revolution Analytics has been named among the "CRN Big Data 100", a collection of 100 companies selected by the CRN editorial team that have demonstrated an ability to innovate in bringing to market products and technologies that help businesses manage big data. Revolution Analytics was included in the Big Data Business Analytics category, along with SAP, SAS, Tableau, Tibco, and other innovative analytics companies. Other notable companes in the Big Data Management and Big Data Infrastructure and Services categories of the Big Data 100 include Cloudera, EMC, Hortonworks, Oracle and Teradata. You can find more information about the CRN Big Data 100 in the press release linked below.

Revolution Analytics News: Revolution Analytics named to inaugural CRN Big Data 100

Open Source software's opportunity to reform government

The results from the 2013 Future of Open Source Survey are in — thanks to everyone who contributed by completing the survey. You can read an overview of the results here, or see the detailed breakdowns in the slides at the end of this post.

For me, one of the most interesting nuggets from the survey is that a plurarity of respondents believed that of all sectors, the public sector will be impacted the most by open source software:

There is huge opportunity to improve the ways that we govern ourselves through the use of open source software, principles and communities. Federal, state and local governments have access to rich and large streams of data, but thus far government hasn’t been as quick as the business sector in building powerful data-based applications and services. But governments have one advantage that the business sector does not: givernment data is, by definition, public. Here are three ways that government can capitalize on rich sources of public data:

Democratize data. By making public data available on the internet (where practical and legal), governments can draw on the expertise of the data scientist community to unlock its value. At the Federal level, the Open Government Initiative is a good start, although many of the data sets made available at data.gov are over-aggregated and too small for truly innovative Big Data applications. Some local governments are making good progress here as well. New York City’s open data policy and new data transparency law has fostered a culture of civic “hacking” on city data, and created handy tools for locals like this iPhone app to find local WiFi-enabled venues suitable for getting out of the house when working “from home”.

Crowdsource data science talent with open tools. The data science community has created powerful open-source tools and resources for the analysis of Big Data. The open-source R language is designed to analyze data common to government applications (like economic, environmental and social data) and is used today by government departments such as NIST, FDA, NOAA, and the CIA. Crowdsourcing platforms and contests are a great way of getting data scientists engaged with public data, as NASA found in this Kaggle contest to find the location of dark matter from radio astronomy data.

Deploy data scientists within departments. Not all government can be made public, of course. There may be privacy issues (think IRS data), or national security issues (CIA, NSA) to consider. So it’s up to data science teams in civil service to work on sensitive data in a secure environment. It’s very encouraging to see departments like the Consumer Protection Bureau openly embrace big-data platforms and open-source tools for their teams. The growing drive towards evidence-based policy making, and the use of statistical experiments to improve government effectiveness, is another very positive sign.

It's great to see that the respondents to the 2013 Future of Open Source Survey recognize this opportunity to revolutionize government. It's up to us, as citizens, to press governments to capitalize on the opportunity, so make your voice heard.

Michael J Skok: 2013 Future of Open Source - 7th Annual Survey results