Revolutions

Statistics vs Data Science vs BI

As someone who trained as a statistician, I've always struggled with that title. I love the rigor and insight that Statistics brings to data analysis, but let's face it: Statistics — the name — has always had a bit of a branding problem. Telling someone I was a statistician was more likely to conjure up images of me counting runs at a baseball (or cricket) game than pursuing serious science. And the image of what Statistics ideally is about — collaborative, interactive, applied, fun — was too often subsumed by the stereotype image — isolated, actuarial, ivory tower, report driven. (And hey, even actuaries can be fun sometimes.)

That's why I'm a fan of the term "data scientist" — it embodies everything that Statistics always should be, without the baggage and tradition of the term "statistician". So I enjoyed participating in yesterday's Kalido webinar "Data Scientist: Your Must-Have Business Investment Now" where I could make the following contrast between the images of Statisticians and Data Scientists:

(A quick aside on the "Data Size" row above: while the unstructured or unaggregated data source data that data scientist work with can be in the terabytes range or even large, by the time it's cleaned and prepared for statistical modeling, a file in the gigabytes range is even more typical — even at "Big Data" companies like Facebook. This is a topic I cover in more detail in my recent Strata talk on real-time predictive analytics.)

So bottom line: while I am a statistician, and I love Statistics dearly, I do prefer to call myself a Data Scientist today, because it better represents to me what Statistics really is to me (if that makes sense). And that's certainly not to diminish the achievements of those who do call themselves Statistician. In particular, I want to recognize George Box: a true hero of mine, coiner of the idiom "all models are wrong, but some are useful", and one of the nicest people I ever met, who sadly passed away in March.

On the other hand, I have no qualms about making a competitive comparison between Data Science and Business Intelligence:

You can get the details of how I differentiate Statistics and Data Science and BI, and hear other perspectives on Data Science from fellow data scientists Carla Gentry and Gregory Piatetsky in the slides and replay of the webinar provided by Kalido at the link below.

 

Kalido: Data Scientist: Your Must-Have Business Investment NOW

Webinar on Data Science, May 14

A quick note that I'll be giving my take on the question, "What is a Data Science" along with Carla Gentry (Analytical Solutions, @data_nerd on Twitter) and Gregory Piatetsky (KDNuggets, @kdnuggets on Twitter) in a free webinar this Tuesday, May 14. You can register here, and I've included the webinar description below. Please join us for what promises to be a lively discussion.

Data Scientist: Your Must-Have Business Investment NOW

Data science has been called the sexiest job of the 21^st century by the Harvard Business Review, and for good reason. Big Data and analytics are growing trends that require experts in interpreting the vast amounts of data being collected and transforming it into actionable insights. Those experts are Data Scientists, and we are lucky to be hosting a few of them for a one hour webinar.

Please join us on May 14th at 3pm EST as we host, Carla Gentry, Data Scientist at Analytical Solution, David Smith, Data Scientist at Revolution Analytics, Gregory Piatetsky, Editor, KDnuggets; co-founder KDD and ACM SIGKDD, Eric Kavanagh, DM Radio Host at Information Management Magazine, and Darren Peirce, CTO at Kalido.

Some of the topics we will explore in this open panel will include:

• The current state of data science and emerging trends
• What’s new on the tricks and tools landscape
• The role (and shortfalls) of traditional data warehouses
• Best practices in deploying data science to production

Register here to sign up for the webinar

Many thanks to Kalido for putting the panel together.

Update May 2: Now available: A roundup of my presentation at the webinar, and the webinar replay

An Evening with Data Scientists

by Joseph Rickert

I had the good fortune to attend the GE hosted event Data Forecast: What’s Next in Data Science held at the chic “Box SF” a couple of weeks ago. (This the close as I get to “elegantly cool”.) The highlight of the evening was a panel discussion very ably led by Jeff Kelly of Wikibon. The panelists were Kaggle’s Anthony Goldbloom, Annika Jimenez of Pivotal (formerly EMC Greenplum) Andreas Weigend of Stanford’s Social Data Lab, Hilary Mason of bitly and Anil Varma of GE. The discussion is online and worth listening to in its entirety.

Video streaming by Ustream

This was a better than average event from the point of view of serious questions and panelists who were willing to give thoughtful, sometimes provocative answers. To Jeff’s first question “What is the role of data science, not just in business but in society in general ... is it only about ad placement”, Anthony replies the data science is about “bringing rigor to decision making” — certainly a positive move away from ad placement — but not as far as Hilary takes it. In the preamble to her response she observes: “For the first time we are able to study human behavior at the scale of human behavior” but then notes that “The accomplishments of data science have out-paced data products.” Annika refers to the “person level impact of data science” and Andreas points out that “data science allows us to observe individual things that individuals do in their natural environment”. This tension between the promise of human understanding implicit in first part of Hilary’s response and the “dark side” of corporations and governments learning much too much about individuals pervades the entire discussion. For example, later in the discussion (29:30) Andreas points out that data scientists are now able to address “questions that didn’t exist before” and suggest that we may be near the “end of insurance as we know it”. It is conceivable that data scientists working for insurance companies could make predictions at a high enough level of granularity to make clear the risk of insuring individuals. He asks: Do we want this as a society?”. And yet, after, this ominous warning, Anil expresses optimism and points to positive examples of companies sharing data for the “greater good” (42:56). The thread of this discussion could not have been planned in the five minutes or so the participants huddled by the hors d'oeuvres before the panel. Magic can happen when good people get together.

The discussion around Jeff’s second question “Is the role of the data scientist to find the right questions” surprised me. Both Annika and Anthony push back insisting that collaboration is key and that the business people should know what questions to ask and how to prioritize. Hilary contributes that an essential skill of a data scientist is to be able to “understand a business problem and translate it into an analysis”, adding in her own unequivocal style that creativity is a and essential quality for a data scientist and the notion that “the data tells us what to do is bullshit”. Later in the discussion (28:46) Annika warns that data scientists and the companies they work for will find themselves in trouble unless that problems that they are working on are attached to a “vision for data science”.

Finally, a major theme pervading the discussion (and probably the reason for the evening) is that everyone there was looking to hire data scientists. There was clear note of desperation here, and how could there not be given that the cumulative description of skills that the panelists attribute to data scientists. One would think that every data-driven organization is trying to hire the the equivalent of the Avengers. Sure, there are a few superheroes out there: “I still believe in heroes” (Nick Fury). But really, do Black Widow or Captain America go looking for work? Data scientists need to be identified, cultivated and mentored, wooed maybe, but rarely merely hired.

A few lists for data scientists and statisticians

Looking for more resources on the web or people to follow on Twitter? Here are some lists you may find useful:

• 100 Savvy Sites on Statistics, which includes 17 sites that focus on R Programming
• Kalido offers this list of 30 influential data scientists on Twitter
• Big Data Republic is taking votes for this list of 100 influential tweeters on Big Data (voting closes on April 15).

 

Facebook profile photo changes reveal trends in support for same-sex marriage

On Tuesday and Wednesday this week, the US Supreme Court heard two landmark cases related to same-sex marriage rights in this country. Simultaneously, to show solidarity for the cause of civil rights for gay members of society, thousands of Facebook users changed their profile pictures to a red equality symbol at the suggestion of the Human Rights Campaign. Most profile pictures looked like the one shown at right, but there were also many variants used as well. (By the way, here's R code use can use to generate the red equality logo, with thanks to Josef Fruehwald.)

The change in Facebook profiles provided some interesting information about the support of advances in gay rights across the USA. Some clever data science from Facebook's Eytan Bakshy reveals that there was indeed a significant increase in the rate of profile picture changes during the oral hearings:

(The chart above is the result of a time-series analysis using the R language, after extracting the normal day-night cycle of profile updates.) Moreover, by looking at the rate of changes of Facebook profiles across every county in the USA in a choropleth, we see those regions where support for gay rights is the strongest:

While it's unlikely that the Justices will consult Facebook profiles as part of their deliberations, the proceedings did seem to be favourable to the advance of gay rights. (Especially for the second case regarding DOMA, which if overturned will be a hugely positive outcome for thousands of couples already married, including me and my husband.) But for those of us for whom this cause means so much personally, it was truly heartwarming to see that sea of red from allies all across the county. What strikes me so much about the map above is that support is coming from all across the country — not one county is a totally unsupportive white. And that's something that all of America can be proud of.

Facebook (Eytan Bakshy): Showing Support for Marriage Equality on Facebook (via Sean J. Taylor)

What does a data scientist do?

The presentation below by Carlos Somohano (founder of Data Science London) provides the best description of a Data Scientist that I've seen in some time:

Highlights include:

• On Slide 14, a history of the Data Science
• On Slide 22, the essential skills of data scientists (and a platypus)
• On Slide 26, 10 things data scientists do
• On Slide 27, a data scientist toolkit (including R, of course)
• On Slides 37-38, the data science process
• On Slides 44-52, examples of data products

Data Science London (SlideShare): Big Data [sorry] & Data Science: What Does a Data Scientist Do (via Milinda Thielbar)

R resources for data journalists

M. Edward Borasky has curated a list of resources for budding data journalists looking to use R. Many of the links relate to data scraping, data mashups and data visualization, and so would be useful for data science applications generally. Check it out. (via Hadley Wickham)

Free e-book on Data Science with R

A new book by Jeffrey Stanton from Syracuse Iniversity School of Information Studies, An Introduction to Data Science, is now available for free download. The book, developed for Syracuse's Certificate for Data Science, is available under a Creative Commons License as a PDF (20Mb) or as an interactive eBook from iTunes.

The book begins with the following clear definition of Data Science:

Data Science refers to an emerging area of work concerned with the collection, preparation, analysis, visualization, management and preservation of large collections of information. Although the name Data Science seems to connect most strongly with areas such as databases and computer science, many different kinds of skills - including non-mathematical skills, are needed.

Throughout the book, you'll find many examples of data science applications implemented in the R language. For R beginners a Getting Started with R chapter is included, but it does get into some fairly in-depth topics including sentiment analysis of Twitter data, working with data in Hadoop via RHadoop, and creating information maps. R code is sprinkled liberally for your own use, and available to download (also under an open-source license) from GitHub.

You can find more details about the book at its companion website, linked below.

Introduction to Data Science: Teach Data Science (via Guillermo Santos)

10 R packages every data scientist should know about

The yhat blog lists 10 R packages they wish they'd known about earlier. Drew Conway calls them "10 reasons to always start your analysis in R". They're all very useful R packages that every data scientist should be aware of. They are:

1. sqldf (for selecting from data frames using SQL)
2. forecast (for easy forecasting of time series)
3. plyr (data aggregation)
4. stringr (string manipulation)
5. Database connection packages RPostgreSQL, RMYSQL, RMongo, RODBC, RSQLite
6. lubridate (time and date manipulation)
7. ggplot2 (data visulization)
8. qcc (statistical quality control and QC charts)
9. reshape2 (data restructuring)
10. randomForest (random forest predictive models)

You can find links to all of these packages and tips on how to use them at link below.

yhat blog: 10 R packages I wish I knew about earlier

Strata's Data Driven Business Day

The tagline for O'Reilly Strata conference series — Making Data Work — has meant that it's always been popular with practitioners, primarily data scientists working with Big Data in real-world environments. Recent Strata events have also attracted more business-oriented attendees, with events focused more on processes and outcomes than on the implementation details. 

On Tuesday February 26, Strata Santa Clara kicks off with Data Driven Business Day (which runs alongside the tutorial track). This series of talks will focus on how to solve business problems with Big Data, and will feature speakers from IBM, Accenture, DDBD and many others. I'm proud to also be included in the lineup: I'll be talking about Real-time Big Data Analytics: From Deployment to Production. This will be an updated, high-level version of my webinar, where I'll focus on defining the buzzwords "big data", "real-time" and "analytics" and how they all fit together. 

If you'd like a preview of what I'll be talking about, check out my conversation with David Pittman at the IBM Big Data Hub. And if you're planning to attend Strata Santa Clara, I look forward to seing you there.