Revolutions

Paris's history, captured in its streets

The following image by Mathieu Rajerison has been doing the rounds of French media recently. It shows the streets of Paris, color-coded by their compass direction. It's been featured in an article in Telerama magazine, and even on French TV Channel LCI (skip ahead to 8:20 in the linked video. which also features an interview with Mathieu).

Mathieu used the R language and OpenStreetMap data to construct the image, which colorizes each street according to the compass direction it points. Orthogonal streets are colored the same, so regular grids appear as swathes of uniform color. A planned city like Chicago, would appear as a largely monochrome grid, but Paris exhibits much more variation. (You can see many other cities in this DataPointed.net article.) As this article in the French edition of Slate explains, the very history of Paris itself is encapsulated in the colored segments. You can easily spot Napoleon's planned boulevards as they cut through the older medieval neighborhoods, and agglomerated villages like Montmartre appear as rainbow-hued nuggets. 

Mathieu explains the process of creating the chart in a blog post written in English. He used the rgdal package to import the OpenStreetMap shapefile and the spatstat package to extract the orientations of the streets. Then, the entire map was interpolated to a grid with the raster package before finally being exported to a TIFF file and rendered within QGIS. The entire chart is created with just 31 lines of R code, which you can find at the link below.

Data and GIS tips: Streets of Paris Colored by Orientation

Microsoft acquires Revolution Analytics - news roundup

There was a lot of news coverage on Friday and over the weekend about the news that Microsoft will acquire Revolution Analytics. Here are some links to just a few of the articles published.

• Wired: Microsoft is "heavily embracing the R programming language"; "the move deepens Microsof's investments in open source".
• TechCrunch: Microsoft's acquisition is to "bolster its analytics services".
• GigaOm: R is "hugely popular amongst data scientists and research types"; "a big deal in the world predictive analytics and machine learning ... an emerging market that Microsoft wants to get in on early".
• Bloomberg BusinessWeek: Microsoft's acquisition is "to help the company build up its cloud-services business".
• ComputerWorld: "In a move to beef up its portfolio of analysis software and services"; "Microsoft is pledging to contribute to the further development of R".
• SiliconAngle: Acquisition "a double win for Microsoft"; "R has soared in popularity over recent years with the rise of data scientists ... an audience that Microsoft is actively trying to court"; "acquisition is as much about data scientists themselves as their algorithms".
• VentureBeat: Deal is "latest proof [Microsoft] is serious about open source".
• DataInformed: Move "strengthens [Microsoft's] commitment to big data analytics".

There's also a lively discussion on the deal in this Hacker News thread, and the Adventures in Advanced Analytics blog reviews Revolution Analytics' contributions to the R community over the last 7 years and the future prospects for the community.

R in Nature, Mashable

R was recently the subject of a feature article in the prestigious science magazine Nature: Programming tools: Adventures with R. 

Besides being free, R is popular partly because it presents different faces to different users. It is, first and foremost, a programming language — requiring input through a command line, which may seem forbidding to non-coders. But beginners can surf over the complexities and call up preset software packages, which come ready-made with commands for statistical analysis and data visualization. These packages create a welcoming middle ground between the comfort of commercial ‘black-box’ solutions and the expert world of code.

The article highlights many of the packages you can use for scientific analysis with R, and also mentions several scientific projects based on R, including BioConductor and ROpenSci. The article also noted that the use of R has increased rapidly in a number of scientific disciplines, as measured by the rate at which R is cited in published articles.

The article also includes quotes from R's co-creator Robert Gentleman ("I can write software that would be good for somebody doing astronomy, but it’s a lot better if someone doing astronomy writes software for other people doing astronomy") and Bob Muenchen, who tracks the popularity of statistical software ("Most likely, R became the top statistics package used during the summer of this year.").

Mashable isn't in the same authoritative league as Nature, but it's read by a lot of people. So it's great that R also got a mention in the recent article, So you wanna be a data scientist?.

"On an average day, I manage a series of dashboards that tell our company about our business — what the users are doing," says Jon Greenberg, a data scientist at Playstudios, a gaming firm. Greenberg is a manager now, so he's programming less than he used to, but he still does his fair share. Usually, he pulls data out of Apache Hadoop storage and runs it through Revolution R, an analytics platform and comes up with some kind of visualization. "It may be how one segment of the population is interacting with a new feature," he explains.

The article also describes the experiences of other data scientists and gives some salary statistics on "2015's Hottest Profession": Data Science.

Ford uses R for data-driven decision making

Mike Cavaretta is Ford Motor Company’s Chief Data Scientist, and was tasked by the incoming CEO Alan Mulally to help change the culture so that "important decisions within the company had to be based on data". In a feature article at Dataconomy, he reveals that R is a big part of this revolution at Ford: 

On the statistical side, we did a lot of stuff in R. ... We’ve done a lot more with R and we’re currently evaluating Pentaho. So we’ve really moved from more point solutions for solving particular problems, to more of a framework and understanding different needs in different areas. For example, there may be certain times when SAS is great for analysis because we already have implementations, and it’s easier to get that into production. There are other times when R is a better choice because it’s got certain packages that makes that analysis a lot easier, so we’re working on trying to put all that together. 

Data Science is revolutionizing Ford's products, too. Among the applications Cavaratta describes:

• Breaking down data silos between analytics groups within Ford;
• Understanding how drivers are using electric cars, based on opt-in telemetry data from the cars themselves; and
• Supporting the development of features in vehicles by looking at social media chatter about them. (He gives a great example related to the "three-blink" turn signal in the Ford Fiesta.)

You can read more about Ford's use of data science and R at the link below, or explore more companies using R.

Dataconomy: How Ford Uses Data Science: Past, Present and Future (via @JulianHi)

The three types of Reddit posts, and how they make it to the front page

Todd Schneider's blog post on solving the traveling salesman problem with R hit the front page of reddit.com. This is a big deal: front-page placement on the popular social news site can drive a ton of traffic (in Todd's case, 1.3 million pageviews). But what factors determine which of reddit's contributed links make it to the front page? (There are 25 front-page slots, but more than 100,000 reddit posts on an average day.)

Todd set out to answer this question using the statistical language R, and reported his results on Mashable. He collected 6 weeks of data including 1.2 million rankings for about 15,000 posts, and looked for commonalities amongst those posts that made the top 25.

Now, you might expect that a post's front page ranking is determined by its score (the number of times it has been "liked" by a reddit user, most likely after having seen it in the "subreddit" special topic area where it was posted), and how long since it was posted (reddit's front page generally contains recent posts). But it turns out that not all subreddits are treated equally. Todd discovered that there are three different types of subreddits when it comes to how posts are promoted to the front page:

• "Viral Candy" subreddits like funny, gifs and todayilearned. Posts from this category dominate page one.
• "Page Two" subreddits", which includes Documentaries, Fitness and personalfinance. As the name suggests, posts in these subreddits almost never make it to page 1, but are often promoted to page 2.
• "The Rest", which includes food, LifeProTips, and sports. Todd's post was in this category, in the subreddit dataisbeautiful. Posts in these subreddits make a small but significant fraction of page 1 posts.

It seems that reddit's front page (and pages 2, 3 and 4 which follow) follow a well-defined mix of posts from each of the three categories, as you can see in the chart below:

Starting from the left of the chart above, you can see the #1 post (on page 1) is from one of the "Viral Candy" subreddits about 97% of the time, but that a "The Rest" post does occasionally make top billing. By contrast, posts from the "Page Two" subreddits almost never appear above #10, but dominate page two (ranks 26-50). There's a pretty consistent mix on pages 3 and 4: about 65% "viral candy", about 15% "page twos" and about 25% "the rest".

As for post scores, Todd noted that posts from "Viral Candy" and "The Rest" subreddits need high scores to get on page 1: about 3500-4500 and 3000-4000 respectively for the top slot. By contrast, posts in "Page 2" reddits only need scores in the 500-1500 range to hit the lower ranks of Page 1 (but are much more likely to appear on Page 2).

If you're interested in the details of what gets a post on reddit's front page, Todd's blog post has lots more information. And if you're an R user and want to do a similar analysis, Todd's data and R code are available on github.

Todd W Schneider: The reddit Front Page is Not a Meritocracy

R leaps to #12 in Tiobe language popularity index

The R language has jumped to number 12 in the November 2014 TIOBE Index of programming language popularity. This is R's highest ranking in the history of the TIOBE index, which has been ranking languages since 2003. A high ranking is an impressive achievement for R given that it is a domain-specific language (designed for data science applications), and puts it in the company of general-purpose languages like C, Java and Python. Here are the top 15 from the latest index:

Other data science / statistics languages in the top 50 include SAS (at #22), Matlab (#24) and Scala (#31).

Tiobe's managing director noted in an Infoworld interview that "R is gaining share for a while now", and you can see R's rise in the history of it's relative ranking:

The TIOBE rankings are based on search engine results and on the number of skilled engineers world-wide, courses and third party vendors. It's interesting to note that while this methodology is very different from that used for the IEEE language popularity rankings, R also ranked very highly in that list (at #9 in July 2014).

TIOBE: TIOBE Index for November 2014

The Open Source R Programming Language is Becoming Pervasive

So says CIO.com, in a recent article 11 Market Trends in Advanced Analytics.

R, an open source programming language for computational statistics, visualization and data is becoming a ubiquitous tool in advanced analytics offerings.

Kirsch says nearly every top vendor of advanced analytics has integrated R into their offering and so that they can now import R models. This allows data scientists, statisticians and other sophisticated enterprise users to leverage R within their analytics package.

One of the big beneficiaries of this trend, Kirsch says, is Revolution Analytics, the leading provider of enterprise support for R. Kaufman and Kirsch also point to advanced analytics firm Predixion, which is focused on extending R beyond data scientists and statisticians to business users through a wizard interface.

The article is based on a recent analyst report, Advanced Analytics: The Hurwitz Victory Index Report 2014. Revolution Analytics was named a Leader in Go To Market Strength in the report.

CIO.com: 11 Market Trends in Advanced Analytics

How data-driven companies use R to compete

The editors at DataInformed invited me to write an article about how R is used in business, and I was pleased to oblige. The article, How Companies use R to Compete in a Data-Driven World, is now live and describes how Facebook, The New York Times, X+1, ANZ Bank and many others successfully use R to analyze their data. You can also find additional details about their use of R, and how many other companies use R, in the slide show below.

 

DataInformed: How Companies use R to Compete in a Data-Driven World

Bank of America uses R for reporting

A recent FastCoLabs article, "The 9 Best Languages For Crunching Data", starts its list with the R language:

It would be downright negligent to start this list with any language other than R. It has been kicking around since 1997 as a free alternative to pricey statistical software, such as Matlab or SAS.

But over the past few years, it’s become the golden child of data science--now a household name not only among nerdy statisticians, but also Wall Street traders, biologists, and Silicon Valley developers. Companies as diverse as Google, Facebook, Bank of America, and the New York Times all use R, as its commercial utility continues to spread.

The article reveals yet another company using R: Bank of America. Journalist Anna Nicolaou interviewed a vice president there:

[R is] also catching on on Wall Street. Traditionally, banking analysts would pore over Excel files late into the night, but now R is increasingly being used for financial modeling, particularly as a visualization tool, says Niall O’Connor, vice president at Bank of America. “R makes our mundane tables stand out,” he says.

You can read much more about R and the other languages for crunching data, including Python and Julia, at the link below.

FastCoLabs: The 9 Best Languages For Crunching Data 

R popularity update

Earlier this month, the results of the 15th annual KDnuggets Software Poll were released and R's popularity continues to grow. 38.5% of respondents reporting using open source R in the last 12 months, up from 37.4% in 2013. R was second only to RapidMiner in popularity, which was used by 44.2% of respondents to this poll. You can see the top 5 responses in the chart below; SAS came in with a 10.9% share, and Revolution R Enterprise with 9.1%

Meanwhile, the latest update to the RedMonk Programming Language Rankings is out, and R has risen two places to #13 since the February 2014. The rankings are based on each language's activity on GitHub and StackOverflow. I've included a detail of the comparison chart below (click for the complete chart):

RedMonk analyst Stephen O'Grady has this to say about R:

Advocates of R will be pleased by the language’s fourth consecutive gain in the rankings. From 18 in January of 2013 to 13 in this run, the R language continues to rise. Astute observers might note by comparing plots that this is in part due to growth on GitHub; while R has always performed well on Stack Overflow due to the volume of questions and answers, it has tended to be under-represented on GitHub. This appears to be slowly changing, however, in spite of competition from Python, issues with the runtime itself and so on.

For other statistics on R's popularity, see this post from April 2014.