Revolutions

Because it's Friday: Wavy Lines Illusion

Another fine illusion: in this one, the pairs of horizontal lines are all smooth sine curves, despite the appearance of the jagged zig-zags:

It's really hard for me at least to tell that the zig-zags in the light grey region are actually curved. Zooming all the way in may help if you want to check for yourself.

That's all from us at the blog for this week. Next week, we'll be taking a break for the week of the US Independence Day holiday, but we'll be back the week of Monday July 9, reporting from the useR! conference in Brisbane. In the meantime, have a great weekend!

 

Global Migration, animated with R

The animation below, by Shanghai University professor Guy Abel, shows migration within and between regions of the world from 1960 to 2015. The data and the methodology behind the chart is described in this paper. The curved bars around the outside represent the peak migrant flows for each region; globally, migration peaked during the 2005-2010 period and the declined in 2010-2015, the latest data available.

This animated chord chart was created entirely using the R language. The chord plot showing the flows between regions was created using the circlize package; the tweenr package created the smooth transitions between time periods, and the magick package created the animated GIF you see above. You can find a tutorial on making this animation, including the complete R code, at the link below.

Guy Abel: Animated Directional Chord Diagrams (via Cal Carrie)

Should you learn R or Python for data science?

One of the most common questions I get asked is, "Should I learn R or Python?". My general response is: it's up to you! Both are popular open source data platforms with active, growing communities; both are are highly sought after by employers, and both have a rich set of capabilities for working with data. It really depends most on your interests and the kind of employer you want to work for.

If your interests lean more towards traditional statistical analysis and inference as used within industries like manufacturing, finance, and the life sciences, I'd lean towards R. If you're more interested in machine learning and artificial intelligence applications, I'd lean towards Python. But even that's not a hard-and-fast rule: R has excellent support for machine learning and deep learning frameworks, and Python is often used for traditional data science applications.

One thing I am quite sure of though: neither Python nor R is inherently better than the other, and arguments on that front are ultimately futile. (Trust me, I've been there.) Which is better for any given person depends on a wide variety of factors, and for some, it may even be worthwhile to learn both. Brian Ray recently posted a good overview of the factors that may lead you towards R or Python for data science: their history, the community, performance, third-party support, use cases, and even how to use them together. It's great food for thought if you're trying to decide which community to invest in. The cartoon below is taken from that post, and sums up things quite nicely.

And what about me, you ask? My needs generally fall on the statistics / data science end of the spectrum, and my interest in deep learning has been served well by the keras support from RStudio. I've been using R for a long time, and I just haven't found anything yet that I, personally, has led me down the Python path. But your needs may vary from mine, and the article linked below is a good guide to that decision.

Brian Ray: Python vs (and) R for Data Science

The Financial Times and BBC use R for publication graphics

While graphics guru Edward Tufte recently claimed that "R coders and users just can't do words on graphics and typography" and need additonal tools to make graphics that aren't "clunky", data journalists at major publications beg to differ. The BBC has been creating graphics "purely in R" for some time, with a typography style matching that of the BBC website. Senior BBC Data Journalist Christine Jeavans offers several examples, including this chart of life expectancy differences between men and women:

... and this chart on gender pay gaps at large British banks:

Meanwhile, the chart below was made for the Financial Times using just R and the ggplot2 package, "down to the custom FT font and the white bar in the top left", according to data journalist John Burn-Murdoch.

There are also entire collections devoted to recreating Tufte's own visualizations in R, presumably meeting his typography standards. Tufte later clarified saying "Problem is not code, problem is published practice", which is true of any programming environment, which is why it was strange that he'd call out R in particular.

Because it's Friday: The lioness sleeps tonight

Handlers for the lion enclosure at San Diego Zoo have developed a novel way to provide stimulation for their big cats: let them play tug-of-war with people outside. People plural that is — it turns out that a young lioness is no match for a trio of pro wrestlers:

🤼‍♂️ How many #NXT #WWE superstar wrestlers does it take to win in tug of war with a 2 1/2 year old lion cub? Apparently more than 3! #NXTSanAntonio #SAZoo pic.twitter.com/avyPVwRYjN

— San Antonio Zoo & Zoo School🦏 (@SanAntonioZoo) May 19, 2018

That's all for this week. Have a great weekend (and a very happy Pride!) and we'll be back next week.

A guide to working with character data in R

R is primarily a language for working with numbers, but we often need to work with text as well. Whether it's formatting text for reports, or analyzing natural language data, R provides a number of facilities for working with character data. Handling Strings with R, a free (CC-BY-NC-SA) e-book by UC Berkeley's Gaston Sanchez, provides an overview of the ways you can manipulate characters and strings with R. 

There are many useful sections in the book, but a few selections include:

• C-style formatting — very useful for preparing tabular data for reports
• String manipulation with the stringr package — which provides some welcome consistency in handling strings with R
• Regular expressions — the savior and/or curse for many data extraction problem

Note that the book does not cover analysis of natural language data, for which you might want to check out the CRAN Task View on Natural Language Processing or the book Text Mining with R: A Tidy Approach. It's also sadly silent on the topic of character encoding in R, a topic that often causes problems when dealing with text data, especially from international sources. Nonetheless, the book is a really useful overview of working with text in R, and has been updated extensively since it was last published in 2014. You can read Handling Strings with R at the link below.

Gaston Sanchez: Handling Strings with R

AI, Machine Learning and Data Science Roundup: June 2018

A monthly roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications I've noted over the past month or so.

Open Source AI, ML & Data Science News

Intel open-sources NLP Architect, a Python library for deep learning with natural language.

Gym Retro, an open source platform for reinforcement learning research on video games.

Facebook open-sources DensePose, a toolkit to transform 2D images of people into a 3-D surface map of the human body.

MLflow, an open source machine learning platform from Databricks, has been released.

Industry News

In a 12-minute documentary video and accompanying Wired article, Facebook describes how it uses Machine Learning to improve quality of the News Feed.

In the PYPL language rankings, Python ranks #1 in popularity and R is #7; both are rising.

Google announces its ethical principles for AI applications, and AI applications it will not pursue.

Wolfram Research launches the Wolfram Neural Network Repository, with implementations of around 70 neural net models.

Google Cloud introduces preemptive pricing for GPUs, with discounts compared to GPUs attached to non-preemptable VMs.

Conversational AIs conducting conversational full-duplex conversations: Microsoft Xiaoce and Google Duplex.

Microsoft News

Microsoft's head of AI research, Harry Shum, on "raising" ethical AI.

Microsoft has acquired Semantic Machines, a startup focused on conversational AI.

Microsoft is developing a bias-detection tool, with the goal of reducing discrimination in applied AI.

Microsoft's Bot Builder SDKv4 further simplifies the process of developing conversational bots.

Cognitive Services Labs, which offers previews of emerging Microsoft Cognitive Services technologies, adds labs for anomaly detection, ink analysis and more.

ML.NET 0.2, a cross-platform open source machine learning framework for .NET developers, has been released.

Microsoft R Open 3.5.0 has been released.

Azure Databricks now provides a machine-learning runtime and GPU support.

Learning resources

A tutorial on visualizing machine learning models with LIME, a package for R and Python.

A visual introduction to Machine Learning, Part II: Model Tuning and the Bias-Variance Tradeoff, with an in-depth and graphically elegant look at decision trees.

Materials for five new AI-oriented courses have been published to the LearnAI Materials site.

A Developer's Guide to Building AI Applications: a free e-book from O'Reilly and Microsoft.

Microsoft Professional Program for Artificial Intelligence, a free, self-paced on-line certification in AI skills.

The Azure AI Lab provides complete worked applications for generative image synthesis, cognitive search, automated drone flight, artistic style transfer and machine reading comprehension.

CVAE-GAN: a new generative algorithm for synthesizing novel realistic images, like faces.

Land cover mapping with aerial images, using deep learning and FPGAs.

Improving medical imaging diagnostics, using the new Azure Machine Learning Python package for Computer Vision.

Find previous editions of the monthly AI roundup here

PYPL Language Rankings: Python ranks #1, R at #7 in popularity

The new PYPL Popularity of Programming Languages (June 2018) index ranks Python at #1 and R at #7.

Like the similar TIOBE language index, the PYPL index uses Google search activity to rank language popularity. PYPL, however, fcouses on people searching for tutorials in the respective languages as a proxy for popularity. By that measure, Python has always been more popular than R (as you'd expect from a more general-purpose language), but both have been growing at similar rates. The chart below includes the three data-oriented languages tracked by the index (and note the vertical scale is logarithmic).

Another language ranking was also released recently: the annual KDnuggets Analytics, Data Science and Machine Learning Poll. These rankings, however, are derived not from search trends but by self-selected poll respondents, which perhaps explains the presence of Rapidminer at the #2 spot.

Because it's Friday: Olive Garden Bot

Comedy writer Keaton Patti claims this commercial script for a US Italian restaurant chain was generated by a bot:

I forced a bot to watch over 1,000 hours of Olive Garden commercials and then asked it to write an Olive Garden commercial of its own. Here is the first page. pic.twitter.com/CKiDQTmLeH

— Keaton Patti (@KeatonPatti) June 13, 2018

Of course this wasn't bot-generated, but "what a bot might write" is fertile ground for comedy:

I forced a bot to read over 1,000 tweets claiming to be scripts written by bots online, then asked it to write a script itself. It wrote these two pages, then hung itself. pic.twitter.com/KbFXStJuAb

— Christine Love (@christinelove) June 1, 2018

That's all from us here at the blog for this week. Have a great weekend, and we'll be back next week.

 

Interpreting machine learning models with the lime package for R

Many types of machine learning classifiers, not least commonly-used techniques like ensemble models and neural networks, are notoriously difficult to interpret. If the model produces a surprising label for any given case, it's difficult to answer the question, "why that label, and not one of the others?".

One approach to this dilemma is the technique known as LIME (Local Interpretable Model-Agnostic Explanations). The basic idea is that while for highly non-linear models it's impossible to give a simple explanation of the relationship between any one variable and the predicted classes at a global level, it might be possible to asses which variables are most influential on the classification at a local level, near the neighborhood of a particular data point. An procedure for doing so is described in this 2016 paper by Ribeiro et al, and implemented in the R package lime by Thomas Lin Pedersen and Michael Benesty (and a port of the Python package of the same name). 

You can read about how the lime package works in the introductory vignette Understanding Lime, but this limerick by Mara Averick sums also things up nicely:

There once was a package called lime,
Whose models were simply sublime,
It gave explanations for their variations,
One observation at a time.

"One observation at a time" is the key there: given a prediction (or a collection of predictions) it will determine the variables that most support (or contradict) the predicted classification.

The lime package also works with text data: for example, you may have a model that classifies a paragraph of text as a sentiment "negative", "neutral" or "positive". In that case, lime will determine the the words in that sentence which are most important to determining (or contradicting) the classification. The package helpfully also provides a shiny app making it easy to test out different sentences and see the local effect of the model.

To learn more about the lime algorithm and how to use the associated R package, a great place to get started is the tutorial Visualizing ML Models with LIME from the University of Cincinnati Business Analytics R Programming Guide. The lime package is available on CRAN now, and you can always find the latest version at the GitHub repository linked below.

GitHub (thomasp): lime (Local Interpretable Model-Agnostic Explanations)