Revolutions

Because it's Friday: The Curiosity Show

This one's all about nostalgia. When I was about 7 or 8 years old back in Australia, my favorite TV show by far was The Curiosity Show, a weekly series hosted by Rob (Morrison) and Deane (Hutton) — they didn't use their last names — featuring segments about science and practical experiments you could do at home. My favorite part was the experiments: I remember pestering my parents for all manner of straws, construction paper, watch batteries and wire to follow along with the TV show (and also the companion books, themed around Earth, Air, Fire and Water). So I was thrilled to find this week that the entire series is now available on YouTube. I still remember the thrill of building my own electric motor:

If you have a young child with a burgeoning interest in science, you might want to check this out. As for me, I always assumed this was a local show (it was filmed in the town where I grew up, and I'm pretty sure my Mum took me to see Rob and Deane do outdoor shows once or twice), so I was pleased to learn that it was shown it was also shown in Asia or Europe.

That's all from us for this week. We'll be back after the long weekend — see you then!

Guide to a high-performance, powerful R installation

R is an amazing language with 25 years of development behind it, but you can make the most from R with additional components. An IDE makes developing in R more convenient; packages extend R's capabilities; and multi-threaded libraries make computations faster. 

Since these additional components aren't included on the official R website, getting the ideal R environment set up can be a bit tricky. Fortunately, there's a handy R installation guide by Mauricio Vargas that explains how to get everything you need set up on Windows, Mac and Ubuntu Linux. On each platform, the guide describes how to install:

• The R language engine
• The RStudio IDE.
• The tidyverse suite of packages
• Multi-threaded math libraries (BLAS). On Windows, Mauricio recommends Microsoft R Open ("what made my R and Windows experience amazing"). For Mac and Unix he suggests installing OpenBLAS, but I'll add that Microsoft R Open provides BLAS acceleration on those platforms as well. It's easy to configure RStudio to use Microsoft R Open, too.

Find all the details in the installation guide, linked below.

DataCamp Community: How to install R on Windows, Mac OS X and Ubuntu

Tips for analyzing Excel data in R

If you're familiar with analyzing data in Excel and want to learn how to work with the same data in R, Alyssa Columbus has put together a very useful guide: How To Use R With Excel. In addition to providing you with a guide for installing and setting up R and the RStudio IDE, it provide a wealth of useful tips for working with Excel data in R, including:

• To import Excel data into R, use the readxl package
• To export Excel data from R, use the openxlsx package
• How to remove symbols like "$" and "%" from currency and percentage columns in Excel, and convert them to numeric variables suitable for analysis in R
• How to do computations on variables in R, and a list of common Excel functions (like RAND and VLOOKUP) with their R equivalents
• How to emulate common Excel chart types (like histograms and line plots) using R plotting functions

Conversely, you can also use R within Excel. The guide suggests BERT (Basic Excel R Toolkit), which allows you to apply R functions to Excel data via the Excel formula interface:

With BERT, you can also open an R console within Excel, and use R commands to manipulate data within the spreadsheet. BERT is open-source and available here, and you can see the detailed guide to using Excel data in R at the link below.

RPubs: How To Use R With Excel (via author Alyssa Columbus)

Videos from NYC R Conference

The videos from the NYC R conference have been published, and there are so many great talks there to explore. I highly recommend checking them out: you'll find a wealth of interesting R applications, informative deep dives on using R (and a few other applications as well), and some very entertaining deliveries. In this post, I wanted to highlight a couple of talks in particular. 

The talk by Jonathan Hersh (Chapman University), Applying Deep Learning to Satellite Images to Estimate Violence in Syria and Poverty in Mexico is both fascinating and a real technical achievement. In the first part of the talk, he uses convolutional neural networks to identify buildings, roads, and vegetation in satellite images, and use that to predict poverty rates across all of Mexico. In the second part, he uses a similar technique to identify violent hotspots in Syria, this time by identifying damaged buildings and roads from the satellite images. In both cases, analyzing satellite images is a lot less expensive (and more consistent) than collecting samples at the ground level. The neural networks were implemented in Theano running on a GPU-enabled Microsoft Azure server.

Moving from applications to platforms, my colleague Marck Vaisman gave a great talk on R for Big Data in the cloud. His talk offers some practical considerations related to working with large data sets in R, and details some cloud technologies for big data that you can connect with R, like Spark and sparklyr (which is easy to use in the cloud with Azure Databricks). 

My talk on R and Minecraft is there as well, if you want to check it out. That talk and all of the rest from the conference are available for viewing at the link below.

New York R Conference: 2018 

Because it's Friday: One Million Integers

This is a visualization of numbers from 1 to 1,000,000, added in order in blocks of 1,000, located in the 2-D plane according to their prime factors:

The animation was created by John Williamson. The mathematical process begins by representing each integer as a vector of length 80,000: the first value is the number of times "2" is used in its prime factorization, the next for "3", and so on through the primes. Then all of those vectors together represent one million points in an 80,000-dimensional space, which is then reduced to two dimensions using a dimension-reducing algorithm. The interesting structure that results is partly an artifact of the algorithm, but also a results of the fractal nature of the primes on the number line.

That's all from the blog for this week. Have a great weekend, and we'll be back next week. See you then!

 

The Chartmaker Directory: Data visualizations in every tool

Working with a new data visualization tool, and wondering how to create a specific type of chart? The Chartmaker Directory (designed by Andy Kirk) indexes more than 35 tools and over 50 charts, and provides links to examples from each combination. Here's an intentionally small detail from the index, where each hollow dot in a column represents a sample chart created a tool (the rightmost column is R, for example), and solid dots also include code. 

The visualization tools include applications like Excel, Power BI and Tableau; languages and libraries including R, Stata, and Python's matplotlib); and frameworks like D3. The data visualizations range from the standard to the esoteric, and follow the taxonomy of the book Data Visualisation (also by Andy Kirk). The chart categories are color coded by row: categorical (including bar charts, dot plots); hierarchical (donut charts, treemaps); relational (scatterplots, sankey diagrams); temporal (line charts, stream graphs) and spatial (choropleths, cartograms). 

Follow the link below to explore the indexed tools and visualization. If you know of an example or solution that isn't listed, you can contribute your own links by clicking the plus sign in the top left of the index.

Visualizing Data: The Chartmaker Directory

Video: Azure Machine Learning in plain English

Data Scientist and author Siraj Raval recently released a 12-minute video overview of Azure Machine Learning (embedded at the end of this post). The video begins with a overview of cloud computing and Microsoft Azure generally, before getting into the details of some specific Azure services for machine learning:

• [at 2:00] Hybrid cloud / on-premises architectures (see for example SQL Server and Azure Stack) 
• [at 3:40] Overview of Azure Machine Learning
• [at 4:25] The MML Spark toolkit 
• [at 5:25] Project Brainwave, inference with FPGAs (field-programmable gate array chips)
• [at 6:05] The Azure Machine Learning Workbench 
• [at 7:15] Visual Studio Tools for AI (also available for open-source Visual Studio Code)
• [at 7:30] Azure Machine Learning Studio, including a demo of using the drag-and-drop UI to predict car prices.

For those of you familiar with similar services from AWS, or just generally baffled by some of the Microsoft service names (and if so you're not alone there — believe me!), Siraj has also produced a plain English codex for Microsoft Azure services, with a description, corresponding service name in other systems, and what the service maybe should have been called in the first place.

Take a look at Siraj's video, below:

 

Using gganimate to illustrate the luminance illusion

Many illusions are based on the fact that our perceptions of color or brightness of an object are highly dependent on the background surrounding the object. For example, in this image (an example of the Cornsweet illusion) the upper and lower blocks are exactly the same color, according to the pixels displayed on your screen.

Mind = blown. These two blocks are exactly the same shade of grey. Hold your finger over the seam and check. pic.twitter.com/OqAnforGqs

— David Smith (@revodavid) December 5, 2013

Here's another simpler representation of the principle, created by Colin Fay (in response to this video made with colored paper). In the animation below, the rectangle moving to the left to right remains the same color throughout (a middling gray). But as the background around it changes, our perception of the color changes as well.

Colin created this animation in R using the gganimate package (available on GitHub from author Thomas Lin Pederson), and the process is delightfully simple. It begins with a chart of 10 "points", each being the same grey square equally spaced across the shaded background. Then, a simple command animates the transitions from one point to the next, and interpolates between them smoothly:

library(gganimate)
gg_animated <- gg + 
  transition_time(t) + 
  ease_aes('linear')

You can find the complete R source code behind the animation at the blog post linked below, along with an interesting discussion of luminance and how you should consider it when choosing color scales for your data visualizations.

RTask: Remaking ‘Luminance-gradient-dependent lightness illusion’ with R

Because it's Friday: A Titanic Brexit

Boris Johnson once declared that Britain leaving the European Union would be a "Titanic success". More than a year ago Comedy Central UK imagined what such a success would look like, and the prospects look much the same today: 

That's all from us for this week. Have a great weekend, and see you later! 

AI, Machine Learning and Data Science Roundup: August 2018

A monthly roundup of news about Artificial Intelligence, Machine Learning and Data Science. This is an eclectic collection of interesting blog posts, software announcements and data applications I've noted over the past month or so.

Open Source AI, ML & Data Science News

ONNX Model Zoo is now available, providing a library of pre-trained state-of-the-art models in deep learning in the ONNX format.

In the 2018 IEEE Spectrum Top Programming Language rankings, Python takes the top spot and R ranks #7.

Julia 1.0 has been released, marking the stabilization of the scientific computing language and promising forwards compatibility.

A retrospective on the R Project, now in its 25th year, from Significance Magazine.

Industry News

Google announces Cloud AutoML, a beta service to train vision, text categorization, or language translation models from provided data. The fast.ai blog evaluates the claim that AutoML works without the need for machine learning skills.

Google announces Edge TPU, a hardware chip and associated software to bring AI capabilities to edge devices.

Oracle open-sources GraphPipe, a network protocol designed to simplify and standardize transmission of machine learning data between remote processes.

AWS Deep Learning AMIs now include ONNX, making it easier to deploy exported deep learning models.

Amazon Rekognition is available in two new regions: Seoul and Mumbai. The video analysis service caused a stir when the ACLU reported it matching pictures of members of Congress to a database of criminal mugshots.

RStudio introduces Package Manager, a repository management server product to organize and centralize R packages.

Data from the O'Reilly Machine Learning Adoption Survey reveal that most machine learning models are built by in-house data science or product development teams, with only 3% adopting cloud-based ML service.

Anaconda Enterprise 5.2 adds capabilities for GPU-accelerated machine learning.

Microsoft News

TechNative's review of Microsoft's AI philosophy and technologies, and the roadmap ahead.

Microsoft R Open 3.5.1 is now available.

Power BI Desktop now offers Python integration (in preview).

AI for Accessibility, a $25M Microsoft program applying artificial intelligence to help people with disabilities.

Azure SQL Database now offers reserved capacity at a discounted rate, and reservation discounts for virtual machines now apply when using differing VM sizes within a group.

Learning resources

The Chartmaker Directory, an index of dozens of data visualization types with examples in more than 30 software tools.

An overview of the benefits and limitations of FPGUs compared to CPUs and GPUs for numeric computing.

A guide to deep learning applied to natural language processing, for those new to the field.

Containerized R and Python Web services with Docker and Microsoft ML Server.

Design, architecture, and implementation of an R-based recommendation system in Azure.

A guide to distributed deep learning in Spark, using Azure Batch and Azure HDInsight.

Google's Machine Learning Guides, with machine learning tips and a guide to text classification.

Field Guide to Machine Learning, a 6-part video series from Facebook Research.

What Data Scientists Really Do, according to a Harvard Business Review article.

Survey analytics company Crunch uses R to provide a data visualization service.

Finding optimal staff composition for a professional services company, with Azure Machine Learning.

Find previous editions of the monthly AI roundup here