Revolutions

Because it's Friday: etaoin shrdlu, the hot-type qwertyuiop

If you've every had cause to complain about your word processor, consider what workers at the New York Times had to go through to print a newspaper in 1978, as shown in this 29-minute movie (via John Chisholm). Columns were typeset by arranging metal letters using a special machine, and column-justification was achieved with wedge-shaped inserts that spread the words across the column. Pages were laid out by moving the metal units around by hand, as a page editor scrutinized the layout in reverse and upside down. The print room was so noisy, that many of the workers were deaf and communicated by sign language. Lots and lots of molten lead was involved.

In 1978, this hot-type typesetting process was replaced by a computerized system that printed columns on paper instead as metal negatives, but even then the layout process still involved literal cutting and pasting with letters and glue! Makes Control-C, Control-V seem like the lap of luxury!

That's all from us for this week. Have a great weekend; we'll be back on Monday. See you then! 

All the R Ladies

Two groups are making and impact in improving the gender diversity of R users worldwide. The R-Ladies organization is creating chapters worldwide to facilitate female R programmers meeting and working together, and the Taskforce on Women in the R Community is working to improve the participation and experience of women in the R community.

R has more participation by women than many programming communities, but there's still a long way to go towards equity: the R Foundation's Women in R Taskforce estimates that between 11% and 15% of R package authors are women. (The count is based on package author first names; some manual corrections were needed because Hadley is categorized by genderizeR as a female name.)

In 2012, Gabriela de Queiroz founded the first women-focused R user group in San Francisco. Since then, "R Ladies" concept has expanded to a global franchise in eleven cities:

• San Francisco, California, USA 
• Twin Cities, Minnesota, USA
• London, England
• Research Triangle Park, North Carolina, USA
• Los Angeles, California, USA
• New York, New York, USA
• Melbourne, Australia
• Madrid, Spain
• Istanbul, Turkey
• Paris, France
• Boston, Massachusetts, USA

This is a great first step to increasing the participation by women in a currently male-dominated field. As the R-Ladies leadership note in their grant application to the R Consortium (which was funded in July):

The R community suffers from an underrepresentation of women in every role and area of participation, whether as leaders, package developers, conference speakers, conference participants, educators, or users. The R community needs to promote the growth of this major untapped demographic by pro­actively supporting women to fulfill their potential, thus enabling and achieving greater participation.

In addition to the groups created by R-Ladies, the Women in R Task Force is making strides towards achieving these goals. For example, the next useR! conference in Brussels will strive for gender balance amongst invited speakers, tutors, and committee and session chairs, and ensure particpation by women on panel discussions. Childcare will also be provided at the conference, and gender statistics will be published on the website. 

For more information on R-Ladies (including contacts to help you create a local chapter in your area), and the Women in R Taskforce, follow the links below.

R-Ladies Global : Women in R Taskforce

Watch: Highlights of the Microsoft Data Science Summit

I just got back from Atlanta, the host of the Microsoft Machine Learning and Data Science Summit. This was the first year for this new conference, and it was a blast: the energy from the 1,000 attendees was palpable. I covered Joseph Sirosh's keynote presentation yesterday, but today I wanted to highlight a few other talks from the program now that the recordings are available to stream.

On deep learning and machine intelligence:

• CNTK: Microsoft’s Open Source Deep Learning Toolkit, by Frank Seide (CNTK architect), on the engine behind many of Microsoft's AI and predictive services
• Cognitive Services: Making AI Easy, by Jennifer Marsman, on emotion detection, sentiment analysis and other intelligence services in Azure

On data science techniques and processes:

• Data Science Doesn’t Just Happen, It Takes a Process. Learn about Ours…, by Jacob Spoelstra and colleagues in the data science team
• Data Science for Absolutely Everybody, by Brandon Rohrer
• What Is Your Data Science Super Power? Tips and tricks to a successful data science project from Danielle Dean and Wee Hyong Tok
• Putting Science into the Business of Data Science, by Rafal Lukawiecki on statistical hypothesis testing for real-world applications

On production-scale data analysis with R:

• Building a Scalable Data Science Platform with R on HDInsight, by Debraj GuhaThakurta
• R and Spark as Yin and Yang of Scalable Machine Learning in Azure HDInsight, by Max Lukiyanov
• Deep Learning in Microsoft R Server Using MXNet on High-Performance GPUs in the Public Cloud, by Max Kaznady and Tao Wu
• and finally, my own talk See, Hear, Move on applications of data science to help people with disabilities, and how R is used to build such applications within the Microsoft stack.

But for me personally, the highlight of the event was the keynote presentation by Dr Edward Tufte on the Future of Data Analysis. I had the distinct honour of introducing Dr Tufte to the stage, who has long been a hero of mine. Please enjoy his presentation, embedded below.

There are many other great talks from the conference available for streaming, too many to mention here. Check out the full program at the link below.

Channel 9: Microsoft Machine Learning & Data Science Summit 2016

Using R to detect fraud at 1 million transactions per second

In Joseph Sirosh's keynote presentation at the Data Science Summit on Monday, Wee Hyong Tok demonstrated using R in SQL Server 2016 to detect fraud in real-time credit card transactions at a rate of 1 million transactions per second. The demo (which starts at the 17:00 minute mark) used a gradient-boosted tree model to predict the probability of a credit card transaction being fraudulent, based on attributes like the charge amount and the country of origin.

Then, a stored procedure in SQL Server 2016 was used to score transactions streaming into the database at a rate of 3.6 billion per hour. If you'd like to try this yourself, a step-by-step tutorial with code to implement the model and scoring is available here.

Later in the keynote (starting at 25:00), John Salch, VP of Technology and Platforms at PROS describes using R to determine prices for airline tickets, hotel rooms, and laptops. PROS has been using R for a while in development, but found running R within SQL Server 2016 to be 100 times (not 100%, 100x!) faster for price optimization. "This really woke us up that we can use R in a production setting ... it's truly amazing," he says.

It's great to see these global-scale applications of R, driving the intelligence of businesses behind the scenes. As Joseph said in the opening, "If there's one language you should learn today ... it's R." 

Channel 9: Microsoft Machine Learning & Data Science Summit 2016

The Financial Times uses R for Quantitative Journalism

At the 2016 EARL London conference senior data-visualisation journalist John Burn-Murdoch, described how the Financial Times uses R to produce high-quality, striking data visualisations. Until recently, charts were the realm of an information designer using tools like Adobe Illustrator: the output was beautiful, but the process was a long and winding one. The FT needed to be able to "audition" several different visual treatments quickly, to be able to create stunning visuals before deadline. That's where R and the ggplot2 package come in.

John presented a case study (you can see the slides with animations here) on creating this FT article, Explore the changing tides of European footballing power. The final work included 128 charts in total, telling the story of dozens of soccer teams in four countries. John also shared these animated versions (along with the ggplot2 code to produce them):

You can find the data and R code behind the animations here.

John B Murdoch: Ggplot2 as a creative engine ... and other ways R is transforming the FT's quantitative journalism 

Deep Learning Part 3: Combining Deep Convolutional Neural Network with Recurrent Neural Network

by Anusua Trivedi, Microsoft Data Scientist

This is part 3 of my series on Deep Learning, where I describe my experiences and go deep into the reasons behind my choices.

In Part 1, I discussed the pros and cons of different symbolic frameworks, and my reasons for choosing Theano (with Lasagne) as my platform of choice. A very recent benchmarking paper compares CNTK with Caffe, Torch & TensorFlow, and CNTK performs substantially better than all the other three frameworks.

In Part 2, I describe Deep Convolutional Neural Network (DCNN) and how Transfer learning and Fine-tuning helps better the training process for domain specific images.

This Part 3 of the series is based on my talk at PAPI 2016. In this blog, I show re-usability of trained DCNN model by combining it with a Long Short-term Memory (LSTM) Recurrent Neural Network (RNN).

I am delivering a Deep Learning webinar on 27^th September 2016, 10:00am-11:00am PST. You can register for live webinar to learn more on Microsoft Azure and Deep Learning. Please feel free to email me at [email protected] if you have questions.

Introduction

Given the role of clothing apparel in society, fashion classification has many applications. For example Liu et.al.’s work on predicting the clothing details in an unlabeled image can facilitate the discovery of the most similar fashion items in an e-commerce database. Yang et.al. shows that real-time clothing recognition can be useful in the surveillance context, where information about individuals’ clothes can be used to identify crime suspects. Depending on the application of fashion classification, the most relevant problems to solve will differ.

We will focus on papers by Wang et.al. and Vinyals et.al. and modify the model for optimizing fashion classification for the purposes of annotating images and predicting clothing tags for the fashion images.

The main inspiration of our work comes from recent advances in machine translation, where the task is to translate words or sentences individually. Recent work has shown that translation can be done in a much simpler way using RNNs and outperform the state-of-the-art performance. An “encoder” RNN reads the source tag/label and transforms it into a rich fixed-length vector representation, which in turn is used as the initial hidden state of a “decoder” RNN that generates the target tag/label. Here, we propose to follow this elegant recipe, replacing the encoder RNN by a DCNN. Over the last few years it has been convincingly shown that DCNNs can produce a rich representation of the input image by embedding it to a fixed-length vector, which can be used for a variety of vision tasks. Hence, it is natural to use a DCNN as an image “encoder”, by first pre-training it for an image classification task and using the last hidden layer as an input to the RNN decoder that generates tags. 

RNN-CNN model for generating Fashion Image Tags

In this work, we use an ImageNet pre-trained GoogLeNet model [Figure 2] for extracting CNN-features from the ACS dataset. Then we use the CNN-features to train a LSTM RNN model for the ACS Tag prediction. 

Architecture for GoogLeNet

The Data

In this work, we use the ACS dataset: a complete pipeline for recognizing and classifying people’s clothing in natural scenes. This has several interesting applications, including e-commerce, event and activity recognition, online advertising, etc. The stages of the pipeline combine several state-of-the-art building blocks such as upper body detectors, various feature channels and visual attributes. ACS defines 15 clothing classes and introduces a benchmark data set for the clothing classification task consisting of over 80, 000 images, which is publicly available. We use ACS dataset to predict new clothing tags for unseen images.

Images from the training and test datasets have very different resolutions, aspect ratios, colors etc. Neural networks require a fixed input size, so each image was resized and/or cropped to fixed dimensions (3 × 224 × 224).

One of the drawbacks of non-regularized neural networks is that they are extremely flexible: they learn both features and noise equally well, increasing the potential for overfitting. In our model, we apply L2 regularization to avoid overfitting. But even after that, we observed a large gap between the performance on the training and validation of ACS images, indicating that the fine-tuning process is overfitting to the training set. To combat this overfitting, we leverage data augmentation the for the ACS image dataset.

There are many ways to do data augmentation, such as the popular horizontally flipping, random crops and color jittering. As the color information in these images is very important, we only rotate the images at different angles – at 0, 90, 180, and 270 degrees.

Augmented ACS images

Finetuning GoogleNet for ACS

For this ACS Tag prediction problem, we fall under fine-tuning scenario 2 (refer to Part 2). The GoogLeNet model that we use here was initially trained on ImageNet. The ImageNet dataset contains about 1 million natural images and 1000 labels/categories. In contrast, our labeled ACS dataset has about 80,000 domain-specific fashion images and 15 labels/ categories. The ACS dataset is insufficient to train a network as complex as GoogLeNet. Thus we use weights from the ImageNet-trained GoogLeNet model. We fine-tune all the layers of the pre-trained GoogLeNet model by continuing the backpropagation.

Replacing the input layer of the pre-trained GoogLeNet network with ACS images.

Training LSTM RNN

In this work, we use a LSTM RNN model, which has shown state-of-the art performance on sequence tasks. The core of the LSTM model is a memory cell, which encodes knowledge of what inputs have been observed at every time step [Figure 5]. The behavior of the cell is controlled by gates, which are layers that are applied multiplicatively. Three gates are being used here:

1. forget the current cell value (forget gate f),
2. read its input (input gate i)
3. output the new cell value (output gate o).

Long-Term Short Memory (LSTM) Recurrent Neural Network

The memory block contains a cell ‘c’ which is controlled by three gates. In blue we show the recurrent connections – the output ‘m’ at time (t – 1) is fed back to the memory at time ‘t’ via the three gates; the cell value is fed back via the forget gate; the predicted word at time (t – 1) is fed back in addition to the memory output ‘m’ at time ‘t’ into the Softmax for tag prediction.

The LSTM model is trained to predict top tags for each image. It takes the ACS DCNN-features (produced using the GoogLeNet) as input. LSTM is then trained on a combination of these DCNN-features and labels for these pre-processed images.

A copy of the LSTM memory is created for each of the LSTM image and each label, such that all LSTMs share the same parameters and the output (m)×(t−1) of the LSTM at time (t – 1) is fed to the LSTM at time (t) [Figure 6]. All recurrent connections are transformed to feed-forward connections in the final version. We saw that feeding the image at each time step as an extra input yields inferior results, as the network can explicitly exploit noise in the image and overfits more easily. The LSTM -RNN loss is minimized w.r.t. all the parameters of the LSTM, the top layer of the image embedder CNN and label embedding.

DCNN-RNN Architecture for ACS Tag prediction

Results

We apply the above LSTM-RNN-CNN model for ACS Tag prediction. Tag prediction accuracy of this model improves quickly during the first part of the iterations and stabilizes after about 20,000 iterations.

Each image in our ACS image dataset consists of only unique types of clothing, there is no image combining the clothing types. In spite of this fact, when we test images with multiple clothing type, our trained model generates tags for these unseen test images quite accurately (~80% accurate).

Below we show a sample of our prediction outputs. Our test image below contains a person in suit and a person in t-shirt. 

Unseen Test Image

The image below describes the GoogLeNet Tag Prediction for our test image using ImageNet-trained GoogleNet model:

Test Image Tag Prediction by ImageNet-trained GoogLeNet model

As we see, the prediction using this model is not very accurate as our test image is tagged as ‘Windsor Tie’. The image below describes ACS Tag Prediction for our test image using the above LSTM-RNN-CNN model. As we see, the prediction using this model is very accurate as our test image is tagged as ‘jersey, T-shirt, upper body, suit of clothes’.

Test Image Tag Prediction by our ACS-trained LSTM-RNN-CNN model

Here our aim is to develop a model which can predict clothing category tags for fashion images with high accuracy.

Conclusion

In this work, we compare the tag prediction accuracy of the GoogLeNet model to our LSTM-RNN-CNN model. The tags predicted by our model are way more accurate than that predicted by the state-of-the-art GoogLeNet model. For training our model, we used a relatively smaller number of training iterations, around ~10,000. Prediction accuracy of our model improves quickly with increasing number of training iterations and stabilizes after about 20,000 iterations. We used only one GPU for this work.

Fine-tuning allows us to bring the power of state-of-the-art DCNN models to new domains. Moreover, combining DCNN-RNN model helps us extend the trained model to solve completely different problem like fashion image tag generation. I hope this blog series helps you train DCNNs for domain-specific images and leads to the re-usability of the trained models.

Because it's Friday: Illusions at the Periphery

This fantastic optical illusion has been doing the rounds recently, after being tweeted by Will Kerslake:

There are twelve black dots in that image, but I bet you can only see one or two of them at a time, directly where you're looking. As IFLScience points out, this is an example of an extinction illusion, caused by the software in our brain that fills in for our poor peripheral vision. We're not usually aware of it, but our vision only has good resolution looking directly ahead: our brains fill in the rest of the frame around the center from our blurry peripheral vision, our memories of looking around the scene, and pattern-matching to fill in the gaps. When the peripheral scene includes repeating elements, our brains can override real (but unseen) details, like the black dots at some of the intersections above.

Here's another famous example of the failure of peripheral vision: the celebrity illusion.

Those celebrity faces haven't been modified in any way, but for some reason when our peripheral-support software kicks in they get distorted into grotesque versions of themselves.

That's all from us here at the blog this week. We'll be back on Monday, reporting from the Machine Learning and Data Science Summit in Atlanta. Have a great weekend!

Microsoft R at the EARL Conference

Slides have now been posted for many of the talks given at the recent Effective Applications of the R Language (London) conference, and I thought I'd highlight a few that featured Microsoft R.

Chris Cole manages the deployment of R at Investec, supporting investment and risk teams worldwide. Despite some initial grumbling about having to use "old" versions of software, the R developers ultimately appreciated the improvements in collaboration and IT support derived from a standardized "R Suite" company-wide.

 

Investec also supports R deployment for other analysts who don't necessarily use R, by integrating R into desktop applications via a centralized DeployR server. This provides managed R-based analytics throughout the company, without the need to manage R on individual desktops.

Adam Rich has been evaluating the in-database capabilities of SQL Server 2016 R Services for Beazley Group. In his presentation, he compared and benchmarked various architectures and processes for applying R to data in SQL Server. The results varied by the type of load applied (calculating a mean of 60,000 rows vs 6,000,000 rows); pulling data from SQL to R was faster for the small loads, but pushing R code to the database was better for heavy loads.

Ben Downe, head of business intelligence at British Car Auctions, deploys R models developed in-house to the production environment using Azure ML. He describes the benefits of using the "one button API" to publish R functions as a Web service as "simple creation and destruction of new containers for new models / new test versions", "doesn't need to use IT resources", and "value for money". He also described using the Custom R Modules feature of Azure ML to simplify the process of updating R functions in deployment.

You can also check out the slides from my own keynote presentation, The R Ecosystem. There, I suggest why R is such a popular language despite being a language designed for data science (rather than a general purpose language), and having some idiosyncratic features (which seem quirky to some programmers, but make it easier to use for those with a background in data science). In addition to the language itself, there's a lot of value in the ecosystem surrounding R: the community resources, governing bodies, and companies (including Microsoft) who provide services and software to make using R more effective.

For more examples of R applications from the EARL London conference, check out the list of presentations (with slides) at the link.

EARL London: Speakers

Update May 22 2017: Fixed some broken links

How to choose the right tool for your data science project

by Brandon Rohrer, Principal Data Scientist, Microsoft

R or Python? Torch or TensorFlow? (or MXNet or CNTK)? Spark or map-reduce?

When we're getting started on a project, the mountain of tools to choose from can be overwhelming. Sometimes it makes me feel small and bewildered, like Alice in Wonderland. Luckily, the Cheshire Cat cut to the heart of the problem:

“Would you tell me, please, which way I ought to go from here?”
“That depends a good deal on where you want to get to,” said the Cat.
“I don’t much care where–” said Alice.
“Then it doesn’t matter which way you go,” said the Cat.
“–so long as I get SOMEWHERE,” Alice added as an explanation.
“Oh, you’re sure to do that,” said the Cat, “if you only walk long enough.”
(Alice’s Adventures in Wonderland, Chapter 6)

The first step to choosing your tools is to choose a goal. Make it clear and keep it firmly in mind.

That’s most of the work. After that there are a few other things to consider and traps to watch out for, but you’re 90% of the way there. Some tools fit some tasks better than others, so it’s just a matter of finding a match. The rest of the details are in this blog post, but if you just let your goal drive your choices, you can’t go far wrong.

Best of luck on your next project!

Data Science and Robots Blog: Which tool should I use?

Welcome to the Tidyverse

Hadley Wickham, co-author (with Garrett Grolemund) of R for Data Science and RStudio's Chief Scientist, has focused much of his R package development on the un-sexy but critically important part of the data science process: data management. In the Tidy Tools Manifesto, he proposes four basic principles for any computer interface for handling data:

1. Reuse existing data structures.

2. Compose simple functions with the pipe.

3. Embrace functional programming.

4. Design for humans.

Those principles are realized in a new collection of his R packages: the tidyverse. Now, with a simple call to library(tidyverse) (after installing the package from CRAN), you can load a suite of tools to make managing data easier into your R session:

• readr, for importing data from files
• tibble, a modern iteration on data frames
• tidyr, functions to rearrange data for analysis
• dplyr, functions to filter, arrange, subset, modify and aggregate data frames

The tidyverse also loads purrr, for functional programming with data, and ggplot2, for data visualization using the grammar of graphics.

Installing the tidyverse package also installs for you (but doesn't automatically load) a raft of other packages to help you work with dates/time, strings, factors (with the new forcats package), and statistical models. It also provides various packages for connecting to remote data sources and data file formats.

Simply put, tidyverse puts a complete suite of modern data-handling tools into your R session, and provides an essential toolbox for any data scientist using R. (Also, it's a lot easier to simply add library(tidyverse) to the top of your script rather than the dozen or so library(...) calls previously required!) Hadley regularly updates these packages, and you can easily update them in your R installation using the provided tidyverse_update() function.

For more on tidyverse, check out Hadley's post on the RStudio blog, linked below.

RStudio Blog: tidyverse 1.0.0