Revolutions

Because it's Friday: F'n railguns, how do they work?

How can this slowly moving ball bearing translate into enough power to launch the can from the table?

Neodymium magnets, that's how. The bearing is rolling towards a very powerful magnet; as it gets close, it's accelerated powerfully, and the momentum is transferred, Newton's cradle style, to the ball at the other end of the cluster (which isn't held as strongly by the magnetic force, being further from the magnet). Repeat that acceleration process a few times, and you get an enormously powerful multiplication of force. This video explains the process in more detail:

 

Unfortunately, you can't repeat the magnet clusters indefinitely to get the ball bearing accelerated to, say, 90% of the speed of light. After 5 or so clusters, the velocity of the ball causes the magnet to shatter (neodymium is quite brittle), so that's one reason why we don't have neodymium-magnet rifles on the battlefield. Another reason is that the energy of the projectile doesn't come for free; it's drawn from the bearing at each stage falling down a magnetic well. You'll have to pay that energy back each time you "reload" the gun -- it takes quite a lot of effort to remove the bearing from the magnet while it's stuck there.

And I know this isn't technically a railgun, but the ICP reference in the title was too much to pass up. Enjoy your weekend!

Browse the in-development R sources at GitHub

As an open-source project, the R source code has always been available to download from the R-project website. You can find source code for the latest released version here, and for the changing-daily new version in progress (R-devel) here.

But if you don't have the R sources handy, and just want to check on the contents of a file or two, a read-only, browsable mirror of the R-devel sources is now available at GitHub. This mirror is updated hourly from the SubVersion repository that the R core team uses to manage the R source code, and so also includes the change history and commit logs for each file. (You'll see a lot of notes marked [ripley] and [hornik] as they and other R-core members update the code.)

Note that this is the source code for the development version of R, and so probably doesn't match the version you're using. But it's a great reference to see what's going on in the R sources right now. Also, it's only includes the files used for development; some human readable files which are built from other source files aren't included. (One example is the HTML version of the NEWS file, but you can always find the daily NEWS file for R-devel here.)

github: Read-only mirror of R source code (via Hadley Wickham)

3-D animation of the changing Antarctic ice sheet

Last month we shared an visualization showing the changing extent of Arctic sea-ice. This visualization by the multinational Commission for the Conservation of Antarctic Marine Living Resources (CCAMLR) switches the view to the Southern pole and takes the visualization to a whole new level, by animating it in 3-D:

The amount of sea ice in the Southern Ocean surrounding Antarctica varies on a seasonal basis and partly depends on the amount of sunlight reaching the region. The area covered by sea ice in the Southern Ocean varies with the seasons, from approximately 3 million km² in February (at the end of the Austral summer), to approximately 20 million km² in October. The areas marked in red indicate the management areas used by CCAMLR in managing Antarctic fisheries.

 

The animation was created in R using the rgl package, which uses OpenGL for the 3-D rendering. Note the depiction of undersea mountains (based on GEBCO bathymetry data) which can hold sea ice in place, and the monthly ice extents (drawn from satellite-derived sea ice concentrations from NSIDC).

CCAMLR: Monthly ice concentration in Antarctica (with thanks to reader DM for the tip)

Using R in production: industry experts share their experiences

I had a great time yesterday moderating the "R in Action" panel discussion at the DataWeek conference in San Francisco. Each of the panelists represented a company that is actively using R and/or Revolution R Enterprise. Here (from memory, since I couldn't take notes) are some the things they shared:

• Jesse Bridgewater from eBay talked about how R is used within eBay's operations to analyze massive volumes of data. 
• Anthony Goldbloom from Kaggle confirmed that more Kaggle competition winners use R than any other software package. Random Forests is a particularly popular predictive modeling technique in the competiitions.
• Jim Porzak from Minted espoused the benefits of using R: the flexibility of the language, the data visualization capabilities, and the extensive user community and CRAN pacakages.
• Hsiu-Khuern Tang from Intuit described how R is used in the creation of the Intuit Small Business Index, and some of the advanced modeling packages (such as glmnet and the Rmetrics suite) used at Intuit.
• John Wallace from Upstream Software talked about dealing with Big Data in R, and using the modeling routines in Revolution R Enterprise to fit regression models for use in production.
• Damien Weldon from Corelogic contrasted using (previously) SAS and (now) R for financial data analysis.

Other topics of conversation included building data science teams to work in modern analytical environments, the strengths and weaknesses of R, and the technology stack and other applications working with R. All in all, it was a lively and entertaining discussion — my sincere thanks go to all of the panelists for sharing their insights and expertise.

Thanks to our guest bloggers

I'm back from a very relaxing holiday in Australia. Many thanks to our guest bloggers for filling in over the last couple of weeks with some great information about R while I was away. If you missed any of the posts, be sure to check them out:

• Douglas McNair, "Population health management with RevoScaleR"
• Yihui Xie, "Integrate data and reporting on the Web with knitr"
• Nathan Yau, "Tutorials for Learning Visualization in R"
• Naomi Robbins, "Effective Graphs with R"
• Garrett Grolemund, "Visualize complex data with subplots"
• Ron Fredericks, "Tips for Making R User Group Videos"
• Andrew Winterman, "Designing Data Apps with R at Periscopic"
• Alex Guazzelli, "The PMML Revolution: Predictive analytics at the speed of business"
• Winston Chang, "How to use your favorite fonts in R charts"
• Luba Gloukhov, "A video map of the hottest songs in the US"

I'm at the DataWeek conference in San Francisco today hosting the R Lab, so normal blogging will resume tomorrow. In the meantime, I thought I'd share some of the wonderful wildlife I saw on my trip. Here's a shot of Australian Fur Seals I took on Kangaroo Island, South Australia.

Coursera's free online R course starts today

Coursera offers a number of on-line courses, all available for free and taught by experts in their fields. Today, the course Computing for Data Analysis begins. Taught by Johns Hopkins Biostatistics professor (and co-author of the Simply Statistics blog) Roger Peng, the course will teach you how to program in R and use the language for data analysis. Here's a brief introduction to the course:

 

The course will run for the next 4 week, with a workload of 3-5 hours per week. You can sign up at the link below.

Coursera: Computing for Data Analysis

A video map of the hottest songs in the US

Today's guest post comes from Revolution Analytics data scientist Luba Gloukhov — ed.

As a big fan of R’s ever-expanding geospatial plotting capabilities, I jumped at the chance to create a map using The Million Song Dataset (MSD).  For each of the 10,000 songs in the million song subset containing non-missing latitudes & longitudes and non-zero artists familiarity scores, I added a song ‘break-out’ score — the ratio of song hotttnesss to artist familiarity (similar that introduced by Echo Nest’s Hottt or Nottt blog post).  I then subset the data further to only include songs with a break-out score of greater than 1.

Exploring the map, I found myself YouTubing songs. Many of them I had never heard of before — a sign of either generally low artist familiarity scores or my own tanking hipppnesss.  I decided against exploring the “why?” fearing it might just prove the latter.  In an effort to simplify my process of exploration (and “what’s hottt”-self-education), I used R to embed YouTube videos in the map’s marker info windows. 

Explore the interactive map here.

For each song in my subset, I added a new variable containing code that embeds the first YouTube video search result of the artist name and song title. For plotting, I used Milan Kilibarda’s plotGoogleMaps package. You can access my code and data via github.

Oakland’s own Del tha Funkee Homosapien’s 1991 hit Mistadobalina came up as a recent break-out song (with a ratio of 1.04).  Since 1991, Del’s had numerous successes – as a member of Hieroglyphics, with the album Deltron 3030 and, perhaps most prominently, as a member of the Gorrilaz.  One would think that, by now, Del’s familiarity would surpass Mistadobalina’s hotttnesss, generating a ratio of less than 1.  How does this ratio compare to those of other Del songs? Is it really the case that Mistadobalina remains Del’s biggest break-out hit or do the hotttness scores of Del’s songs in general surpass his familiarity?  Perhaps more enlightening than momentary snapshots of these metrics would be an investigation of how the variables have changed over time.  

So much data exploration, so little time! I’ll have to save these topics for another R play date.

Luba Gloukhov is a Pre-Sales Engineer at Revolution Analytics. When not playing with R or helping others do the same, she can be found lifting heavy objects, thinking light thoughts or eating delicious food.

 

How to use your favorite fonts in R charts

Today's guest post comes from Winston Chang, a software developer at RStudio — ed.

When it comes to making figures in R, you can use any font you like, as long as it's Helvetica, Times, or Courier. Using other fonts that are installed on your computer can seem an impossible task, especially if you want to save the output to PDF.

Fortunately, the extrafont package makes this process much easier. With it, you can create beautiful, professional-looking results like this:

When using fonts in PDF files, there are two challenges: First you must tell R that the font is available to use. Next, you must embed the font into the PDF file to make it render properly on another computer or printer that doesn't already have the font.

Here's an example of what a PDF using Garamond might look like when it's not embedded, and printed or viewed on a device that lacks the font. It will substitute some other font in the place of Garamond:

Using extrafont in three easy steps

The first step is to install extrafont, and then import the fonts from your system into the extrafont database:

Installation

install.packages("extrafont")

library(extrafont)
font_import()

You may see some warnings, but you should be able to ignore them. After the fonts are imported, you can view the available fonts by running fonts() or fonttable():

fonts()
##  [1] "Andale Mono"                  "AppleMyungjo"                
##  [3] "Arial Black"                  "Arial"                       
##  [5] "Arial Narrow"                 "Arial Rounded MT Bold"       
##  [7] "Arial Unicode MS"             "Bangla Sangam MN"            
##  [9] "Brush Script MT"              "Comic Sans MS"               
## [11] "Courier New"                  "Georgia"                     
## [13] "Gujarati Sangam MN"           "Impact"        
##  ...

# This will show more detailed information about fonts
fonttable()

Creating PDF files with fonts

Once you've imported the fonts from your system to the extrafont database, they must be registered with R as being available for the PDF output device. This must be run once in each R session where you want to use the fonts:

library(extrafont)
loadfonts()
# If you want to output to .ps files instead of .pdf, use:
# loadfonts(device="postscript")

After the fonts are registered with R's PDF device, you can create figures with them. Here's an example using base graphics:

pdf("plot_garamond.pdf", family="Garamond", width=4, height=4.5)

plot(mtcars$mpg, mtcars$wt, 
     main = "Fuel Efficiency of 32 Cars",
     xlab = "Weight (x1000 lb)",
     ylab = "Miles per Gallon")

dev.off()

Again, you may see some warnings, but they shouldn't cause any problems.

And with ggplot2:

library(ggplot2)
p <- ggplot(mtcars, aes(x=wt, y=mpg)) + geom_point() +
  ggtitle("Fuel Efficiency of 32 Cars") +
  xlab("Weight (x1000 lb)") + ylab("Miles per Gallon") +
  theme_bw() +
  theme(text=element_text(family="Garamond", size=14))

ggsave("ggplot_garamond.pdf", p, width=3.5, height=3.5)

Embedding fonts

Extrafont uses GhostScript, a free PostScript interpreter, to embed the fonts. You'll need to make sure it's installed on your computer (note that GhostScript is not an R package). If you're using Windows, you'll also need to tell R where the Ghostscript executable is:

# For Windows - in each session
# Adjust the path to match your installation of Ghostscript
Sys.setenv(R_GSCMD = "C:/Program Files/gs/gs9.05/bin/gswin32c.exe")

To embed the fonts, use embed_fonts():

#  If you don't specify 'outfile', it will overwrite the original file
embed_fonts("plot_garamond.pdf", outfile="plot_garamond_embed.pdf")

embed_fonts("ggplot_garamond.pdf", outfile="ggplot_garamond_embed.pdf")

If you have many PDF figures that are being put into a single PDF document, it can be more space-efficient to embed the fonts in the final document, instead of in each figure. That way, there will be a single copy of the font in the document instead of one for each figure.

Using Computer Modern (with math symbols)

If you are creating a document with TeX, there's a good chance that your document uses the default font, Computer Modern. Some R users use TikZ (with the tikzDevice package) to create figures that use Computer Modern, to match the appearance of the document text.

With extrafont, you can use Computer Modern in a PDF figure without using TikZ. This is done by installing the font package called fontcm. (Extrafont supports special font packages that contain a PostScript type 1 font stored in a particular way.)

font_install("fontcm")

It will ask you to download the fontcm package from CRAN, and then it will import the font into the extrafont database. Once that's done, run loadfonts() to register the fonts with R's PDF device. If you run fonts(), they will be listed as “CM Roman”, “CM Sans”, and “CM Typewriter”.

Now you can create PDF figures with these fonts:

# Base graphics
pdf("plot_cm.pdf", family="CM Roman", width=5.5, height=5)

curve(dnorm, from=-3, to=3, main="Normal Distribution")
text(x=0, y=0.1, cex=1.5, expression(italic(y == frac(1, sqrt(2 * pi)) *
                                     e ^ {-frac(x^2, 2)} )))

dev.off()
embed_fonts("plot_cm.pdf", outfile="plot_cm_embed.pdf")

And with ggplot2:

# ggplot2
exp_text <- "italic(y == frac(1, sqrt(2 * pi)) * e ^ {-frac(x^2, 2)} )"

p <- ggplot(data.frame(x=c(-3,3)), aes(x=x)) + ggtitle("Normal Distribution") +
  stat_function(fun=dnorm, geom="line") +
  theme_bw() +
  theme(text=element_text(family="CM Roman")) +
  annotate("text", x=0, y=0.1, label=exp_text, parse=TRUE, family="CM Roman")

ggsave("ggplot_cm.pdf", p, width=4, height=3.5)
embed_fonts("ggplot_cm.pdf", outfile="ggplot_cm_embed.pdf")

Note that font packages installed this way can only be used for PDF and PostScript figures generated with R. They aren't available for on-screen output, nor are they available for use for other programs.

Output to bitmap files (PNG, TIFF)

If you output graphics using Linux or Mac, you are probably using the Cairo device or Quartz output devices. With these output devices, system fonts are generally available without having to jump through any of these hoops.

On Windows, however, the default output device, “windows”, requires registering fonts to work with it. Usually this is a bit of a chore, but extrafont makes it easy. After the fonts have been imported:

loadfonts(device = "win")

This registers the fonts with R's windows output device. Then you can make bitmap figures with system fonts on screen, or saved to a file.

Experimental support for OTF and TTC files

The released version of extrafont can only import TTF (TrueType) fonts. It doesn't work with with OTF (OpenType) and TTC (TrueType font collection) files, which are increasingly common. There is an in-development version supports these file formats. It can be installed from GitHub, using the devtools package. This requires that you have a C compiler installed on your system, and the proper R development environment set up:

library(devtools)
install_github("Rttf2pt1", "wch")
install_github("extrafont", "wch")

Once it's installed, import the fonts as before:

library(extrafont)
font_import()

Support for these file types is experimental. If you find any issues, please report them on the issues page for extrafont:

https://github.com/wch/extrafont/issues

Learning more

For more information, see:

• extrafont project page
• fontcm project page
• A writeup by Paul Murrell about using symbol fonts with R. (The fontcm package should take care of the issues.)

The current version of extrafont is 0.11. Previous version had problems with character spacing when you use hyphens and minus signs, but it should be fixed in the latest version.

Winston Chang graduated in 2012 from Northwestern University with a PhD in Psychology. He is now a software developer for RStudio, working primarily on developing the ggplot2 package for data visualization. He created and maintains the "Cookbook for R" website, and is the author of the forthcoming book "R Graphics Cookbook", to be published by O'Reilly Media.

The PMML Revolution: Predictive analytics at the speed of business

This guest post is by Alex Guazzelli, VP of Analytics at Zementis Inc. -- ed.

PMML, the Predictive Model Markup Language, is the de facto standard to represent predictive analytics and data mining models. With PMML, it is extremely easy to move a predictive solution from one system to another, since it avoids proprietary issues and incompatibilities.

Companies around the globe are benefiting from PMML to make instant use of their predictive solutions. With PMML, there is no need for custom coding: you can easily move your solution from the scientist’s desktop, where it was built, to the production environment, where it is operationally deployed. Companies also use PMML as the common language between service providers and external vendors. In this way, it defines a single and clear process for the exchange of predictive solutions. It becomes the bridge not only between data analysis, model building, and deployment systems, but also between all the people and teams involved in the analytical process. This is extremely important, since PMML is used to disseminate knowledge and best practices, and to ensure transparency.

All the top analytical tools, commercial and open-source, support PMML. And, the language itself has reached a great level of maturity and refinement. PMML 4.1, its latest version, makes it extremely easy for predictive solutions to be represented in an open and standard way. With PMML, you can represent a myriad of pre- and post-processing steps, besides the predictive modeling techniques per se. PMML 4.1 allows for multiple models (model composition, chaining, segmentation, and ensemble, which includes random forest models), to be represented by a single and concise language element. It also allows for model outputs to be transformed into business decisions. Therefore, a PMML file is able to represent the entire solution, from raw data to business decision, with one or multiple predictive models.

The availability of a standard such as PMML combined with scoring solutions in the cloud, for Hadoop, and in-database make it possible for predictive analytics to fulfill its promise and crack the big data code. Zementis, Inc. has been in the forefront of PMML-based scoring, first through its ADAPA Scoring Engine, which is available for on-site deployment or as a service on cloud (Amazon and IBM), and lately through its Universal PMML Plug-in which is offered for a range of databases and for Hadoop. Zementis has partnered with Revolution Analytics, so that predictive solutions built in R can benefit from the vast scoring infrastructure already in place. I am proud to be associated with Zementis and excited to be part of an ever-growing PMML community.

A PMML package for R that exports all kinds of predictive models is available directly from CRAN.

Traditionally, the PMML Package offered support for the following data mining algorithms:

• ksvm (kernlab): Support Vector Machines
• nnet: Neural Networks
• rpart: C&RT Decision Trees 
• lm & glm (stats): Linear and Binary Logistic Regression Models 
• arules: Association Rules
• kmeans and hclust: Clustering Models 

Recently, it has been expanded to support: 

• multinom (nnet): Multinomial Logistic Regression Models;
• glm (stats): Generalized Linear Models for classification and regression with a wide variety of link functions 
• randomForest: Random Forest Models for classification and regression (click HERE for examples);
• rsf (randomSurvivalForest): Random Survival Forest Models;

And, this expansion is still on-going as the R community implements support for other packages and techniques. For more on the PMML package, please take a look at the paper we published with Graham Williams from Togaware in “The R Journal”. For that just follow the link below:

PMML: An Open Standard for Sharing Models

There may be quite a few reasons for you to move your predictive solution from R to an independent deployment platform. Among them, you may want parallel execution on big data or real-time scoring for applications such as fraud detection or recommender systems. With PMML you can easily move your model to the cloud or inside the database for scoring. Or, even have it executed on Hadoop. It is really up to you! On top of that, PMML allows for side-by-side deployment of predictive assets from R as well as other commercial data mining tools, supporting a multi-vendor environment as well as platform independent deployment.

More and more companies and individuals are using the PMML standard for the obvious benefits it provides, putting their predictive solutions on the fast track. With PMML, the speed of predictive solutions can be on par with the speed of business.

Dr. Alex Guazzelli is the VP of Analytics at Zementis Inc. where he is responsible for developing core technology and predictive solutions under ADAPA, a PMML-based decisioning platform. With more than 20 years of experience in predictive analytics, Dr. Guazzelli holds a PhD in Computer Science from the University of Southern California and has co-authored the book PMML in Action: Unleashing the Power of Open Standards for Data Mining and Predictive Analytics, now in its second edition (paperback and kindle). You can follow him at @DrAlexGuazzelli.

Designing Data Apps with R at Periscopic

Today's guest post comes to us from Andrew Winterman, Data Designer at data visualization company Persiscopic. He shares with us the process of using the R language and other tools to create an interactive data application for a client — ed. 

The Hewlett Foundation contacted us a few months ago because they were interested in exploring ways to visualize the distribution and impact of their grantmaking efforts over the last ten years. They hoped to make a tool with three functions: It would provide insight into where the Foundation has made the largest impact; provide grant seekers context for their applications; and help the Foundation’s officers make decisions about new grantmaking efforts, based on their existing portfolio. They had one request: No maps.

The data arrived, as it so often does, in the rough: An Excel document compiled quickly, by hand, with the primary goal of providing an overview, rather than complete accuracy. At this point in the process, we paint with broad brushes. We learn the data’s characteristics, determine which facets are interesting, and prototype visualization ideas.

At the beginning of a project, I always explore a few simple visualization techniques to get a feel for the data. For example, simple bar charts as shown in Figure 1, scatter plots, and choropleths, are great ways to get a visual sense of what the data is saying. 

 

Figure 1

My main tools for this process are d3.js, R — ggplot2 in particular — and Tableau. For this project I used ggplot2 (version 0.9 came out halfway through) and the CRAN package 'beanplot'.

 

Once we have a feel for the data, we start brainstorming, and trying out ideas. For example, an early idea led us to explore using concentric circles to represent the tree of geographic categories (Hemisphere, Continent, Country, Region, County, City), and then filling an arc of a circle with a scatter plot to show individual grants. You can see this idea sketched, with mostly fake data in Figure 2. We ultimately decided the technique didn’t use space effectively enough for what we needed to convey. 

Figure 2

Our next idea was to use modified beanplots [Figure 3] to succinctly describe the distribution of various quantities at the same time. These were made with the beanplot package available on CRAN. We denormalized them — meaning we hacked the beanplot function to make the total area of the beanplot proportional to volume. With traditional beanplots, the total area of each bean is always the same, since they represent probability distributions rather than counts. This is counter-intuitive if the viewer is unfamiliar with statistics. We actually went as far as developing a working tool using these modified bean plots. 

Figure 3 : Beanplot

The width of the bean at a given dollar amount shows the probability the next dollar falls at the given amount. After extensive user testing, this proved too high a cognitive hurdle for the casual viewer. Users liked the visual presentation, but were confused as to their meaning, even with a detailed page showing how to interpret the beanplot.

We decided to consider alternatives to the beanplot that still accomplish the same goals. We also wanted a very simple technique that could be explained in a phrase. After a few iterations, we agreed that interactive heatmaps [Figure 4] would be a good solution. You will be able to see them in action at Persiscopic.com when the final product launches at later this year.

Figure 4: Heatmap

R provides an ideal toolkit to explore methods to visualize data distributions. Between specialized packages and comprehensive toolkits like ggplot2, a wide range of techniques are available to the analyst. In particular, the transparent structure of most R functions make them easy to pull apart and put back together again, lending great flexibility to the patient programmer.

Andrew Winterman does Data Design for Periscopic. An inquisitive humanist, he is motivated by the promise of making ours a more rational society. He applies his skills to the problem of converting data into information, a process requiring scripting and research into the relevant fields of study. He holds a B.A. in Mathematics from Reed College, and patiently pursues a Masters of Science in Biostatistics at the Oregon Health and Sciences University. He greatly enjoys his daily bicycle commute, Portland’s artisanal culture, searing vegetables in cast iron, and thinking about epidemiology.