Revolutions

Learning Time Series with R

by Joseph Rickert

Late last Saturday afternoon I was reading in my usual spot at the Dana Street Coffee House in Mt. View. A stranger walking by my table noticed my copy of Madsen’s Time Series Analysis (sitting there untouched again) said he needed to learn something about time series and asked if I could recommend a book. He looked serious so I asked him if he knew any R. To my delight the guy replied: “Of course,... I learned some R for my Ph.D.”. I suggested Cowpertwait’s Introductory Time series with R. The fellow pulled out his smartphone and bought a copy right then and there. As he left, the stranger said: “God I love this coffee shop”. Yes. I love the place too: funky Dana Street for sure, but I mean the whole Bay Area: R people are everywhere.

Anyway, because this was the second time in less than a week that someone asked me about time series, I thought it would be useful to collect some information on how one might go about learning time series with R. This is by no means a comprehensive survey. I have just gathered some low hanging fruit and listed a few books that have been helpful to me.

First off: Why R? Well, attempting to learn time series without good computational tools would be madness, and unless you are already a SAS, Stata or even a MatLab expert there would be little reason to even consider these systems. The three of them together can’t match the scope and depth of the time series tools listed in R’s Time Series Task View. Like all R task views, this one organizes the libraries of available R functions by topic and is the place to start if you what you are looking for.

Otherwise, learning time series comes down to matching you learning style and experience with the available R resources. If you are a book person looking for a general introduction to R that has some time series material, then I would suggest Paul Teetor’s R Cookbook. Chapter 14 is very good. It begins by making the case for using zoo and time series object and then moves briskly showing how to manipulate time series and the basics of ARIMA models.

If you are looking for a first book devoted entirely to time series then in addition to Cowperwait you might want to look at Time Series Analysis with Applications in R by Jonathan D. Cryer and Kung-Sik Chan. Cowperwait and Metcalfe cover more ground: in addition to the basics he has chapters on non-stationary series, long memory processes, spectral analysis, multivariate models and state space models in less than 250 pages. Cryer and Chan take things a little slower but they still cover ARIMA models, ARCH and GARCH models, regression models and spectral analysis. The TSA package that goes with the book is also helpful.

If you are looking for one book that will see you through graduate school then I don’t think you can do better than the new R friendly edition of Shumway and Stoffer’s Time Series Analysis and Its Applications with R Examples. Previous editions of this book have reached the stature of having defined time series for a generation of students.

If you are interested in time series for Finance then a good place to start would be David Ruppert’s book Statistics and Data Analysis for Financial Engineering. This book has several very nice chapters on ARIMA, GARCH and regression models and even has a discussion on fitting ARMA models with Bayesian techniques.  Also, anyone interested in time series for Finance would find the ebooks on the RMetrics site are a valuable resource.

If forecasting is your focus, then  Forecasting: principles and practice by Rob Hyndman and George Athanasopoulos is online and free. This book would be a good deal at full hardback prices, and it is very generous of the authors to promise to keep the online version free even after a print version becomes available on Amazon. Hyndman and Athanasopoulos are focused on basic principles and forecast accuracy, recommending simple tools when they are right for the job.The following illustrates the kind of big picture context setting knowledge that is not often explicitly mentioned by other authors:

It is a com­mon myth that ARIMA mod­els are more gen­eral than expo­nen­tial smooth­ing. While lin­ear expo­nen­tial smooth­ing mod­els are all spe­cial cases of ARIMA mod­els, the non-linear expo­nen­tial smooth­ing mod­els have no equiv­a­lent ARIMA coun­ter­parts. ...

There is a considerable amount of time series tutorial material online that is worth a look. If you only have a few minutes then the Quick R page Time Series and Forecasting might get you off to a good start. Coghlan’s A Little Book of R For Time Series and the Time Series Tutorial with R Part I from Zucchini and Nenadic both provide accessible elementary presentations covering seasonal decomposition and forecasting with ARIMA models.

The paper by  McLeod, Yu, and Mahdi, Time Series Analysis with R, provides an overview of time series topics at a more demanding level of mathematical sophistication. For example, there is a section on stochastic differential equations. The paper contains an extensive bibliography and many links to further reading.

At the risk at oversimplifying things, here are a just few lines of R code that take you from fetching real data to automatically fitting an ARIMA model. 

library(xts)
library(forecast)
# Fetch IBM stock data from Yahoo Finance
# Go to http://finance.yahoo.com/q/hp?s=IBM+Historical+Prices and copy the link to the table
url <- "http://ichart.finance.yahoo.com/table.csv?s=IBM&a=00&b=2&c=1962&d=05&e=26&f=2013&g=d&ignore=.csv"
IBM.df <- read.table(url,header=TRUE,sep=",")			# Read data into a data frame
head(IBM.df)							# Look at the first few lines
IBM <- xts(IBM.df$Close,as.Date(IBM.df$Date))			# Make a time series object
plot(IBM)							# Plot the time series
IBM.1982 <- window(IBM,start="1982-01-01", end="2013-01-23")	# Select at a subset
fit <- auto.arima(IBM.1982)					# Auto fit an ARIMA model
fit                                                             # Examine the model

Created by Pretty R at inside-R.org

This is an example of how R places powerful tool at you finger tips without getting in the way. For sure, there is real work in making the effort to develop the understanding and intuition to build meaningful time series models, but there is not much of a learning curve to climb to get a handle on the required R functions. All of the authors cited have made conscientious efforts to help you to become time series fluent through using powerful, easy-to-use R tools that will enhance your learning experience.

Oh, - about Madsen's book: there is no R code here. However, Madsen's approach, emphasizing the similiarity between linear systems and time series, is interesting and unusual. For the mahtematically inclined it is a great reference for topics like Kalman Filters.

Two forthcoming R books

Today I learned about two forthcoming R books that I'm now looking forward to.

The first is Applied Predictive Modeling by Max Kuhn and Kjell Johnson. Max Kuhn is the author of the caret package, an extremely useful and powerful R package for fitting and optimizing all kinds of predictive models in R. It's available now on Amazon Kindle and will be published in hardcover by Springer in July.

The second is Dynamic Documents with R and knitr by Yihui Xie, the author of the knitr package. With knitr you can easily create beautiful documents and reports, with text, tables and figures all dynamically generated by R. It will also be available in July.

If you already have either of these books, let us know in the comments.

A Review of the R Graphics Cookbook

A common criticism of R, especially from data scientists who are new to R but proficient in multiple programming languages, is that R is “quirky” and annoying because there is almost always more than one way to do simple things.  I usually counter that they are trying to say that R is “flexible” and “rich”, but by the time we get around to talking about plots I have to concede that my data scientist friends have a point. There are, after all, three completely different major plotting systems in R:  the base graphics system, lattice plots and the ggplot2. Each of these systems comprises its own little universe of functions, form, feel and things you just have to remember (or look up every time). They all do the basic plots really well.  So, when you come to a new plot that might be a little way out of the ordinary for you what is the best way to go? Now, I know. In the preface to his recent source book for producing plots in R, the R Graphics Cookbook Winston Chang writes. “Each recipe in this book lists a problem and a solution. In most cases, the solutions I offer aren’t the only way to do things in R, but they are, in my opinion, the best way.” – no mincing words here. I only got to page one and I knew I was going to like this book!

All right, so Winston Chang  is an accomplished ggplot2 developer , and who am I to argue whether the solutions he picked are the “best” solutions, but how good is the book itself? Let’s begin with the appendix. (When reading a technical book, I almost always jump from the beginning to somewhere near the end to get an idea of how much work I am signing up for.) Appendix A is a succinct, lucid and well-written introduction to ggplot2. In my opinion, it provides sufficient information, presented with exemplary clarity, to enable someone unfamiliar with ggplot2 to work through the rest of the book.  Since a good many of Winston’s examples are based on ggplot2 I recommend that anyone who is new to this package read appendix first.

The R Graphics Cookbook is mostly written in what might be called “web page style”: short sentences, bullet points, code snippets and plots integrated in a manner that, I think, facilitates comprehension. The book is organized into 15 chapters, eleven of which are either concerned with a particular kind of plot (e.g. Chapter5:  Scatter Plots) or some potentially vexing task that is often required to create a satisfying plot (e.g. Chapter 7: Annotations). The remaining chapters are devoted to supporting material such as Chapter 1: “R Basics” which is a minimalist account on getting information into R and Chapter 15 “Getting Your Data into Shape” which is a short, but fairly comprehensive account of transforming data into the “long form” used by ggplot().  Each chapter is organized as a series of “problems” with sections entitled “Problem”, “Solution” and “Discussion”. The Solution section contains the recommended code while the Discussion section provides some elaboration. In simple cases, the discussion may be just a few bullet points. At other times, the Discussion anticipates difficulties the reader may have. For example, the Discussion section “Making a Basic Bar Graph” introduces factors.

The following plot Download Stacked Plot from the discussion section of making a stacked graph illustrates is typical of the many simple examples presented in the book. It is a variation on the basic theme that shows the sophisticated look and feel that is the hallmark of ggplot2.

 

This next plot Download Vector field which shows a vector field superimposed on a map is one of the more elaborate examples presented in the book. It gives an idea of the data manipulation that is often required to make a useful plot.

The R Graphics Cookbook contains a few surprises too. The Note on the last page of the appendix reads:

“Some introductions to ggplot2 make use of a function called qplot(), which is intended as a convenient interface for making graphs. It does require a little less typing than using ggplot() plus a geom, but I’ve found it a bit confusing to use because it has a slightly different way of specifying certain graphing parameters. I think it’s simpler and easier to just use ggplot().”

I was pleasantly surprised to read this ( I have always been confused by qplot() ), but then I was surprised again to see that the second plot in the book, and several others, are done with qplot()! Clearly, there is more going on with these plots than meets the eye. Then, bang!, another surprise : In the Discussion section of “Making A Scatter Plot Matrix” Winston writes: “We didn’t use ggplot2 here because it doesn’t make scatter plot matrices (at least, not well). “ Damn! For two years now, I have been trying to make ggplot2 produce scatter plot matrices that color points by group . (I bet I am not the only one.) The problem is you start out doing a small project and decide to make everything look really nice with ggplot2 then, the thought pops in your head to do a plot matrix. Both the base R graphics system plot(iris[,1:4],col=iris$Species) and lattice  splom(iris[,1:4],col=iris$Species)do this easily, but there is plotmatrix() lurking in in the depths of ggplot2 and mocking my aesthetic efforts.  Personal problems aside, the point of all this is that the R Graphics Cookbook is not just a manual for ggplot2(). It is something much more valuable, a guide to making good plots in R written by a knowledgeable developer with strong feelings, a sense of style, and a gift for exposition. The book obviously doesn’t cover everything, and it would have been nice to see some lattice examples. The R Graphics Cookbook is, however, a very nice example of the kind of systematic, task orientated documentation that we need more of for R. I highly recommend it, especially for people who are new to R. I might even buy a couple of copies for my multi-language, “need the one best way to do things” friends.

 

 

Review of 'R in Action' by Robert I. Kabacoff

By Joseph Rickert

Yesterday, the cosmic randomizer placed me next to a newly minter lawyer in a crowed Los Gatos coffee shop. In three minutes of conversation I learned that that the fellow was interested in corporate law, was about to take a job that would give him a seat in the great VC/start-up game and that he had some understanding of statistical models: outstanding! A lawyer with a quantitative background; so, we started talking about R (I always talk about R). But seriously, in today’s data-driven decision making environment every professional: doctor, lawyer and maybe even Indian chief ought to have some statistical skills and the tools to reveal the picture that the data are painting. There is no doubt in my mind either that R is the right tool for these professionals, or that with the right introduction they can learn enough R to resurrect the basic statistical knowledge they acquired at some time in their professional training and use it to visualize a simple data set.

For the class of professionals I have in mind, and for anyone who has some background in statistics and who would like to do useful data analysis without becoming a programmer, Robert I. Kabacoff’s book, R in Action: Data Analysis and Graphics with R, is the right introduction. After making the case for R and providing some graphics for motivation, the book starts slowly with setting up the R environment and developing the necessary data handling skills. The analytic techniques presented midway through the book are mostly limited to elementary statistics, linear regression and the analysis of variance; but these are precisely the topics that professionals with engineering, bio-science and business backgrounds are most likely to be familiar. The extraordinary value in Rob’s book, however, is the emphasis on producing elegant, interpretable plots. With the clarity that has become trademark of Rob’s Quick-R website, R in Action presents multiple examples of simple, elementary R code that may be adapted to produce insightful, report-ready work. One of my favorite examples from Chapter 11 uses sample data in the car library to show the relationship between miles per gallon and weight segmented by the number of cylinders. With just a few lines of code, carefully explained in the text,

library(car)
scatterplot(mpg ~ wt | cyl, data=mtcars, lwd=2,
main="Scatter Plot of MPG vs. Weight by # Cylinders",
xlab="Weight of Car (lbs/1000)",
ylab="Miles Per Gallon",
legend.plot=TRUE,
id.method="identify",
labels=row.names(mtcars),
boxplots="xy"
)

Rob leverages the default capabilities of the scatter plot function to make a plot worth looking at. This example is most effective in showing R can be harnessed for useful work with very little programming. It is not necessary to climb any steep learning curve to use R at this level. Anyone with an interest in squeezing information from data with and the technical savvy to download R should be able use Rob’s book to make R work for them.

Rob goes on to explain several sophisticated two and three dimensional sample plots, including hexagonal binning for large data sets, and correlograms that are nice to see in an elementary text.

There are now well over 100 books either about R itself, or that use R for computations and graphics. Someone not familiar with this powerful and empowering language might question the need for still another introduction. The truth is that among all of these texts there is considerable repetition and overlap. However, Rob’s book shows that there is always room for simplicity, clarity and elegance that follows from function.

Review of "The Art of R Programming" by Norman Matloff

By Joseph Rickert

Anyone seeking to learn R faces two major challenges: (1) learning how to swim in the sea of information: R packages, books, websites, blog posts, message boards etc. that threatens to drown a newbie and (2) and coming to grips with the structure, syntax and features of the language itself. Having some idea of what one wants to do with R is clearly an important first step that will set the path of learning. R, an open source computer language, is the premier software system for statistical computing. Not only can any statistical idea be expressed in R, it is likely that someone in the open source community has already written a function to accomplish or at least facilitate any statistical analysis a working statistician or data scientist might be contemplating.

R functions are organized into libraries or packages that usually relate to some particular statistical task. Assuming something like an average of 20 functions per package, the 3400 available contributed packages[1] offer over 68,000 routines to read in data, manipulate it analyze it and visualize the results. No one could possibly become familiar with all of these. But, because R is an interpreted (instant feedback) language that encourages experimentation, some serious, sophisticated statistical analyses can be accomplished by stringing together the appropriate functions into a script. If interest in R is to only perform some particular analysis then a beginner’s best bet might be to select one of 100 or so books or blogs on doing statistics with R that provides relevant sample code and cut and paste to get a workable script. There is no shame in this. That is why all the open source authors went to the trouble of packaging up their work.

However, if a person really wants to be able to speak the R language and become a competent R programmer then, at the present time, one can find no better guide than Norman Matloff’s The Art of R Programming. Professor Matloff is a statistician and a computer scientist with a considerable amount of teaching experience. His book is no mere programming reference guide. It is a carefully crafted sequence of lessons that start at the beginning and work up to some fairly advanced topics including a lucid account of object-oriented programming in R, a presentation of the rudiments of TCP/IP operations and a discussion of R programming for the internet, examples of parallel programming with R, and a discussion spanning several chapters of how to write production-level R code that includes methods and advice on debugging R code, writing efficient R code, and interfacing R with other languages. Other distinguishing features of the book are brief examples showcasing a large number of functions (including rare gems such as D() for symbolic differentiation) that indicate the power and scope of R, and over thirty “Extended Examples” each of which is a credible study in writing careful, professional code. The most captivating aspect of the book, however, is Matloff’s thoughtful manner of exposition. R’s rich, compact syntax can be challenging the first time around. Matloff knows where the difficulties are. His presentations of R’s various features and functions begin from a point of view that anticipates obstacles that likely to confound someone going down the R path for the first time and guides the novice around them. I expect that The Art of R Programming will appeal to diverse audience of aspiring R programmers.

---

[1] As of 11/27/11

DecisionStats review of Revolution R Enterprise 5.0

At the DecisionStats blog, Ajay Ohri has published his review of Revolution R Enterprise 5.0. The review includes a slideshow highlighting some of the features of the new release, including the expanded code snippets manager and the new cluster job manager. It's well worth checking out if you'd like a quick overview of what's new in the latest release. Thanks, Ajay!

DecisionStats: Product Review – Revolution R 5.0

Three free books on R for Statistics

Avril Coghlan, a lecturer at University College Cork in Ireland, has written and made available for free three books ideal for students or practitioners new to R who want to use it for multivariate analysis, time series analysis or biomedical statistics. Each book begins with practical advice for installing and using R in general, before diving into their specialized topics:

• A Little Book of R for Multivariate Analysis (pdf, 49 pages) is a simple introduction to multivariate analysis using the R statistics software. It covers topics such as reading and plotting multivariate data, principal components analysis, and linear discriminant analysis.
• A Little Book of R for Biomedical Statistics (pdf, 33 pages) is a simple introduction to biomedical statistics using the R statistics software, with sections on relative risks and odds ratios, dose-response analysis, clinical trial design and meta-analysis.
• A Little Book of R for Time Series (pdf, 71 pages) is a simple introduction to time series analysis using the R statistics software (have you spotted the pattern yet?). It includes instruction on how to read and plot time series, time series decomposition, forecasting, and ARIMA models.

All three books are free to use, share and remix under a Creative Commons license, and are available from Dr Coghlan's home page linked below.

Dr Avril Coghlan: avrilomics

Data Science: a literature review

Just what is Data Science, anyway? Here's one take:

Ever since the term "Data Scientist" was coined by DJ Patil and Jeff Hammerbacker in 2009, there's been a vigorous debate on what the team actually means. More than 80% of statisticians consider themselves data scientists, but Data Science is more than just Statistics. (My own take is that Data Science is a valuable rebranding of computer science and applied statistics skills.)

To help bring clarity to the issue, Data Scientist and R user Harlan Harris has published a great presentation he gave at the Data Science DC meetup group, "What is Data Science anyway?". The presentation recaps the key data science discussions over the last few years, from Hal Varian ("the sexy job in the next 10 years will be Statistics"), Mike Driscoll ("sexy skills of data geeks"), Nathan Yau ("data scientists: people who can do it all"), Mike Loukides ("Data science enables the creation of data products"), Hilary Mason ("Data science is clearly a blend of the hackers"), Drew Conway ("The Data Science Venn Diagram") and many others.

In fact, the entire presentation servers as a literature review for the birth of "Data Science" as a concept, and would make excellent fodder for the "Data Science" page on Wikipedia which, sadly, is still a blank page.

One thing that seems certain: Data Science is here to stay. Companies are clamoring to hire people with data science skills, and the excitement at data science events like the recent Strata conference in New York is palpable. (This review of the conference even says that "Data Scientists are the new rock stars of the technology world.") As with all new concepts, the definition of "data science" may seem a bit cloudy now, but I'd wager that in 5 years or less "Data Scientist" will be as natural a job title as "Product Manager" or "Engineer". Haran notes  a most apposite analogy made by John Cook, who looked at the "computer programmer" role:

"Take an expert programmer back in time 100 years. What are his skills? Maybe he’s pretty good at math. He has good general problem solving skills, especially logic. He has dabbled a little in linguistics, physics, psychology, business, and art. He has an interesting assortment of knowledge, but he’s not a master of any recognized trade."

Apply the same quote to "data scientists", and I'd wager that just 5 years or so we'll look at all the data scientists working around us and wonder how companies ever survived without them.

Harlan adds additional thoughts to his presentation at the link below.

Harlan Harris: Data Science, Moore’s Law, and Moneyball

A free book on probability and statistics with R

G Jay Kerns has published a 400+ page introductory text on Probability and Statistics. All of the examples and illustrations are done using R (as Jay puts it, "The people at the party are Probability and Statistics; the handshake is R") so if you want to brush up on your probability and learn R at the same time, this might be a good resource. It would also be great for teaching: Jay wrote the book based on an undergraduate course he gave at Youngstown State University. There's also a plug-in for R Commander to access some of the methods via dialogs.

Jay's book is free, in both senses of the word. You can download the PDF for free from Lulu, or purchase a printed copy for just over $30. Jay has also published all of the LaTeX sources if you want to build the book yourself. And if you're already using R, you can read the book with just three commands:

install.packages("IPSUR")
library(IPSUR)
read(IPSUR)

I haven't read the entire book, but glancing through it, it looks like a comprehensive overview of the basics of Statistics: distributions, hypothesis testing, estimation, linear regression, and even touches on resampling and nonparametric methods.

I did want to point out one minor error on page xiii, though. It's "We're not in Kansas any more, Toto", not "We're not in Kansas any more, Dorothy". They'd take away my Rainbow Card if I didn't mention that. 

G Jay Kearns: Introduction to Probability and Statistics using R

Learning R

When R is brought up as a possibility for doing statistics or data mining or any sort of predictive analytics among non R users, someone will invariably point out that R has a “steep learning curve”, and the response among those gathered usually includes a significant amount of head nodding. Even those who have put in heroic efforts to help people learn R sometimes say scary things: e.g. in his introduction to Rattle, a Data Mining GUI for R, Graham Williams writes:

 R offers a breadth and depth in statistical computing beyond what is available in commercial closed source products. Yet R remains, primarily, a programming language for the highly skilled statistician, and out of the reach of many. (The R Journal Vol. ½, December 2009)

Are things really that bad? Is R that difficult to learn? I think not - unless you take the statement to refer to the absolutely worst case scenario that you can think of: the effort that would be involved in learning R by a person who doesn’t have any background in statistics or data analysis and who has absolutely no interest in learning R. In this context the statement that R has a steep learning curve conveys the same truth as the assertion that Chinese or Italian or Japanese or English or any other language is difficult to learn if you don’t know anything about the people who speak these languages, don’t want to know, and wouldn’t have an opportunity to practice the language anyway. There is some truth to this, but so what? Context is everything. If you have some background in statistics and a desire to learn some more, learning R is not going to be an insurmountable problem. Once you have some version of R installed on your favorite computing platform, any book that provides carefully worked examples of the kinds of statistical analyses that interest you should be all that is required to make you productive. These days, there are dozens or maybe even hundreds of statistics books that use R. Two of my personal favorites are John Fox’s classic An R and S Plus Companion to Applied Regression  (Sage Publications, 2002) and Data Analysis and Graphics Using R: An Example-Based Approach (Cambridge Series in Statistical and Probabilistic Mathematics) by John Maindonald and W. John Braun (2010). Books, of course, are only the tip of the iceberg of the resources available for the statistical cognoscenti to learn R. Have a look at the links on Inside-R as places to start on the web.

This discussion does raise the issue of what one means by learning a language. Knowing how to run the R scripts required to get some simple analysis done may not entitle you to say that you know R (significantly more is required to become an R developer) anymore than understanding enough Italian to get around Rome while eating well entitles you to say that you know Italian – but again, so what? It is possible to do some pretty impressive statistics using R without any deeper understanding of the R language than what is required to run the applicable models. On the other hand, if you don’t know any statistics and you don’t want to know more than you have to, then that might be a big problem. But, it’s the statistics part that has the steep learning curve, not R. Maybe this is the point of William’s comment: R remains the language of highly skilled statisticians because it is the language most capable of expressing what highly trained statisticians think about, and R is out of the reach of the many who just don’t care much about statistics. But, if you are reading this post this many probably doesn’t include you.

So suppose you are not a statistician but belong to some other data analysis culture, data mining, for example, and you want to learn R. Well, like any of the world’s great natural languages R approachable from many different starting points. The entry point may be different but the path to knowledge is going to be similar. Find something you care about and start formulating simple sentences. There are fewer R language guides available for people who have some data mining skills, but that is changing. Seni and Elder have recently published a very nice little book on ensemble methods: Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions (Synthesis Lectures on Data Mining and Knowledge Discovery) that I highly recommend.  And, of course the masters Hastie, Tibshirani and Freidman authors of the classic text, Elements of Statistical Learning, have made both the text and much of the their code available. Moreover, if you are a data miner with a background in computer science you can probably code rings around the highly trained statisticians. If so, you may find the books by Chambers, Software for Data Analysis and Gentlemen, R Programming for Bioinformatics of great value.

Finally, to be perfectly fair, I should acknowledge Graham Williams’ quote in the context in which he wrote it. Like any other language, R is much easier to learn when you have access to the best learning tools, CDs, dictionaries, grammars etc. Rattle is one such high value learning tool and so is Revolution’s R Productivity Environment. But, more about these on another day.