Revolutions

Putting the R in Hallowe'en

PhD student Caroline Tucker created a Hallowe'en card using the R graphics system:

You can find the R code to create the above image here. Note that you'll need to install the fields and MBA packages first. One interesting aspect of the code is the use of the mba.surf function to create the shading that gives the Jack-o'-lantern its depth.

The EEB and Flow: halloween caRd

R 2.14.0 is released

As scheduled, the first release of the new R 2.14 series is now available for download in source code form. As of this writing, pre-compiled binaries for Windows, Linux and MacOS aren't yet available, but will appear on your local CRAN mirror in the next couple of days.

One of the biggest changes in 2.14 is the introduction of the new "parallel" package as a standard part of R. As the NEWS file explains:

[The parallel package] incorporates (slightly revised) copies of packages multicore and snow (excluding MPI, PVM and NWS clusters).  Code written to use the higher-level API functions in those packages should work unchanged (apart from changing any references to their namespaces to a reference to parallel, and links explicitly to multicore or snow on help pages).

In addition, all of the standard (base and recommended) R packages are now byte-compiled on installation, which improves R's performance in several situations. 

Other improvements include better alignment of math and text in graphics, Rao's efficient score test for GLM's, the ability to draw curves from functions with plot.function, a new L'Ecuyer random number generator, improved access to documentation (especially vignettes), and several minor bug-fixes.

With R now on an annual (rather than six-monthly) update cycle, R 2.15 is not expected until October 2012 (with point releases for the 2.14.x series likely in the interim).

R-announce mailing list: R 2.14.0 is released

 

Last chance to enter the $20,000 "Applications of R" contest

Don't forget that midnight tonight (Pacific Daylight Time) is the deadline to submit your draft entry for the Applications of R in Business Contest, with $20,000 in prizes up for grabs from Revolution Analytics. You'll still be able to edit your entry during the public review period (until November 30), but no new entries will be accepted for the competition after tonight's deadline. 

We've already got some great entries with applications in manufacturing, finance, sports, IT, social network analysis and other industry-specific uses of R, but with seven cash prizes available there's a good chance of winning. We don't have too many entries yet making use of the unique capabilities of Revolution R Enterprise (like big data analysis, Hadoop, or Web Services integration), so that's something to consider given there's a $5,000 prize for such entries.

So get your entries -- at least a rough draft -- in today to qualify. All submissions will be viewable publicly at noon PDT tomorrow, at which point you can make comments on the entries for the benefit of the judges.

inside-R.org: Applications of R in Business Competition

Because it's Friday: Anti-Gravity Slinky

Here's an interesting quirk of Physics: if you hold a Slinky spring-toy in mid-air and then drop it, the base of the Slinky becomes momentarily suspended in mid-air, apparently defying gravity: 

Physics Student Dan Walsh provides a detailed explanation of this phenomenon, but the basic explanation is that while the center of gravity of the Slinky does indeed accelerate towards the ground at one gee, the base of the slinky is simultaneously accelerated upwards by one gee: the contracting force from the spring itself that, until the moment of release, was counteracting the gravitational force. The net result: the base remains stationary. You can see more in a longer version of the above clip, and a subsequent experiment with an even longer slinky (which exhibits the same behaviour).

Dan's Geometrical Curiosities: Explaining an astonishing slinky

New R User Group in Dublin, Ireland

Update Nov 8: Dublin-R now has a home on MeetUp.com.

There have been several requests to an R User Group in Ireland, so thanks to Kevin O'Brien for stepping up to co-ordinate the Dublin-R group. Kevin invites all R users in the area to the first meeting on November 17: 

The Dublin R users group will be holding a series of monthly meetings. On the agenda is the development of user group, recruiting new members, and to continue with the theme of our last meeting "Use of R in an industrial setting".

The next meeting is at 6.30 on Thursday November 17th. The location is The Bank Pub on College Green (look out for the R symbol ). We hope to see you there.

Meetings will continue monthly, and you can find the meeting schedule on the R Community Calendar.

Google Groups: Dublin-R

Building diversified portfolios with R

A common approach to reducing risk associated with financial portfolios is diversification. A portfolio made of components that are all highly correlated with each other -- a portfolio composed solely of financial stocks, for example -- is risky, because if there's a wide-spread crisis that affects the banking sector, all components of the portfolio will tank at once, together. This is bad. A way to avoid risks like this is to try and choose components that are as uncorrelated (or even anti-correlated) as possible: that way, if one sector tanks, the entire portfolio isn't brought down. 

The classical way to deal with this is problem is Markowitz mean-variance portfolio optimization: for a given level of risk (say, 12%), find the portfolio that maximizes the expected return, given the historic correlations between the different potential components (treasury bonds, equities, index funds, commodities, etc). Choosing a higher or lower level of risk will result in a different mix of components: generally more on the equities side for the higher risk levels, more in treasuries for the lower risk levels.

Portfolio managers often set constraints on the amount of stocks to be allocated specific sectors (say, 20% in finance equities and 10% in municipal bonds). Given those constraints, the classical mean-variance optimization process can still be used, but the set of solutions is constrained to those portfolios that meet the sector allocations. Nonetheless, the individual assets in those sectors are still considered independently in the optimization process.

A recent paper suggests a better approach might be to minimize not overall risk, but instead the average correlation of the components within each sector. The Systematic Investor blog shows that it's easy to implement a criterion like this in the R language:

portfolio.sigma = sqrt( t(weight) %*% assets.cov %*% weight )
mean( ( weight %*% assets.cov ) / ( assets.sigma * portfolio.sigma ) )

You can then use one of R's nonlinear solvers -- they use Rdonlp2 -- to maximize the equations and return the optimal portfolios for different levels of risk. (The R code to do this is available at github.) Here are their results for standard mean-variance portfolios (at the top), and minimum average correlation portfolios at the bottom:

In each case, read the vertical line above a given level of risk to see how the optimal portfolio is allocated. At the lower risk levels, the average-correlation portfolio includes gold (GLD) and 20-year treasuries (TLT); at higher risk levels emerging markets securities (EEM) get mixed in as well.

For the full details of average-correlation portfolios and their implementation in R, see the blog post at Systematic Investor linked below.

Systematic Investor: The Most Diversified or The Least Correlated Efficient Frontier

Mixed-Effects Models in R with Quantum Forest

For anyone who wants to estimate linear or nonlinear mixed-effects models (aka random-effects models, hierarchical models or multilevel models) using the R language, the Quantum Forest blog has several recent posts that will be of interest. Written by Luis Apiolaza from the School of Forestry at the University of Canterbury in New Zealand, the blog includes a number of illustrated examples of using the open-source lme4 package and the proprietary asreml-r package. 

Recent posts include an overview of linear mixed models in R, and worked examples of analyzing forestry yields using a family model and visualizing correlations (using the lattice and ggplot2 packages) -- and it's well worth browsing the archives for many other interesting posts about mixed-effects modeling. (By the way, if you're looking even more in-depth resources, the 2009 edition of book Mixed-Effects Modeling in S and S-PLUS is, despite the name, the best reference for using the lme4 package in R. Draft chapters from the forthcoming book lme4: Mixed Effects Modeling in R are also available online. If you have ASReml, Dr Apiolaza's asreml-r cookbook is a handy supplement to the standard documentation.)

The Quantum Forest blog also has several posts on more general R-related topics as will, such as how to find the solution to general maximum likelihood problems in R, and how to simulate data with a defined correlation structure. Also, as someone who has switched from using SAS to R for teaching, Dr Apiolaza has some interesting perpectives on SAS versus R. His summary of reasons why he chooses R over SAS for academic and commercial use is a good one, and I reproduce it verbatim here:

• There is good integration between the programming language and the statistical functions. Both SAS macros and IML are poorly integrated with the data step and procs.
• R is highly conducive to exploratory data analysis; visualization functions (either the lattice or the ggplot 2 packages) produce high quality plots that really help developing ideas to build models.
• Statistics is not defined by the software. If someone develops a new methodology or algorithm chances are that there will be an R implementation almost immediately. If I want to test a new idea I can scramble to write some code that connects packages developed by other researchers.
• It is relatively easy to integrate R with other languages, for example Python, to glue a variety of systems.
• asreml-r!
• I can exchange ideas with a huge number of people, because slowly R is becoming the de facto standard for many disciplines that make use of statistics.

Thanks to Dr Apiolaza for sharing such useful information; I've added Quantum Forest blog to my RSS feed and I look forward to more.

Quantum Forest: A shoebox for scribbles on data analysis

"Anyone planning to work with Big Data ought to learn Hadoop and R"

Dan Woods at Forbes interviewed LinkedIn's Daniel Tunkelang about the rise of data science and on building data science teams. When asked how students today should prepare themselves to be data scientists, Tunkelang gives some good advice:

When we built the data science team at LinkedIn a few years ago, we looked for raw talent, assuming that smart people could pick up the needed technical skills on the job. Now that the field has matured, it’s a good idea to learn some of those technical skills in school. Anyone planning to work with big data ought to learn Hadoop and R, the two open-source tools most used by data scientists. It’s also a good idea to take courses in statistics in machine learning. Beyond that, find every opportunity to work with real data sets. Struggling with the warts of real data is a key part of a data scientist’s job — in fact, some would say that the struggle is our “day job.”

(Emphasis mine.) Any student thinking about working with Hadoop and R should check out the RHadoop project, a collection of R packages that make it easy to write map-reduce jobs for Hadoop data stores in the R langauge.

Forbes: LinkedIn's Daniel Tunkelang On "What Is a Data Scientist?"

One week left to enter the $20,000 "Applications of R" contest

The deadline to enter the "Applications of R in Business" contest is just a week away. To qualify for $20,000 in prizes from Revolution Analytics, your entry must be submitted to inside-r.org by midnight PST on October 31. 

Note that this doesn't have to be your final submission: as long as you've entered a draft version, you can still modify your entry during the public review period, which runs through November 30. 

A prize of $10,000 will be awarded to the entry that best showcases an application of R, as selected by our judges. An additional $5,000 prize is available to the best entry using a feature specific to Revolution R (such as big data, Hadoop or web deployment), plus an additional five $1,000 honorable mention prizes.

Here are some resources to help you get started:

• Overview of the contest
• Detailed information on how to make a submission
• A sample entry (to give you an idea of the style judges will be looking for)
• Some public data sets you can use in your entry (any publicly-available data is fine)

If you've ever written a paper about R, written a report using R, or given a presentation about R and you've thought, "Someone at work will think this is cool", then you have a shot at winning some great prizes. Follow the link below to get started -- even submitting a skeleton by October 31 will put you in the running until judging begins on November 30.

inside-R.org: Applications of R in Business Competition

NYT on Big Data and R

In the New York Times' "Bits" blog today, Quentin Hardy offers recollections on Big Data talks at the recent Web 2.0 Summit. He begins with a definition of Big Data:

Big Data is really about ... the benefits we will gain by cleverly sifting through it to find and exploit new patterns and relationships. You see it now in things like Facebook ads, which are put in front of you because the posts you have read and contributed to (which Facebook’s algorithms get to examine as the price of this “free” service) indicate you might be ready to buy the advertised good.

The article includes applications of big data analytics at various companies: ad placement at Google; credit card transaction analysis (according the CEO of TrialPay, the value of the transaction data exceeds the transaction fee credit companies like Visa charge merchants); and inferring information from the semantic web at search start-up Domo. (By the way, here's a great presentation on using R to mine the semantic web, from Chris Davis and Alfredas Chmieliauskas of the Amsterdam R Users Group.)

Speaking of R, the NYT article ended with a mention of new tools for Big Data analyyics: MapReduce, NoSQL and R:

There are an uncountable number of data-mining start-ups in the field: MapReduce and NoSQL for managing the stuff; and the open-source R statistical programming language, for making predictions about what is likely to happen next, based on what has happened before. 

For example, R is used at social networking sites like Foursquare and OMGPOP to make predictions based on user transaction data.

Bits Blog: The Big Business of ‘Big Data’