Revolutions

On the Road in July

Starting today and for most of the rest of this month, I'll be on the road to attend conferences and talk about R and REvolution Computing. This week, I'll be in Rennes for User! 2009 (talking about foreach); next week, in Copenhagen for DSC (talking about a debugging IDE for R); the following week in San Jose for OSCON (evangelizing R to the broader open-source community), and then finally in Seattle for BioC 2009 (for a practical session using foreach). 

As a result, expect a lighter posting rate than usual here on Revolutions in the month of July. However, with the blessings of the gods of internet connectivity, I do plan to bring you news and reports from the road -- keep an eye out for posts at odd hours. And if you see me or any of the other REvolutionaries at any of these events, come and say hi!

Temporal illusions

I love optical illusions, and how a well-designed illusion can provide real insights into how sight works. Not just in terms of light and eyes and lenses but also mentally, as our brain assembles a representation of the world from imperfect information delivered to it by the eyes and other senses. (Here's a great 60-second video that demonstrates how the brain adapts to missing information from the blind spot.) Our personal sense of time can be fooled, too, and these "temporal illusions" give similar insights into how the brain assembles events into our perception of time. David M Eagleman, director of Baylor College of Medicine's Laboratory for Perception and Action writes in the essay Brain Time:

The days of thinking of time as a river—evenly flowing, always advancing—are over. Time perception, just like vision, is a construction of the brain and is shockingly easy to manipulate experimentally. We all know about optical illusions, in which things appear different from how they really are; less well known is the world of temporal illusions. When you begin to look for temporal illusions, they appear everywhere. In the movie theater, you perceive a series of static images as a smoothly flowing scene. Or perhaps you've noticed when glancing at a clock that the second hand sometimes appears to take longer than normal to move to its next position—as though the clock were momentarily frozen.

Try this exercise: Put this book down and go look in a mirror. Now move your eyes back and forth, so that you're looking at your left eye, then at your right eye, then at your left eye again. When your eyes shift from one position to the other, they take time to move and land on the other location. But here's the kicker: you never see your eyes move. What is happening to the time gaps during which your eyes are moving? Why do you feel as though there is no break in time while you're changing your eye position? (Remember that it's easy to detect someone else's eyes moving, so the answer cannot be that eye movements are too fast to see.)

Try it, it really works. Your eyes do move, but your brain never perceives it. The essay talks about other experiments investigating these split seconds in time, including the invention of a "perceptual chronometer" to measure how we perceive time, and dropping students off a fifteen-story building for science:

To accomplish this, we engineered a device (the perceptual chronometer) that alternated randomized digital numbers and their negative images at adjustable rates. Using this, we measured participants' threshold frequencies under normal, relaxed circumstances. Next, we harnessed participants to a platform that was then winched fifteen stories above the ground. The perceptual chronometer, strapped to the participant's forearm like a wristwatch, displayed random numbers and their negative images alternating just a bit faster than the participant's determined threshold. Participants were released and experienced free fall for three seconds before landing (safely!) in a net. During the fall, they attempted to read the digits. If higher temporal resolution were experienced during the free fall, the alternation rate should appear slowed, allowing for the accurate reporting of numbers that would otherwise be unreadable.

The experiment failed to observe that the subjects had greater perceptual capabilities in their sudden state of terror, but it's still the coolest experiment I've heard about in a while. There are many other interesting insights in the article -- including the claim that tall people live further in the past than short people -- and even though it has nothing to do with R or statistics, it's well worth a read.

Edge: Brain Time by David M. Eagleman

Because it's Friday: Ferrofluid

Round out your Friday with a relaxing look at a bizarre substance in action: Ferrofluid.

Basically a suspension of tiny iron particles in oil, ferrofluids produce strange and  beautiful arrangements when subjected to magnetic fields. If you want to learn more about these and other strange materials (and don't mind some not-suitable-for-work language), check out the amusing and informative "7 Man-Made Substances that Laugh in the Face of Physics". 

More R at Google

Reader HW noticed this graphic (created by Gmail developers at Google) today on the Google Code blog, with the comment: "Post doesn't mention anything about R, but that's clearly an R boxplot!"

Thanks for the tip, HW!

Package Update Roundup: Jun 2009

This is a list of new or updated packages that were released for R in June, as announced on the r-packages mailing list. To include other updates on this list, please email David Smith. For a complete list of all updates on CRAN, see the CRANberries archive for June 2009. 

deSolve, a package to solve initial value problems of differential equations, has been updated with improved functionality, performance, and documentation.

metafor, a new package for meta-analysis, for has been released.

JM, a package for joint modelling of longitudinal and time-to-event data, has been updated with support for additional models and diagnostics.

plyr, a fast, memory-efficient, and consistent set of tools for processing subsegments of massive data structures, has been updated with bug fixes and improved functionality.

rimage, an image processing module for R, has been updated to work with the latest versions of R. Until a maintainer can be found, this packages is orphaned.

RHIPE, an integration between R and Hadoop for implementing map-reduce algorithms, has been updated. Red Hat 8 AMIs are available to test this functionality on Amazon's EC2 service.

RImageJ, a new package that provides the image processing and analysis tools of ImageJ to R, has been released.

RODBC, a package linking R to databases on Windows and other platforms supporting ODBC, has been updated with bug fixes and improved documentation.

WriteXLS, a package that creates Excel spreadsheets from data frames, has been updated to fix the order in which data frames appear in the spreadsheet and to support encoding schemes.

In case you missed it: June roundup

In case you missed them, here are some articles from last month of particular interest to R users.

This post announced the release of the "foreach" and "iterators" packages on CRAN, for simple scalable parallel programming in R.

This post linked to Thomas Levin's Joy-Division-esque band T-shirt made with R

This post discussed R's involvement in the Netflix Prize

This post revealed that the New York Times uses R to create graphics, including an interactive chart of Michael Jackson's musical career

This post described how to run batch jobs using R CMD BATCH

This post reviewed "Data Mashups in R", an extended example of corralling foreclosure data from various sources

This post announced a new "statistical learning web service" implemented with R

This post was one of a series of posts looking at fraud in the Iranian election (including a now-famous analysis done in R)

This post linked to a widely-reproduced R graph on gay marriage support

This post linked to free source code of many R graphs on Wikimedia Commons

This post demonstrated animation of R graphics using Flash

This post linked to a comparison of various statistical analyses done in R, SAS and SPSS

This post showed how Twitter users can "tweet" using R code

This post linked to an analysis of basketball plays done in R

This post linked to a video walkthrough and code for bagged decision trees in R

This post gave some tips, tricks and pitfalls on working with dates and time zones in R

This post brought the news that R was used in a winning entry of KDD 2009

Other non-R-specific stories in June covered Simpson's Paradox in polling data, Google Squared, real random numbers, the chances of a meteor bringing down an aircraft, amusing article titles from PubMed, and open-source user interfaces.

June was a record-breaking traffic month for the blog. The "Data Mashups" article and the discussion of the Iranian Election were highly visited. Thanks to everyone who provided comments and tips and
please keep them coming to david@revolution-computing.com.

Simple, scalable parallel computing in R

REvolution Computing has just released three new packages for R to CRAN (under the open-source Apache 2.0 license): foreach, iterators, and doMC. Together, they provide a simple, scalable parallel computing framework for R that lets you take advantage of your multicore or multiprocessor workstation to program loops that run faster than traditional loops in R.

The three packages build on each other to implement a new loop construct for R -- foreach -- where iterations may run in parallel on separate cores or processors.

iterators implements the iterator object familiar to users of languages like Java, C# and Python to make it easy to program useful sequences - from all the prime numbers to the columns of a matrix or the rows of an external database. Iterators objects are used as the index variable for the parallel loops.

 

foreach builds on the "iterators" package to introduce a new way of programming loops in R. Unlike the traditional for loop, a foreach loop can run multiple iterations simultaneously, in parallel. If you only have a single-processor machine, foreach runs iterations sequentially. But with a multiprocessor workstation and a connection to a parallel programming backend, multiple iterations of the loop will run in parallel. This means that without any changes to your code, the loop will run faster (and with a speedup scaled by the number of available cores or processors, potentially much faster).

 

doMC provides such a link between "foreach" and a parallel programming backend -- in this case, the multicore package (from Simon Urbanek). With this connection, foreach loops on MacOS and Unix/Linux systems will make use of all the available cores/processors on the local workstation to run iterations in parallel.

The "iterators" and "foreach" packages were previously available only in REvolution R Enterprise as part of the ParallelR suite: now, we've made versions available on CRAN for all R users. 

In REvolution R Enterprise, "iterators" and "foreach" connect with our NetworkSpaces parallel backend (which comes pre-installed and pre-configured in our distribution). Not only does this additionally enable parallel programming on Windows 32-bit and Windows 64-bit systems, it also allows iterations of foreach loops to run on separate machines on a cluster, or in a cloud environment like Amazon EC2. The cluster can even be a mixed collection of Windows, MacOS and/or Unix/Linux machines. REvolution R Enterprise with NetworkSpaces also provides some other benefits, including fault tolerance (so a parallel computation will complete correctly even if some nodes in the cluster fail), and high-performance mathematics libraries to make the computations run even faster.

You can see an example of foreach in action in this financial backtesting example. That example will run as written in REvolution R Enterprise, and in the latest version of R by installing the "multicore" package and replacing the call to registerDoNWS() with registerDoMC(). You'll need the relevant packages installed, which you can do easily in R as follows:

install.packages("foreach")
install.packages("doMC")

I'll be posting more examples using the "foreach" and "iterators" packages in the coming weeks. I'll also be talking about them at UseR! later this month. You can also read more in today's press release. In the meantime, feedback and comments are always welcome to packages@revolution-computing.com.