Revolutions

Where should you publish that next paper?

Like many academics, Arthur Charpentier thinks a lot about publishing papers in journals. Specifically, we wondered if there was a way to figure out which journal was the best place to publish his next paper and have it accepted: 

I was wondering if there were clusters of journals, i.e. journals that publish almost the same kind of articles (so that next time one of my paper is rejected by the editor, I just go to for some journal in the same cluster).

By using statistical analysis of the titles of articles that have been published in various journals, he was able to figure out which journals are most alike, and group them together in clusters as displayed in this tree graph:

Check out the full post in the Freakonometrics blog for the details of the analyses and the R code that created them.

Freakonometrics: Think academic journals look the same ? Well, some do...

Google Summer of Code advances R

For the third year running, the Google Summer of Code program has sponsored a number of students working with R, and has again resulted in several new contributions expanding R in various fields. Dirk Eddelbuettel, who coordinated the R-related projects for GSoC in 2010, summarized the results, with details about the new packages now available for R thanks to the students and their mentors:

• Two packages for automated differentiation in R (calculating numeric derivatives of multivariate functions)
• Adding graphical capabilities (based on ggplot2) to the Deducer GUI for R
• An R wrapper for GEOS, adding 2-D and 3-D geometry features to R
• A new package for social relations analyses in R

For more information about these projects, and links to download the new packages, check Dirk's post linked below.

Thinking Inside the Box: R Project and Google Summer of Code: Wrapping up 

Revolution R Enterprise 4.0 free download for academics

The Windows version of our latest enterprise distribution of R, Revolution R Enterprise 4.0, is now being delivered to subscribers and is also available for free download for members of the academic community. Revolution R Enterprise 4.0 is a major update, and includes many new and improved features:

• Based on R 2.11.1, the current release of the R. See the list of new features in R's language engine since 2.9.2 (upon which the previous release of Revolution R Enterprise was based) in the NEWS file on CRAN.
• Includes a new package, RevoScaleR, that provides scalable, fast (multicore), and extensible data analysis  for large datasets with R.
• Includes the doSMP package (previously only available in Revolution R Community) for parallel processing on multi-processor Windows machines.
• Includes several improvements for R programmers in the R Productivity Environment, including: visual indicators for script status (not run, loaded, debugging); new code snippets for RevoScaleR; integration with Subversion for source-code control; load and/or debug multiple scripts at once; easily create solutions from a folder of R scripts.
• Available as Windows 32-bit and 64-bit versions, and 100% compatible with all packages available on CRAN.

If you're at a university anywhere in the world and want to use Revolution R Enterprise for research or teaching, you can download it free of charge. (If you're at a commercial or non-profit business, you'll need to purchase a subscription to get additional features included with Revolution R Enterprise.)

At the moment only Windows versions are available for download; the Red Hat version will be available later in the year. (Unfortunately for Mac users like me, we don't plan to produce a MacOS X version of Revolution R Enterprise until there's more demand from businesses. But we do offer the open-source Revolution R Community distribution on Macs, but it doesn't include any of the proprietary Revolution features.)

Revolution Analytics: Revolution R Enterprise Free Academic Software

A free book on probability and statistics with R

G Jay Kerns has published a 400+ page introductory text on Probability and Statistics. All of the examples and illustrations are done using R (as Jay puts it, "The people at the party are Probability and Statistics; the handshake is R") so if you want to brush up on your probability and learn R at the same time, this might be a good resource. It would also be great for teaching: Jay wrote the book based on an undergraduate course he gave at Youngstown State University. There's also a plug-in for R Commander to access some of the methods via dialogs.

Jay's book is free, in both senses of the word. You can download the PDF for free from Lulu, or purchase a printed copy for just over $30. Jay has also published all of the LaTeX sources if you want to build the book yourself. And if you're already using R, you can read the book with just three commands:

install.packages("IPSUR")
library(IPSUR)
read(IPSUR)

I haven't read the entire book, but glancing through it, it looks like a comprehensive overview of the basics of Statistics: distributions, hypothesis testing, estimation, linear regression, and even touches on resampling and nonparametric methods.

I did want to point out one minor error on page xiii, though. It's "We're not in Kansas any more, Toto", not "We're not in Kansas any more, Dorothy". They'd take away my Rainbow Card if I didn't mention that. 

G Jay Kearns: Introduction to Probability and Statistics using R

MMDS 2010

The 2010 Workshop on Algorithms for Modern Massive Data Sets (MMDS 2010) finished up this past Friday (June 18th) at Stanford. This was an exceptionally well organized conference: four days of mind-stretching talks on algorithm development and the challenges of working with massive data sets approached from almost every conceivable angle. The approximately 100 attendees were a diverse group of computer scientists, developers and quantitative marketing people from industry and academia. The mix of academics and industry types appeared to be pretty well balanced with a number of speakers listing themselves as having both university and company affiliations. This mashup of people, backgrounds and ideas reflected what I think was the overall theme and rationale for the workshop: moving deep theory ideas to practical solutions.

The level of the talks ranged from Peter Norvig’s (head of Google research) opening tutorial on the practical problems of running Google’s data center that is so massive that shark attacks, blasphemy outbreaks and other 6+ sigma events are real worries -- to Konstantin Mischaikow’s (Rutgers university) presentation on a “Combinatorial Framework for Nonlinear Dynamics” for which a good background in algebraic topology would have been helpful. Most of talks were fairly technical, but many of the speakers came across as being very skilled at communicating to non-specialists. (For a short time, I even had the illusion that I understood some algebraic topology!)

Maybe it was the perfect weather, or the beautiful Stanford campus, or the fact that the coffee was always hot, but it seemed to me that people were just in the mood for talking openly about their work.  Too much and too varied a body of material was presented for me, an outsider to this field, to summarize here, but the following is my take on some of the major themes -- the recurring melodies if you will -- that played as background music through a good many of the presentations:

• Many modern data sets of practical interest, those containing social networking information for example, are best described by graphs not flat tables. The need to look at networked data has triggered a shift in perspective were researchers are looking at almost all massive data sets as graphs, and are investigating aspects of network structure that might be useful in developing new algorithms. There are deep theoretical connections between linear algebra and graph theory; however, it is not clear to what extent these connections can be exploited to create practical, efficient algorithms. John Gilbert (UC Santa Barbara), referred to linear algebra as the “middle ware” of scientific computing. So, finding analogous computational tools for graphical data sets or augmenting linear algebraic methods with new techniques could indeed have profound, practical implications.

• The analysis of massive data is tied to the underlying hardware in at least two ways. First there is the matter of capacity. There seemed to be some agreement that data sets are small to medium if they can fit on a laptop, medium to large if they can fit on a single server, and massive if they are so big that they need to be spread among multiple servers. Hence, both the tools and techniques to analyze data will depend on the underlying hardware architecture. So for example, memory bound languages such as MatLab and R, viable choices for small to large data sets, are not be feasible for massive data sets without extensions to blow through the memory constraint problem. Then, there is the matter of parallelism. As several speakers remarked, parallel computing is no longer an option. These days, even low end laptops are designed around multiple cores. It is very likely that the next generation of middleware algorithms will take maximum advantage of parallel computing.

• Growth in the number of records to be processed is not the only contributor to the size and complexity of modern massive data sets. The drive to bring all of the relevant data to bear on a problem is leading statisticians to seek new methods of analyzing heterogeneous data. Susan Holmes (Stanford) presented work that integrated data describing networks of gene interactions, microarray data and image data into a single framework for analysis. Flexible tools for working with multiple data types and perhaps new data types altogether will be necessary to advance this program. For example, David Bader (George Tech College of Computing) described work on a new data type for dealing with streaming graphs with vertices that change over time, while several speakers presented work to develop structures and algorithms to deal with streaming data.

• Computer scientists and statisticians working with massive data sets are pursuing dual strategies for dealing with the challenge of size. On the one hand there is the continuing search for algorithms that scale in less than exponential time. On the other, there is the strategy of reducing the size of the problem either through sampling or dimensionality reduction techniques. A favorite trick of algorithm developers is to approximate a large matrix M, with a smaller matrix M’ that is “close” to M according to some relevant norm. However, as Petros Drineas (Rensselaer Polytechnic Institute) pointed out in his tutorial on randomized algorithms, clever algebra won’t always work if the resulting structures do not have straightforward interpretations. In describing work he did with SNPs, Petros pointed out that the geneticists he was working with were not happy with linear combinations of the SNPs that contained the maximum amount of relevant information; they wanted to know the individual SNPs that were most important

The papers presented at the workshop should be posted at the link above in the near future. I highly recommend them not only as a snapshot of the state of the science, but also as a way for non-specialists to learn something about the field.

Revolution R Enterprise now free to academics

Unlike Revolution R Community which is 100% free to everyone, our commercial-grade Revolution R Enterprise distribution bundles R with proprietary components from our development team, which are normally available only to paying subscribers. (Those subscriptions are the way we get income to keep the company going.) Those components include our full ParallelR libraries for parallel programming, enhanced for 64-bit Windows, and our R Productivity Environment with full code editing and visual debugging on Windows ... and there's more to come soon in our roadmap. 

Today, we're making the full Revolution R Enterprise distribution available to anyone in the academic community, free of charge. You can read all the details in this page about the free academic download program. All you need to do is fill out a form attesting that you're a professor or a student at an accredited degree-granting institution, and you can download the Windows 32-bit, Windows 64-bit or RHEL 5 version, free of charge.

The only restrictions are that it's for use on a single-user workstation, and you won't have access to live technical support. (For universities/schools that full support on a server or cluster, subscriptions are still available.)

Since there's so much overlap between the R community and the academic community, this is just one more way for us to give back to the community. We want to make sure everyone in academia gets access to both R and all the extensions we've made to it, and will make in the future.

Revolution Analytics: Free Single-User Subscription to Revolution R Enterprise for the Academic Community

Seven reasons to use R

Biological anthropologist James Holland Jones is asked a simple question by his professor colleagues at Stanford: Why use R? Of course, it's free, but are there other reasons to choose R for research and teaching over the commercial alternatives. In his essay, Jones not only offers a defense of free but also offers seven other excellent reasons to be using R:

• R is used by the majority of academic statisticians; 
• R is platform-independent;
• R has unrivaled help resources; 
• R makes the best graphics; 
• The command-line interface is better for teaching;  
• R is a full programming language, not just a statistics application; and 
• The online distribution system beats anything out there. 

See the link below for detailed explanations of these points. But his final point needs no further explanation: "Let's face it: all the cool kids use it."

Monkey's Uncle: Why Use R?