Revolutions

How to generate correlated random numbers

We've covered how to generate random numbers in R before, but what if you want to go beyond generating one random number at a time? What if you want to generate two, or three or more random numbers, and what's more, you want them to be correlated?

JD Long lays out the way in a couple of posts at his Cerebral Mastication blog. If you want to generate bivariate (or trivariate, or more) random multivariate Normal variates, it's pretty easy, as JD points out. Just use the mvrnorm function from the MASS package in R, specify the covariance matrix, and you're all set.

But what if the random variables need to follow some other marginal distribution than Normal? This is a common task for simulations, where you may know the distributions you want for each variable, but need a way to specify the covariance structure. This is where copulas come in handy, as explained in JD's second post, complete with R code implementing a simulation. 

Cerebral Mastication: Even Simpler Multivariate Correlated Simulations

R is indispensable, because it's reproducible

Maria Wolters, self-styled "Science-Mum of two" and speech and language technology researcher, has a great blog post about the one tool she couldn't live without: R. Maria says R is her "favourite tool for analysing experimental results and modelling the resulting patterns of behaviour and preferences", and explains why:

R is a programming language for everything statistical. It’s free, it’s open source, and it’s being maintained by statisticians for statisticians. Its origin means that it is a pain to learn. It takes a while until one has cleared a path through the data structures, including the various conventions for extracting information from objects that store the results of painstaking statistical analyses, and I am still often baffled myself.

But the payoff is magnificent. Clear (modulo coding ability), open, replicable analyses. R is the ultimate in replicable research. If you give people your data set and your source code, they can repeat every single step of your reasoning. There are no paywalls, no limits of affordability, no packages that are indispensable for the analysis, but that your department hasn’t paid for.

This issue of "replicable analysis" is an important one: the ability to know that you can re-run your analysis at any time in the future (assuming you still have access to the same hardware, or at least a virtual instance of it) and verify the results, without having to worry about the software no longer being available, is crucial. It also means that third parties can reproduce your results where necessary. The fact that it really is necessary to support good science is the topic that Fritz Leisch covered in this excellent keynote speech at this year's UseR! conference.

Speech and Science: The one tool I couldn’t live without

Wanted: R Analysis of New Scientist Covers

Peter Aldhous and Jim Giles -- from New Scientist's San Francisco bureau -- are looking for a statistician and R user to take part in an interesting data analysis challenge, and also be part of a future article in the magazine. They were inspired by this rather tongue-in-cheek presentation where Sebastian Wernicke analyzed videos, transcripts and ratings of TED talks to conclude, for example, that a talk about how "French coffee spreads happiness in your brain" would be the "ultimate TED talk".

As the hook for an upcoming article about analytics and predictions, New Scientist will soon be running a competition along similar lines: can you use information extracted from the covers of New Scientist magazines (headline, subheadline, photograph, main colors, article titles, etc.) to predict newsstand sales? Competitors will be given historical sales data to build models, and can use the extracted cover details (or extract their own details from provided cover images), plus any other publicly-accessible data (weather records for examples), to help boost their predictions. Teams will be ranked on a weekly basis on their ability to generate the best predictions for future newsstand sales in the leadup to the published article.

Several teams with expertise in domains ranging from machine learning to sentiment analysis and neural networks -- and even a pair of trained pigeons! -- have been assembled already, but as yet no-one's using the power of R for the analysis. This is a problem that R is ideally suited for, so if you'd like to give it a shot, and can spend some time over the next few months building novel models that can outperform the other teams, New Scientist would like to hear from you. I've offered to help Peter and Jim find a candidate (or team) for the competition, so send me an email if you'd like to participate with some notes about how you'd tackle the problem. C'mon, you just know you can beat those pigeons.

Incidentally, New Scientist is no stranger to R: previous articles have featured analyses or graphics done in the R environment. A few examples appear after the jump; most of the links require a New Scientist subscription to view.

Because it's Friday: How Machines Work

Ever wondered how a sewing machine seemingly manages to knot stitches without ever releasing the thread? Well, wonder no more:

 

Find this and other animations of marvels of engineering, including the universal velocity joint and the rotary engine, at the link below.

World Of Technology: Complicated Mechanisms Explained in simple animations (via)
 

http://mytechnologyworld9.blogspot.com/2010/08/complicated-mechanisms-explained-in.html

Poll: Half of SAS users considering a switch

A recent poll of KD Nuggets readers suggests that of those using SAS today, almost half (49.6%) are considering switching to a different system for statistical analysis.

The poll was prompted by the recent high court decision in the UK, that affirmed that "WPS is lawful clone of SAS system" (as stated in a WPS press release). The exact poll question was, "Are you a current SAS user and are you considering a switch, in light of a recent WPS court victory vs SAS?". The poll also allowed respondents to indicate one or more solutions they're considering switching to, with the following responses:

• 58.6% to WPS
• 32.8% to R
• 27.6% to other systems

KD Nuggets Polls: Switching from SAS to WPS, R, ...

New R User Groups in Seoul, Denver

We have two new local R user groups to report this week.

In Seoul, South Korea R user Chel Hee Lee is the organizer of the GNU R User's Group and Open Statistics Project in Korea. My translate-fu isn't quite up to figuring out when the next meeting is, but you can contact the group organizers here or here.

In Denver, Colorado computational statistician Ryan Elmore (recently featured here on the blog) has teamed up with several other local R users to form the Denver R User Group. The first meetup on September 14 will feature Ryan talking about using R to analyze baseball data, in addition to planning for future meetups.

R users in Denver and Seoul -- get to it!

Revolutions: Local R User Group Directory

Online Certificate Courses in Computational Finance with R

As announced on the R-Sig-Finance mailing list, the University of Washington is now offering a three-course online certificate in computational finance, using the R programming language as the software base. The three courses are:

• Investment Science
• R Computing for Computational Finance
• Portfolio Construction and Risk Management

The course is presented by Doug Martin (founder of S-PLUS and longtime computational finance expert), Eric Zivot (creator of the S-PLUS module S+FinMetrics and co-author of one of the best computational finance books), and Guy Yollin (one of the lead developers for S+FinMetrics and crack hedge-fund analyst). I worked with all three of these fine fellows for many years, and they really know their stuff. At $3300 per course this on-line certificate might seem pricey, but the calibre of the instructors can't be beat.

University of Washington: Certificate in Computational Finance

Slides and replay for "Big Data with Revolution R"

Thanks to everyone who attended our webinar this morning, Big Data Analysis for R Using Revolution R Enterprise, and in particular thanks for all the thoughtful questions during the Q&A session.

If you missed the live broadcast, a replay is now available (requires the ability to view WMV files), and you can also download the slides in PDF format.

Revolution Analytics: Big Data Analysis for R Using Revolution R Enterprise

R and Analytics: A good career choice

According to Microsoft, the hottest three new tech majors are:

• Data Mining/Machine Learning/AI/Natural Language Processing 
• Business Intelligence/Competitive Intelligence
• Analytics/Statistics – specifically Web Analytics, A/B Testing and statistical analysis

Tim O'Reilly concurs. Of course, R features prominently in most of these areas -- follow the links I added above for examples -- and is growing rapidly, so learning R makes for a smart career choice. 

Microsoft Careers JobsBlog: The Top Three hottest new majors for a career in technology

Webinar: Big Data Analysis with Revolution R

Don't forget that I'll be hosting a webinar tomorrow talking about the new RevoScaleR package included with the forthcoming Revolution R Enterprise 4.0. The webinar will also feature a live demonstration from Joseph Rickert. The full details are below, and you can register for the webinar here.

Big Data Analysis for R Using Revolution R Enterprise
Date: Wednesday, Aug 25, 2010
Time: 9am - 10am Pacific Time

Presenters:
David Smith, Revolution Analytics
Joseph Rickert, Revolution Analytics

The R language is well-established as the modern language for predictive analytics. However, given the deluge of data that must be processed and analyzed today, some organizations have been reluctant to deploy R beyond research into production applications. Additionally, R's in-memory design offers great flexibility, but can be limiting when processing multi-gigabyte or terabyte-class datasets.

Attend this webinar to see how Revolution R Enterprise now extends the reach of R into the realm of 'Big Data' data analysis!

• Unprecedented levels of performance and capacity for statistical analysis of very large data sets in the R environment
• See a demonstration of how Revolution R Enterprise can process, visualize and model this scale of data in a fraction of the time of legacy systems
• Cut out the need for expensive or specialized hardware

Revolution Analytics webinars: Big Data Analysis for R using Revolution R Enterprise