Revolutions

Because it's Friday: Copyright vs creativity

When the USA codified its copyright and patent laws in the 18th century, the goal was to create a robust public domain that all of society could benefit from, while still giving inventors and content creators a window in which to capitalize on their creative investment. Filmmaker Kirby Ferguson concludes his excellent Everything is a Remix series with a look at how over the years the laws have turned to preventing content from entering the public domain and protecting creators indefinitely:

 

Much of the change in the laws is a result of the influence of media companies, usually backed by the claim that copying (legal or otherwise) costs billions of dollars and thousands of jobs. Rob Reid takes a look at the numbers behind those claims in his humorous TED talk, Copyright Math (TM):

See the wind

The image below isn't a bearskin rug in the shape of the USA. It's fact, it's a visualization of the wind flowing over the United States, as of 4PM EDT today, March 30.

You can click through to see the current wind conditions, based on latest data from the National Digital Forecast Database. But more importantly, as long as you have a modern browser like Chrome you'll also see the winds animate. It's quote hypnotic watching the various jetstreams collide and intermingle and interact with geographic features like the Rockies and the Great Lakes. The wind patterns change regularly, as this gallery of past wind maps shows. You can easily create static wind-vector maps in R, but this is one of the somewhat rare cases where adding animation to instantaneous data really benefits the quality of the visualization.

hint.fm: wind map

R 2.15.0 "Easter Beagle" is released

The R core group today announced the availability of R 2.15.0, codenamed "Easter Beagle". If you build R yourself, the new source distribution (including packages for Debian Linux) is available now, and binaries for Windows, MacOS and Linux will be available from your local CRAN mirror over the next couple of days. (As of this writing, Windows binaries are already propagating.)

This release includes many new features, but highlights include:

• Parallelized versions of the mapply and Map functions
• Load-balancing support for parallel programming on clusters 
• Improved performance when serializing R objects to/from disk (which will in turn improve performance of R integrated with systems like Hadoop)
• High-performance functions for calculating row/column statistics on matrix and data frame objects
• Support for simulating random matrixes using the Wishart distribution

R 2.15.0 represents the first release in the R Project's new annual release cycle. R 2.16 is expected about the same time next year, with patch updates in the interim as required. Many thanks go to the R core team for their hard work and dedication continuing to improve and advance the state of the R engine, core packages, and build infrastructure.

R-announce mailing list: R 2.15.0 is released

See how Deloitte uses R for actuarial analysis

Many thanks to Jim Guszcza (Predictive Analytics lead at Deloitte Consulting and Assistant Professor at UW-Madison) who gave a great webinar presentation yesterday on actuarial analysis with R. 

Jim's demo (starting at the 20 minute mark in the video replay below) is a great way to get a sense of how R is used for exploratory data analysis and modeling, with a live examples of fitting a mixute distribution to bimodal claims data, and calculating loss reserves using Poisson regression.

 

Many actuaries use Excel to make calculations like these, but Jim makes a great point point about the benefits of programming with data instead of spreadsheet cells at the 32:00 mark:

Just one simple line of [R] code that would work just as well for a 100-by-100 loss triangle as it would for a 10-by-10 triangle. No hidden cells in the spreadsheet, no risk of spreadsheet error. It's a little bit of code you could look at in one screen, it's replicable ... and this does all the work that a spreadsheet would do. 

Also, check Jim's final case study at the 47-minute mark for a sneak preview of the next version of Revolution R Enterprise -- big-data Generalized Linear Models. He uses the Allstate Claim Prediction Challenge data (from a recent Kaggle competition) to fit a Tweedie model to 13 million records of claim data. (The Tweedie distribution is often used to model insurance claims, where many claims are exactly zero, and non-zero claims follow a continuous Gamma-like distribution.) Using the forthcoming rxGLM function, he fit the model to this large data set in just over two minutes (140.22 seconds) using a single quad-core PC.

You can download the slides from Jim's presentation and an a replay for offline viewing at the link below.

Revolution Analytics Webinars: Actuarial Analytics in R

Bigger data, better intelligence for government

Tomorrow (at 2PM Eastern on Thursday), the White House White House Office of Science and Technology Policy will host a 90-minute forum on Challenges and Opportunities in Big Data. You can watch the event live at the previous link, and see federal government science heads from OSTP, NSF, NIH, DoE, DoD, DARPA and USGS outline how their agencies are engaged in Big Data research. The event will also include a discussion by a panel of thought leaders from academia and industry, moderated by Steve Lohr of the New York Times.

Naturally, the event has prompted some speculation about the government's "big data agenda", but this in fact a direction the administration has been pushing ever since the Obama appointed Aneesh Chopra as the nation's first CTO in 2009. (Chopra passed the mantle to current CTO Todd Park just this this month.) One of the first initatives was the creation of data.gov, a vehicle to make government data available in a timely and accessible format for a community of citizen data scientists to make sense of it all. For example, independent analysis of data from data.gov (using Big Data statistical analysis techniques) revealed a flaw in the 2000 Census results that went unnoticed by the Census Bureau.

And there's the rub: while much of the talk around Big Data has been around storage and search for massive data sets, the real value comes from analysis and inference on the data themselves. I'd expect to hear more from the administration on this topic in today's forum. The administration is already pushing to use more open-source software in government, the open-source R language (along with the big-data enhanced distribution from Revolution Analytics) is already widely used at NIST, FDA, NOAA, CIA, and many other government agencies. Given that the data storage and data analysis infrastructure is already well in place, I expect we'll hear a vision for new kinds of applications, citizen services and data-driven strategies from the participating federal agencies in today's forum.

You can join the forum in the live-streaming webcast at the link below. 

NSF: OSTP and NSF Present: Challenges and Opportunities in Big Data (2PM EST, Thu Mar 29 2012)

Updated Mar 28 to correct the time of the forum.

Students: Help improve R with the Google Summer of Code

The Google Summer of Code project provides funding for students to write and contribute code to open-source projects of interest to Google. And given that R is widely used at Google, it's certainly a project of significant interest. So for the fifth year running the R Project will mentor students as part of GSOC 2012 to develop packages and improvements for R.

Tal Galili of R-bloggers provides a list of potential projects for students to work on, which include developing tools for open science with R, exporting R graphics as interactive applets for the Web, extending the knitR package to generate web-based documents, and much more. If you're a student with R and coding skills, this is a great way to improve the R ecosystem while earning some cash over the summer:

If you are a student looking to write some code for the R community, Google is willing to pay you $5000 for 3 months during this summer of 2012.  A good starting point is the R GSoC wiki. Interested students or mentors are encouraged to discuss other project ideas on the gsoc-r Google group.

For more information about R's involvement in GSOC 2012, see Tal's post below.

R-bloggers: Google summer of code 2012 – and R – a call for students

Take the 2012 Future of Open Source survey

Every year for the past five years, a consortium of software companies has collaborated to conduct a survey on attitudes and adoption of open-source software in business. Now in its sixth year, the 2012 Future of Open Source survey is now open. Open source analyst firm 451 Research is a collaborator for the first time this year, along with Revolution Analytics, Red Hat, Couchbase, Eucalyptus and several other firms.

If you work for a company that is using (or plans to use) open-source software, or for a company that produces or supports open-source software, please take the survey. (It should take around 10 minutes to complete.) The results and conclusions from the survey will be presented at the 2012 OSBC conference in San Francisco (May 21-22), and shared online. (Survey participants will also receive a 20% discount code to attend the conference.) The conclusions from the 2011 survey can be found below.

 

2012 Future of Open Source Survey: Take the survey

Because it's Friday: Please don't write like a scientist

When Adam Rubin was a PhD student, his advisor accused him of "writing like a poet" for using a non-scientific word in this thesis:

The word was “lone,” as in “PvPlm is the lone plasmepsin in the food vacuole of Plasmodium vivax.” It was a filthy word. A non-scientific word. A flowery word, a lyrical word, a word worthy of — ugh — an MFA student.

Rubin goes on to provide 12 tips to follow if you want to avoid accusations of literary or poetic writing in your scientific papers, such as:

Some journals, such as Science, officially eschew the passive voice. Others print only the passive voice. So find a healthy compromise by writing in semi-passive voice.

ACTIVE VOICE: We did this experiment.
PASSIVE VOICE: This experiment was done by us.
SEMI-PASSIVE VOICE: Done by us, this experiment was.

Yes, for the semi-passive voice, you’ll want to emulate Yoda. Yoda, you’ll want to emulate.

... and ...

Most journals use the past tense. To add flair to your writing, try writing your entire article in the Third Conditional Progressive Interrogative tense. Instead of, “We did this experiment,” you’d write, “Would we have been doing this experiment?” This may seem more convoluted than simple writing, but your article probably won’t be any less comprehensible than most other scientific journal articles.

You can read the rest of the tips in his Science Careers article.

But in all seriousness, from someone from outside the academic community who still has to read scientific articles, please don't follow this advice, tongue-in-cheek though it may be. I really don't understand why science (and yes, that include mathematics and statistics) can't be written in a more natural and easy-to-read style. It's not as though a more literary style necessarily comes at the expense of clarity and accuracy.

If you've ever read any of Isaac Asimov's non-fiction popular science books, you'll know what I mean. I devoured his books as a teenager, and despite having no background in astronomy, chemistry, ecomomics, religion or any of the other myriad topics Asimov wrote about. I learned a lot about topics I'd never got to study, and had a good read while doing so. (If you have a teenager yourself, the books still make a great read, even though the science is a bit dated now. Bill Bryson appears to have taken up the eclecticism mantle for the 21st century.) So it is possible to make esoteric topics easy to understand, and without assuming extensive background knowledge on behalf of the reader. Popular science, of course, can't get into the depth of a journal article, but even there Asimov made the effort ... and ran into familiar problems. In his 1979 autobiography he wrote, 

The trouble was that I hated writing research papers … writing a research paper is a tedious and stylized job. You cannot write as you wish; you cannot use English; you cannot have fun...

It really did spoil my fun and erased any pleasure I could have had in the dissertation. Nor did it really help. Once it was done and under consideration, one professor was reported to have said, "It reads like a mystery story." And he didn't say it with approval.

Isaac Asimov did not publish another research paper after 1953.

Science Careers: "How to Write Like a Scientist" (via @peteskomoroch)

R, Twitter and McDonald's

Ed Chen is a data scientist at Twitter, so he's accustomed to working with big data and complex models. In an interview with MIT Technology Review, he describes his data science toolbox:

A common pattern for me is that I'll code a MapReduce job in Scala, do some simple command-line munging on the results, pass the data into Python or R for further analysis, pull from a database to grab some extra fields, and so on, often integrating what I find into some machine learning models in the end.

He put this toolbox to great use in a recent blog post, Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process. After using simulation in Ruby and Python to generate some test data, he used the R language to create a novel classification model which can group "similar" members from a data set, without needing to specify the number of groups in advance. He used this model to categorize the McDonald's member into a number of "food groups" containing products with similar nutritional content.

One cluster contained all the desserts: Baked Hot Apple Pie, Snack Size McFlurry, and others including the three below: 

As you can see, the foods in the "dessert" group cluster together because of high trans fat content, low fiber, and other similar nutritional attributes. Other groups identified by the model include salads, burgers and other fried food, three categories of sauces, and (in a cluster all on its own) Fruit and Maple Oatmeal: the only high-fibre item on the menu.

For an astounding amount of detail about the analysis, visit Ed Chen's blog at the link below.

Edwin Chen's Blog: Infinite Mixture Models with Nonparametric Bayes and the Dirichlet Process

Casino outsmarted at probability

There's a great story in The Atlantic about a man who broke the bank at an Atlantic City casino by winning $6m at Blackjack. He won without counting cards: the practice of remembering previously-played cards that allows the player's expected return to exceed 100%. (Casinos consider this practice cheating, and will ban players whose play reflects card-counting strategy.) Don Johnson's carefully-observed plays were dependent only on the cards played that hand, and so he was allowed to continue his winning streak.

And this was no mere lucky streak: Johnson had also taken home multi-million dollar winnings in other nearby casinos in the last month. He did it by exploiting the odds ... and the desperation of hard-up casinos eaget to compete for the business of (hopefully) lucrative high-rollers. By convincing casino bosses to offer him loss discounts and lenient game rules, he had tuned the tables on the house edge: with enough time, money and skill, he was guaranteed to win. 

For example, at the Trop, he was willing to play with a 20 percent discount after his losses hit $500,000, but only if the casino structured the rules of the game to shave away some of the house advantage. Johnson could calculate exactly how much of an advantage he would gain with each small adjustment in the rules of play. He won’t say what all the adjustments were in the final e-mailed agreement with the Trop, but they included playing with a hand-shuffled six-deck shoe; the right to split and double down on up to four hands at once; and a “soft 17” (the player can draw another card on a hand totaling six plus an ace, counting the ace as either a one or an 11, while the dealer must stand, counting the ace as an 11). When Johnson and the Trop finally agreed, he had whittled the house edge down to one-fourth of 1 percent, by his figuring. In effect, he was playing a 50-50 game against the house, and with the discount, he was risking only 80 cents of every dollar he played. He had to pony up $1 million of his own money to start, but, as he would say later: “You’d never lose the million. If you got to [$500,000 in losses], you would stop and take your 20 percent discount. You’d owe them only $400,000.”

Figuring the odds of a table game is fairly straighforward: in a standard Blackjack, for example, the house expects to rake in a profit of 1% of all bets laid over the long run. The key there is the long run: i.e. over millions of hands played by players of varying skill levels and strategies. When it comes to figuring the expected profit that comes from attracting a skilled high-roller, things get much trickier. Casinos have computer models for to estimate the expected wins (or risk of losses) from high rollers, incorporating not just long-run probability and expected returns, but also aspects of game theory (will he quit or continue if he gets $1,000,000 up? What about $2,000,000 down?) and even physiology (will he play differently when he gets sleepy or hungry?). In this case, with the aggressive incentives and the modified rules, it seems like Johnson had the better model: “I just think somebody missed the math when they did the numbers on it,” he said.

Read the full, fascinating story at The Atlantic, linked below.

The Atlantic: The Man Who Broke Atlantic City