Revolutions

Swine Flu: Epidemiology scores a point over politics

Vice-President Biden got into a bit of trouble today, saying he wouldn't advise his family to travel by plane or by subway to limit the chances of their being infected with swine flu. Some, like the United States Travel Association trade group, weren't happy. But the science may be on Biden's side, says Ezra Klein:

Biden may be right to induce a bit of panic. The United States Travel Association won't think so. It's their profit stream on the line, after all. But epidemiologists are probably quietly relieved by the Vice President's comments.

The danger of swine flu lies in whether or not it becomes a full-blown epidemic, and that in turn depends on the base reproduction rate: the number of secondary infections generated by a primary infection. As shown in this chart (which unless I am very much mistaken was generated with R), anything we can do to reduce the reproduction rate R₀ can dramatically reduce the infection attack rate.

So score one for science.

Ezra Klein: Is Joe Biden right about swine flu?

A better way of choosing colors

I posted already about how you can choose specific colors in R by name, but judging from the some of comments I got I have a lot to learn about color selection. I got several emails recommending the RColorBrewer package as a way of selecting ranges of colors for various purposes that are not only aesthetically pleasing, but also lend themselves to better and fairer interpretations of data. This 2007 post about RColorBrewer at the Statistical Modeling blog is a great introduction to how RColorBrewer works, and the comments are also interesting.

Aleks Jakulin: Assistance in picking colors and charts

LinkedIn's chief scientist talks big data, analytics

Ben Lorica over at O'Reilly Radar interviews LinkedIn's chief scientist DJ Patel. His view: with the wealth of open-source and commodity scalable data analysis platforms now available, we're set to see an explosion in the application of statistical and data mining methods applied to new data sets. Can't say I disagree. Sadly no mention of R, but with Google and Facebook using R can LinkedIn be far behind?

O'Reilly Radar: How big data impacts analytics 

Upcoming R events in San Francisco

A quick personal and uber-local update: I'll be speaking at a couple of events in San Francisco in the next few weeks. On May 13, I'll be speaking at the Bay Area User's Group meeting, talking about distributed computing with R and the new foreach function in ParallelR. (Regulars: please note the earlier start time and change of venue.) 

Also, on June 3 I will be presenting a one-day introductory R training course, "What's all the fuss about R?". If you have any friends or colleagues who need to get started with R, I'd be happy to set them on the right and true path. :)

Both events are in downtown San Francisco. Looking forward to meeting more R users (or the R-curious) in the area. 

UseR! 2009 Program Announced

The program and schedule for the premier R conference UseR! 2009 has been announced, jam-packed with almost 200 talks over three days, plus an extensive poster session. The growing interest and enthusiasm around R is clearly evident here.

I'm happy to report that several of us from REvolution Computing will be in attendance. I'll be talking about Parallel Computing with Iterators, Bryan will look at speeding up the singular-value decomposition algorithm, and Danese will introduce a collaborative documentation system. Also, our friend and collaborator Jay Emerson will talk about analysing large data sets with the bigmemory package.

If you've been on the fence about going to France this year, this tremendous agenda is likely to sway you. Registrations are still open. See you there!

Open-Source Credit Ratings

When I was working with analysts in the financial sector in the late 90's and early 2000's I found an interesting contrast between the credit ratings agencies (S&P, Moody's etc.) and the investment banks in their methods for estimating probability of default in portfolios of loans (using, at the time, S-PLUS). Or rather, a lack of contrast: although the details differed, it seemed to me that they were using fundamentally the same modeling techniques and -- crucially -- the same data sources and histories to make their estimates. Perhaps if the banks had known at the time that the ratings agencies weren't really giving them anything like independent estimates of risk, our problems with credit-default swaps wouldn't have been quite so severe. (Also: I want a pony.)

So maybe it's a good sign that European regulators appear to be moving towards an open-source model for calculating ratings: 

New rules for Credit Ratings Agencies include the following:

Credit rating agencies may not provide advisory services.

They will not be allowed to rate financial instruments if they do not have sufficient quality information to base their ratings on.

They must disclose the models, methodologies and key assumptions on which they base their ratings.

They must differentiate the ratings of more complex products by adding a specific symbol.

(My bold.) More here:

Asymptotix: Is The European Union Legislating The Ratings Process Towards An Open Source Model?

R/Finance 2009 roundup

The first international conference dedicated to the use of R in the finance industry, R/Finance 2009, was a great success. With over 150 attendees (my poor estimation skills notwithstanding), sold-out tutorials, and an outstanding lineup of invited and contributing speakers from around the world, this event really demonstrated the importance of R in the world of financial analysis.

Some of the highlights of the event for me were:

Robert Grossman kicked off the event with style, declaring that "R is mainstream!". Not only are unrelated companies promoting R, but R is now a major application of the fastest-growing meme of 2009: cloud computing. Robert has an excellent demonstration of using R and Hadoop on Amazon's EC2 cloud computing service (although due to technical difficulties couldn't show it live). 

Diethelm Wuertz gave an amazing talk about portfolio optimization in R with Rmetrics. If you haven't taken a look at Rmetrics before, it is the premier open-source software for financial market analysis and financial instrument valuation, and includes over 32 R packages comprising hundreds of functions. I was truly blown away with the capabilities of Rmetrics for portfolio analysis, and it seems to me that with the capabilities Wuertz described, R now represents the state of the art in portfolio optimization technology. (And given that he uses these tools to manage his own fund, they must be really good!)

Guy Yollin (Rotella Capital Mangement) also spoke about portfolio optimization, and in particular demonstrated the flexibility of R by modifying the standard solve.QP function to introduce custom constraints. This flexibility is what makes R so powerful for financial analysis, and Guy showed an example of optimizing a portfolio where all weights must either be zero, or greater than 3%. He also showed applications in Conditional Value at Risk (CVaR), and maximum drawdown optimization. 

My colleague Bryan Lewis (REvolution Computing) introduced the new foreach function from ParallelR 2.0, and turned more than a few heads by using it to dramatically speed up large-scale backtesting of a portfolio trading strategy. He also showed a very neat animation (generated using the spatstat package) 

David Kane and Patrick Burns gave complementary presentations on assessing portfolio performance. David gave a nice overview of the characteristics approach for finding matching portfolios for performance assessment, noting that while it is widely cited in the academic literature it's little-used in practice. As an alternative, he suggests using the MatchIt package (as described in this vignette) to find matching portfolios, and showed how this method outperforms the characteristics method to match a recommended Starmine portfolio. 

In a similar vein (but in his own inimitable style), Patrick Burns focused on generating random portfolios: not just for portfolio assessment, but also for testing trading strategies, evaluating constraints and validating risk models. For assessment, Patrick demonstrated convincingly that random portfolios are superior to using a benchmark or peer groups. To me, an even more compelling application Patrick showed was using random portfolios for evaluating the effect of constraints on the portfolio. Although constraints are supposed to be "insurance against the portfolio doing anything TOO stupid" you may be surprised at what you're paying for this insurance: As Patrick showed using random portfolios, even simple constraints may be sacrificing positive returns without any elimination of lower-tail risks.

Brian Rowe (Merrill Lynch / Bank of America) drew on insights from the world of physics to filter noise in correlation matrices using Random Matrix Theory. His methods are implemented and available in the R package Tawny.

Roger Koenker suggested that quantile regression, currently an under-used technique in finance, has application in the modeling of "bubble-like" phenomena in finance. In addition to allowing parametric models for quantities like Value at Risk defined by quantiles, quantile regression also allows modeling of what he called "pessimistic portfolios", where the probability of "good" events are downplayed while those of "bad" events are amplified. Also: betting on college basketball. The techniques he described are available in the R package quantreg.

Eric Zivot (University of Washington) offered the insights of a recent convert from S-PLUS advising a fund of funds using R for its quantitative analysis. As the author of a book on financial modeling with S-PLUS, it was interesting to hear him say that overall R now has more financial functionality than S-PLUS (while still lacking a few functions that S-PLUS has). It was also interesting to hear his experiences working with an IT department: their reluctance to support R or let it talk directly to the data warehouse (meaning that all data I/O was via Excel).

Wow, that's a lot of highlights for a single conference! If you missed the event, I've heard that the presentations from these and the other excellent speakers will soon be available from the R/Finance 2009 website. [Update Mar 4: presentations now available!] Kudos and thanks go out to the organizing committee for putting together such an amazing event. I'm very proud that REvolution Computing was a sponsor of this groundbreaking conference, and I look forward to the next one. It's already been confirmed that a second R/Finance conference will be held -- watch this space for further details.

[Update Apr 29: Corrected link for David Kane and quote of Patrick Burns.]

Swine Flu: What makes this outbreak different?

What makes this outbreak different from the usual cases of influenza expected at this time of year? The science blog Effect Measures breaks it down and asks, "What did you expect?":

If there is normally so much respiratory disease around, why is this an outbreak or even an epidemic? It's a more difficult question than it appears, and it relates to "what did you expect?" An epidemic is an increase in the number of new cases beyond what you would expect. Four or five cases of human rabies in an area in the US would be an outbreak or even an epidemic. Hundreds of colds or even serious pneumonias in an urban area is normal. It's not an epidemic. What makes the swine flu an outbreak is that it is an infection with a virus we haven't seen before and which we believe may be new. Hence these cases are not what we expect and it is an outbreak. If it turned out that there had been the same number of cases in Mexico but from many different known viruses we might look for another explanation, for example, a change in insurance that changed care seeking behavior so cases were counted that weren't counted before.

I expect epidemiology (an application R excels at) will be at the forefront of the news very soon.

Update 1:50PM: You can now track reported cases of swine flu in Google Maps.

Effect Measures: Swine Flu: What did you expect?

Tweeting R/Finance 2009

I'll be at the Applied Finance with R conference in Chicago today and tomorrow, and I'll post updates on Twitter as it goes on. You can follow me for the updates as revodavid.

Quote of the day

A respected Columbia professor disputes the findings that trailing at halftime makes a team more likely to win. Andrew Gelman's rebuttal is over my head, but seems to boil down to different ways to interpret the same data, and he includes this instant classic of a line: "they do some ridiculous overfitting with a 5th-degree polynomial. Don't even get me started on this sort of thing." Someone had to say it, and he said it.

ESPN (!): Wednesday Bullets