Revolutions

Rewriting SAS Programs for Financial Data Manipulation in R

by Lixun Zhang, Data Scientist at Microsoft

For financial institutions currently using SAS, R is an alternative statistical software that is free and has been widedly used in academia and industry. Technical support for R is included with Microsoft R Server, which in addition to being 100% compatible with R, also has improved speed and the capability to work with large datasets.

As corporations switch from SAS to R, they might need to rewrite some of their legacy SAS programs in the new language. The good news is that for most of SAS data manipulation steps, R has a straightforward corresponding function. For example, merging can be easily done with R’s merge function. As another example, R’s colSums function, just like the SAS 'sum' function, calculates the sum of two or more variables while allowing missing values to be omitted.

The purpose of this write-up is to help programmers rewriting SAS programs in R understand an important concept in SAS: the “retain” statement, which is used to remember information from one row to the next. This article assumes some familiarity with R and basic understanding of SAS. As a reminder, SAS processes data row by row: it finishes processing the current row then processes the next row. This is different from R where operations are column-based.

Creating Data

The dataset shown below will be used throughout this article. In this dataset there are 2 credit cards, each having 3 rows corresponding to 3 months’ expense information.

Sample data

We can load the data with the code below, with the SAS code first, followed by the R code.

Now, let's look at some examples of working with this data in SAS and R.

Reshaping data

In this first example we’ll look at how the SAS “retain” statement works by examining the code below.

This diagram explains how the sample dataset’s first 3 rows are processed by the above SAS code.

Row 1: Code Block A tells SAS to create 3 variables Expense1, Expense2, and Expense3 (abbreviated as Expense1 – Expense3) because this is the 1^st row (n = 1 is TRUE). By default, SAS assigns missing values (“.”) to these new variables. In addition it creates an array named “Expensex” for the 3 variables. Code Block B uses the “retain” statement and tells SAS to keep the value of 3 variables Expense1, Expense2, and Expense3 from the previous row. Since this is the 1^st row this code block does not affect the values of the 3 variables. Code Block C assigns value “.” to these three variables because CardID 1 appears for the first time (first.CardID is TRUE). Then Code Block D assigns Expense3 = 100 because the variable month has value 3 and Expense is 100. Code Block E does not output the current row because this row is not the last occurrence of CardID 1 (last.CardID is FALSE).

Row 2: Because of the statement “retain Expense1 – Expense3” in Code Block B, the values of the variables Expense1 – Expense3 from the previous row are carried over. So Expense3 has value 100 and Expense1 and Expense2 have missing values. Code Block C does not affect this row because this is not the first time CardID 1 appears. Code Block D then assigns Expense2 = 50 because the value of month is 2 and the value of Expense is 50. Code Block E does not output the current row because this is not the last occurrence of CardID.

Row 3: Code Block B copies the values of Expense1 – Expense3 from Row 2. Code Block C does not affect this row because this is not the first time CardID 1 appears. Then Code Block D assigns Expense1 = 70 because the value of month is 1 and the value of Expense is 70. Because this is the last occurrence of CardID 1, Code Block E exports this row to the dataset bank_mod.

Then the code starts processing Row 4 for CardID 2 and repeat the same process as for CardID 1. It’s worth noting that Code Block C assigns values “.” to the three expense variables at Row 4 because this is the first time CardID 2 appears.

Here's how the data look before and after the transformation (redundant variables such as month and Expense have been removed by the “drop” option in the SAS code):

R users will recognize that this transformation can be accomplished with the reshape function, as the code below shows.

If we had not used the “retain” statement in this example, SAS would not have carried over the values for Expense1 – Expense3 from one row to the next. This means, for example, that when we export Row 3, Expense1 would have value 70 whereas Expense2 and Expense3 would have missing values.

Sum by ID

The “retain” statement can be used for other purposes. In the following example, we are trying to calculate the total expense per CardID. Here the retain variable “total_expense” is first copied from the previous row and then used in calculating a new value through the line:

total_expense = total_expense + Expense ;

We can do these calculations with the aggregate function in R. The diagram below describes how the data look like before and after the transformation and is followed by the SAS and R programs for this task. Only variables 'CardID' and 'total_expense' are kept through the “keep” option in the SAS code.

Count by ID

The “retain” statement can also be used for counting purposes. For example, we may want to count the number of times that the expense was higher than 40. The diagram below describes how the data look like before and after the transformation, followed by the SAS and R programs for this task. Only variables 'CardID' and 'over_spend' are kept through the “keep” option in the SAS code. This is similar to the example for calculating sums by ID except that here the variable has values 0 or 1.

Conclusion

In this write-up we have shown that it’s possible to make the conversion for SAS “retain” statement even though R does not provide a 1-to-1 match function. In such scenarios, the best approach is to understand what SAS is doing and then find the R function that does the same thing.

Some Impressions from R Finance 2016

by Joseph Rickert

R / Finance 2016 lived up to expectations and provided the quality networking and learning experience that longtime participants have come to value.  Eight years is a long time for a conference to keep its sparkle and pizzazz.  But, the conference organizers and the UIC have managed to create a vibe that keeps people coming back. The fact that invited keynote speakers (e.g. Bernhard Pfaff 2012, Sanjiv Das 2013, and Robert McDonald 2014) regularly submit papers in subsequent years is a testimony to the quality and networking importance of the event. My guess is that the single track format, quality presentations, intense compact schedule and pleasant venues comprise a winning formula.

Since I have recently written about the content of this year's conference in preparation for the event, and since most of the presentations are already online for you to examine directly I'll just present a few personal highlights here. 

My favorite single visual from the conference is Bryan Lewis' depiction of corporate "Big Data" architectures as a manifestation of the impulse for completeness, control and dominance that once drove Soviet style central planning. (If you don't read Russian, run google translate on the text in the first panel.)

In his presentation, R in practice, at scale, Bryan presents a lightweight, R-centric architecture built around Redis that is adequate fro many "big data" tasks.

Matt Dziubinski's talk on Getting the Most out of Rcpp, High-Performance C++ in Practice, is probably not a talk I would have elected to attend in a multi-track conference, and I would have missed seeing a virtuoso performance. Matt got through over 120 of his 153 prepared slides in a single, lucid stream of clear (but loud) explanations in only 20 minutes. Never stopping to pause, he gave a mini-course in computer science performance evaluation (both hardware and software aspects) that addressed the Why, What and How of it all.

Ryan Hafen's presentation, Interactively Exploring Financial Trades in R, showed how to use a tool chain built around Tessera and the NxCore R package to perform exploratory data analysis on a large NxCore data set containing approximately 1.25 billion records of 47 variables without leaving the R environment. The following slide provides an example of the kinds of insights that are possible.

 

In his presentation, Quantitative Analysis of Dual Moving Average Indicators in Automated Trading, Douglas Service showed how to use stochastic differential equations and the Itô calculus to derive a closed form solution for expected Log returns under the Luxor trading strategy and a baseline set of simplifying assumptions. If you like seeing the Math you will be pleased to see that Doug provides all of the details.

Michael Kane (glmnetlib: A Low-Level Library for Regularized Regression) discussed the motivations for continuing to improve linear models and showed the progress he is making on re-implementing glmnet which, although very efficient, does not support arbitrary link family combinations or out of memory calculations and is written in the obscure Mortran flavor of Fortran. Kane's goal with his new package (renamed pirls: Penalized, Iteratively Reweighted Least Squares Regression) is to rectify these deficiencies while producing something fast enough to use. 

In his presentation Community Finance Teaching Resources with R/Shiny, Matt Brigida showed off some opensource resources for teaching quantitative finance that are based on the new paradigm of GitHub as the place for tech savvy people to hang out and Shiny as the teaching / presentation tool for making calculations come alive. Check out some of Matt's 5 minute mini lessons. Here is an example from his What is Risk module:

There is much more than I have presented here on the R / Finance conference site. If you are interested in deep Finance and not just the tools I have highlighted, be sure to check out the presentations by Sanjiv Das, Bernhard Pfaff, Matthew Dixon and others. There is plenty of useful R code to be mined in these presentations too.

I would be remiss without mentioning Patrick Burns' keynote presentation which was highly entertaining, novel and thought provoking on many levels: everything a keynote should be. Pat launched his talk by referring to the Sapir-Worf hypothesis which posits that language controls how we think and assigned a similar role to model building. He went on to describe his Agent inspired R simulation model and showed how he calibrated this model to provide a useful tool for investigating ideas such as risk parity, variance targeting and strategies for taxing market pollution. The code for Pat's model is available here, but since his slides are not up on the conference site, and I was apparently too mesmerized to take useful notes, we will have to wait for Pat to post more on his website. (Pat's slides should be available soon.)

Finally, I would like to note that Doug Service and Sanjiv Das won the best paper prizes. This is the second year in a row for Sanjiv to win an R / Finance award. Congratulations to both Doug and Sanjiv!

Get ready for R/Finance 2016

by Joseph Rickert

R/Finance 2016 is less than a month away and, as always, I am very much looking forward to it. In past years, I have elaborated on what puts it among my favorite conferences even though I am not a finance guy. R/Finance is small, single track and intense with almost no fluff. And scattered among the esoterica of finance and trading there has, so far, always been a rich mix of mathematics, time series applications, R programming, stimulating conversation and attitude. When it comes down to it, it’s the people, the organizers and participants who make a conference. Looking over the agenda for this year, I am sure that once again, for two days at least, Chicago will be the center of the R world.

This year, however, I am going to be ready for R/Finance. I am going to do my homework. If I had done a little prep last year I would have had a copy of Arthur Koestler’s The Sleepwalkers in my bag. So when Emanuel Derman went deep philosophy I could have gone through that looking-glass with him.

So what’s on the line up this year? Rishi Narang will lead off for the keynote speakers with a talk provocatively entitled “Rage Against the Machine Learning”. There is not much online for and industry outsider to latch onto, but it probably wouldn’t hurt to have a look at one of his three books on quantitative trading.

Tarek Eldin will deliver the second keynote entitled ‘Random Pricing Errors and Systematic Returns: The Flaw in Fundamental Prices” My guess is that this online paper might provide some relevant preparatory reading.

Frank Diebold’s keynote is entitled “Estimating Global Bank Network Connectedness”. I think it’s a safe bet that his recent paper with Mert Demirer, Laura Liu and Kami Yilmaz will indeed be relevant.

Batting cleanup for the keynote speakers will be none other than the R Inferno himself, who vaguely and possibly misleadingly suggests that preparation for his talk, “Some Linguistics of Quantitative Finance” might begin with Yucatan.

Satellite image of Península de Yucatán, México. Credit: Nasa

For preparation on more solid ground, I am going to look into the R packages explicitly called out in the agenda. Of course, there will be Rcpp. Chicago is Eddelbuettel country and no doubt much of the conversation over coffee will revolve around high performance computing. But, even R users who are not particularly interested in writing high performance code themselves ought to know something about this package. With a reverse table listing hundreds of packages it is becoming the foundation for much of R.

In addition to Dirk’s tutorial on Rcpp and RcppArmadillo, Matt Dziubinski will talk about getting the most out of Rcpp in practice and Jason foster will talk about using RcppParallel for multi-asset principal component regression. Look here for some older talks by Matt.

Robert McDonald will describe the derivmkts package which contains functions that support his book Derivatives Markets.

Eran Raviv will talk about combining multiple forecasts using R’s  ForecastCombinations package.

Kjell Konis will describe how to compare Fitted Factor Models with his fit.models package.

Steven Pav will speak of madness, package for multivariate automatic differentiation. There is a very nice vignette that describes the mathematics of madness.  

Qiang Kou will talk about deep learning in R using the MxNet package which makes use of GPUs.

Mario Annau will talk about the h5 package, an S4 interface to the HDF5 storage format.

Robert Krzyzanowski will describe the Syberia development framework for R.

Dirk Eddelbuettel will revisit the Rblapi package for connecting R to Bloomberg.

Michael Kane will talk about a new package he is writing glmnetlib which is intended to be a low-level library for Regularized Regression.

Matt Brigida use a Shiny implementation to talk about Community Finance Teaching Resources.

When I registered for the conference I saw that the preconference tutorial by Harte and Weylandt on modern Bayesian tools for time series analysis is going to use STAN. So, I need to add rstan to the list.

In his tutorial on leveraging the Azure cloud from R, Doug Service will show how to use the foreach package in the Azure environment.

And then, for some serious preparation it might be helpful to take a look at the math underlying some of the presentations. For example, Klaus Spanderen will talk about calibrating Heston Local Stochastic Volatility Models. Sida Yang will discuss using Latent Dirichlet Allocation to discover distributions underlying financial news topics and Pedro Alexander will discuss portfolio selection with support vector regression.

All that I have listed won't even cover half of what will be presented at the conference, however, I hope some of it will be helpful in preparing for R/Finance. But, most importantly, don’t forget to register! Unfortunately, this year many, if not most, of the people who would like to go to the useR! conference will not be able to attend. Don’t get locked out of R/Finance too!

Reading Financial Time Series Data with R

by Joseph Rickert

In a recent post focused on plotting time series with the new dygraphs package, I did not show how easy it is to read financial data into R. However, in a thoughtful comment to the post, Achim Zeileis pointed out a number of features built into the basic R time series packages that everyone ought to know. In this post, I will just elaborate a little on what Achim sketched out. First off, I began the previous post with url strings that point to stock data for IBM and LinkedIn. Yahoo Finance make this sort of thing easy. A quick search for a stock will bring you to a page with historical stock prices. Then, its is only a matter of copying the url to the associated csv file, and as Achim points out:

If you already have the download URLs ibm_url and lnkd_url, then you can also simply use zoo::read.zoo() and merge the resulting closing prices:

library("zoo")
z <- merge(
  ibm = read.zoo(ibm_url, header = TRUE, sep = ",")$Close,
  lnkd = read.zoo(lnkd_url, header = TRUE, sep = ",")$Close
)

And the ggplot2 figure can just be drawn with the autoplot() method for zoo series:

library("ggplot2")
autoplot(z, facets = NULL)

The resulting plot takes you most of the way to the ggplot produced in the post.

The final formatting can be accomplished with the additional ggplot commands used in my post. This is just delightful: 3 lines of code that fetch the data and prepare it for plotting, and half-line to get a sophisticated default plot.

Of course, the manual step of hunting for urls is completely unnecessary. The get.historic.quote function in the tseries package will fetch a time series object for you that can also be plotted with the zoo autoplot function.

ibm2 <- get.hist.quote("IBM", quote="Close", provider="yahoo",
            start="2010-08-24", end = "2015-09-02", retclass="zoo")

If you are really interested in working like a quant then the defaults in the quantmod package will give you the look and feel of a traders screen. The function:

getSymbols("IBM",src="google")

will bring the "xts" / "zoo" time series object, IBM, containing historic IBM stock data directly into your work space with no need even to make an assignment!

head(IBM)
           IBM.Open IBM.High IBM.Low IBM.Close IBM.Volume
2007-01-03    97.18    98.40   96.26     97.27    9199500
2007-01-04    97.25    98.79   96.88     98.31   10557200
2007-01-05    97.60    97.95   96.91     97.42    7222900
2007-01-08    98.50    99.50   98.35     98.90   10340100
2007-01-09    99.08   100.33   99.07    100.07   11108900
2007-01-10    98.50    99.05   97.93     98.89    8744900

Moreover, IBM is an "OHLC" object that, with the right plotting function like chartSeries from the quantmod package, will produce the kind of open-high-low-close charts favored by stock analysts for charting financial instruments. (There is even an R function to determine if you have an OHLC object.)

is.OHLC(IBM)
#[1] TRUE
chartSeries(IBM,type="candle",subset='2010-08-24::2015-09-02')

The getSymbols function will fetch data from the Yahoo, Google, FRED and oanda financial services sites and also read as well as reading from MySQL data bases and .csv and RData files.

Quandl, however, is probably the best place to go for free (and premium) financial data. Once you signup for an account with Quandl the following code will get data frame with several columns of IBM stock information.

token <- "your_token_string"
Quandl.auth(token)	# Authenticate your token
ibmQ = Quandl("WIKI/IBM", start_date="2010-08-24", end_date="2015-09-03")
head(ibmQ)
Date   Open   High    Low  Close  Volume Ex-Dividend Split Ratio Adj. Open Adj. High Adj. Low Adj. Close Adj. Volume
1 2015-09-02 144.92 145.08 143.18 145.05 4243473           0           1    144.92    145.08   143.18     145.05     4243473
2 2015-09-01 144.84 144.98 141.85 142.68 5258877           0           1    144.84    144.98   141.85     142.68     5258877
3 2015-08-31 147.26 148.40 146.26 147.89 4093078           0           1    147.26    148.40   146.26     147.89     4093078
4 2015-08-28 147.75 148.20 147.18 147.98 4058832           0           1    147.75    148.20   147.18     147.98     4058832
5 2015-08-27 148.63 148.97 145.66 148.54 4762003           0           1    148.63    148.97   145.66     148.54     4762003
6 2015-08-26 144.09 146.98 142.14 146.70 6186742           0           1    144.09    146.98   142.14     146.70     6186742

Here, I have presented just some of the very basics, still not coming close to describing all that R offers for acquiring and manipulating financial time series information.

As a final note: a couple of years ago I posted a short tutorial for getting started with the Quandl R API. Unfortunately, since that time Quandl has changed it's coding scheme so my R code from that tutorial will not run without changes. The code in the file Quandl_code, however, produces the following plot of Asian Currency exchange rates and may serve as an updated example.

Look here to decipher the Quandl currency codes.

 

Plotting Time Series in R using Yahoo Finance data

by Joseph Rickert

I recently rediscovered the Timely Portfolio post on R Financial Time Series Plotting. If you are not familiar with this gem, it is well-worth the time to stop and have a look at it now. Not only does it contain some useful examples of time series plots mixing different combinations of time series packages (ts, zoo, xts) with multiple plotting systems (base R, lattice, etc.) but it provides an instructive, historical perspective that illustrates the non linear nature of progress in software development: new code is written to solve certain technical problems with the current software. Progress is made, and the new code makes it possible to do some things that couldn't be done before, but there were tradeoffs. Design choices for the new system make it a little more difficult to do something that was easy before. The net result: all of the software continues to advance in a messy mix, confusing the newcomer and providing critics with the opportunity to complain that there is not just one way to solve a problem.

Because this is turning out to be a week when more than a few people are likely lo be plotting financial time series, I thought I would be helpful to call attention to this time series resource and also take a look at the current state of the R art for performing a relatively simple task: plotting closing prices for two stocks on the same chart.

The following code just reads stock price data from Yahoo Finance for both IBM and LinkedIn from 8/24/2010 through 8/24/2015 and picks out the closing prices. I cheated a little here because I already knew the urls for the two series. I picked these two stocks because they both traded at about the same range for the period in question and because I wanted to see if the fact that one stock, LinkedIn, wasn't trading when at the beginning of the selected period caused any problems.

# Time Series Plotting
library(ggplot2)
library(xts)
library(dygraphs)
 
# Get IBM and Linkedin stock data from Yahoo Finance
ibm_url <- "http://real-chart.finance.yahoo.com/table.csv?s=IBM&a=07&b=24&c=2010&d=07&e=24&f=2015&g=d&ignore=.csv"
lnkd_url <- "http://real-chart.finance.yahoo.com/table.csv?s=LNKD&a=07&b=24&c=2010&d=07&e=24&f=2015&g=d&ignore=.csv"
 
yahoo.read <- function(url){
   dat <- read.table(url,header=TRUE,sep=",")
   df <- dat[,c(1,5)]
   df$Date <- as.Date(as.character(df$Date))
   return(df)}
 
ibm  <- yahoo.read(ibm_url)
lnkd <- yahoo.read(lnkd_url)

To my mind, the "go to" method for simple plotting that you will show to someone else is ggplot(). The following code suggested by Didzis Elferts, in answer to a StackOverflow question, accomplishes the task with great economy, using just a few more features than what the defaults would give you.

ggplot(ibm,aes(Date,Close)) + 
  geom_line(aes(color="ibm")) +
  geom_line(data=lnkd,aes(color="lnkd")) +
  labs(color="Legend") +
  scale_colour_manual("", breaks = c("ibm", "lnkd"),
                          values = c("blue", "brown")) +
  ggtitle("Closing Stock Prices: IBM & Linkedin") + 
  theme(plot.title = element_text(lineheight=.7, face="bold"))

 

This next plot, which uses the dygraphs package, represents the new frontier for creating interactive time series plots in R.

# Plot with the htmlwidget dygraphs
# dygraph() needs xts time series objects
ibm_xts <- xts(ibm$Close,order.by=ibm$Date,frequency=365)
lnkd_xts <- xts(lnkd$Close,order.by=lnkd$Date,frequency=365)
 
stocks <- cbind(ibm_xts,lnkd_xts)
 
dygraph(stocks,ylab="Close", 
        main="IBM and Linkedin Closing Stock Prices") %>%
  dySeries("..1",label="IBM") %>%
  dySeries("..2",label="LNKD") %>%
  dyOptions(colors = c("blue","brown")) %>%
  dyRangeSelector()

Building on the work done for rCharts profiled in the Timely Portfolio piece, the dygraphs R package provides an interface to the dygraphs javascript library. With just a few lines of R code, it is now possible to produce charts that approach the polished look of the professional stock charting services - and no knowledge of JavaScript.

 

 

Connect R to Bloomberg with the RBlpapi package

For anyone who works with financial data and has access to a Bloomberg terminal, there is a new R package to interface to Bloomberg data services: RBlpapi. (If you had searched for an R connection to Bloomberg you wouldn’t have found this one — Bloomberg is happy to have software that connects to its public API, but not to use its name, apparently.)

One of the authors, Dirk Eddelbuettel, gave a presentation at at the R/Finance conference last month (you can take a look at the slides here). Here are a few examples of the types of queries you can perform, which generate time series data objects in R:

bdp(c("ESA Index", "SPY US Equity"), c("PX_LAST", "VOLUME"))
bds("GOOG US Equity", "TOP_20_HOLDERS_PUBLIC_FILINGS")
bdh("SPY US Equity", c("PX_LAST", "VOLUME"), start.date=Sys.Date()-31)
getBars("ESA Index", startTime=ISOdatetime(2015,1,1,0,0,0))
getTicks("ESA Index", "TRADE", Sys.time()-60*60))
fieldSearch("VWAP")

The package is fast and lightweight (no Java required), and works well on Linux-based systems. Dirk reports that it's currently tricky to build on Windows, though: suggestions welcome in the comments or via the Github project linked below.

Github: RBlpapi, R Access to Bloomberg API

Some Impressions from R Finance 2015

by Joseph Rickert

The R/Finance 2015 Conference wrapped up last Saturday at UIC. It has been seven years already, but R/Finance still has the magic! - mostly very high quality presentations and the opportunity to interact and talk shop with some of the most accomplished R developers, financial modelers and even a few industry legends such as Emanuel Derman and Blair Hull.

Emanuel Derman led off with a provocative but extraordinary keynote talk. Derman began way out there, somewhere well beyond the left field wall recounting the struggle of Johannes Kepler to formulate his three laws of planetary motion and closed with some practical advice on how to go about the business of financial modeling. Along the way he shared some profound, original thinking in an attempt to provide a theoretical context for evaluating and understanding the limitations of financial models. His argument hinged on making and defending the distinction between theories and models. Theories such as physical theories of Kepler, Newton and Einstein are ontological: they attempt to say something about how the world is. A theory attempts to provide "absolute knowledge of the world". A model, on the other hand, "tells you about what some aspect of the world is like". Theories can be wrong, but they are not the kinds of things you can interrogate with "why" questions.

Models work through analogies and similarities. They compare something we understand to something we don't. Spinoza's Theory of emotions is a theory because it attempts to explain human emotions axiomatically from first principles.

The Black Scholes equation, by contrast, is a model that tries to provide insight through the analogy with Brownian motion. As I understood it, the practical advice from all of this is to avoid the twin traps of attempting to axiomatize financial models as if they directly captured reality, and of believing that analyzing data, no matter how many terabytes you plow through, is a substitute for an educated intuition about how the world is.

The following table lists the remaining talks in alphabetical order by speaker.

	Presentation	Package	Package Location
1	Rohit Arora:   Inefficiency of Modified VaR and ES
2	Kyle Balkissoon: A Framework for Integrating Portfolio-level Backtesting with   Price and Quantity Information	PortFolioAnalytics
3	Mark Bennett:   Gaussian Mixture Models for Extreme Events
4	Oleg Bondarenko: High-Frequency Trading Invariants for Equity Index Futures
5	Matt Brigida:   Markov Regime-Switching (and some State Space) Models in Energy Markets	code for regime   switching	GitHub
6	John Burkett:   Portfolio Optimization: Price Predictability, Utility Functions,   Computational Methods, and Applications	DEoptim	CRAN
7	Matthew Clegg:   The partialAR Package for Modeling Time Series with both Permanent and   Transient Components	partialAR	CRAN
8	Yuanchu Dang:   Credit Default Swaps with R (with Zijie Zhu)	CDS	GitHub
9	Gergely Daroczi: Network analysi​s of the Hungarian interbank lending market
10	Sanjiv Das:   Efficient Rebalancing of Taxable Portfolios
11	Sanjiv Das:   Matrix Metrics: Network-Based Systemic Risk Scoring
12	Emanuel Derman:   Understanding the World
13	Matthew Dixon:   Risk Decomposition for Fund Managers
14	Matt Dowle:   Fast automatic indexing with data.table	data.table	CRAN
15	Dirk Eddelbuettel: Rblpapi: Connecting R to the data service that shall not be   named	Rblpapi	GitHub
16	Markus Gesmann:   Communicating risk - a perspective from an insurer
17	Vincenzo Giordano: Quantifying the Risk and Price Impact of Energy Policy Events   on Natural Gas Markets Using R (with Soumya Kalra)
18	Chris Green:   Detecting Multivariate Financial Data Outliers using Calibrated Robust   Mahalanobis Distances	CerioliOutlierDetection	CRAN
19	Rohini Grover:   The informational role of algorithmic traders in the option market
20	Marius Hofert:   Parallel and other simulations in R made easy: An end-to-end study	simsalapar	CRAN
21	Nicholas James:   Efficient Multivariate Analysis of Change Points	ecp	CRAN
22	Kresimir Kalafatic: Financial network analysis using SWIFT and R
23	Michael Kapler:   Follow the Leader - the application of time-lag series analysis to discover   leaders in S&P 500	SIT	other
24	Ilya Kipnis:   Flexible Asset Allocation With Stepwise Correlation Rank
25	Rob Krzyzanowski: Building Better Credit Models through Deployable Analytics in   R
26	Bryan Lewis:   More thoughts on the SVD and Finance
27	Yujia Liu and Guy Yollin: Fundamental Factor Model DataBrowser using Tableau and R	factorAnalytics	RFORGE
28	Louis Marascio:   An Outsider's Education in Quantitative Trading
29	Doug Martin:   Nonparametric vs Parametric Shortfall: What are the Differences?
30	Alexander McNeil: R Tools for Understanding Credit Risk Modelling
31	William Nicholson: Structured Regularization for Large Vector Autoregression	BigVAR	GitHub
32	Steven Pav:   Portfolio Cramer-Rao Bounds (why bad things happen to good quants)	SharpeR	CRAN
33	Jerzy Pawlowksi: Are High Frequency Traders Prudent and Temperate?	HighFreq	GitHub
34	Bernhard Pfaff:   The sequel of cccp: Solving cone constrained convex programs	cccp	CRAN
35	Stephen Rush:   Information Diffusion in Equity Markets
36	Mark Seligman:   The Arborist: a High-Performance Random Forest Implementation	Rborist	CRAN
37	Majeed Simaan:   Global Minimum Variance Portfolio: a Horse Race of Volatilities
38	Anthoney Tsou:   Implementation of Quality Minus Junk	qmj	GitHub
39	Marjan Wauters:   Characteristic-based equity portfolios: economic value and dynamic style   allocation
40	Hadley Wickham:   Data ingest in R	readr	CRAN
41	Eric Zivot:   Price Discovery Share-An Order Invariant Measure of Price Discovery with   Application to Exchange-Traded Funds

I particularly enjoyed Sanjiv Das' talks on Efficient Rebalancing of Taxable Portfolios and Matrix Metrics: Network Based Systemic Risk Scoring, both of which are approachable by non-specialists. Sanjiv became the first person to present two talks at an R/Finance conference, and thus the first person to win one of the best presentation prizes with the judges unwilling to say which of his two presentations secured the award.

Bryan Lewis' talk: More thoughts on the SVD and Finance was also notable for its exposition. Listening to Bryan you can almost fool yourself into believing that you could develop a love for numerical analysis and willingly spend an inordinate amount of your time contemplating the stark elegance of matrix decompositions.

Alexander McNeil's talk: R Tools for Understanding Credit Risk Modeling was a concise and exceptionally coherent tutorial on the subject, an unusual format for a keynote talk, but something that I think will be valued by students when the slides for all of the presentations become available.

Going out on a limb a bit, I offer a few un-researched, but strong impressions of the conference. This year, to a greater extent than I remember in previous years, talks were built around particular packages; talks 5, 7 and 8 for example. Also, it seemed that authors were more comfortable hightlighting  and sharing packages that are work in progress; residing not on CRAN but on GitHub, R-Forge and other platforms. This may reflect a larger trend in R culture.

This is the year that cointegration replaced correlation as the operative concept in many models. The quants are way out ahead of the statisticians and data scientists on this one. Follow the money!

Speaking of data scientists: if you are a Random Forests fan do check out Mark Seligman's Rborist package, a high-performance and extensible implementation of the Random Forests algorithm.

Network analysis also seemed to be an essential element of many presentations. Gergely Daróczi's Shiny app for his analysis of the Hungarian interbank lending network is a spectacular example of how interactive graphics can enhance an analysis.

Finally, I'll finish up with some suggested reading in preparation for studying the slides of the presentations when they become available.

Sanjiv Das: Efficient Rebalancing of Taxable Portfolios
Sanjiv Das: Matrix Metrics: Network-based Systematic Risk Scoring
Emanuel Derman: Models.Behaving.Badly
Jurgen A. Doornik and R.J. O'Brien: Numerically Stable Cointegration Analysis (A recommendation from Bryan Lewis)
Arthur Koestler: The Sleepwalkers (I am certain this is the book whose title Derman forgot.)
Alexander J. McNeil and Rudiger Frey: Quantitative Risk Management Concepts, Techniques and Tools
Bernhard Pfaff: Analysis of Integrated and Cointegrated Time Series with R (Use R!)

Introduction to Applied Econometrics with R

Bruno Rodrigues teaches a class on applied econometrics at the University of Strasbourg, with a focus on implementing econometric concepts in the R language. Since many of the students don't have any previous programming background, he's put together a tutorial on the basics of applied econometrics with R. The first two chapters serve as a general-purpose beginners' introduction to R, while chapter 3 explores basic applied econometrics with R (primarily data summaries and linear models). A fourth chapter to come promises a focus on reproducible research, so check back for updates to this free document at the link below. (And if you're looking for a more advanced tutorial on econometrics with R, check out Econometrics in R by Grant Farnsworth.)

Bruno Rodrigues: Introduction to Programming Econometrics with R

Quandl Chapter 2: The Democratization of Commercial Data

by Tammer Kamel
Quandl's Founder

About 22 months ago I had the privilege of introducing Quandl to the world on this blog. At that time Quandl had about 2 million datasets and a few hundred users. (And we thought that was fabulous.) Now, at the end of 2014, we have some 12 million datasets on the site and tens of thousands of registered users. On most days we serve about 1 million API requests.

One thing that has not changed however, is the simplicity with which R users can access Quandl. Joseph’s post last year, and Ilya’s this year both demonstrated the ease of connecting to Quandl via R.

Adoption of Quandl in the R community was perhaps the biggest factor in our early success. Thus it is fitting that I am back guest-blogging here at this moment in time because we are actually at the dawn of a new chapter at Quandl: We’re adding commercial data to the site. We are going to make hundreds of commercial databases from domain experts like Zacks, ORATS, OptionWorks, Corre Group, MP Maritime, DelphX, Benzinga and many others available via the same simple API.

What makes this new foray interesting is that we won't be playing by the rules that the incumbent oligarchy of data distributors have established. Their decades-old model has not served consumers well: it keeps data prices artificially high, it cripples innovation, and it is antithetical to modern patterns of data consumption and usage. In fact, the business models around commercial data predate the internet itself. They can and should be disrupted. So we're going to give that a go.

Our plan is nothing less than democratizing supply and demand of commercial data. Anyone will be able to buy data on Quandl. There will be no compulsory bundling, forcing you to pay for extra services you don’t need; no lock-in to expensive long-term contracts; no opaque pricing; no usage monitoring or consumption limits; no artificial scarcity or degradation. Users will be able to buy just the datasets they need, a la carte, as and when they need them. They will get their data delivered precisely the way they want, with generous free previews, minimal usage restrictions and all the advantages of the Quandl platform. And of course, the data itself will be of the highest quality; professional grade data manufactured by the best curators in the world.

We will also democratize the supply of data. Anyone, from existing data vendors and primary data producers to individuals and entrepreneurs, will have equal access to the Quandl platform and the unmet demand of the Quandl user base. We want to create a situation where anyone capable of curating and maintaining a database can monetize their work. In time, we hope that competition among vendors will force prices to their economic minimum. This is the best possible way to deliver the lowest possible prices to our users.

At the same time this democratization should empower capable curators to realize the full value of their skills: If someone can build and maintain a database that commands $25 a month from 1000 people, then Quandl can be the vehicle that transforms that person from skilled analyst to successful data vendor.

If you were to characterize what we are doing as a marketplace for data you would be absolutely correct. We are convinced that fair and open competition will do great things, both for data consumers who are, frankly, being gouged, and for existing and aspirational data vendors who are disempowered. Open and fair competition is a panacea for both ills: it effects lower prices, wider distribution, better data quality, better documentation and better customer service.

Our foray into commercial data has already started with 6 pilot vendors. They range from entrepreneurially-minded analysts who are building databases to rival what the incumbents currently sell for exorbitant fees, to long-established data vendors progressive enough to embrace Quandl’s modern paradigm. We have no less than 25 vendors coming online in Q1 2015.

So, Quandl in 2015 should very quickly become everything an analyst needs: A free and unlimited API, dozens of package connections including to R, 12 million (and growing) free and open datasets, and access to commercial data from the best companies in the world at ever decreasing prices. Wish us luck!

Using R to simulate the finances of public sector pension funds

by Don Boyd, Senior Fellow, Rockefeller Institute of Government

The Rockefeller Institute of Government is excited to be developing models to simulate the finances of public pension funds, using R.

Public pension funds invest contributions from governments and public sector workers in an effort to ensure that they can pay all promised benefits when due. State and local government pension funds in the United States currently have more than $3 trillion invested, more than $2 trillion of which is in equity-like investments. For example, NYC, has over $158 billion invested. Governments usually act as a backstop: if pension fund investment returns do better than expected, governments will be able to contribute less, but if investment returns fall short they will have to contribute more. When that happens, politicians must raise taxes or cut spending programs. These risks often are not well understood or widely discussed. (For a discussion of many of the most significant issues, see Strengthening the Security of Public Sector Defined Benefit Plans.)

We are building stochastic simulation models in R to help quantify the investment risks and their potential consequences. We are modeling the finances of specific pension plans, taking into account all of the main flows such as current and expected benefit payouts to workers, contributions from governments and from workers, and investment returns, and how they affect liabilities and investible assets. The models will take into account the changing demographics of the workforce and retiree populations. We are modeling investment returns stochastically, examining different return scenarios and different economic environments, as well as different governmental contribution policies. We will use these models to evaluate the risks currently being taken and to help provide policy advice to governments, pension funds, and others. (For a full description of our approach, see Modeling and Disclosing Public Pension Fund Risk, and Consequences for Pension Funding Security)

We have chosen R because:

• It is extremely flexible, allowing us to do data collection, data management, exploratory data analysis, and other essential non-modeling tasks.
• Manipulating matrices is easy.
• It has sophisticated tools for modeling investment returns and for analyzing and presenting results of simulations. And it has great tools for visualizing results.
• The work can be completely open and reproducible, which is essential to the success of this project.

All programming languages have weaknesses. R’s great flexibility means that it is easy to write ill-organized programs that are hard to understand and debug. And poorly written programs that do not take advantage of R’s strengths can be extremely slow. We believe we can compensate for these weaknesses by making our programs modular, using a consistent programming style with appropriate documentation, and by using R features smartly and speed-testing where appropriate.

R analysts and programmers interested in learning about the opportunity to work on this project should examine the programmer/analyst position description and related materials at the Rockefeller Institute’s web site.