Revolutions

Importing public data with SAS instructions into R

Many public agencies release data in a fixed-format ASCII (FWF) format. But with the data all packed together without separators, you need a "data dictionary" defining the column widths (and metadata about the variables) to make sense of them. Unfortunately, many agencies make such information available only as a SAS script, with the column information embedded in a PROC IMPORT statement.

To solve this problem, R user Anthony Damico (who's also the energetic voice behind the excellent R Twotorials series) created the SAScii package. It parses the SAS script (or even an unstructured text instructions file, with a PROC IMPORT statement included), and uses that information to read the associated fix-format ASCII file. Just provide the name or URL of the script/instructions file and the data file, and SAScii does the rest. There are several examples in the read.SAScii helpfile, including the R command to read the following public data sets:

• 2009 Medical Expenditure Panel Survey Emergency Room Visits File (Medical Expenditure Panel Survey)
• 2010 National Health Interview Survey Persons file (CDC)
• IPUMS - American Community Survey Extract (Minnesota Population Center)
• 2008 Survey of Income and Program Participation Wave 1 (US Census)

Many thanks to Anthony for creating this package and pointing it out to me at the useR!2102 conference last month. It unlocks many useful public data sets for those of us without SAS licenses.

SAScii package: read.SAScii documentation

'Moneyball' for government policy

The UK government proposes to use data and evidence as the basis for deciding which policy initiatives to implement, in this Cabinet Office paper. The paper, co-authored by Ben Goldacre (who we've mentioned here before), proposes using randomized trials to evaluate the efficacy of government programs. It's like applying the 'Moneyball' principles to government policy: rather than using anecdotes and ideology to decide how to spend government money, why not try policies out on a small scale first and see what works? 

The US government has recently begun applying such trials to evaluate programs at the federal and state level, as well.

 

 

 

 

FDA: R OK for drug trials

In a poster (PDF) presented at the UseR 2012 conference, FDA biostatistician Jae Brodsky reiterated the FDA policy regarding software used to prepare submissions for drug approvals with clinical trials: Sponsors may use R in their submissions.

The FDA does not endorse or require any particular software to be used for clinical trial submissions, and there are no regulations that restrict the use of open source software (including R) at the FDA. Nonetheless, any software (R included) used to prepare data analysis from clinical trials must comply with the various FDA regulations and guidances. The R Foundation helpfully provides a guidance document for the use of R in regulated clinical trial environments, which provides details of the specific FDA regulations and how R complies with them.

Despite some mistaken conceptions in the pharmaceutical industry, SAS is not required to be used for clinical trials. This origin of this fallacy is probably related to the fact that data must be submitted in the XPT "transport format" (which was originally created by SAS). This data format is now an open standard: XPT files can be read into R with the standard read.xport function, and exported from R with the write.xport function in the SASxport package. (And if you have legacy data in other SAS formats, there's a handy SAS macro to export XPT files.)

R is widely used at the FDA. (Full disclosure: the FDA is a Revolution Analytics customer.) The poster describes how FDA offices "use R on a daily basis", and that FDA scientists have written R packages for use by other scientists (within and outside the FDA) involved in the submissions process. FDA-contributed packages include popPK, MAVTgsa and QT, which was written specifically for the FDA's QT Interdisciplinary Review Team.

Jae Brodsky also provides some personal views on the use of R at the FDA. She notes:

• All results from data should be reproducible independent of software used.
• In submissions, FDA reviewers would like to know: which R functions were accessed; how they were accessed; and that the results are accurate, reliable and consistent.
• The FDA has used R for its own reports. (Interesting side-note: as a government agency, its reports must be Section 508 compliant, one part of which demands that reports be accessible to readers with limited color vision. Jae mentioned to me that she uses R's Rcolorbrewer package to generate 508-compliant pallettes for charts in FDA reports, as the poster excerpt below demonstrates.)

In Jae's experience, "R use at the FDA is completely acceptable and has not caused any problems." So as more and more companies adopt R for statistical analysis, we can expect to see more FDA clinical trial submissions using R for statistical analysis, reporting, and data visualization.

Applications of R in Government

Following the announcement of the US Government Big Data Initiative, I was asked to write a small article about applications of R in government. The article has just appeared in Government Security News (and I believe will appear in their daily newsletter tomorrow). In the article, I highlighted several R applications that been highlighted here in the blog:

• In the early days of the Deepwater Horizon disaster, NIST used uncertainty analysis in R to harmonize spill estimates from various sources, and to provide ranges of estimates to other agencies and the media.
• Before new drugs are allowed on the market, the FDA works with pharmaceutical companies to verify safety and efficacy through clinical trials. Despite a false perception that only commercial software may be used, many pharmaceutical companies are now using open-source R to analyze data from clinical trials.
• The National Weather Service uses R for research and development of models to predict river flooding.
• The newly-formed Consumer Financial Protection Bureau -- freed from the restrictions of a legacy IT infrastructure -- is championing the use of open-source technologies in government.
• Local governments are also building data-based applications. The SF Estuary Institute uses R and Google Maps to provide a tool to track pollution in the San Francisco Bay area.

For more on the emergive governmment drive to make better use of analytics, open source and Big Data read the full article at the link below.

Government Security News: Open-Source R software driving Big Data analytics in government

How NOAA uses R to forecast river flooding

Thanks to the lower-than-usual snowfall over most of the US this past winter, there's low risk of major flooding as the snow melts this Spring (for the first time in four years!). Nonetheless, being able to forecast river flood events is of critical importance to local emergency managers, water & electric utilities, river navigation companies, and the US Army Corps of Engineers -- not to mention residents in flood-prone areas. That's why the National Weather Service (a division of NOAA) is charged with forecasting flood risk throughout the United States.

As noted by Tom Adams, Development & Operations Hydrologist for the NWS's Ohio River Forecast Center, R is currently used at 4 out of 13 River Forecast Centers to generate graphics representing real-time hydrologic ensemble (probabilistic) forecasts. (Five additional regions are expected to also use R later this year.) For example, here's a recent analysis showing that Clay City on Illinois' Little Wabash River has a small risk of minor flooding in the next couple of weeks:

R is also used for research and development at NWS for forecast verification and analyses for the calibration of distributed hydrologic models. You can see these models in action for the NWS's Eastern Region by exploring the interactive maps at the link below.

National Weather Service: NWS Experimental Short-term Hydrologic Ensembles (MMEFS)

R at the Consumer Financial Protection Bureau

The O'Reilly Radar blog has a lengthy and very interesting interview with the lead and deputy CIOs of the Consumer Financial Protection Bureau, the new US government agency devoted to consumer protections in the financial markets. In that interview, they talk about the many open-source tools used in the agency (and the parent Treasury Department): Linux, WordPress, Splunk, Django, Git and  yes -- R for data analysis with Big Data. (The CFPB has also made public its policy on use of open-source software.) They reveal an interesting cultural shift related to the use of R in this question at the end of the interview:

How do you see open source contributing to your ability to get insights from large amounts of data? If you're recruiting developers, can they actually make a difference in helping their fellow citizens?

[...] R is an interesting example. What we're finding is that as more people are coming out of academia into the professional world, they're actually used to using R in school. And then they have to come out and learn a different tool and they're actually working in the marketplace.

It's similar with the Mac versus the PC. You get people using the Mac in college — and suddenly they have to go to a Windows interface. Why impose that on them? If they're going to be extremely productive with a tool like R, why not allow that to be used?

Here at Revolution Analytics, we're hearing much the same story from commercial organizations who have data analysis platforms build around legacy platforms like SAS. When all of the new recruits coming out of school are already trained in R, the choice is between spending a year training the up in SAS (and incurring both the expense and the opportunity cost of a year's work in the process), or setting up a platform based around a commercially-supported R distribution. And more and more often, that second choice is the more compelling one.

O'Reilly Radar: Open source is interoperable with smarter government at the CFPB

Bigger data, better intelligence for government

Tomorrow (at 2PM Eastern on Thursday), the White House White House Office of Science and Technology Policy will host a 90-minute forum on Challenges and Opportunities in Big Data. You can watch the event live at the previous link, and see federal government science heads from OSTP, NSF, NIH, DoE, DoD, DARPA and USGS outline how their agencies are engaged in Big Data research. The event will also include a discussion by a panel of thought leaders from academia and industry, moderated by Steve Lohr of the New York Times.

Naturally, the event has prompted some speculation about the government's "big data agenda", but this in fact a direction the administration has been pushing ever since the Obama appointed Aneesh Chopra as the nation's first CTO in 2009. (Chopra passed the mantle to current CTO Todd Park just this this month.) One of the first initatives was the creation of data.gov, a vehicle to make government data available in a timely and accessible format for a community of citizen data scientists to make sense of it all. For example, independent analysis of data from data.gov (using Big Data statistical analysis techniques) revealed a flaw in the 2000 Census results that went unnoticed by the Census Bureau.

And there's the rub: while much of the talk around Big Data has been around storage and search for massive data sets, the real value comes from analysis and inference on the data themselves. I'd expect to hear more from the administration on this topic in today's forum. The administration is already pushing to use more open-source software in government, the open-source R language (along with the big-data enhanced distribution from Revolution Analytics) is already widely used at NIST, FDA, NOAA, CIA, and many other government agencies. Given that the data storage and data analysis infrastructure is already well in place, I expect we'll hear a vision for new kinds of applications, citizen services and data-driven strategies from the participating federal agencies in today's forum.

You can join the forum in the live-streaming webcast at the link below. 

NSF: OSTP and NSF Present: Challenges and Opportunities in Big Data (2PM EST, Thu Mar 29 2012)

Updated Mar 28 to correct the time of the forum.

R's role in the national response to the BP Oil Spill

In the early days of the Deepwater Horizon oil spill in the Gulf of Mexico, the rate of flow of oil from the spill was of great concern: estimating it accurately was key to coordinating the scale and scope of the response to the emergency. Unfortunately, estimates from independent sources varied widely, and BP's own estimates varied widely over time. 

Antonio Possolo, Division Chief of Statistical Engineering at the National Institute of Science and Technology (NIST), was charged with making sense of these varied estimates to help the government coordinate the national response to the spill. As described in this video testimonial (starting at 2:20), Possolo was sitting in the company of the Secretaries of Energy and the Interior, when he broke out R on his laptop to run uncertainty analysis and harmonize the estimates from the various sources. In the video, Possolo is adamant in his confidence in his statistical analysis using the open-source R system. As he says to the crowd (including many of the developers of R), "The quality that you have built into R, through public, open examination, is the greatest strength and source of confidence I could have asked for." 

Blip.tv: useR talks, video #1

Pollution from the BP oil spill

There's been a lot of talk about the slicks and plumes of oil from the Deepwater Horizon disaster, but how does the presence of that oil translate into measurable pollution in the air, water, and sediment? The EPA is now releasing pollutants and making the data available for analysis. Because the data are online, it's a simple process to read it into R and do visualization and analysis. The blog R-Chart shows us how to map the sampling locations around the Louisiana coastline, for example:

This is a simple analysis, but demonstrates the principle of working with this public data in R. It would be interesting to see some more in-depth analyses of the data.

R-Chart: Plotting BP Oil Spill Testing Data using R

Forecasting the weather with R

The US National Centers for Environment Prediction (NCEP) produces weather forecasts for the entire world from a model that's updated every 6 hours. The data is made freely available, and with a couple of free tools to convert the data and R you can easily produce am unpdated global weather forecast like this (click to enlarge):

(Check out the heat-wave in Australia!) Physicist and climate scientist Joe Wheatley provides all the details, including R code. All you need is R, the ncdf package to process the data file, and a free command-line tool to convert the file from the format provided by NCEP.

Biospherica: NCEP Global Forecast System