Revolutions

Promoting open government in Brazil with R

I met R hacker Eduardo Leoni at the DC R User Group (which is still looking for a new organizer, by the way) last month, where he gave an impromptu lightning talk on the open-government website he created in his spare time, CongressoAberto.com.br. With a name that means "Open Congress" in Portuguese, it is an independent website that aims to increase transparency and contribute to debates about the Brazilian legislature, facilitating access to information and analysis on the Brazilian political process. Naturally, the site is in Portuguese, but Google Translate does a good job of rendering the site in English for non-native speakers like me.

Much of the site is devoted to sharing official government data with Brazilians in an easily-understood format, giving insight into how and why political decisions are made. For example, Eduardo has written scripts in R that analyzes the votes on particular bills (for example, this one) and displays the legislators' votes by party and on a regional map:

Although other open-government sites exist in many countries (for example, the Sunlight Foundation here in the US), the Brazilian political system is different in that there are many important political parties instead of just two major ones. As a result, Eduardo had to create all of the analyzes and charts for Brazilian government from scratch. Luckily, the tools provided by R to scrape data from unwieldy government websites and to create color-coded maps (to name just a couple of features) make this process easier than it would be with closed-source tools.

CongressoAberto.com.br: Home Page
 

Open Government and R

Our own Open Source Diva at REvolution, Danese Cooper, gives an interview to O'Reilly Radar and talks about her background in the Open Source world, her involvement in the Obama campaign, and how R is likely to play a role in the Open Government initiative (aka Government 2.0) of the current Administration. 

Danese also gives a preview of the R talk we'll be giving at OSCON next week, and predicts the future of R by analogy with the history of Linux:

Linux was a toy until, all of the sudden, it was good enough to run financial services houses that didn't want to pay the premium for Solaris anymore. And we're right at that inflection point with R where it's just finally gotten good enough that it can go up against some of these well established, expensive tools.

It's a great interview, check out the transcript or listen to the podcast version at the link below.

O'Reilly Radar: Making Government Transparent with R.

Find yourself a safer place to swim or fish in the Bay Area

Another example of a web dashboard built using R comes from John Oram, a scientist at the San Francisco Estuary Institute. SFEI collects and monitors environmental data from the waters and wetlands of the Bay Area. By sampling the waters, sediments and fish of the San Francisco Bay and testing for toxins, pollutants and other hazards, they monitor the health of the environment for Bay Area residents.

Making sense of such data could be a challenge: how do you translate random point-samples of pollutants taken over a long period of time into an overall picture of the environmental impact? John Oram has met this challenge in two ways: by linking the point data to a Google map of the region, where you can see the individual measurements in context; and by using kriging analysis and contour charts from R to extrapolate those samples to the entire Bay. Using his Web Query Tool you can select whether to review sediment, water, or fish data, choose the period over which the samples were collected to view, and select a specific toxin or pollutant to analyze. Here's a screenshot (click to enlarge) showing the distribution of all PCBs in the sediments of the bay based on samples from 2002-2007, but you can try it yourself and produce maps for other pollutants.

 

 

 

Amazingly, John created this application on basically zero budget: it uses all free tools including Google Maps, Rapache, and of course R. John said during the meeting that he couldn't have created this application without R: not only is it free, but the advanced statistical tools that analyze the data and even help select the sampling locations simply aren't available anywhere except in R.

As for the future, John is looking for other agencies that may be interested in using the application he's built to provide environmental analysis to everyone through a Web-based interface. I can certainly imagine the day when your local surf-safety bulletin or fishing recommendations come from a tool like John's.

San Francisco Estuary Institute: Web Query Tool (Update: a newer version is now available.)

R, the FDA, and clinical trials

There's been an interesting discussion recently on the R Project group on LinkedIn (sub. req.) on the topic of using R for clinical trial analysis in the pharmaceutical industry. The questions asked were:

Is the number of [FDA] submissions featuring R growing? Are companies happy to use R for submission work yet? Is the use of R within regulatory agencies growing? If not, how do we move the discussion forward both internally within companies AND externally with regulators?

This is a topic we've covered here before, but Marc Schwartz from MedNet Solutions posted such an excellent reply to that thread that I repost it here in its entirety (with his permission):

The exact answer is that there is nothing, including 21CFR11, preventing you from using R for a submission. There are those who believe that 21CFR11 does not apply to R or any other application, including SAS, used purely for statistical analysis and presentation. 21CFR11 is intended to be used in the domain of source electronic records and their management. If R, or SAS for that matter, are not used for that purpose (eg. you use Oracle for data acquisition and management), then there is the basis to believe that 21CFR11 is not relevant.

That being said, there are other FDA documents that would still apply, including GxP and the others relative to the validation of software, which are listed in the "R-FDA" document that has been referred to.

This means that you (not R Core nor the R Foundation) still have an obligation to implement and document SOPs and related IQ/OQ/PQ processes relative to R. How you do this is up to you and your organization based upon existing SOPs and any relevant internal legal and regulatory guidance.

The "R-FDA" document provides the basis to express the SDLC for R (Base and Recommended packages). There is also documentation provided as to how R meets the relevant 21CFR11 components, so that those folks who are not convinced that 21CFR11 is not relevant, can understand how R fits within that framework.

With respect to contributed packages, the obligation for those is no different than if you were to use your own code. Provide documentation and SOPs for IQ/OQ/PQ based upon your own processes. The key is that you can document how you have established that the results from R and any other contributed code are consistent and reliable. See section 5.2 "Validation" in the R-FDA document.

The question keeps coming up because of FUD, perpetuated by those who are loathe to open their minds or by those with a vested fiduciary interest in seeing other applications used. Bear in mind, there are those who still believe that SAS is the only application that can be used for FDA submissions. The consulting companies whose business model is dependent upon making MS Access and Excel 21CFR11 compliant and validated would have an argument there... :-)

The FDA will not come out formally with a position that endorses any application, so you will not get an answer from them formally stating anything. They are precluded from doing that as a government agency.

That the FDA is using R internally and for rather high profile cases (eg. Avandia) should be cause for increasing one's comfort level.

This does not mean that the challenge for R is not present. Risk averse internal management and decision makers have to be sold on the use of R and that it is free may not be sufficient, given the costs of converting legacy code, processes and training. Even in a time of cost containment, the potential savings of not paying significant sums for annual SAS licenses may not be sufficient motivation.

If the organization is already using open source software internally (eg. Linux, Apache, SVN, Firefox, Thunderbird, OpenOffice) then there is at least an open mind to the potential use of R. If not, then the challenge may be more profound.

If R is already being used for non-clinical applications, which is presently the case even in large pharma, then there is the potential opportunity to educate the clinical statisticians to improve their comfort level. This would provide a bottom up strategy for R, if you can gain the comfort and support of the folks doing the work. That can help to make the argument to senior management somewhat easier. However, at the senior management level, you need to be able to make a solid business case for the change.

This situation is not going to shift overnight. It is part and parcel of the fundamental resistance to change by all of us. It will take time and people willing to step up and do the work within their own organizations to change behavior and overcome the resistance. 

I think Marc's response is spot-on. The key is that the burden falls on the organization using R to verify GxP and 21CFR11 compliance, insofar as they may apply to the specific context in which R is being used for clinical trial analysis. Fortunately, this is relatively straightforward with R. REvolution Computing helps companies with this process of validating their use of R in context, and by providing a distribution of R (REvolution R Enterprise) with a documented build process ready for audit. 

Analyzing clinical trial data for FDA submissions with R

R is well-known for its use in pre-clinical data analysis in drug research. In the relatively unregulated phases before human subjects are exposed to new compounds, researchers have the opportunity to explore data with a variety of techniques. R's flexibility and breadth of standard and novel methods for data analysis and graphics makes it an ideal choice to make new discoveries and guide the direction of subsequent clinical trials.

It's in the clinical phases, where human subjects are exposed to prospective new drugs for the first time, where R is less well-known. SAS is currently the dominant software tool for data preparation and statistical analysis in this domain. There is a misconception that SAS is the preferred software of the FDA, the regulatory agency that oversees clinical trials, but in fact the FDA does not endorse or require use of any specific software. 

I'm often asked whether R is "validated" for use in clinical trials, but in fact neither SAS nor R can be considered validated in isolation.  That's because it's not software that's validated, it's the entire process of creating the submission to the FDA. Validation is defined by the FDA as:

Establishing documented evidence which provides a high degree of assurance that a specific process will consistently produce a product meeting its predetermined specifications and quality attributes.

Clearly, the software used for the data analysis portion of this process must adhere to these same standards, so how can you established the required documented evidence for R?  To that end, the R Foundation has created the document R: Regulatory Compliance and Validation Issues. Described as "a guidance document for the use of R in regulated clinical trial environments", the purpose of this document is to demonstrate that R, when used in a qualified fashion, can support the various regulatory requirements for validated systems, most notably 21 CFR Part 11 and the relevant "good practice" documents (GxP) and industry guidance documents. 

The document describes the rigorous software development life cycle for R. The software for the core R engine, plus the base and recommended packages included in the R distribution, is all managed under a secure source code management system accessible only by the R Core Group (a widely recognized, international team of experts) and the changes between half-yearly releases are clearly documented. To verify that R works as designed and documented, a suite of validation tests compares the output of R against known data and known results.

Of course, the fact that R itself is maintained with a compliant development process doesn't complete the whole validation process, since R is always part of a larger process to collect, analyze, and report clinical trial data. (The data collection process usually happens independently of the statistical analysis process, and must usually be validated independently, so I won't touch on it further here.)

When any software is installed for analysis, it must be validated to meet IQ/OQ/PQ criteria. IQ refers to Installation Qualification: has the software been installed correctly? OQ refers to Operational Qualification: is the software operating correctly? Finally, PQ refers to Performance Qualification: is the data analysis process effective and reproducible? (Reproducible analysis in clinical trials is an active area for the R community -- see this presentation from Frank Harrell outlining the tools available, and here's an example of a report created with R using a reproducible analysis process.)

REvolution Computing offers a distribution of R designed to comply with established IQ/OQ/PQ criteria, and provides process documentation and services to support the validation process. This enables our clients in the pharmaceutical sector validate their installation of REvolution R, and if required any associated scripts and in-house or third-party packages being used, within the clinical analysis process.

Clear guidance from the R Foundation, sophisticated tools to create reproducible reports, and the support of vendors like REvolution Computing to validate the entire process, have resulted in growing use of R for clinical trial analysis. FDA employees have presented at statistical conferences on the use of open-source software for clinical trial analysis, have used R for internal FDA analyses, and have described the use of R in the FDA submission review process. Statisticians at companies like Novartis have described using R for pharmaceutical development. And with pharmaceutical companies embracing modern statistical methods and graphical techniques in support of FDA submissions, the use of R is only likely to grow further.

(Note: Special thanks go to Marc Schwartz for this r-help post, which was the source for many of the resources linked in this article.)