Revolutions

Get your questions answered about Open Data

The OpenData StackExchange site has just launched in beta, and looks to be a great resource for open data sources. Like StackOverflow for programming and CrossValidated for statistics,  OpenData is is a question and answer site for developers and researchers interested in open data. There's no R tag yet (though that would be nice for data sources specifically compatible with R), but there are already some useful tags for government data, APIs, and tools (to name just a few). If you have expertise on open data to share, or just have a question you need an answer to, follow the link below and check it out.

StackExchange: Open Data beta (via JM)

Open Source software's opportunity to reform government

The results from the 2013 Future of Open Source Survey are in — thanks to everyone who contributed by completing the survey. You can read an overview of the results here, or see the detailed breakdowns in the slides at the end of this post.

For me, one of the most interesting nuggets from the survey is that a plurarity of respondents believed that of all sectors, the public sector will be impacted the most by open source software:

There is huge opportunity to improve the ways that we govern ourselves through the use of open source software, principles and communities. Federal, state and local governments have access to rich and large streams of data, but thus far government hasn’t been as quick as the business sector in building powerful data-based applications and services. But governments have one advantage that the business sector does not: givernment data is, by definition, public. Here are three ways that government can capitalize on rich sources of public data:

Democratize data. By making public data available on the internet (where practical and legal), governments can draw on the expertise of the data scientist community to unlock its value. At the Federal level, the Open Government Initiative is a good start, although many of the data sets made available at data.gov are over-aggregated and too small for truly innovative Big Data applications. Some local governments are making good progress here as well. New York City’s open data policy and new data transparency law has fostered a culture of civic “hacking” on city data, and created handy tools for locals like this iPhone app to find local WiFi-enabled venues suitable for getting out of the house when working “from home”.

Crowdsource data science talent with open tools. The data science community has created powerful open-source tools and resources for the analysis of Big Data. The open-source R language is designed to analyze data common to government applications (like economic, environmental and social data) and is used today by government departments such as NIST, FDA, NOAA, and the CIA. Crowdsourcing platforms and contests are a great way of getting data scientists engaged with public data, as NASA found in this Kaggle contest to find the location of dark matter from radio astronomy data.

Deploy data scientists within departments. Not all government can be made public, of course. There may be privacy issues (think IRS data), or national security issues (CIA, NSA) to consider. So it’s up to data science teams in civil service to work on sensitive data in a secure environment. It’s very encouraging to see departments like the Consumer Protection Bureau openly embrace big-data platforms and open-source tools for their teams. The growing drive towards evidence-based policy making, and the use of statistical experiments to improve government effectiveness, is another very positive sign.

It's great to see that the respondents to the 2013 Future of Open Source Survey recognize this opportunity to revolutionize government. It's up to us, as citizens, to press governments to capitalize on the opportunity, so make your voice heard.

Michael J Skok: 2013 Future of Open Source - 7th Annual Survey results

Take the 2013 Future of Open Source Survey

Every year for the past six years, a consortium of software companies has collaborated to conduct a survey on attitudes and adoption of open-source software in business. The 2013 Future of Open Source survey is now open and taking responses until March 28. Open source analyst group 451 Research is a collaborator again this year, along with Revolution Analytics, Red Hat, Hortonworks, and several other open-source software firms.

If you work for a company that is using (or plans to use) open-source software, or for a company that produces or supports open-source software, please take the survey. (It should take around 10 minutes to complete.) The results and conclusions from the survey will be presented in a live webinar on April 17. The conclusions from the 2012 survey can be found below.

Please take the 2013 survey before March 28 by following the link below.

2013 Future of Open Source Survey: Take the survey

Vendor news: TIBCO's proprietary R runtime; Teradata's appliance integrates R

In a webinar today previewing Spotfire 5 (scheduled for release this November), TIBCO announced that it will include TERR: The Tibco Enterprise Runtime for R. TERR is a closed-source reimplementation of the R language engine, and not based on the GPL-licensed R project from the R Foundation. Here's the relevant slide from the webinar:

By making the TERR engine TIBCO intellectual property (IP), rather than using the open-source R engine, TIBCO claimed in the webinar to have been able to improve performance. Apparently, while some packages will run at about the same speed, others may run at 10x speed or even faster. This performance comes at the expense of compatibility: not all R functions or CRAN packages will work with the TERR engine, and it's not clear whether or at what rate TERR will follow R's development path.

The TERR engine will be included in the Spotfire Professional Client, and some new statistical interfaces (point-and-click regression and classification modeling dialogs) will make use of the engine. But if you want to use TERR-compatible R code in the Spotfire Web Player (to deploy beyond the local desktop), you'll also need a license for Spotfire Statistics Services.

Coinceidentally, a second enterprise analytics vendor also announced integration with R today. Teradata's Big Data Appliance, which combines high-performance hardware with open-source Hadoop and Teradata Aster software, will include integration between the Hadoop engine and R. From the data sheet:

The SQL-MapReduce framework, created by Teradata Aster, allows developers to write powerful and highly expressive SQL-MapReduce functions in languages such as Java, c#, Python, c++, and R, and push them into the discovery platform for advanced in-database analytics.

Like the similar RHadoop project, Teradata's R integration works with the open-source R engine.

The fact that more enterprise software vendors are integrating with R is generally a good thing for the R community: it validates the power of R within organizations, and adds more options for bringing advanced analytical methods developed in R to production environments. I just hope that the introduction of a prorpietary R language engine doesn't promote the fracturing of the vibrant R community — one of R's greatest strengths — and result in a fork of the R language into incompatible dialects.

Designing real-world 3-D objects with R

The Maker Movement has led to the production of open-source 3-D printers and other manufacturing machines that allow hobbyists to design, create and produce real-world objects affordably. Now R user Ian Walker, in a post at the Psychological Statistics blog, shows how to use the R language to transform 3-D surfaces into real-world physical objects with a 3-D printer. Ever had trouble visualizing exactly what a 3-D surface looks like when constrained by a boring old 2-D monitor screen? Now you can print it, and rotate and inspect it in realspace:

Walker's open-source R code works much like the standard R persp function, but instead of translating a grid of (x,y,z) points to a mesh surface on the screen, it creates a stereolithography file for use with 3-D printers, which in turn can be used to create an object like that shown above. Now, if only I had a 3-D printer to play with...

Psychological Statistics: Visualizing data using a 3D printer (via Dirk Eddelbuettel)

Value of Open Source: Lower costs, freedom from lock-in

The results of the 2012 Future of Open Source Survey were presented at last week's OSBC conference in San Francisco. (Revolution Analytics is a collaborator in the Survey.) You can read a summary of the results in this press release, but one interesting trend is evident from Slide 13 of the presentation:

With the same question, "What makes Open Source Software Attractive" having been asked in four successive surveys, a trend emerges. Among the top three reasons each year are:

• Lower costs
• Freedom from vendor lock-in

The rank changes from year to year, but these two reasons are among the top concerns. Some other interesting conclusions from the survey:

• Open source is leading, not following, in Big Data;
• 44% of respondents identified data management as the segment most likely to be affected by open-source software in 2012.

For other highlights of the survey results, follow the link below.

BlackDuck Software: Sixth Annual Future Of Open Source Survey Results Point to Era of "Open Innovation On Demand"

EU court's SAS ruling conflicts with Oracle v Google

In a blow to SAS's efforts to litigate competitor and low-cost SAS clone WPS out of existence, the European Union High Court has ruled that programming languages can't be copyrighted. SAS Institute (Cary, NC) had claimed that the WPS software — which allows users to process SAS data files and SAS "data step" scripts without SAS software — breached SAS's copyright in its re-implementation of SAS functionality. But given that WPS did not study SAS source code, and merely reimplemented its interfaces and behaviour based on observation, the court ruled that this was not a violation. In a press release following the decision, the court stated:

 ... neither the functionality of a computer program nor the programming language and the format of data files used in a computer program in order to exploit certain of its functions constitute a form of expression. Accordingly, they do not enjoy copyright protection.  

(Emphasis in press release.) SAS has issued no comment on the ruling to date.

The case has implications beyond SAS and WPS. In Europe, at least, it implies that developers of both proprietary and open-source software have the right to duplicate the functionality and exposed interfaces (in this case, the structure of SAS procedure calls) of a proprietary language provided they don't copy the source code of the implementation itself.

The EU ruling conflicts with that of the jury in the Google v Oracle trial, where it was decided that Google's re-implementation of Java APIs in the Android OS did infringe Oracle's copyright. Implementing a SAS procedure and a Java API are similar programming efforts, and the EFF has argued convincingly that preventing programmers from re-implementing APIs harms innovation. Nonetheless, the US-based jury did not decide if Google's Android implementation was a fair use infringement of copyright; if that question were to be resolved in Google's failure the outcome would have a similar practical effect as the EU decision.

JDSupra: European Union Court Rules that Software Functions Cannot Be Copyrighted 

 

 

 

Take the 2012 Future of Open Source survey

Every year for the past five years, a consortium of software companies has collaborated to conduct a survey on attitudes and adoption of open-source software in business. Now in its sixth year, the 2012 Future of Open Source survey is now open. Open source analyst firm 451 Research is a collaborator for the first time this year, along with Revolution Analytics, Red Hat, Couchbase, Eucalyptus and several other firms.

If you work for a company that is using (or plans to use) open-source software, or for a company that produces or supports open-source software, please take the survey. (It should take around 10 minutes to complete.) The results and conclusions from the survey will be presented at the 2012 OSBC conference in San Francisco (May 21-22), and shared online. (Survey participants will also receive a 20% discount code to attend the conference.) The conclusions from the 2011 survey can be found below.

 

2012 Future of Open Source Survey: Take the survey

Discussion thread on R vs SAS for businesses

There's an interesting discussion thread on LinkedIn going on now on the relative benefits of R versus SAS in the commercial sector. Oleg Okun kicks off the discussion with this question:

Did anyone have to justify to a prospect/customer why R is better than SAS? What arguments did you provide? Did your prospect/customer agree with them? Why do you think, despite being free and having a lot of packages, R is still not a favorite in Data Mining/Predictive Analytics in the corporate world?

What follows is an in-depth discussion (more than 130 comments so far) comparing the two statistical software systems. Steve Miller condenses the discussion in a great post at the Information Management blog. Themes covered include: the benefits and purported risks of using open-source software vs commercial software; dealing with large data sets (one R user notes: "I've used a very fast (~16Tb RAM) computer to run simulations on hundreds of billions of observations"); availability of skills for new hires ("Many of our customers have the problem of needing to spend the time and money to train new hires in SAS because their new hires have only used R"); availibility of support for R (Revolution Analytics provides support for R); and many other topics. One sub-thread focused on quality in open-source software, for which Steve had an excellent riposte:

There's little argument that the vast international R community provides access to the latest statistical models and procedures before they're available in proprietary SAS. But SAS proponents counter that R users assume  more risk with software quality than do those of SAS. In fact, an oft-quoted comment from a SAS executive on the "benefits" of  R goes something like “I think it addresses a niche market for high-end data analysts that want free, readily available code. We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet.” My take after 8 years of heavy R usage is that I've never worked with a more stable, bug-free piece of software.

Check out the full thread and contribute to the discussion on LinkedIn.

Information Management: LinkedIn Advanced Business Analytics – SAS Vs. R

Future of Open Source Survey 2011

If you work for a company that uses open-source software, or for a company that has a business based on open-source company, please take take a few minutes to respond to the the 2011 Future of Open Source Software Survey. This annual survey, now in its fifth year, is run by North Bridge Venture Partners, its open-source survey collaborators (including Revolution Analytics) abd open-source analyst firm The 451 Group to track trends in open-source issues as they relates to business.

A new emphasis in the survey this year is the connections between Open Source and the Cloud and Mobile applications, and on how revenue and licensing models are evolving accordingly. As always, the results from the survey will be shared at the open-source business conference, OSBC 2011. (Here are the results of the 2010 survey.) This survey will also be used to drive the discussion at the North Bridge keynote panel at OSBC on May 16, which this year will feature leaders from Acquia, Cloudera, Red Hat and J.P. Morgan.

The survey takes about 5 minutes to complete, and there's more information at the link below.

2011 Future of Open Source Forum: 5th Annual North Bridge Future of Open Source Survey 2011