Revolutions

More on the IBM vs SAS battle

Businessweek.com also enters the fray, with an online article shedding more light on SAS's reaction to IBM's entry into the data analytics market with its acquisition of SPSS:

In the past two years, IBM (IBM), SAP (SAP), and Oracle (ORCL) have bought companies that compete with SAS in providing software that can predict trends, identify profitable customers, reduce risk, or shave costs. [SAS CEO] Goodnight views the mergers as a chance to make hay. "We always look for any company that's acquired to be messed up for a year," Goodnight says in an interview.

The article also brings some interesting data about SAS pricing and market share:

SAS's growth comes partly because it can charge premium prices for its software, which lets statisticians predict future scenarios based on historical data. Versions of SAS products tailored for specific industries can start at $1 million to run a unit of a company, with annual subscription fees of 20% to 30% of the initial cost. "That makes the product somewhat pricey," compared with other vendors', says Bill Hostmann, an analyst at market research company Gartner (IT).

Sales of data analysis software increased 22%, to $8.8 billion, in 2008, according to Gartner. SAS had about 15% of the market, the same as Oracle's share. SAP held a 24% share of the market and IBM 11%.

As is becoming routine in these stories, R gets a mention too:

At the low end of the market, SAS also is contending with the popularity of an open source programming language called R, which is used by Google (GOOG), Stanford University, and other prominent organizations to build statistical applications.

BusinessWeek.com: IBM vs. SAS: The Battle over Data Analysis Software

Wondrous oddities of R

R is a distinctive language, and one of the ways that is distinguishes itself from other language is the way function calls are handled. Function arguments are named, and can be provided out of order or even not at all (in which case default values can be used). Lazy evaluation and R's scoping rules can result in some elegant (or surprising, depending on your point of view) code. Tom Moertel delves into all these aspects of R's function call semantics in this interesting post from 2006. Some fascinating nuggets in there: did you know you can abbreviate argument names ambiguously as long as the other clashing argument names are bound?

Tom Moertel's Weblog: Wondrous oddities: R's function-call semantics

Happy Thanksgiving

To all our readers in the US, have a happy Thanksgiving day. Revolutions is taking the day off for the holiday. And to all our readers outside the US, apologies for all the national holidays we've failed to recognize to date and in the future. :)

Competition to SAS from yet another angle

Steve Miller at Information Management discusses the NYT story about SAS from last weekend, and notes that open-source R and major new players in the commercial BI space aren't the only threats to SAS's revenues. SAS is also facing competition from a low-priced commercial clone: WPS, created by a small software company based in the UK. WPS allows analysts to run SAS code and replicate the outputs without the need for a SAS license. And much like SAS has recently added facilities to call out to R, WPS can also interface with R thanks to the "Bridge to R" software created by third-party (or is it fourth-party by this stage?) software firm.

SAS clearly sees WPS as a threat: they recently sued WPS in the High Court in London, and the suit seeks to halt the sales of the WPS software. Interestingly, the lawsuit regards not the duplication of the SAS language itself, but instead relates to a violation of the terms of a student license of SAS:

“The lawsuit alleges that World Programming has used the SAS Learning Edition product for purposes of developing, testing and benchmarking its World Programming System software in breach of the terms of the SAS Learning Edition Licensing Agreement,” said John Boswell, Vice President and General Counsel at SAS. 

WPS is fighting back:

"SAS are claiming that the way in which WP has used a $100 learning package bought on Amazon was not permitted by the terms of the click wrap license.

SAS is alleging that WP have run "the SAS Learning Edition software for the purposes of observing its operation and functions in order to develop the WPS software in a way which reproduces the operation and functions of the SAS System". European law provides that one can lawfully observe the operation and functioning of a computer program and develop interoperable or alternative solutions which perform the same function. So far as we can see, WP have honoured both the spirit and the letter of the law. We are surprised that a company of SAS’s repute would want to pursue this action"

The case continues.

Information Management: More on the Statistical Revolution – a SAS Story (continued)

Mythbusters at the White House

I'm really happy to hear that the Mythbusters team got some presidential recognition yesterday. It's just nice to see promotion of science education (and the Mythbusters' entertaining ways of illustrating the scientific method) getting such prominence.

Wired: Mythbusters at the White House!

Promoting open government in Brazil with R

I met R hacker Eduardo Leoni at the DC R User Group (which is still looking for a new organizer, by the way) last month, where he gave an impromptu lightning talk on the open-government website he created in his spare time, CongressoAberto.com.br. With a name that means "Open Congress" in Portuguese, it is an independent website that aims to increase transparency and contribute to debates about the Brazilian legislature, facilitating access to information and analysis on the Brazilian political process. Naturally, the site is in Portuguese, but Google Translate does a good job of rendering the site in English for non-native speakers like me.

Much of the site is devoted to sharing official government data with Brazilians in an easily-understood format, giving insight into how and why political decisions are made. For example, Eduardo has written scripts in R that analyzes the votes on particular bills (for example, this one) and displays the legislators' votes by party and on a regional map:

Although other open-government sites exist in many countries (for example, the Sunlight Foundation here in the US), the Brazilian political system is different in that there are many important political parties instead of just two major ones. As a result, Eduardo had to create all of the analyzes and charts for Brazilian government from scratch. Luckily, the tools provided by R to scrape data from unwieldy government websites and to create color-coded maps (to name just a couple of features) make this process easier than it would be with closed-source tools.

CongressoAberto.com.br: Home Page
 

NYT: SAS facing stiff competition

In a front-page article in Sunday's business section, the New York Times takes a look at SAS and how it is increasingly facing competition from both proprietary and open-source alternatives. On the commercial side, more and more vendors are moving into the predictive analytics market eyeing SAS's enviable recurring subscription revenues:

The company also faces the classic challenge of being the innovative pioneer — enjoying rich profit margins but facing new competition from rivals seeking to gain market share with lower prices and substitute technology. [...]

The competitive thrust that really grabbed SAS’s attention came in late July, when I.B.M. announced that it planned to pay $1.2 billion for SPSS, a maker of predictive modeling software. I.B.M. has placed SPSS and Cognos into a new business analytics and optimization group. That business will be supported by 200 scientists, and the company has said it will retrain or hire 4,000 consultants and analysts to work in the group.

But competition is coming from the other direction, too: open-source. The article specifically mentions R, which competes with SAS to offer high-end statistical and predictive analyses. (Incidentally, the New York Times covered R in detail back in January.) Until recently, SAS dismissed R as a threat:

To be sure, the corporate cocoon in Cary can breed insularity. SAS, for example, was slow to recognize the brewing challenge from free, open-source alternatives to some of its products. A free programming language and set of software tools for statistical computing, called R, has become increasingly popular at universities and labs.

The company shifted course earlier this year and modified its software so programs written with R work seamlessly with SAS technology. “Shame on us for not engaging more with the open-source community,” says Keith Collins, senior vice president and chief technology officer. “But we’re committed to doing that now.”

In my experience at REvolution, that's true: I've definitely seen SAS take much more of an interest in R over the past year or so.

New York Times: At a Software Powerhouse, the Good Life Is Under Siege

The Senate, ObamaCare, the NYT, and R

The New York Times on Wednesday published an editorial by Andrew Gelman, Nate Silver and Daniel Lee on the motivations of the 100 US Senators with regard to their opinion on the healthcare legislation currently under debate. Using multilevel regression and post stratification, they reveal that many senators' motivations may be linked more closely to the President's popularity in their state (and, by proxy, their chances of re-election) than the wishes or needs of their constituents.

On his blog, Gelman has posted an interesting behind-the-scenes look on what it takes to make a statistical analysis into a political op-ed. I was particularly interested to see how the graphics department converted this chart created with R:

 
 into this graphic:

 

Usually it's a bit more obvious when R is used as the basis of an NYT graphic, but it's interesting here how they've taken the raw data and presented it in a more accessible fashion. It's also very cool to see how widely R is being used in political science discussions -- see also this analysis of the controversial NY-23 congressional race by Boris Shor that was cited in the NYT and Mother Jones.

Update: Awesome! Boris Shor's NY-23 analysis was done in REvolution R Enterprise for 64-bit Windows. It was the ability to handle larger data sets made the analysis possible.

New York Times: The Senate's Health Care Calculations

Because it's Friday: Brute Force

Can a brute-force solution to the Riemann Hypothesis ever be sufficient?

Abstruse Goose: We Give Up

Breast cancer and Bayes' Rule

Mike Anderson at Harvard's Social Science Statistics Blog looks at the recent change in recommendations for women with regard to breast cancer screening from the perspective of Bayes' Theorem. This calculation was particularly interesting:

So if you're a woman in your 40s, and you get a positive result from your latest mammogram, you'll still only have an 11.9% probability of having breast cancer -- basically the same as a woman's lifetime risk of 12%. 

Of course none of this takes into account the myriad other factors (cost of a false positive or negative, improved understanding of risk factors, etc.) but it does shed some light on the controversial recommendation change.

Update Nov 20: As noted here in the comments, the linked post has been removed, possibly due to an error in its calculations.

Update Dec 3: The post has since been replaced.

Social Science Statistics Blog: breast cancer, mammograms, rare diseases, and bayes rule (via @drewconway)