Tutorial: Parallel programming with foreach

Exegetic Analytics extols the wonders of foreach package for iterative operations that go beyond the standard "for" loop in R. For example, here's a neat (if not optimally efficient) construct using filters to calculate the primes less than 100:

foreach(n = 1:100, .combine = c) %:% when (isPrime(n)) %do% n

The open-source team at Revolution Analytics created the foreach package with parallelization in mind, and the tutorial goes on to use the doMC and doSNOW packages to speed up the foreach-based loops by running the iterations in parallel on a cluster. You can find the code used to do so, and benchmarks of the results, at the blog post linked below.

Exegetic Analytics: The Wonders of foreach

Comments

[mike]

*parallelization

[David Smith]

@mike, thanks, corrected above

[Andy]

I found doSNOW very useful as a way to send multiple SQL queries with minor changes (date ranges, cohort definitions, etc) to a database simultaneously.

library(doSNOW)
cl <- makeCluster(c("localhost","localhost"), type = "SOCK")
registerDoSNOW(cl)

i = list(TYdates,LYdates)

result <- foreach(j = 1:2, .packages="RODBC")%dopar%{
ch=odbcConnect("data warehouse")
sqlQuery(ch, paste('Your SQL here', i[j]))
}

Instead of writing a script that runs everything in series.