« November 2017 | Main | January 2018 »
The Revolutions team is celebrating Christmas today, and we're taking a break with family and enjoying good food. And given the number of Eggnogs that are being prepared — thanks to Hadley Wickham's eggnogr Shiny app — it might be a good idea to take the rest of the week off as well. (You can find the R source behind the eggnogr app here.)
We'll be back again in the New Year, on January 2. Enjoy the holiday season, and a happy New Year to all.
Sure, this is a promo for a movie, but I'd love to have a full-length single of this:
Relatedly, if you want to settle an argument about which pop diva has the greatest vocal range, Giora Simchoni used R to perform frequency analysis of their hits:
That's all from us here at the blog for this week, and in fact for a little while: we're taking a break for the holidays. We'll be back on January 2, and in the meantime, enjoy!
The main international R user conference, useR!, is typically hosted in Europe every other year. Back in 2016 when useR! was held in the USA, an alternative conference was held in Poland. Next year (when the useR! conference will be in Australia) the tradition continues, and the 2018 eRum (European R Users Meeting) will be held in Budapest, Hungary on May 14-16.
Around 400-500 attendees are expected, and the program committee is sure to put together an excellent collection of workshops and talks. If you'd like to contribute, workshop proposals are due on January 14, and abstracts for talks and posters are due on February 25. You can submit your proposals here.
Conference registration will open in early January, but you can follow @erum2018 on Twitter to be notified or simply check at the URL below for ticket information.
eRum 2018: About the Conference
The winners of the American Statistical Association Police Data Challenge have been announced. The ASA teamed up with the Police Data Initiative, which provides open data from local law enforcement agencies in the US, to create a competition for high school and college students to analyze crime data from Baltimore, Seattle and Cincinnati. In this post, we take a look at the winners of the visualization category.
The winners of the Best Visualization for college students were Julia Nguyen, Katherine Qian, Youbeen Shim, Catherine Sun from University of Virginia. Their entry included several visualizations of crime data in Baltimore, including the crime density map shown below. The team used R for all of the data cleaning, manipulation, and visualizations. The tidyverse suite of packages was used for data pipelining (including stringr and lubridate for merging data on latitude/longitude and date), and the ggmap package for visualization.
The winners of the Best Visualization for high school students were Alex Lapuente, Ana Kenefick and Sara Kenefick from Charlotte Latin School (Charlotte, N.C.). They used Microsoft Excel to look at overall trends Seattle crime data, the impact of employment and poverty on crime, and this visualization of the frequency of traffic-related incidents (note the "pedestrian violation" segment — I can attest from experience that jaywalking is strictly enforced in Seattle!):
For more on the Police Data Challenge and the winners in the Overall and Best Use of External Data categories, follow the link below.
This is Statistics: Police Data Challenge: Congratulations to our Winners!
by John Mount, Win-Vector LLC
seplyr is an R package
that supplies improved standard evaluation interfaces for many common data wrangling tasks.
The core of seplyr is a
re-skinning of dplyr's
functionality to seplyr conventions (similar to how stringr
re-skins the implementing package stringi).
"Standard evaluation" is the name we are using for the value oriented calling convention found in many programming languages. The idea is: functions are only allowed to look at the values of their arguments and not how those values arise (i.e., they can not look at source code or variable names). This evaluation principle allows one to transform, optimize, and reason about code.
It is what lets us say the following two snippets of code are equivalent.
x <- 4; sqrt(x)x <- 4; sqrt(4)The mantra is:
"variables can be replaced with their values."
Which is called referential transparency.
"Non-standard evaluation" is the name used for code that more aggressively inspects its environment. It is often used for harmless tasks such as conveniently setting axis labels on plots. For example, notice the following two plots have different y-axis labels (despite plotting identical values).
plot(x = 1:3)
plot(x = c(1,2,3))
dplyr and seplyrThe dplyr authors appear to strongly prefer a non-standard evaluation interface. Many in the dplyr community have come to think a package such as dplyr requires a non-standard interface. seplyr started as an experiment to show this is not actually the case.
Syntactically the packages are deliberately similar.
We can take a dplyr pipeline:
suppressPackageStartupMessages(library("dplyr"))
starwars %>%
select(name, height, mass) %>%
arrange(desc(height)) %>%
head()
## # A tibble: 6 x 3
## name height mass
## <chr> <int> <dbl>
## 1 Yarael Poof 264 NA
## 2 Tarfful 234 136
## 3 Lama Su 229 88
## 4 Chewbacca 228 112
## 5 Roos Tarpals 224 82
## 6 Grievous 216 159
And re-write it in seplyr notation:
library("seplyr")
starwars %.>%
select_se(., c("name", "height", "mass")) %.>%
arrange_se(., "desc(height)") %.>%
head(.)
## # A tibble: 6 x 3
## name height mass
## <chr> <int> <dbl>
## 1 Yarael Poof 264 NA
## 2 Tarfful 234 136
## 3 Lama Su 229 88
## 4 Chewbacca 228 112
## 5 Roos Tarpals 224 82
## 6 Grievous 216 159
For the common dplyr-verbs (excluding mutate(), which we will discuss next) all the non-standard evaluation is saving us is a few quote marks and array designations (and we have ways of getting rid of the need for quote marks). In exchange for this small benefit the non-standard evaluation is needlessly hard to program over. For instance in the seplyr pipeline it is easy to accept the list of columns from an outside source as a simple array of names.
Until you introduce a substitution system such as rlang or wrapr::let() (which we recommend over rlang and publicly pre-dates the public release of rlang) you have some difficulty writing re-usable programs that use the dplyr verbs over "to be specified later" column names.
We are presumably not the only ones who considered this a limitation:
seplyr is an attempt to make programming a primary concern by
making the value-oriented (standard) interfaces the primary interfaces.
mutate()The earlier "standard evaluation costs just a few quotes" becomes a bit strained when we talk about the dplyr::mutate() operator. It doesn't seem worth the effort unless you get something more in return. In seplyr 0.5.0 we introduced "the something more": planning over and optimizing dplyr::mutate() sequences.
A seplyr mutate looks like the following:
select_se(., c("name", "height", "mass")) %.>%
mutate_se(., c(
"height" := "height + 1",
"mass" := "mass + 1",
"height" := "height + 2",
"mass" := "mass + 2",
"height" := "height + 3",
"mass" := "mass + 3"
)) %.>%
arrange_se(., "name") %.>%
head(.)
## # A tibble: 6 x 3
## name height mass
## <chr> <dbl> <dbl>
## 1 Ackbar 186 89
## 2 Adi Gallia 190 56
## 3 Anakin Skywalker 194 90
## 4 Arvel Crynyd NA NA
## 5 Ayla Secura 184 61
## 6 Bail Prestor Organa 197 NA
seplyr::mutate_se() always uses ":=" to denote assignment (dplyr::mutate() prefers "=" for assignment, except in cases where ":=" is required).
The advantage is: once we are go to the trouble to capture the mutate expressions we can treat them as data and apply procedures to them. For example we can re-group and optimize the mutate assignments.
plan <- partition_mutate_se(
c("name" := "tolower(name)",
"height" := "height + 0.5",
"height" := "floor(height)",
"mass" := "mass + 0.5",
"mass" := "floor(mass)"))
print(plan)
## $group00001
## name height mass
## "tolower(name)" "height + 0.5" "mass + 0.5"
##
## $group00002
## height mass
## "floor(height)" "floor(mass)"
Notice seplyr::partition_mutate_se() re-ordered and re-grouped the assignments so that:
The "safe block" assignments can then be used in a pipeline:
starwars %.>%
select_se(., c("name", "height", "mass")) %.>%
mutate_seb(., plan) %.>%
arrange_se(., "name") %.>%
head(.)
## # A tibble: 6 x 3
## name height mass
## <chr> <dbl> <dbl>
## 1 ackbar 180 83
## 2 adi gallia 184 50
## 3 anakin skywalker 188 84
## 4 arvel crynyd NA NA
## 5 ayla secura 178 55
## 6 bail prestor organa 191 NA
This may not seem like much. However, when using dplyr with a SQL database (such as PostgreSQL or even Sparklyr) keeping the number of dependencies in a block low is critical for correct calculation (which is why I recommend keeping dependencies low). Furthermore, on Sparklyr sequences of mutates are simulated by nesting of SQL statements, so you must also keep the number of mutates at a moderate level (i.e., you want a minimal number of blocks or groups).
Because we are representing mutate assignments as user manipulable data we can also enjoy the benefit of machine generated code. seplyr 0.5.* uses this opportunity to introduce a simple function named if_else_device(). This device uses R's ifelse() statement (which conditionally chooses values in a vectorized form) to implement a more powerful block-if/else statement (which conditionally simultaneously controls blocks of values and assignments; SAS has such a feature).
For example: suppose we want to NA-out one of height or mass for each row of the starwars data uniformly at random. This can be written naturally using the if_else_device.
if_else_device(
testexpr = "runif(n())>=0.5",
thenexprs = "height" := "NA",
elseexprs = "mass" := "NA")
## ifebtest_30etsitqqutk
## "runif(n())>=0.5"
## height
## "ifelse( ifebtest_30etsitqqutk, NA, height)"
## mass
## "ifelse( !( ifebtest_30etsitqqutk ), NA, mass)"
Notice the if_else_device translates the user code into a sequence of dplyr::mutate() expressions (using only the weaker operator ifelse()). Obviously the user could perform this translation, but if_else_device automates the record keeping and can even be nested. Also many such steps can be chained together and broken into a minimal sequence of blocks by partition_mutate_se() (not forcing a new dplyr::mutate() step for each if-block encountered).
When we combine the device with the partitioned we get performant database-safe code where the number of blocks is only the level of variable dependence (and not the possibly much larger number of initial value uses that a straightforward non-reordering split would give; note: seplyr::mutate_se() 0.5.1 and later incorporate the partition_mutate_se() in mutate_se()).
starwars %.>%
select_se(., c("name", "height", "mass")) %.>%
mutate_se(., if_else_device(
testexpr = "runif(n())>=0.5",
thenexprs = "height" := "NA",
elseexprs = "mass" := "NA")) %.>%
arrange_se(., "name") %.>%
head(.)
## # A tibble: 6 x 4
## name height mass ifebtest_wwr9k0bq4v04
## <chr> <int> <dbl> <lgl>
## 1 Ackbar NA 83 TRUE
## 2 Adi Gallia 184 NA FALSE
## 3 Anakin Skywalker NA 84 TRUE
## 4 Arvel Crynyd NA NA TRUE
## 5 Ayla Secura 178 NA FALSE
## 6 Bail Prestor Organa 191 NA FALSE
The value oriented notation is a bit clunkier, but this is offset by it's greater flexibility in terms of composition and working parametrically.
Our group has been using seplyr::if_else_device() and seplyr::partition_mutate_se() to greatly simplify porting powerful SAS procedures to R/Sparklyr/Apache Spark clusters.
This is new code, but we are striving to supply sufficient initial documentation and examples.
R has been available for Windows since the very beginning, but if you have a Windows machine and want to use R within a Linux ecosystem, that's easy to do with the new Fall Creator's Update (version 1709). If you need access to the gcc toolchain for building R packages, or simply prefer the bash environment, it's easy to get things up and running.
Once you have things set up, you can launch a bash shell and run R at the terminal like you would in any Linux system. And that's because this is a Linux system: the Windows Subsystem for Linux is a complete Linux distribution running within Windows. This page provides the details on installing Linux on Windows, but here are the basic steps you need and how to get the latest version of R up and running within it.
First, Enable the Windows Subsystem for Linux option. Go to Control Panel > Programs > Turn Windows Features on or off (or just type "Windows Features" into the search box), and select the "Windows Subsystem for Linux" option. You'll need to reboot, just this once.
Next, you'll need to install your preferred distribution of Linux from the Microsoft Store. If you search for "Linux" in the store, you'll find an entry "Run Linux on Windows" which will provide you with the available distributions. I'm using "Ubuntu", which as of this writing is Ubuntu 16.04 (Xenial Xerus).
Once that's installed you can launch Ubuntu from the Start menu (just like any other app) to open a new bash shell window. The first time you launch, it will take a few minutes to install various components, and you'll also need to create a username and password. This is your Linux username, different from your Windows username. You'll automatically log in when you launch new Ubuntu sessions, but make sure you remember the password — you'll need it later.
From here you can go ahead and install R, but if you use the default Ubuntu repository you'll get an old version of R (R 3.2.3, from 2015). You probably want the latest version of R, so add CRAN as a new package repository for Ubuntu. You'll need to run these three commands as root, so enter the password you created above here if requested:
sudo echo "deb http://cloud.r-project.org/bin/linux/ubuntu xenial/" | sudo tee -a /etc/apt/sources.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E084DAB9
sudo apt-get update
(Don't be surprised by the message key E084DAB9: public key "Michael Rutter <[email protected]>" imported. That's how Ubuntu signs the R packages.)
Now you're all set to install the latest version of R, which can be done with:
sudo apt-get install r-baseAnd that's it! (Once all the dependencies install, anyway, which can take a while the first time.) Now you're all ready to run R from the Linux command line:
Note that you can access files on your Windows system from R — you'll find them at /mnt/c/Users/<your-Windows-username>. This FAQ on the WSL provides other useful tips, and for complete details refer to the Windows for Linux Subsystem Documentation.
Update Dec 14: Jeroen Ooms shares that this works for running RStudio Server and OpenCPU Server as well.
Wow the Windows Subsystem for Linux totally works. Installed both opencpu-server and rstudio-server on Windows 10 using standard Ubuntu installers from https://t.co/clTdvM7ezt. Thanks to @revodavid for pointing this out at https://t.co/iTea5cc5hZ pic.twitter.com/2gyLPVquT0
— Jeroen Ooms (@opencpu) December 14, 2017
A movie is said to satisfy the Bechdel Test if it satisfies the following three criteria:
The website BechdelTest.com scores movies accordingly, granting one point for each of the criteria above for a maximum of three points. The recent Wonder Woman movie scores the full three points (Diana and her mother discuss war, for example), while Dunkirk, which features no named female characters, gets zero. (It was still a great film, however.)
The website also offers an API, which enabled data and analytics director Austin Wehrwein to create this time series chart of Bechdel scores for movies listed on BechdelTest.com:
This chart only includes ratings for that subset of movies listed on Bechdeltest.com, so it's not clear whether this is representative of movies as a whole. Austin suggests combining these data with the broader data from IDMB, so maybe someone wants to give that a try. Austin's R code for generating the above chart and several others is available at the link below, so click through for the full analysis.
Austin Wehrwein: A quick look at Bechdel test data (& an awtools update) (via Mara Averick)
Follow David on Twitter: @revodavid
