Revolutions

Forecasting gentrification in city neighborhoods, with R

If you've lived in a big city, you're likely familiar with the impact of gentrification. For longtime residents of a neighbourhood, it can represent a decline in the culture and vibrancy of your community; for recent or prospective residents, it can represent a financial opportunity in rising home prices. For those that live in a gentrifying neighbourhood, it's one of those you-know-it-when-you-see-it things, but for economists and urban planners it can be difficult to identify. So a team of analysts at Urban Spatial to build a longitudinal model based on census tract data to quantify gentrification. Their motivation?

Neighborhoods change because people and capital are mobile and when new neighborhood demand emerges, incumbent residents rightfully worry about displacement.

Acknowledging these economic and social realities, policy makers have a responsibility balance economic development and equity. To that end, analytics can help us  understand how the wave of reinvestment moves across space and time and how to pinpoint neighborhoods where active interventions are needed today in order to avoid negative outcomes in the future.

They also provide a detailed data visualization tutorial showing how they used R to visualize the results, like this sequence of panels showing the dramatic rise in prices for homes in the Mission District of San Francisco over the last eight years.

The R full code behind this and other interesting charts is included, often making extensive use of the ggmap package.  

Zooming out from neighbourhoods to cities, the full report describes the analysts built a model to predict gentrification within "legacy" US cities (and even produced detailed maps within those cities showing where gentrification was likely to occur). At the city level, the impact is shown in rising or declining average housing values:

If you're into the economic aspect, check out the full report for lots of interesting analysis of housing trends in US cities. Or, if you just want to learn how all the charts and maps were made, click the link below to see the tutorial.

Urban Spatial: #Dataviz tutorial: Mapping San Francisco home prices using R (via Sharon Machlis)

ggraph: ggplot for graphs

A graph, a collection of nodes connected by edges, is just data. Whether it's a social network (where nodes are people, and edges are friend relationships), or a decision tree (where nodes are branch criteria or values, and edges decisions), the nature of the graph is easily represented in a data object. It might be represented as a matrix (where rows and columns are nodes, and elements mark whether an edge between them is present) or as a data frame (where each row is an edge, with columns representing the pair of connected nodes).

The trick comes in how you represent a graph visually; there are many different options each with strengths and weaknesses when it comes to interpretation. A graph with many nodes and edges may become an unintelligible hairball without careful arrangement, and including directionality or other attributes of edges or nodes can reveal insights about the data that wouldn't be apparent otherwise. There are many R packages for creating and displaying graphs (igraph is a popular one, and this CRAN task view lists many others) but that's a problem in its own right: an important part of the data exploration process is trying and comparing different visualization options, and the myriad packages and interfaces makes that process difficult for graph data.

Now, there's the new ggraph package,  recently published to CRAN by author Thomas Lin Pederson, which promises to make exploring graph data easier. Unlike other graphing packages, ggraph uses the grammar of graphics paradigm of the ggplot2 package, unifying the data structures and attributes associated with graphics. It also includes a wide range of visual representations of graphs — layouts — and makes it easy to switch between them. The basic "mesh" visualization of nodes and edges provides 11 different options for arranging the nodes:

Other types of visualizations are supported, too: hive plots, dendrograms, treemaps, and circle plots, to name just a few. Note that only static graphs are available, though: unlike igraph and some other packages, you can't rearrange the location of the nodes or otherwise manipulate the graphics with a mouse.

For the R programmer, most of the work is done by the ggraph function. It's analagous to the ggplot function, except that you don't provide data for the locations of the nodes; their position is selected by an algorithm. (Similarly, layout choices are automatically made for visualization types other than the mesh.) There are also various themes suited to graphs you can use to style your chart: goodbye gridlines and axes; hello labels, annotations and edge arrows.

The ggraph package is available on CRAN now, and works with R version 2.10 and later. For more on the ggraph package, see the announcement blog post linked below.

Data Imaginist: Announcing ggraph: A grammar of graphics for relational data

Because it's Friday: Looking down the Glory Hole

California, which has been in drought for for the past five years, has found some respite in heavy rains this winter. But while those same rains have caused flooding, mudslides, and a near-catastrophic release of water from a failing dam spillway, they have nonetheless replenished reservoirs and aquifers. As one example, take a look at this picture of Lake Berryessa in Northern California I took on January 17, 2011:

Photo credit: David Smith. CC BY-SA

That cylindrical object you see on the left is the bellmouth spillway of the Monticello Dam, known to locals as the "Glory Hole". It's essentially a big drainpipe that drains the lake to an outflow at the bottom of the dam when it exceeds its maximum level, which last happened in 2006. That is, until this month when the lake reached capacity once again. This drone footage shows the engineering of the Glory Hole in action:

That's all from us here at the blog for this week. We'll be back on Monday, but in the meantime have a great weekend!

Prophet: How Facebook operationalizes time series forecasting at scale

Facebook is a famously data-driven organization, and an important goal in any data science activity is forecasting. Now, Facebook has released Prophet, an open-source package for R and Python that implements the time-series methodology that Facebook uses in production for forecasting at scale.

Prophet has a very simple interface: you pass it a column of dates and a column of numbers, and is produces a forecast for the time series, like this:

The black dots are the number of views of Peyton Manning's Wikipedia page through the end of 2016; the blue region is a forecast (with uncertainty interval) into 2017. As you can see, the Prophet forecast automatically detects the seasonal cycles (presumably related to NFL seasons). The prophet function also provides options to explicitly model weekly and/or yearly seasonality, account for holidays, and to specify changepoints where discontinuities in the time series are expected. It also supports modeling logistic growth, where each data point is measured against a maximum possible capacity.

The underlying model behing the forecast is described in this paper. It's not your traditional ARIMA-style time series model. It's closer in spirit to a Bayesian-influenced generalized additive model, a regression of smooth terms. The model is resistant to the effects of outliers, and supports data collected over an irregular time scale (ingliding presence of missing data) without the need for interpolation. The underlying calculation engine is Stan; the R and Python packages simply provide a convenient interface.

Prophet is billed as a platform for "forecasting at scale", but here "scale" does not refer to data size or computational speed (as we are accustomed to in this domain). As authors Sean Taylor and Ben Letham write in the aforementioned paper:

The actual problems of scale we have observed in practice involve the complexity introduced by the variety of forecasting problems and building trust in a large number of forecasts once they have been produced.

With its simple interface that automates much of the process of finding a "best fit" characterization of a complex time series, Prophet scales easily to a large number of users who need many forecasts of qualitatively many different kinds of data, without in-depth expertise in time series modeling techniques. 

You can find out more about Prophet, including how to install it for R and Python, at the link below.

Facebook Open Source: Prophet

Preview: R Tools for Visual Studio 1.0

After more than a year in preview R Tools for Visual Studio, the open-source extension to the Visual Studio IDE for R programming, is nearing its official release. RTVS Release Candidate 2 1 is now available for download, giving you the opportunity to try out the new features ahead of the official announcement.

We'll cover the features in detail with the general availability release of RTVS 1.0, but in summary the new features include:

• Remote Execution: type R code in your local RTVS instance, but have the computations performed on a remote R server. You can also switch between local and remote workspaces at will.
• SQL Server Integration: work with database connections and SQL queries, and create stored procedures with embedded R code.
• Enhanced R Graphics Support: multiple floating and dockable plot windows, each with plot history.

RTVS works with all flavours of R on Windows: CRAN R, Microsoft R Open, and Microsoft R Client & Server. It requires Visual Studio 2015 (including the free Community edition). The RTVS team welcomes your feedback: you can report issues or offer suggestions via the RTVS Github repository. To get started with RTVS, follow the link below.

Update March 2: Release Candidate 2 is now available!

R Tools for Visual Studio: Welcome to R Tools for Visual Studio Preview!

The difference between R and Excel

If you're an Excel user (or any other spreadsheet, really), adapting to learn R can be hard. As this blog post by Gordon Shotwell explains, one of the reasons is that simple things can be harder to do in R than Excel. But it's worth perservering, because complex things can be easier.

While Excel (ahem) excels at things like arithmetic and tabulations (and some complex things too, like presentation), R's programmatic focus introduces concepts like data structures, iteration, and functions. Once you've made the investment in learning R, these abstractions make reducing complex tasks into discrete steps possible, and automating repeated similar tasks much easier. For the full, compelling argument, follow the link to the Yhat blog, below.

Yhat blog: R for Excel Users

Finding Radiohead's most depressing song, with R

Radiohead is known for having some fairly maudlin songs, but of all of their tracks, which is the most depressing? Data scientist and R enthusiast Charlie Thompson ranked all of their tracks according to a "gloom index", and created the following chart of gloominess for each of the band's nine studio albums. (Click for the interactive version, crated with with highcharter package for R, which allows you to explore individual tracks.)

If you're familiar with the albums, this looks pretty reasonable. Radiohead's debut, "Pablo Honey" was fairly poppy musically, but contained some pretty dark lyrics (especially in the break-out hit, Creep). Their most recent album "A Moon Shaped Pool" is a fantastic listen, but it isn't exactly going to get the party started. 

The "Gloom Index" charted above is a combination of three quantities, and then scaled from 1 (Radiohead's gloomiest song, True Love Waits), to 100 (the cheeriest, 15 Step).

The first quantity is Spotify's "valence score", which Spotify describes as a "quantity describing the musical positiveness of a track". Valence scores range from 0 (songs that sound depressed or angry) to 1 (songs that are happy or euphoric). Charlie extracted the list of Radiohead's 101 singles and the valence score for each from the Spotify developer API, using the httr package for R. This is useful in its own right, but several songs have the same valence score, so Charlie also looks at song lyrics to further differentiate them.

The second quantity is the percentage of words in the lyrics that that are "sad". Charlie scraped the song lyrics from Genius using the rvest package, and then used the tidytext package to break the lyrics into words, eliminate common "stop words" like 'the' and 'a', and count the number with negative sentiment.

The third quantity is the "lyrical density" (following a method described by Myles Harrison): the number of words per second, easily calculated from the Spotify track length data and the word counts calculated in the prior step.

The three quantities are combined together to create the "gloom index" as follows:

$$ \mathrm{gloomIndex} = \frac{1-\mathrm{valence}}{2} + \frac{\mathrm{pctSad}*(1+\mathrm{lyricalDensity})}{2} $$

Roughly, this is the average of the valence score and (almost) the number of sad words per second. (I'm guessing Charlie adds 1 to the lyrical density to get the two terms to about the same scale, so that both have about equal weight.)

It would be interesting to compare the "Gloom Index" for Radiohead with that for other famously downbeat artists (say, Bon Iver or Low). You'd need to so away with scaling the Gloom Index from 1 to 100 as Charlie has done here, but the formula could easily be adapted to make a universal score. If you'd like to give it a try, all of the R code is included in Charlie's blog post, linked below.

RCharlie: fitteR happieR

 

Because it's Friday: Et tu?

I spent 6 years learning to speak French as a student in Australia, so naturally I was excited to try out my language skills on my first visit to France. Inevitably, I could understand no-one, and no-one could understand me. (The Australian accent doesn't translate well, as it turns out.) But even after some months of practice, there were still two aspects of the language I dreaded every time I spoke: mixing up feminine and masculine articles (using le instead of la) or — much worse! — using the informal pronoun when I should have used the formal variant. In standard English, we refer to just about everyone as "you". (One exception is God, who gets the archaic 'thee'.) In French, whether you refer to someone as tu or vous is an indication of your relative status to them, and if you get it wrong, well ... quelle horreur! 

So I wish I'd had at the time this handy flowchart from the LA Times (via the French Embassy in NY):

That's all from us for this week. It's a long weekend for many of us here in the US, so we'll be back on Tuesday. Enjoy your (long) weekend!

Catterplots: Plots with cats

As a devotee of Tufte, I'm generally against chartjunk. Graphical elements that obscure interpretation of the data occasionally have a useful role to play, but more often than not that role is to entertain the expense of enlightenment, or worse, to actively mislead. So it's with mixed feelings that I refer you to catterplots, an R package by David Gibbs to create scatterplots using outlines of cats as the symbols.

But hey, it's the Friday before a long weekend, so here's a poorly annotated catterplot of households with dogs vs households with cats in the 48 continental US states (plus DC):

The data behind this chart I found at data.world, which looks to be an interesting new resource for data sets. (Registration is required to download data, however.) The code behind the chart is below. (If you want a variant featuring Nyan Cat, try cat=11.)

dogscats <- catplot(dogs, cats, size=0.1, catcolor=c(0.7,0.7,0.7,1),
                    cat=3,
                    xlab="Households with dogs",
                    ylab="Households with cats",
                    main="Household Pet Ownership in US States")

The catterplot package is available for on GitHub at the link below.

GitHub (GibbsDavidl): Catterplots

Six Articles on using R with SQL Server

Tomaž Kaštrun is developer and data analyst working for the IT group at SPAR (the ubiquitous European chain of convenience stores) in Austria. He blogs regularly about using Microsoft R and SQL Server for data analyis, and recently published a roundup of his articles about R and SQL Server.

Follow the link below for the an overview of the articles, which cover:

• using Microsoft R in enterprise environments,
• an introduction to SQL Server R Services,
• how to install R packages in SQL Server R Services,
• a tutorial on using SQL Server R Services to analyze sales data,
• using R to visualize data in Power BI and SQL Server Reporting Services,
• and using R to analyze data generated by SQL Server DBA activities.

TomazTSQL: R and SQL Server articles