Revolutions

Where to find the worst weather in the US

Which US city has the worst weather? To answer that question, data analyst Taras Kaduk counted the number of pleasant days in each city and ranked them accordingly. For this analysis, a "pleasant" day is one where the average temperature was in the range 55°F-75°F, the maximum was in the range 60°-90°, the minimum was in the range 40°-70°, and there was no significant rain or snow. (The weather data was provided by the NOAA Global Surface Summary of the Day dataset and downloaded to R with the rnoaa package.)

With those criteria and focusing just on the metro regions with more than 1 million people, the cities with the fewest pleasant days are:

1. San Juan / Carolina / Caguas, Puerto Rico (hot year-round)
2. Rochester, NY (cold in the winter, rain in the summer)
3. Detroit / Warren / Dearborn, MI (cold in the winter, rain in the summer)

You can see the top (bottom?) 25 cities in this list in the elegant chart below (also by Tara Kaduk), which shows each city as a polar bar chart and with one ring for each of the 6 years of data analyzed. 

And if you're wondering which cities have the best weather, here's the corresponding chart for the 25 cities with the most pleasant days. San Diego / Carlsbad (CA) tops that list.

You can find the R code behind the analysis and charts in this Github repository. (The polar charts above required a surprisingly small amount of code: it's a simple transformation of a regular bar chart with the ggplot2 coord_polar transformation — quite appropriate given the annual cycle of weather data.) And for the full description of the analysis including some other nice graphical representations of the data, check out the blog post linked below.

Taras Kaduk: Where are the places with the best (and the worst) weather in the United States

Working with US Census Data in R

If you need data about the American populace, there's no source more canonical than the US Census Bureau. The bureau publishes a wide range of public sets, and not just from the main Census conducted every 10 years: there are more than 100 additional surveys and programs published as well. To help R users access this rich source of data, Ari Lamstein and Logan Powell have published A Guide to Working with US Census Data in R, a publication of the R Consortium Census Working Group.

The guide provides an overview of the data available from the US census bureau and various tools available in R to access and analyze it. The guide notes that there are 22 R packages for working with census data, and cites as being particularly useful:

• tigris, for working with shape files of census regions (census data is may be aggregated to any of a number of levels as shown int the diagram below)
• acs, for downloading and managing data from the decennial census and the American Community Survey
• choroplethr and choroplethrMaps, for mapping data (including census data) by region
• tidycensus, to extract census data as tidy data frames
• censusapi, for extracting data using the Census API
• ipumsr, to extract US census data in a form that can be compared with data from other countries 

Standard Hierarchy of Census Geographic Entities, US Census Bureau.

You can find the complete guide at the link below.

R Consortium: A Guide to Working with US Census Data in R

The City of Chicago uses R to issue beach safety alerts

Among the many interesting talks I saw a the Domino Data Science Pop-Up in Chicago earlier this week was the presentation by Gene Lynes and Nick Lucius from the City of Chicago. The City of Chicago Tech Plan encourages smart communities and open government, and as part of that initiative the city has undertaken dozens of open-source, open-data projects in areas such as food safety inspections, preventing the spread of West Nile virus, and keeping sidewalks clear of snow. 

This talk was on the Clear Water initiative, a project to monitor the water quality of Chicago's many public beaches on Lake Michigan, and to issue safety alerts (or in serious cases, beach closures) when E Coli levels in the water get too high. The problem is that E Coli levels can change rapidly: water levels can be normal for weeks, and then spike for a single day. But traditional culture tests take many days to return results, and while rapid DNA-based tests do exist, it's not possible conduct these tests daily at every beach.

The solution is to build a predictive model, which uses meteorological data and rapid DNA tests for some beaches, combined with historical (culture-based) evaluations of water quality, to predict E Coli levels at all beaches every day. The analysis is performed using R (you can find the R code at this Github repository).

The analysis was developed in conjunction with citizen scientists at Chi Hack Night and statisticians from DePaul University. In 2017, the model was piloted in production in Chicago to issue beach safety alerts and to create a live map of beach water quality. This new R-based model predicted 60 additional occurrences of poor water quality, compared with the process used in prior years.

Still, water quality is hard to predict: once you have the slower test data and an actual result to compare with, that's an accuracy rate of 38%, with fewer than 2% false alarms. (The city plans to use clustering techniques to further improve that number.) That model uses rapid DNA testing at five beaches to predict all beaches along Lake Michigan. A Shiny app (linked below) lets you explore the impact of testing at a different set of beaches, and adjusting the false positive rate, on the overall accuracy of the model.

You can find more details about the City of Chicago Clear Water initiative at the link below.

City of Chicago: Clear Water

Hurricane Irma's rains, visualized with R

The USGS has followed up their visualization of Hurricane Harvey rainfalls with an updated version of the animation, this time showing the rain and flooding from Hurricane Irma in Florida:

Another #rstats #dataviz! Precip and #flooding from #HurricaneIrma 💧 #opensource code: https://t.co/rpocPQe7zR #openscience pic.twitter.com/rGX1SNiYEM

— USGS R Community (@USGS_R) September 15, 2017

This iteration improves the Harvey version by displaying rainfall in a fine grid over the state rather than on a county-by-county basis. Once again you can find the R code and data on Github, and an interactive version of the chart is available at the link below.

USGS Vizlab: Hurricane Irma's Water Footprint

UK government using R to modernize reporting of official statistics

Like all governments, the UK government is responsible for producing reports of official statistics on an ongoing basis. That process has traditionally been a highly manual one: extract data from government systems, load it into a mainframe statistical analysis tool and run models and forecasts, extract the results to a spreadsheet to prepare data for presentation, and ultimately combine it all in a manual document editing tool to produce the final report. The process in the UK looks much like this today:

Matt Upson, a Data Scientist at the UK Government Digital Service, is looking to modernize this process with a reproducible analytical pipeline. This new process, based on the UK Government's Technology Service Manual for new IT deployments, aims to simplify the process by using R — the open-source programming language for statistical analysis — to automate the data extraction, analysis, and document generation tasks.

The development of the new process is underway now, with the first target being a report on culture and sporting impacts on the UK economy:

Towards the end of 2016 we embarked on a project with a team in the Department for Culture, Media, and Sport (DCMS) who are responsible for the production of the Economic Estimates for DCMS Sectors Statistical First Release (SFR). Currently this publication is produced with a mix of manual and semi-manual processes. Our aim was to see if we could speed up production of the SFR, whilst maintaining the high standard of the publication and QA.

Central to the process is automating the report with Rmarkdown, a system for programmatically laying out a document while encapsulating all of the R code for data preparation, analysis, tabulation and charting into a collaborative document. As a comprehensive language, R has the functionality to handle all of these tasks natively, without the need to bring other systems into the chain. And it's an ideal tool for the statistical data analysis required for these reports, which must follow the guidance of the Aqua Book, the UK government's manual for producing quality analysis. (The Aqua Book is well worth reading for any data scientist, with excellent guidance on designing studies in the face of uncertain data.) 

Naturally, the government has specific standards for the presentation of data, and the R-based process needs to be able to reproduce that look in the final report. The GDS team has produced a package for R, called govstyle, which standardizes the look of data charts while upgrading them to more modern design principles. For example this chart from a report on truancy in UK schools:

has been upgraded and reproduced in R to look like this:

An automated workflow allows for the inclusion of modern devops processes, too. Dependency management for R packages is handled with packrat. Test-driven development comes courtesy of the testthat package, and automated testing (including data verification and code coverage analysis) is provided by Travis CI. And source code control and collaboration is provided by Github, which is also where the entire process is documented as R package: the eesectors package.

For more on the UK government's use of R for official statistics reporting, see the post below from the UK Government Digital Service. 

Data at GDS: Reproducible Analytical Pipeline

Education Analytics with R and Cortana Intelligence Suite

By Fang Zhou, Microsoft Data Scientist; Hong Ooi, Microsoft Senior Data Scientist; and Graham Williams, Microsoft Director of Data Science

Education is a relatively late adopter of predictive analytics and machine learning as a management tool. A keen desire for improving educational outcomes for society is now leading universities and governments to perform student predictive analytics to provide better-informed and timely decision making.

Student predictive analytics often aims to solve two key problems:

1. Predict student academic outcomes so as to better target support.
2. Predict students at risk of dropping out so as to prevent attrition.

Education systems face enormous diversity across regions and countries. Two case studies demonstrate the novel and unique landscape for machine learning in the education world.

• A mixed effects regression model has been developed in conjunction with an Australian education department to measure the influence of student characteristics and to predict student test scores in the presence of variation across students and schools. The model was implemented using R and then integrated with Azure Machine Learning for deployment to production through Power BI.
• A predictive model for student drop out has been developed in conjunction with an Indian state government using machine learning two-class boosted decision trees. For deployment an end-to-end pipeline was built using Azure services including Azure SQL Database, Azure ML and Azure Data Factory

Microsoft Data Scientists assisted with the analysis in both cases and we present details below with R code provided in a git repository to replicate the modelling on artificial data.

Student Performance

 

How the State of Indiana uses R and Azure to forecast employment

"Big Data" generates a lot of news these days, but sometimes small data still means big computation.

Indiana's Department of Workforce Development has the responsibility to forecast future employment rates in the State of Indiana. And not just the number of jobs available: the department also needs to forecast the types of jobs that will be available, so the administration can link up future job requirements with training and education policies and make sure a skilled workforce can fill the future demand. 

The State of Indiana contracted with analytics professional services firm Inquidia Consulting to develop a system to generate the forecast, as part of the "Demand-Driven Workforce System" project. After assembling a database of employment data, the team quickly found that this wasn't exactly a big-data problem: each forecast was based on just 1kb to 3kb of data, and could easily be computed using the R language in less than 2 seconds. The problem was that one the combinations of counties and selected variables were taken into account, millions of different forecasts needed to be made, immediately making this a problem that demanded significant of computing time.

Fortunately, this is exactly the kind of problem that can be tackled using parallel programming: since each of the models can be computed independently, a large number of machines can compute models simultaneously and dramatically reduce the required computation time. Using Microsoft Azure, the team could spin up hundreds of virtual machines to perform the R processing, orchestrated by a PDI/carte process to implement a simple (yet reliable) fault-tolerant distributed system. (The team considered using HDInsight and Spark for the distribution framework, but decided given the small data sizes that it would be overkill for their needs.)

For the actual forecasts, the Inquidia team used R's forecast package, noting that it offers "a robust set of options for projecting time series" and "allows for easy projections from any set of models for an arbitrarily large forecast period".  However, vector autoregressive models can only accomodate a relatively small number of variables and there are a large number of macroeconomic variables to choose from to forecast employment. This created a difficult variable-selection problem, which the team solved with a brute-force model forecasting and evaluation process.

Finally, to present the forecasts to State of Indiana administrators, the team created a shinydashboard library. This dashboard gave administrators flexbility to customize the forecasts by adjusting for seasonality, combining multiple models, and including new employment data. 

As a result, the Indiana DWD now receives a much more finely grained and accurate analysis of job skills and needs in Indiana, and can now view details at the county and metro level, without waiting for months. This is thanks to the system build with R on Microsoft Azure: a computation engine capable of producing more than 1,000 models per second at a fraction of the cost of an on-premises configuration.

To learn more about the forecasting system Inquidia Consulting build for the Indiana DWD, you can watch this archived webinar presented by the project leads. Or check out the white paper linked below, which gives details about the underlying architectute and implementation.

Microsoft Advanced Analytics and IoT: White Paper: Dive deep into small data with big data techniques 

Using R to simulate the finances of public sector pension funds

by Don Boyd, Senior Fellow, Rockefeller Institute of Government

The Rockefeller Institute of Government is excited to be developing models to simulate the finances of public pension funds, using R.

Public pension funds invest contributions from governments and public sector workers in an effort to ensure that they can pay all promised benefits when due. State and local government pension funds in the United States currently have more than $3 trillion invested, more than $2 trillion of which is in equity-like investments. For example, NYC, has over $158 billion invested. Governments usually act as a backstop: if pension fund investment returns do better than expected, governments will be able to contribute less, but if investment returns fall short they will have to contribute more. When that happens, politicians must raise taxes or cut spending programs. These risks often are not well understood or widely discussed. (For a discussion of many of the most significant issues, see Strengthening the Security of Public Sector Defined Benefit Plans.)

We are building stochastic simulation models in R to help quantify the investment risks and their potential consequences. We are modeling the finances of specific pension plans, taking into account all of the main flows such as current and expected benefit payouts to workers, contributions from governments and from workers, and investment returns, and how they affect liabilities and investible assets. The models will take into account the changing demographics of the workforce and retiree populations. We are modeling investment returns stochastically, examining different return scenarios and different economic environments, as well as different governmental contribution policies. We will use these models to evaluate the risks currently being taken and to help provide policy advice to governments, pension funds, and others. (For a full description of our approach, see Modeling and Disclosing Public Pension Fund Risk, and Consequences for Pension Funding Security)

We have chosen R because:

• It is extremely flexible, allowing us to do data collection, data management, exploratory data analysis, and other essential non-modeling tasks.
• Manipulating matrices is easy.
• It has sophisticated tools for modeling investment returns and for analyzing and presenting results of simulations. And it has great tools for visualizing results.
• The work can be completely open and reproducible, which is essential to the success of this project.

All programming languages have weaknesses. R’s great flexibility means that it is easy to write ill-organized programs that are hard to understand and debug. And poorly written programs that do not take advantage of R’s strengths can be extremely slow. We believe we can compensate for these weaknesses by making our programs modular, using a consistent programming style with appropriate documentation, and by using R features smartly and speed-testing where appropriate.

R analysts and programmers interested in learning about the opportunity to work on this project should examine the programmer/analyst position description and related materials at the Rockefeller Institute’s web site. 

Open Source software's opportunity to reform government

The results from the 2013 Future of Open Source Survey are in — thanks to everyone who contributed by completing the survey. You can read an overview of the results here, or see the detailed breakdowns in the slides at the end of this post.

For me, one of the most interesting nuggets from the survey is that a plurarity of respondents believed that of all sectors, the public sector will be impacted the most by open source software:

There is huge opportunity to improve the ways that we govern ourselves through the use of open source software, principles and communities. Federal, state and local governments have access to rich and large streams of data, but thus far government hasn’t been as quick as the business sector in building powerful data-based applications and services. But governments have one advantage that the business sector does not: givernment data is, by definition, public. Here are three ways that government can capitalize on rich sources of public data:

Democratize data. By making public data available on the internet (where practical and legal), governments can draw on the expertise of the data scientist community to unlock its value. At the Federal level, the Open Government Initiative is a good start, although many of the data sets made available at data.gov are over-aggregated and too small for truly innovative Big Data applications. Some local governments are making good progress here as well. New York City’s open data policy and new data transparency law has fostered a culture of civic “hacking” on city data, and created handy tools for locals like this iPhone app to find local WiFi-enabled venues suitable for getting out of the house when working “from home”.

Crowdsource data science talent with open tools. The data science community has created powerful open-source tools and resources for the analysis of Big Data. The open-source R language is designed to analyze data common to government applications (like economic, environmental and social data) and is used today by government departments such as NIST, FDA, NOAA, and the CIA. Crowdsourcing platforms and contests are a great way of getting data scientists engaged with public data, as NASA found in this Kaggle contest to find the location of dark matter from radio astronomy data.

Deploy data scientists within departments. Not all government can be made public, of course. There may be privacy issues (think IRS data), or national security issues (CIA, NSA) to consider. So it’s up to data science teams in civil service to work on sensitive data in a secure environment. It’s very encouraging to see departments like the Consumer Protection Bureau openly embrace big-data platforms and open-source tools for their teams. The growing drive towards evidence-based policy making, and the use of statistical experiments to improve government effectiveness, is another very positive sign.

It's great to see that the respondents to the 2013 Future of Open Source Survey recognize this opportunity to revolutionize government. It's up to us, as citizens, to press governments to capitalize on the opportunity, so make your voice heard.

Michael J Skok: 2013 Future of Open Source - 7th Annual Survey results

Preparing public data for analysis with R

In most data science applications, preparing the data is at least half the job. Finding where the data lives, figuring out how to access it, finding the right records, filtering, cleaning and transforming the data ... all of this has to be done before the statistical analysis can even begin.

Fortunately, the R language has many tools for data processing, and there's a great example of those tools in action at the PremierSoccerStats blog. There you can see R code that accesses public data (climate data from the Ontario Ministry of the Environment) and performs several data processing steps to create the chart below.

PremierSoccerStats: Processing Public Data with R