Hi Cardiff!

• I’m Jasmine Dumas
  • Insurance Data Scientist and useR living in Connecticut, USA
  • M.S. Candidate in Predictive Analytics at DePaul University
  • Beginning Home brewer!
• Slides and Materials available at: jasdumas/talks/caRdiff-uk-open-gov-beer

What this presentation is about (and not about)

• I’m going to talk about the state of open government data in relation to beer
  • difficulties in finding and exploring analysis-ready data in a never ending sea of data
• This is not about “drunk politics”
  • not about regulations, hacking, or scandals regarding data

not

The search for datasets

• Finding interesting and available datasets is a concern for all those involved in data analysis
• Where to look?
  • Google public data explorer
  • Github
  • kaggle data repository
  • Open Government Data Portals: UK, Connecticut, SF,
  • Machine Learning Repositories at Universities: UCI

The search continues for analysis-ready datasets

• Many people say that data wrangling (or cleaning) is ~70% of the data science process
  • the desire for analysis-ready data is a high priority after addressing the business questions of a project and before the data wrangling step
• analysis-ready data also needs to include some type of documentation (data wiki’s FTW)

The U.S. Government Open Data Portal

The “clearinghouse” for open U.S. government data is located at data.gov. It also contains tools, and resources to conduct research, develop web and mobile applications, design data visualizations.

Example Application of the U.S. Government open data portal

• datausa.io: an online application developed by a team of data scientists at MIT Media Lab and Datawheel, backed by Deloitte is helping Americans visualize demographic and economic data using an open source platform

Just because it’s open doesn’t mean it’s accessible!

Good examples of datasets

• “Good” datasets are “Good” if they can be read in to R with functions like:
  • read.csv, read_csv or read.table
• “Great” datasets take minimal cleaning aside from manipulation and have interpretable data fields

Bad examples of datasets

• “Bad” datasets are “bad” if they are in inconsistent formats, have mixed columns, and cryptic data fields
• “Bad” = difficult for analysis

The landscape of beer datasets

• There are limited amounts of datasets for beer analytics. Here are some examples:
  • Datasets: Craft Beer Sample Data, SNAP-RateBeer* - which is provided to academic researchers only
  • API’s: RateBeer, Untapped, The Beer Mapping Project, BreweryDB | API’s that are available have restrictions on usage or require a user account
  • Data analysis projects of beer: yhat Beer Advocate Recommender, Crowd-sourced Beer Distribution Map

The reality and struggle of beer datasets

• … are real!
• This can lead to:
  • frustration
  • losing interest and passion for the domain
  • choosing another dataset
  • abandoning the data analysis project
  • breaking Terms of Service (ToS) with web-scraping

Necessity breeds innovation

• I was inspired to become better at finding datasets and web scraping from the road-blocks of finding analysis-ready datasets!
• Here is a tutorial I wrote to get meta data on brewery ratings in Connecticut from Beer Advocate using Import.io

Introducing ttbbeer

• An R data package of beer statistics from U.S. Department of the Treasury, Alcohol and Tobacco Tax and Trade Bureau (TTB)
  • The plan: liberate more beer statistics datasets from open U.S. government data portals to analysis-ready data frames for R and beyond
  • The method: web-scraping with rvest (the first dataset was copy/pasted to excel due to the limited R ecosystem of PDF parsing packages)
  • The dream: increase awareness of beer analytics and promote analysis-ready datasets from data.gov
• available on CRAN and Github

Questions & Cheers!

+