Skip to contents

Rebuilding the Data

Source: vignettes/Rebuilding-the-Data.Rmd
Rebuilding-the-Data.Rmd

Overview

Primary data - show listings, venue coordinates, tag/duration data, and discography metadata - lives in this package’s own inst/extdata/, not in a separate package. The companion package fugazibase is downstream of Repeatr: it’s a generated export of Repeatr’s own cleaned data, not a source Repeatr reads from. See vignette("Data-Catalogue", package = "fugazibase") for what it contains.

This vignette covers the three-stage process of refreshing inst/extdata/’s raw sources, rebuilding every dataset in data/*.rda, exporting the fugazibase subset, and getting the rebuilt package and Shiny app back out into the world. data-raw/build_data.R is the canonical, runnable version of the process below - this vignette explains the why, that script is what to actually run.

1. Refreshing the raw sources (stage A)

Update whichever of inst/extdata/’s raw sources has new material:

  • Show data - a fresh scrape_fls_shows() run, writing to inst/extdata/fls_data.csv.
  • Venue coordinates - a synced export of the private Google Sheet used to look up/confirm venue locations, overwriting inst/extdata/fls_venue_geocoding_v2.csv directly. This file is the single source of truth for coordinates in both Repeatr and fugazibase.
  • Tags/duration data - re-exported from kid3 against the personally-tagged MP3 collection, overwriting inst/extdata/fls_tags.txt. If a new show’s album string doesn’t parse cleanly, the hand-written corrections for specific albums/venues live in the “process tags data” section of R/Repeatr_1.R.
  • Song/release/duration and tempo data - hand-maintain inst/extdata/releases_songs_durations_wikipedia.csv, inst/extdata/releases.csv, and inst/extdata/song_tempo_bpm_data.csv directly against their sources (Wikipedia, rateyourmusic.com, personal BPM readings).

None of these five need refreshing together - run whichever one has new material.

2. Rebuilding Repeatr’s own data (stage B)

Once the raw sources are refreshed, rebuild Repeatr’s own data objects:

Repeatr_Updatr(really = "really", update_stacks = TRUE)

This runs Repeatr_1() -> Repeatr_2() -> Repeatr_3() -> Repeatr_4() -> Repeatr_5() and saves every downstream dataset (othervariables, Repeatr0, Repeatr1, gid_sound_quality, played_with, shows_data, xray, fls_tags, fls_tags_show, songvarslookup, song_tempo_bpm_data, the choice-model outputs, and more) into data/. It’s the really = "not_really" default that stops this running by accident - always pass really = "really" explicitly. It can take a while (the choice model fit in Repeatr_4() is the slow part), so it’s worth checking Repeatr_1() on its own first if you only want to sanity-check a raw-source refresh.

Repeatr_1() also does two small pieces of cleaning on othervariables/shows_data beyond typed/joined raw data: it splits the raw scraped door-price text into a numeric price and an ISO 4217 currency (via inst/extdata/fls_doorprice_currency_lookup.csv), and it fills in Brazilian state codes for subdivision (the FLS site’s own scrape never populates a subdivision outside the US/Canada/Australia) and standardizes any remaining blank subdivision to NA.

update_stacks = TRUE also regenerates gid_initial_gid_sound_quality (the data behind the Shiny app’s “stock” pages, via Repeatr_6()) - data-raw/build_data.R’s rebuild call passes this by default, so a normal rebuild keeps it current; omit it only if you deliberately want to skip the (slower) stacks regeneration.

Repeatr_1() (and Repeatr_Updatr(), which threads it through) accepts an output_dir argument if you want to redirect the rebuilt data/*.rda objects somewhere other than the package root’s own data/ - useful for testing a rebuild without touching your working checkout.

3. Exporting fugazibase (stage C)

Once Repeatr’s own data/*.rda objects are rebuilt, compose fugazibase’s tables from them:

export_fugazibase_data(fugazibase_dir = "../fugazibase")

This reads Repeatr’s own already-saved data/*.rda objects (no re-derivation) plus inst/extdata/fls_venue_geocoding_v2.csv directly, and writes fugazibase_dir/data/*.rda - six tables in all (shows, locations, durations, discography, songs, bands). locations’s coordinates are named latitude/longitude; discography/songs hold what used to be releases/discography respectively (renamed so discography means what it says). fugazibase_dir has no default - point it at a local fugazibase checkout. This is a one-way generation step: it writes files, but does not commit or push anything in that checkout - review and commit fugazibase’s own changes separately, on its own schedule. Nothing in Repeatr’s own pipeline depends on fugazibase being refreshed or reinstalled.

4. Reinstalling and redeploying

  1. Commit and push the regenerated data/*.rda files.
  2. Reinstall the package from the updated source (devtools::install(), or devtools::install_github("alexmitrani/Repeatr") once pushed).
  3. Run the Shiny app locally to check nothing broke: shiny::runApp("inst/shiny/Fugazetteer").
  4. Redeploy to shinyapps.io (rsconnect::deployApp("inst/shiny/Fugazetteer"), or the RStudio “Publish” button from app.R).

The app reads the package’s lazy-loaded data objects for everything except venue coordinates (fetched live from a Google Sheet - see vignette("Data-Provenance")), so as long as stages 1-2 above ran cleanly there’s nothing else to change in app.R itself.