Skip to contents

Updating the Data

Source: vignettes/Updating the Data.Rmd
Updating the Data.Rmd

Overview

The package’s data comes from five independent sources, updated on five different schedules. See vignette("Data Provenance") for how every dataset in data/ maps onto these sources and the pipeline stages below.

  1. Show data - dates, venues, attendance, sound quality, played-with bands, notes, and tracklists, scraped from the Fugazi Live Series website. Fully automated via scrape_fls_shows().
  2. Tags/duration data - song-level timings, taken from my own personally-tagged MP3 collection. Manual, done outside this repository.
  3. Venue coordinates - latitude/longitude for each venue. Manual, looked up on Google Maps as new venues show up.
  4. Song/release/duration data - release, track number, instrumental/vocalist, and duration metadata per song, hand-maintained in inst/extdata/releases_songs_durations_wikipedia.csv against the Wikipedia Fugazi discography page. Manual, done outside this repository.
  5. Song tempo data - beats-per-minute per song, hand-compiled in inst/extdata/song_tempo_bpm_data.csv. Manual, done outside this repository; also read directly by the Shiny app.

All five feed into Repeatr_1() (song tempo data is read directly by the Shiny app instead), which combines them (plus a fair amount of hand-written, gid-keyed correction code accumulated over time) into the package’s data/*.rda objects. Repeatr_Updatr(really = "really") runs Repeatr_1() and the rest of the modelling pipeline (Repeatr_2 through Repeatr_5) in sequence, and the Fugazetteer Shiny app is just a consumer of those same rebuilt data/*.rda objects - it never reads a CSV directly (song tempo data is the one exception, read straight from inst/extdata/song_tempo_bpm_data.csv), so once the package data is rebuilt and reinstalled the app picks it up automatically.

None of steps 1-4 need to happen together. Run whichever one has new material, then rebuild (step 5) and redeploy (step 6).

data-raw/build_data.R is the canonical, runnable version of steps 1-5 below - this vignette explains the why, that script is what to actually run.

1. Updating show data

scrape_fls_shows() (R/scrape_fls_shows.R) discovers the current show listing directly from the site - including any pages added since the last run - and scrapes each show’s detail page for date, venue, door price, attendance, recorded/mastered by, original source, sound quality, played-with, any official notes, and the tracklist. It also attaches tour (e.g. “1988 Fall European Tour”) from the listing pages’ own tour headings - that’s the only field here that doesn’t come from the show’s own detail page, so a run with gids supplied directly (bypassing listing discovery) won’t be able to fill it in.

Test on a small slice first:

test_shows <- scrape_fls_shows(max_listing_pages = 1, max_shows = 3, sleepseconds = 2)

By default it’s incremental: it only scrapes shows not already in the existing dataset (fls_data.csv unless you point existing_data elsewhere), so routine re-runs are quick. It also picks up on shows whose recording has newly gone from unavailable to available (detect_changes, on by default) without needing a full re-scrape - this is how ~50 shows that had gone from “no tracklist yet” to fully available got picked up in one run without touching the ~1000 shows that hadn’t changed.

To actually refresh the packaged file, for real, against the live site:

fls_data <- scrape_fls_shows(
  update_existing = TRUE,
  sleepseconds = 2,
  mycsvfilename = "inst/extdata/fls_data.csv"
)

A few things worth knowing:

  • It respects a delay (sleepseconds, default 2) between every request, listing pages and show pages alike - don’t reduce this to hammer the site.
  • A full run pages through every listing page and scrapes every target show, which can be tens of minutes depending on how many shows are new. max_shows/max_listing_pages cap it for testing.
  • fugazi-live-all-access (FLS0000) is a standing all-access download bundle, not a show, and is always excluded automatically.
  • fls_data.csv supersedes the older fugotcha.csv, gid_fls_id_sound_quality.csv and gid_fls_id_played_with.csv - those older files are left in place for reference but nothing reads them anymore.
  • If fls_data.csv ever needs tour/city/state/country backfilled without a full show-by-show re-scrape (e.g. after adding one of these columns, or if some rows are missing it), scrape_fls_listing_data() crawls just the listing pages - a few dozen requests instead of 1000+ - and returns a gid + all four: fls_data <- fls_data %>% select(-tour) %>% left_join(scrape_fls_listing_data() %>% select(gid, tour), by = "gid"). Repeatr_1() reads tour straight from fls_data.csv/Repeatr0 now, not from the older fugazi-small.csv file - city/state/country are scraped the same way but haven’t been migrated over to source from fls_data.csv yet.

2. Updating tags/duration data

Song-level durations (fls_tags, fls_tags_show, and everything derived from them like duration_summary, cumulative_duration_counts) come from inst/extdata/fls_tags.txt, which isn’t produced by any script - it’s exported from the mp3 files using kid3. Each show’s MP3s get tagged with the track name and album set to YYYYMMDD Venue, City, State, Country, and duration comes along for free from the file itself.

To update: open the new mp3 files in kid3, export the tag data as format “CSV quoted” with header track; artist; album; name; duration and tracks "%{track}";"%{artist}";"%{album}";"%{title}";"%{duration}". Open the resulting file in a text editor, remove the quotation marks and add spaces after the semi-colons using find and replace, then and add the new data to the end of any existing inst/extdata/fls_tags.txt file, or create the file if it does not exist. Save as UTF-8, not the editor’s default ANSI/system codepage - a venue or city name with an accent (é, ö, ã, …) saved in the wrong encoding will parse fine at first but crash Repeatr_1() with an invalid multibyte string error once it reaches nchar()/gregexpr() further down. fls_tags_importer() parses it, and Repeatr_1() calls it automatically - no separate function to run. If a new show’s album string doesn’t parse cleanly (typos happen), Repeatr_1()’s “process tags data” section is where the hand-written corrections for specific albums/venues live - add a new mutate(album = ifelse(...)) line there following the existing pattern if needed.

3. Updating venue coordinates

Venue coordinates now come primarily from inst/extdata/fls_venue_geocoding_v2.csv, a local snapshot of a private Google Sheet the coordinates are actually maintained in - Repeatr_1() reads this file directly (matched to shows by country/city/venue), not the live sheet itself, since that sheet isn’t reliably available and isn’t part of the formal package workflow. To update: look up new/corrected venues on Google Maps in the sheet as usual, then download it and overwrite inst/extdata/fls_venue_geocoding_v2.csv with the current export.

inst/extdata/fugazi-small.csv is still consulted as a fallback for any venue fls_venue_geocoding_v2.csv doesn’t cover, so a show doesn’t lose its coordinates entirely just because a new venue hasn’t been added to the sheet yet.

The older fls_venue_geocoding.csv file and nscmov()’s (R/nscmov.R) to-do-list bookkeeping around it are no longer part of this pipeline - Repeatr_1() stopped reading that file once fls_venue_geocoding_v2.csv became the source of truth. They’re left in place for reference but nothing in Repeatr_1() calls nscmov() or reads fls_venue_geocoding.csv anymore; treat that workflow as retired unless it gets revived deliberately.

4. Updating song/release/duration and tempo data

inst/extdata/releases_songs_durations_wikipedia.csv carries release, track number, instrumental/vocalist, and duration metadata per song, hand-maintained against the Wikipedia Fugazi discography page. Repeatr_1() reads it into songvarslookup and joins it onto the live, classified song set by song title (song) - not by a hardcoded id column, precisely so this file can’t silently drift out of sync with Repeatr_1()’s classification rules the way it once did. If you add, rename, or remove a song here, Repeatr_1() will warning() at build time listing any song names that don’t match between the live classification and this file - see the songid reconciliation check in R/Repeatr_1.R for what that looks like, and fix the mismatch in this CSV’s song column before trusting the rebuilt data.

inst/extdata/song_tempo_bpm_data.csv carries hand-compiled tempo (BPM) per song. Unlike the other four sources, it is not read by Repeatr_1() at all - the Shiny app (inst/shiny/Fugazetteer/app.R) reads it directly - so there is no automated check tying its song names to the live classification; when a song’s title changes elsewhere, update it here by hand too.

5. Rebuilding everything

Once any of sources 1-4 is updated, rebuild the package’s data objects:

Repeatr_Updatr(really = "really", update_stacks = TRUE)

This runs Repeatr_1() -> Repeatr_2() -> Repeatr_3() -> Repeatr_4() -> Repeatr_5() and saves every downstream dataset (othervariables, Repeatr0, Repeatr1, gid_sound_quality, played_with, shows_data, xray, fls_tags, fls_tags_show, the choice-model outputs, and more) into data/. It’s the really = "not_really" default that stops this running by accident - always pass really = "really" explicitly. It can take a while (the choice model fit in Repeatr_4() is the slow part), so it’s worth checking Repeatr_1() on its own first if you only want to sanity-check the data-ingestion changes.

update_stacks = TRUE also regenerates gid_initial_gid_sound_quality (the data behind the Shiny app’s “stock” pages, via Repeatr_6()) - data-raw/build_data.R’s rebuild call passes this by default, so a normal rebuild keeps it current; omit it only if you deliberately want to skip the (slower) stacks regeneration.

6. Reinstalling and redeploying

  1. Commit and push the updated inst/extdata/* source files and the regenerated data/*.rda files.
  2. Reinstall the package from the updated source (devtools::install(), or devtools::install_github("alexmitrani/Repeatr") once pushed).
  3. Run the Shiny app locally to check nothing broke: shiny::runApp("inst/shiny/Fugazetteer").
  4. Redeploy to shinyapps.io (rsconnect::deployApp("inst/shiny/Fugazetteer"), or the RStudio “Publish” button from app.R).

The app reads the package’s lazy-loaded data objects for everything except song_tempo_bpm_data (read directly from its source CSV, see step 4), so as long as steps 1-5 above ran cleanly there’s nothing else to change in app.R itself.