Data Provenance

Introduction

Every dataset in this package is somewhere on a spectrum from “raw, exactly as scraped or hand-entered from an external source” to “the output of a statistical model.” That distinction matters: raw and hand-curated data can only be wrong (a typo, a stale lookup), while derived and modelled data can also be inconsistent with itself if two different steps compute the same thing two different ways - which is exactly what happened to songid (see ?songidlookup and the Provenance section of its help page). This vignette catalogues every dataset produced in this package against that spectrum.

Primary/raw data - the “raw, exactly as scraped or hand-entered” end of the spectrum - lives in this package’s own inst/extdata/ (fls_data.csv, fls_tags.txt, releases.csv, releases_songs_durations_wikipedia.csv, song_tempo_bpm_data.csv, fls_venue_geocoding_v2.csv). Repeatr_1() turns those six raw sources into everything below.

The companion package fugazibase is downstream of this package, not upstream of it: composes a subset of Repeatr’s own Derived-cleaned tier (corrected, reformatted, and keyed for joining, but with no joined/summarized/modeled columns and no copyrighted free-text show notes) and writes it into a local fugazibase checkout. See vignette("Data-Catalogue", package = "fugazibase") for that package’s own catalogue and join keys.

Before adding a new dataset or reclassifying an existing one, check inst/shiny/Fugazetteer/app.R and vignettes/*.Rmd for consumers, not just R/ - the Shiny app and several vignettes read package data directly without that being visible from a search of R/ alone, and some of them locally rebuild an object under the same name as a package dataset without ever touching the lazy-loaded original (see the Frozen-legacy table below for concrete examples of both traps).

Types of data

Raw-scraped / Raw-hand-curated - primary data, as scraped () or hand-curated directly into this package’s own inst/extdata/.
Derived-cleaned - mechanically produced from inst/extdata/’s raw sources by Repeatr_1(), with no judgment calls beyond straightforward joins/typing/renaming.
Derived-classified - mechanically produced by Repeatr_1(), but shaped by the hand-written grepl()-based song-title recoding and tracktype classification rules - i.e. the layer where a rule change can shift which songs exist and how they’re numbered.
Derived-modeled - depends on the mlogit choice model fit in Repeatr_4(), or on the min_song_count modelling-eligibility filter applied from Repeatr_2() onward.
Frozen-legacy - present in data/ but not written by any current pipeline stage (no save() call anywhere in R/, not touched by data-raw/build_data.R). Confirmed to have no consumers in R/, vignettes/, or app.R either, as of this vignette’s last update.

Data processing sequence

inst/extdata/ (primary data - fls_data.csv, fls_tags.txt, releases.csv,
               releases_songs_durations_wikipedia.csv,
               song_tempo_bpm_data.csv, fls_venue_geocoding_v2.csv)
        │
        ▼
Repeatr_1()
├─ Repeatr0, othervariables, gid_sound_quality, played_with*, shows_data,
│  fls_tags*, duration_data_da, duration_summary, cumulative_*, xray,
│  releasesdatalookup, releases_menu_list, releaseid_variable_colour_code,
│  transitions_data_da, last_performance_data, songvarslookup,
│  song_tempo_bpm_data                          [Derived-cleaned]
└─ Repeatr1, songidlookup                        [Derived-classified]
        │
        ▼
Repeatr_2() ─▶ Repeatr_3() ─▶ Repeatr_4() ─▶ Repeatr_5()
├─ Repeatr2, Repeatr3, altlookup,
│  fugazi_song_performance_intensity              [Derived-modeled: min_song_count filter + alt]
├─ fugazi_song_counts                              [Derived-modeled: covers every classified song]
├─ results_ml_Repeatr4, vcovmat_ml_Repeatr4        [Derived-modeled: mlogit fit]
└─ fugazi_song_choice_model, fugazi_song_preferences,
   releases_rated, releases_summary, releases_data_input, summary
                                                    [Derived-modeled: model output]
        │
        ▼ (update_stacks = TRUE)
Repeatr_6()
└─ gid_initial_gid_sound_quality                  [Derived-modeled: depends on summary]

        │ (from the Derived-cleaned tier only, minus fls_notes)
        ▼
export_fugazibase_data()
└─ fugazibase: shows, locations, durations, discography,
   songs, bands

songvarslookup is joined into Repeatr1 by title text, not carried forward with its own songid column - the hand-maintained CSV behind it (inst/extdata/releases_songs_durations_wikipedia.csv) doesn’t carry one, precisely so it can’t silently drift out of sync with the songid Repeatr_1() computes; see ?songidlookup for that mapping.

Dataset catalogue

`data/*.rda` objects

Dataset	Tier	Produced by
`Repeatr0`	Derived-cleaned	`Repeatr_1()`, from `inst/extdata/fls_data.csv`
`othervariables`	Derived-cleaned	`Repeatr_1()`, joining `inst/extdata/fls_data.csv` with `inst/extdata/fls_venue_geocoding_v2.csv`; also read directly by `app.R`
`gid_sound_quality`	Derived-cleaned	`Repeatr_1()`
`gid_initial_gid_sound_quality`	Derived-modeled	`Repeatr_6()`, via `sweepstack()`/`stacks()`, whenever `Repeatr_Updatr()` is run with `update_stacks = TRUE`; read directly by `app.R`’s “stock” pages
`played_with`, `played_with_summary`	Derived-cleaned	`Repeatr_1()`
`shows_data`	Derived-cleaned	`Repeatr_1()`; also read directly by `app.R`
`fls_tags`, `fls_tags_show`	Derived-cleaned	`Repeatr_1()`, from `inst/extdata/fls_tags.txt` (via )
`duration_data_da`, `duration_summary`, `cumulative_duration_counts`, `cumulative_song_counts`	Derived-cleaned	`Repeatr_1()`
`last_performance_data`, `xray`, `transitions_data_da`	Derived-cleaned	`Repeatr_1()`
`releasesdatalookup`, `releases_menu_list`, `releaseid_variable_colour_code`	Derived-cleaned	`Repeatr_1()`, from `inst/extdata/releases.csv`
`songvarslookup`	Derived-cleaned/Raw-hand-curated	`Repeatr_1()` reads it as-is from `inst/extdata/releases_songs_durations_wikipedia.csv`
`song_tempo_bpm_data`	Raw-hand-curated	`Repeatr_1()` reads it as-is from `inst/extdata/song_tempo_bpm_data.csv`
`Repeatr1`	Derived-classified	`Repeatr_1()`
`songidlookup`	Derived-classified	`Repeatr_1()`; the single source of truth for song identity
`Repeatr2`, `Repeatr3`	Derived-modeled	`Repeatr_2()` / `Repeatr_3()` - applies the `min_song_count` filter and builds `alt`
`altlookup`	Derived-modeled	`Repeatr_2()`; the `alt` <-> `songid`/`title` translation table used by `Repeatr_5()`/`rankr()`
`fugazi_song_counts`	Derived-modeled	`Repeatr_2()`; covers every classified song, not just the `min_song_count`-eligible ones
`fugazi_song_performance_intensity`	Derived-modeled	`Repeatr_2()`; `min_song_count`-eligible songs only
`results_ml_Repeatr4`, `vcovmat_ml_Repeatr4`	Derived-modeled	`Repeatr_4()`, saved together so they always describe the same fit
`fugazi_song_choice_model`, `fugazi_song_preferences`, `releases_rated`, `releases_summary`, `releases_data_input`, `summary`	Derived-modeled	`Repeatr_5()`; `summary` is also read directly by `app.R`

For the full column-by-column description of any dataset, see its help page (e.g. ?summary). For fugazibase’s own tables (shows, locations, durations, discography, songs, bands), see vignette("Data-Catalogue", package = "fugazibase").

Note on app.R: it reads song_tempo_bpm_data and shows_data/othervariables directly (via library(Repeatr)’s lazy-loaded data), but for venue coordinates specifically it does not use the package’s own x/y - at startup it re-fetches coordinates live from a Google Sheet (gsheet2tbl()) and overwrites whatever othervariables/shows_data provided. So the deployed app’s map coordinates track that live sheet, not any package release.

Introduction

Types of data

Data processing sequence

Dataset catalogue

data/*.rda objects

`data/*.rda` objects