src/generate.py: autoregressive generation with top-p sampling, grammar
masking (ROOT→QUAL→EXT→BASS; EOS only at bar boundary), key transposition,
and optional chord prefix. Partial bars on context truncation are padded
with HOLDs rather than discarded.
scripts/generate.py: CLI wrapping generate_period — accepts mode, key,
time, subdivision, style, function, prefix, temperature, top-p, seed,
tempo; writes .chord and optional MIDI.
src/tokenizer.py: fix docstring vocab size (81→84); normalize redundant
BASS_<note>==root to no slash in _tokens_to_symbol.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Re-run pre-training results with the corrected 84-token vocabulary and
max_seq_len=320. Previous checkpoint was trained on stale data with BAR
tokens and a corrupted tokenizer.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bar boundaries are now implicit — the detokenizer counts positions per bar
using TIME × SUB, and the generator gates EOS to bar boundaries only.
Removing the deterministic BAR token reduces vocab size from 85 to 84 and
lets the model focus on meaningful predictions.
- src/tokenizer.py: drop BAR from VOCAB (85→84); replace BAR-based
detokenize_to_period with position-counting logic; add write_chord_file;
fix _tokens_to_symbol for add9/m(add9) qualities
- tests/test_tokenizer.py: update vocab-size assertions to 84, structural
token test, remove bar-count test, add test_no_bar_token_in_vocab
- docs/chord_format_spec.md: bump to v2.3; document BAR removal in §5.2,
§5.3, §5.4, §5.5, §5.6, §6.2, and changelog
- CLAUDE.md: remove stale BAR reference, update vocab size to 84
- scripts/pretrain.py: raise max_seq_len 256→320 to cover regenerated
McGill data (mean=83, max=283 tokens with BAR-free tokenizer)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
scripts/make_colab_zip.py packages src/, scripts/pretrain.py,
requirements.txt, and processed .pt files into hamori_colab.zip,
remapping data/processed/{train,val}/ -> data/processed/mcgill/{train,val}/
so pretrain.py finds the data without modification.
notebooks/colab_pretrain.ipynb guides through upload, extraction,
dependency install, training run, report display, and results download.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
README:
- processed/ tree now shows mcgill/ and user/ subdirs
- --style user -> --style H1K0 in quick-start prefix example
- pretrained.report.txt and finetuned.report.txt added to artifact tables
architecture.md (-> v1.1):
- remove stale music21 fallback mention from chord_parser section
- fix ChordDataset: on-demand loading, not eager; remove non-existent
make_dataloader from public interface
- fix train function name: train_model -> train
- update logging description: report goes to .report.txt, not stdout
- note that scripts use max_seq_len=256 (sequences top out at 195 tokens)
requirements.md (-> v1.1):
- FT-12: update from unified script to pretrain.py + train.py pair
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pretrain.py -> checkpoints/pretrained.report.txt
train.py -> checkpoints/finetuned.report.txt
Single-line [report] saved -> <path> printed to stdout instead.
Also fix arrow character incompatible with Windows cp1251 console.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- scripts/run_pretrain.py -> scripts/pretrain.py: pre-trains on McGill
corpus (data/processed/mcgill/), saves checkpoints/pretrained.pt.
- scripts/train.py: rewritten as high-level fine-tune wrapper; loads
pretrained.pt, trains on data/processed/user/, saves finetuned.pt.
Both scripts include timing estimate, loss-curve plot, per-epoch report,
and --skip-training flag.
- README: updated section 7 to reflect new script names and separate
data directories.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- scripts/run_pretrain.py: single-command pre-training runner with
timing estimate, loss-curve plot (matplotlib), and per-epoch report.
Sets max_seq_len=256 (McGill sequences max out at 195 tokens, ~4x
faster attention than the 512 default).
- src/train.py: normalise --output so pretrained.pt and pretrained both
produce pretrained.pt + pretrained.log.csv (not pretrained.pt.log.csv).
Serialize Path fields as strings in checkpoint to satisfy weights_only.
- requirements.txt: drop unused pandas/music21, add mido (pretty_midi dep).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AdamW + cosine-with-warmup schedule, PAD-ignoring cross-entropy, per-epoch
CSV logging, best-val-loss checkpointing, early stopping (patience=5).
Same script handles both pre-training and fine-tuning via --init-from.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds src/model.py with a weight-tied autoregressive transformer and
tests/test_model.py with shape, weight-tying, and causal-masking checks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the YYYY_NNN_kebab-case scheme with title_in_snake_case-function.chord.
Snake_case makes the title double-click-selectable; dash unambiguously
separates the title from the optional function suffix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add 5/4, 7/4, 7/8, 9/8 to _VALID_TIMES and VOCAB (TIME_* tokens).
Vocab size grows from 81 to 85 tokens. _parse_metre in the McGill
converter assigns subdivision=8 to 7/8 and 9/8. Spec bumped to v2.2.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- STYLE_user renamed to STYLE_H1K0 in VOCAB (author's personal tag)
- Style field now accepts any [A-Za-z][A-Za-z0-9_]* identifier in .chord files
- Unknown styles fall back to STYLE_other at tokenization time with a log warning
- Test fixtures updated to style: other; drop closed _VALID_STYLES frozenset
- Spec bumped to v2.1: documents open style field, fallback behaviour, and §5.7
guide on registering a new style token
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- src/dataset.py: ChordDataset wrapping .pt files with pad/truncate
- scripts/prepare_data.py: tokenize .chord to .pt with train/val/holdout
split, logs token length stats and style/function distributions
- src/external_converters/mcgill_to_chord.py: rewrite parser for real
McGill v2 format (2-column annotation, each bar in its own pipe group,
interval bass notation e.g. /5 and /b3)
- .gitignore: exclude data/processed/train, val, holdout subdirectories
- tests: 37 new tests for ChordDataset and converter (260 total, all pass)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds src/external_converters/mcgill_to_chord.py with two public functions:
- convert_song(song_dir, output_dir) — converts one salami_chords.txt to
per-section .chord files (4–16 bars each, style=other)
- convert_dataset(dataset_dir, output_dir) — batch converts all songs
Key decisions:
- Harte qualities mapped to our 18-quality vocabulary; hdim7 → m7b5,
parenthetical alterations (e.g. 7(b9)) handled via regex
- Bar duration estimated from median non-trivial chord duration
- Mode (major/minor) inferred from tonic chord quality distribution
- Sections with <4 or >16 bars are skipped with a logged reason
- Unrecognized Harte chords skip the whole section (no silent corruption)
48 new tests in tests/test_mcgill_converter.py; total suite 223 passed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
chord_file_to_midi() parses the period in the user's original key (no
transposition), accumulates held-chord segments, then writes two pretty_midi
tracks: chords with root anchored at octave 4 (MIDI 60–71 + intervals) and
bass at octave 2 (MIDI 36–47). Extension notes are added as a fifth voice
at their standard interval above the root. Tempo is parameterised; the CLI
wrapper (python -m src.midi_export) supports --tempo BPM.
10 tests cover: file creation, parseability, instrument count and names,
chord/bass note counts for a 4-chord C-major fixture (14 chord + 4 bass),
octave placement assertions, and tempo affecting total duration.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds VOCAB (81 tokens), TOKEN_TO_ID, and ID_TO_TOKEN per spec §5.2.
tokenize_period() transposes to C/Am then emits BOS + metadata tokens +
per-bar chord/HOLD/NC tokens + BAR + EOS. detokenize_to_period() is the
exact inverse, returning a ChordPeriod in canonical key. The m(add9)
quality maps to QUAL_m_add9 in the vocab (parentheses not valid in token
names) via _qual_token/_token_qual helpers.
36 new tests cover vocabulary integrity, token sequence structure,
and full round-trip fidelity for all four valid fixture files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Six factual corrections: slash chord definition (inversions vs on-chords),
logits described as unnormalized, #11 = eleventh not fourth, duplicate
Polukadentsiya sentence removed, pre-training LR corrected to 1e-4,
unverified PAD index claim removed. All ML term explanations rewritten
with analogies accessible to a near-beginner.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
src/tokenizer.py:
- parse_chord_file(Path) → ChordPeriod: reads header + bar body, strips //
comments, validates bar position counts and chord symbols, raises
ChordFormatError with filename and bar number on any violation.
- transpose_to_canonical(ChordPeriod) → ChordPeriod: shifts all chord roots
and bass notes by the semitone offset to C major / A minor; fast-path
returns the original object when shift == 0.
tests/test_chord_file_parser.py: 39 tests covering parsing of 4 valid fixtures
(C major, F# major, B minor, G# minor), error messages for 2 invalid
fixtures, and transposition correctness including slash chord root+bass.
tests/fixtures/: 6 .chord fixture files (4 valid, 2 invalid).
requirements.txt: pinned to current latest stable versions
(torch 2.12.0, music21 10.1.0, pretty_midi 0.2.11, matplotlib 3.10.9,
numpy 2.4.6, pandas 3.0.3, pytest 9.0.3); Python >= 3.11 noted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Revise and expand the authoritative .chord format specification:
version bump 1.0 → 2.0, clarified period scope, updated tokenization
rules, grammar tables, and key normalization details.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Authoritative .chord file format spec covering header fields, body
syntax, chord symbol grammar, tokenization rules, and key normalization.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
H1K0