tech/hamori
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat: remove BAR token; bump spec to v2.3; fix max_seq_len

Browse Source 
Bar boundaries are now implicit — the detokenizer counts positions per bar
using TIME × SUB, and the generator gates EOS to bar boundaries only.
Removing the deterministic BAR token reduces vocab size from 85 to 84 and
lets the model focus on meaningful predictions.

- src/tokenizer.py: drop BAR from VOCAB (85→84); replace BAR-based
  detokenize_to_period with position-counting logic; add write_chord_file;
  fix _tokens_to_symbol for add9/m(add9) qualities
- tests/test_tokenizer.py: update vocab-size assertions to 84, structural
  token test, remove bar-count test, add test_no_bar_token_in_vocab
- docs/chord_format_spec.md: bump to v2.3; document BAR removal in §5.2,
  §5.3, §5.4, §5.5, §5.6, §6.2, and changelog
- CLAUDE.md: remove stale BAR reference, update vocab size to 84
- scripts/pretrain.py: raise max_seq_len 256→320 to cover regenerated
  McGill data (mean=83, max=283 tokens with BAR-free tokenizer)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Masahiko AMANO
2026-05-20 13:56:34 +03:00
parent 329952b02e
commit 4aead2ea20
5 changed files with 92 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+2 -2
View File
@@ -90,9 +90,9 @@ The authoritative specification is in `docs/chord_format_spec.md`. **Always read
- Body: bars separated by `|`, exactly `subdivision` positions per bar (for 4/4), positions separated by single spaces.
- A position holds: chord symbol, `.` (hold previous), `NC` (no chord), or `?` (unknown).
- Chord symbols: `<root><quality?><extension?>(/<bass>)?`. 18 qualities, 7 extensions, slash inversions are mandatory and meaningful.
- Tokenization: each new chord becomes exactly 4 tokens (`ROOT_x`, `QUAL_x`, `EXT_x`, `BASS_x`). Hold = `HOLD`. Bar end = `BAR`. Plus metadata tokens at the start.
- Tokenization: each new chord becomes exactly 4 tokens (`ROOT_x`, `QUAL_x`, `EXT_x`, `BASS_x`). Hold = `HOLD`. Bar boundaries are **not tokens** — the detokenizer reconstructs them by counting positions (`TIME` × `SUB`). Plus metadata tokens at the start.
- **Keys are normalized.** Before tokenization, the entire period is transposed: majors → C major, minors → A minor. The model never sees absolute keys. The vocabulary contains `MODE_major`/`MODE_minor` but no `KEY_x` tokens.
- Vocabulary size: ~81 tokens.
- Vocabulary size: 84 tokens.
## Model