Troubleshooting index
Every failure described across this guide, in one table. Find your error text, read the cause, apply the fix.
| Error / symptom | Where | Cause | Fix |
|---|---|---|---|
Warning: lengths of stems are different for path… |
Metadata collection, dataset_type 1 |
Stems inside a song folder do not end at the same sample. | Trim or pad the stems so every file in the folder has identical length. |
ValueError: operands could not be broadcast together with shapes |
Validation stage | mixture, stems and other files in a validation folder have different lengths. |
Re-export them from the same session, or trim the longest file to match. |
RuntimeError: output with shape [1, 2, …] doesn't match with the broadcast shape [2, 2, …] |
Training a 4→2-stem conversion (e.g. SCNet) | The config still declares stems the dataset no longer provides. | Set target_instrument: null and list only the instruments you actually train. |
CUDA_OutOfMemoryError |
Training | Batch size, chunk size or model size exceeds available VRAM. | Enable use_torch_checkpoint, lower chunk_size or batch_size, and use gradient_accumulation_steps to keep the effective batch large. |
No module named … |
Cloud instance, after installing requirements | Some dependencies were never installed by requirements.txt. |
pip install the module and re-run the training command until metadata collection starts. See the module list below. |
ERROR: Failed to build installable wheels … (wxpython, diffq, pesq) |
End of the requirements install on Linux | No C build toolchain, or a numpy version incompatible with pesq. |
sudo apt-get install build-essential plus pip install numpy==1.26.4, or remove wx-python from requirements.txt. |
| OpenBLAS threading error | Cloud training | OpenBLAS spawning more threads than the instance's CPU allocation. | export OPENBLAS_NUM_THREADS=1 in the same terminal session before training. |
| Impossible metric values after resuming, e.g. “inst other fullness: 49” | Training, second phase | The learning rate was too high for continued training — the model learned too fast. | Lower the learning rate and resume again; compare against a healthy spectrogram (Figure 9.4). |
| Validation files rejected outright | Validation stage | Files were exported as FLAC or at the wrong bit depth. | Re-export the validation set as 16-bit WAV. |
| Files silently ignored | Dataset loading | MP3 or another lossy format in the dataset. | Convert to FLAC or WAV. Lossy formats are not supported. |
| Config opens in the browser instead of downloading | Repository setup | The browser renders YAML as plain text. | Copy the contents into a text editor and save the file yourself with a .yaml extension. |
| Install commands fail or partially apply | Local setup on Windows | The shell lacks administrator rights. | Open Command Prompt or PowerShell as administrator before running installs. |
| Fusion script refuses to combine checkpoints | Model fusion | The checkpoints target different stems, or their base architecture settings differ. | Only fuse checkpoints with the same target (both other for two instrumentals) and the same base settings. |
The missing-module list
Modules reported missing on a clean cloud instance, in the order they were encountered: wandb, soundfile, auraloss, audiomentations, pedalboard, ml_collections, omegaconf, einops, beartype, rotary_embedding_torch.
pip install wandb soundfile auraloss audiomentations pedalboard \
ml_collections omegaconf einops beartype rotary_embedding_torch
Appendix A · Glossary
The vocabulary used throughout the guide, in plain language.
results/.other that holds everything else.chunk_size = (dim_t − 1) × hop_length.num_steps updates. The trainer validates and stores weights at the end of each one.num_steps steps.use_amp.use_torch_checkpoint).null when you enumerate instruments explicitly.1 during training.Appendix B · Metric reference
The numbers the trainer prints. They exist to tell you which direction a run is moving — not to declare a model finished.
| Metric | What it measures | Read it as |
|---|---|---|
sdr |
Signal-to-Distortion Ratio: the classic overall quality score for a separation. | higher is better |
l1_freq |
L1 Frequency: a spectrogram-domain distance, similar in spirit to SDR. | compare run to run |
si_sdr |
Scale-Invariant SDR: like SDR but ignoring the scaling between target and noise. | higher is better |
neg_log_wmse |
Negative Log Weighted Mean Square Error: a loss-style error measurement. | compare run to run |
aura_stft |
Aura Short-Time Fourier Transform: weighted towards perceptual quality of the result. | perceptual |
aura_mrstft |
Aura Multi-Resolution STFT: the same idea computed across several resolutions. | perceptual |
bleedless |
How much of one output leaks into the other — for example vocal bleed inside the instrumental. Spectrogram based. | higher = less bleed |
fullness |
How complete the target stem sounds. Spectrogram based. | higher = fuller |
Fullness and bleedless move against each other. Push the target stem to sound fuller and more of the other stem tends to leak in; push bleed down and the target starts sounding thin. Pick the end of that scale your material needs — an instrumental for a karaoke track wants bleedless, a stem for remixing may prefer fullness.
A checkpoint sits somewhere on this line, not at the end of it. Decide which way your material leans before you chase a number.
Choosing the scheduler metric
--metric_for_scheduler decides which of these numbers drives the learning-rate scheduler: while the chosen metric keeps improving, the rate holds; when it stalls for patience epochs, the rate is multiplied by reduce_factor. sdr is the usual choice, with fullness or bleedless as alternatives when you care about one specific behaviour.
Use metrics for a quick evaluation and for spotting trend changes. They will not tell you whether a song sounds good. Export a few tracks with the checkpoint, listen on the material you actually care about, and treat the numbers as a smoke detector rather than a verdict.
Appendix C · Notes on this edition
This is a rewritten edition of the original PDF, not a re-upload. Every technical instruction is preserved; the structure, the language and the references were rebuilt, and a few known errors were corrected.
What changed
- Structure. The original's loosely numbered sections were reorganised into 13 chapters and 4 appendices, each with a consistent shape: overview, steps, caveats, cross-references.
- Navigation. One page per stage of the workflow, a table of contents per page, a searchable error index, and cross-links between chapters.
- Reference material. Arguments, metrics and configuration keys are now tables you can scan instead of prose you have to read.
- Error index. All failure modes were collected into a single cause-and-fix table (Chapter 13), including the ones originally buried inside unrelated chapters.
- Figures. Screenshots from the original document were reproduced with numbered captions explaining what to look at.
- Language. Typos and unclear sentences were corrected (for example “weary” → “wary”), and the tone made consistent throughout.
Corrections and flagged claims
| Item | Original | This edition |
|---|---|---|
--pin_memory |
Described as “controls the number of parallel data-loading workers” — identical to --num_workers. |
Corrected: keeps host memory page-locked so GPU transfers are faster. |
| dim_t 1333 → duration | Listed as “13,22 seconds” next to 587412 samples. | Corrected to 13.32 s: 587412 ÷ 44100 = 13.32, and the other three rows of the same table already divide exactly (2.55, 8.00, 11.00 s). |
| Fusion constraints | Marked “NOT CONFIRMED” regarding matching base architecture settings. | Kept as unconfirmed, and labelled as such rather than stated as fact. |
| OpenBLAS fix | Noted as appearing at the metadata stage but turning out to be unrelated. | Kept in the cloud sequence with that caveat intact. |
| Learning-rate → fullness 49 | Offered with a hedge (“I think”). | Presented as “the most likely cause”, matching the uncertainty. |
The original document invites more experienced users to modify it as they see fit. That invitation still stands: if something here is wrong or outdated, correct it rather than working around it.
Appendix D · Credits & links
Nobody learns this alone. The original guide exists because several people answered questions they did not have to answer.
Bas Curtiz
Helped with the metal dataset from the very start, and wrote the dataset-creation guide linked from Chapter 03.
unwa
Aided the repository setup on Bas's advice, diagnosed that the MaskEstimator is what changes when the target changes, and provided the transfer-learning patch in Chapter 09.
ZFTurbo
Author of Music-Source-Separation-Training — without it there would be nothing to train — and a source of help on validation datasets.
Kimberley Jensen
Provided the best base Roformer to fine-tune, and helped work out the cloud training process.
jarredou & becruily
Additional help with errors — becruily supplied the cloud install commands, jarredou the note about wx-python breaking requirements installs.
Whoever is reading this
The original document ends by thanking the newbie reading it. That is still the point: this guide exists to shorten someone else's learning curve.
All links in one place
| Resource | Why you need it | Link |
|---|---|---|
| Training repository | The code that trains models | github.com/ZFTurbo/Music-Source-Separation-Training |
| Dataset types | Defines types 1–4 for --dataset_type | docs/dataset_types.md |
| PyTorch install | Pick the command for your CUDA version | pytorch.org/get-started/locally |
| Python | Required for local training | python.org/downloads |
| Dataset creation video | Full walkthrough of dataset building | youtube.com/watch?v=Wmt_0zu94L8 |
| Model fusion script | Merges two compatible checkpoints | huggingface.co/Sucial · model_fusion.py |
| Cloud GPU rental | Hardware for real training runs | vast.ai · runpod.io |