Diarization Evaluation
This document describes the recommended way for partners and users to evaluate Sortformer and Nemotron diarization models using NVIDIA NeMo Speech.
Use the NeMo Speech evaluation script as the source of truth:
The script runs model inference, converts frame-level speaker activity predictions into diarization segments, and reports diarization error rate (DER) using NeMo Speech's diarization scoring utilities.
Recommended Evaluation Command
Install and run from a checkout of NeMo Speech:
git clone https://github.com/NVIDIA-NeMo/Speech.git
cd Speech
export NEMO_ROOT="$PWD"
Evaluate a downloaded .nemo checkpoint:
python ${NEMO_ROOT}/examples/speaker_tasks/diarization/neural_diarizer/e2e_diarize_speech.py \
model_path="/path/to/Nemotron-3-Diarization-preview.nemo" \
dataset_manifest="/path/to/diarization_manifest.json" \
batch_size=32 \
collar=0 \
ignore_overlap=false \
precision=bf16 \
compile_encoder=false \
spkcache_len=264 \
chunk_len=340 \
chunk_right_context=40 \
fifo_len=40 \
spkcache_update_period=300
For a Hugging Face model repository, first download the .nemo checkpoint and
then pass its local path to model_path.
The input manifest must contain one JSON object per line. For DER evaluation, each record must include an RTTM reference:
{"audio_filepath": "/path/to/audio_001.wav", "offset": 0, "duration": 600, "rttm_filepath": "/path/to/audio_001.rttm"}
{"audio_filepath": "/path/to/audio_002.wav", "offset": 0, "duration": 580, "rttm_filepath": "/path/to/audio_002.rttm"}
If a UEM file is required by the benchmark protocol, include uem_filepath in
the manifest record. Otherwise, NeMo derives the evaluated region from the
reference and hypothesis extents and clamps it to the manifest offset and
duration when those fields are present.
Metrics
Diarization Error Rate
Diarization Error Rate (DER) is the primary diarization metric. It is the sum of three error components, normalized by the scored reference speaker time:
DER = false alarm + missed speech + speaker confusion
The NeMo Speech script reports:
FA, MISS, CER, DER
In the script output, CER means speaker confusion error rate, not character
error rate.
Speaker Counting Accuracy
Speaker Counting Accuracy (SCA) measures whether the predicted number of speakers exactly matches the reference number of speakers for each recording:
SCA = number_of_recordings_with_correct_speaker_count / number_of_recordings
Report SCA as a simple percentage:
SCA (%) = 100 * SCA
NeMo Speech logs this value as Spk. Count Acc..
Speaker Counting Mean Absolute Error
NeMo Speech currently reports Speaker Counting Mean Absolute Error (MAE):
MAE = mean(abs(predicted_speaker_count - reference_speaker_count))
The NeMo Speech scorer logs this value as Spk. Count MAE.
Required Reporting Convention
When reporting DER for Sortformer or Nemotron diarization models, include the evaluation protocol details with the number. DER is not fully interpretable without these settings.
Report at least:
| Field | What to report |
|---|---|
| Model | Model name and checkpoint path or revision |
| Evaluation code | NeMo Speech repository URL and commit hash |
| Dataset | Dataset name, split, and manifest path or release identifier |
| Reference annotation | RTTM source and citation, such as paper, dataset URL, or forced-alignment reference |
| UEM | Whether UEM regions were used, and the UEM source if applicable |
| Overlap | Whether overlapping speech was included in scoring (ignore_overlap=false) or excluded (ignore_overlap=true) |
| Collar | Collar half-width in seconds, for example 0.0 or 0.25 |
| Post-processing | Whether post-processing was bypassed or which post-processing YAML was used |
| Output resolution | Native or overridden output subsampling factor |
| Streaming settings | spkcache_len, chunk_len, chunk_left_context, chunk_right_context, fifo_len, and spkcache_update_period |
| Precision | For example bf16, bf16-mixed, or 32 |
| Hardware | GPU model and batch size |
Convert NeMo's DER, FA, MISS, CER, and Spk. Count Acc. rates to
percentages by multiplying them by 100; report Spk. Count MAE unchanged.
Recommended metric table columns:
| Dataset | DER (%) | FA (%) | Miss (%) | Confusion (%) | SCA (%) | Speaker Count MAE |
|---|---|---|---|---|---|---|
| Example split | 0.00 | 0.00 | 0.00 | 0.00 | 100.00 | 0.0000 |
Notes on Collar and Overlap
The NeMo Speech DER scorer follows NIST md-eval-22.pl collar semantics:
collar is the no-score half-width around each reference boundary. For example,
collar=0.25 excludes 0.25 seconds to the left and 0.25 seconds to the right of
each reference boundary.
The ignore_overlap flag controls whether overlapped speech contributes to DER:
ignore_overlap=false # include overlap in DER
ignore_overlap=true # exclude overlap from DER
For model-card style reporting, prefer the benchmark's official protocol. If the benchmark does not specify a protocol, report DER with overlap included and state the collar value explicitly.
Where the Metrics Come From in NeMo Speech
e2e_diarize_speech.py restores the diarization model, sets the test manifest, applies the requested streaming parameters, runs inference, converts predictions to timestamped speaker segments, and calls:
score_labels(
AUDIO_RTTM_MAP=infer_audio_rttm_dict,
all_reference=all_refs,
all_hypothesis=all_hyps,
all_uem=all_uems,
collar=cfg.collar,
ignore_overlap=cfg.ignore_overlap,
)
The NeMo Speech scoring utility computes cumulative FA, MISS, speaker
confusion (CER), and DER. It also computes speaker-count accuracy and
speaker-count MAE by comparing the number of unique speakers in the reference
and hypothesis for each manifest region.
Stand-alone scoring script
score_diarization.py is a simple user-facing script for diarization scoring.
python ${NEMO_ROOT}/scripts/speaker_tasks/score_diarization.py \
-r $REFERENCE \
-h $HYPOTHESIS \
-c $COLLAR
The reference and hypothesis may each be:
- A compound RTTM file
- A directory containing per-recording RTTM files
- A JSON/JSONL diarization manifest
When both inputs are directories, RTTM filenames must match exactly.
The scorer includes overlapping speech regions (ignore_overlap=False).