Benchmark¶

common datasets · shared splits · one evaluation protocol

Once a model is registered, it can be evaluated against every baseline on shared datasets under a common protocol. The headline comparison uses per-event negative log-likelihood (NLL), and the predictive and distributional metric profiles round out the picture, which matters for generative models that do not admit an exact likelihood. The command below runs your preset alongside several baselines and writes a comparable table.

preset COVID Earthquakes Citibike

your modelmy_preset

auto_stpp

deep_stpp

poisson_gmm

smash

diffusion_stpp

exact NLL · directly comparable approximate NLL · a bound (ELBO, score-matching) your model

Each cell is one (preset × dataset × seed) run, trained and evaluated under the same protocol. Results are written to report.html and the NLL tables.

python -m seahorse bench \
  --presets my_preset auto_stpp deep_stpp poisson_gmm hawkes_gmm \
  --splits_dir splits \
  --seeds 1 2 3 \
  --out runs/bench

Replace my_preset with your registered name, and point --splits_dir at the core benchmark datasets (COVID, Earthquakes, Citibike). For a first run, the small benchmark walkthrough is a good place to start.

The route¶

from a single run to reported results

first run Run a small benchmark A few presets on one dataset and seed: the walkthrough and the files it produces. after a campaign Compare results Read the NLL tables and line up models on individual sequences, within a comparable tier. selecting hyperparameters Tune hyperparameters Ray Tune HPO inside a campaign, with an explicit tuning dataset held out from the comparison. reporting results Artifacts & reproducibility Every run directory, config, and table on disk, so a result reproduces months later.

Comparability¶

what every preset shares before its own code runs

Every bench run applies a shared contract: all presets receive the same train, validation, and test splits, the same normalization policy, and a per-event NLL computed by the same harness. A difference in the table therefore reflects the models rather than the experimental setup. The execution contract records exactly what is held fixed.

Models differ in how they express likelihood. Likelihood-based models report an exact NLL; generative families such as score-matching and diffusion models report an approximation, typically a bound on the log-likelihood (for example an ELBO). Both are informative once the distinction is kept in view, and they remain comparable when read in that light. Beyond NLL, the predictive and distributional metric profiles give a fuller account of model quality, which is especially useful for models whose likelihood is only approximate.

Exactdirectly comparable

auto_stppdeep_stppnsmppnjsdeneural_*poisson_*hawkes_*rmtpp_gmmthp_gmm

Approximatea bound on the likelihood

smash · score-matchingdiffusion_stpp · ELBO

How a model declares its NLL type is covered in capabilities.

What a campaign leaves behind¶

Every run can be reproduced from disk. Alongside each fit's config, metrics, and checkpoint, a campaign writes:

File	What it's for
`report.html`	Self-contained comparison report, viewable in a browser
`table_test_nll_all.csv`	Test NLL for every cell
`table_test_nll_exact.csv`	The exact-NLL tier only
`cell_index.json`	Maps each cell to its saved run directory

Full layout and per-run files: Artifacts and run directories.

If you arrived from Add your model, your model now appears in the same tables as the established baselines. From here, read the comparison or reproduce the published results.