Benchmark¶

common datasets · shared splits · one evaluation protocol

Once a model is registered, it can be evaluated against every baseline on shared datasets under a common protocol. The headline comparison uses per-event negative log-likelihood (NLL), and the predictive and distributional metric profiles round out the picture, which matters for generative models that do not admit an exact likelihood. The command below runs your preset alongside several baselines and writes a comparable table.

preset ComCat Austin 311 NYC collisions

your modelmy_preset

auto_stpp

deep_stpp

poisson_gmm

smash

diffusion_stpp

exact NLL · directly comparable approximate NLL · sample-based estimate, not a bound your model

Each cell is one (preset × dataset × seed) run, trained and evaluated under the same protocol. Results are written to report.html and the NLL tables.

python -m seahorse bench \
  --presets my_preset auto_stpp deep_stpp poisson_gmm hawkes_gmm \
  --splits_dir splits \
  --seeds 1 2 3 \
  --out runs/bench

Replace my_preset with your registered name, and point --splits_dir at a directory of released benchmark datasets (see the dataset catalog). For a first run, the small benchmark walkthrough is a good place to start.

The route¶

from a single run to reported results

first run Run a small benchmark A few presets on one dataset and seed: the walkthrough and the files it produces. after a campaign Compare results Read the NLL tables and line up models on individual sequences, within a comparable tier. selecting hyperparameters Tune hyperparameters Ray Tune HPO inside a campaign, with an explicit tuning dataset held out from the comparison. reporting results Artifacts & reproducibility Every run directory, config, and table on disk, so a result reproduces months later.

Comparability¶

what every preset shares before its own code runs

Every bench run applies a shared contract: all presets receive the same train, validation, and test splits, the same normalization policy, and a per-event NLL computed by the same harness. A difference in the table therefore reflects the models rather than the experimental setup. The execution contract records exactly what is held fixed.

Models differ in how they express likelihood. Likelihood-based models report an exact NLL (closed-form or ODE-integrated compensator); NSMPP and THP-GMM evaluate their density directly and integrate the compensator by numerical quadrature; the generative families SMASH and DSTPP report a sample-based approximation (a KDE on their native samples), which is not a likelihood bound. The three labels are kept visible in every table. Beyond NLL, the predictive and distributional metric profiles give a fuller account of model quality, which is especially useful for models whose likelihood is only approximate.

Exactdirectly comparable

auto_stppdeep_stppnjsdeneural_*poisson_*hawkes_*selfcorrecting_*rmtpp_gmmnsmpp · quadraturethp_gmm · quadrature

Approximatesample-based estimate, not a bound

smash · sample KDEdiffusion_stpp · sample KDE

How a model declares its NLL type is covered in capabilities.

What a campaign leaves behind¶

Every run can be reproduced from disk. Alongside each fit's config, metrics, and checkpoint, a campaign writes:

File	What it's for
`report.html`	Self-contained comparison report, viewable in a browser
`table_test_nll_all.csv`	Test NLL for every cell
`table_test_nll_exact.csv`	The exact-NLL tier only
`cell_index.json`	Maps each cell to its saved run directory

Full layout and per-run files: Artifacts and run directories.

If you arrived from Add your model, your model now appears in the same tables as the established baselines. From here, read the comparison or reproduce the published results.