Declare Capabilities¶

Seahorse evaluation profiles are gated by capability flags. Before adding a preset to benchmark examples, decide which capabilities it supports and verify them with explicit tests.

What each capability unlocks¶

Each capability is a method you implement. Implement it → its metrics turn on. Skip it → those metrics report a clean skip, never a wrong number.

Exact NLLexact

declared by log_prob() returning an exact log-likelihood unlocks

corenllbenchmark NLL tables

Bounded NLLbound

declared by log_prob() returning a surrogate (score-matching, ELBO) unlocks

nll tables · non-exact

Next-event samplingoptional

declared by implementing sample_next() unlocks

predictivepredict_next

Generative rolloutsoptional

declared by a sampling path that supports multi-step rollout unlocks

generativeautoregressive

Intensity surfaceoptional

declared by implementing intensity_grid() unlocks

surfaceevaluate surface

Save/load and re-evaluation from disk come for free through the runner — there is no capability to declare.

Declaring in EventModel¶

Capabilities are declared by which methods are implemented (not raising NotImplementedError):

class MyEventModel(nn.Module):

    def log_prob(self, times, locations, state, mask):
        """Exact log-likelihood — enables NLL metrics."""
        ...

    def sample_next(self, state, t_last, n_samples=1):
        """Implement for next-event sampling; raise NotImplementedError otherwise."""
        ...

    def intensity_grid(self, state, t_query, s_query):
        """Implement for surface diagnostics; raise NotImplementedError otherwise."""
        ...

When a method raises NotImplementedError, the corresponding metric is marked available: false in metrics.json with a clear reason — it is not treated as a failure.

Exact vs approximate NLL¶

NLL is only comparable across families that compute it the same way. Seahorse keeps the two tiers visibly separate in benchmark tables.

Exactdirectly comparable

auto_stppdeep_stppnsmppnjsdeneural_*poisson_*hawkes_*rmtpp_gmmthp_gmm

Approximatea bound on the likelihood

smash · score-matchingdiffusion_stpp · ELBO

Capability Testing¶

Before adding your preset to benchmark examples, verify each claimed capability:

from seahorse import STPPEstimator, load_jsonl

train = load_jsonl("data/tiny/train.jsonl")
val   = load_jsonl("data/tiny/val.jsonl")
test  = load_jsonl("data/tiny/test.jsonl")

model = STPPEstimator("my_preset", device="cpu")
model.fit(train, val, test, epochs=1, batch_size=4)

# Test NLL
scores = model.evaluate(test)
assert "test_nll" in scores

# Test sampling (if claimed)
samples = model.predict_next(test[:2], n_samples=4)
assert "next_times" in samples

# Test save/load
save_dir = model.save("runs/test_save")
loaded = STPPEstimator.load(save_dir)
scores2 = loaded.evaluate(test)
assert abs(scores["test_nll"] - scores2["test_nll"]) < 1e-4

See the Testing Checklist for the full test suite.