Quickstart with scikit-learn¶

The full workflow on real data: fit a classifier, hand it to rocci, read the band. We use the diabetes dataset that ships with scikit-learn, predicting whether a patient's disease progresses above the cohort median from three routine measurements — blood pressure and two blood-serum panels. This is a genuinely hard task (AUC around 0.75), which is exactly where a confidence band earns its keep: the uncertainty is real and worth quantifying.

In [1]:

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

data = load_diabetes(as_frame=True)
X = data.data[["bp", "s3", "s4"]]
y = (data.target > data.target.median()).astype(int)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, stratify=y, random_state=0
)
clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
clf.fit(X_train, y_train)

from sklearn.datasets import load_diabetes from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler data = load_diabetes(as_frame=True) X = data.data[["bp", "s3", "s4"]] y = (data.target > data.target.median()).astype(int) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.5, stratify=y, random_state=0 ) clf = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)) clf.fit(X_train, y_train)

Out[1]:

Pipeline(steps=[('standardscaler', StandardScaler()),
                ('logisticregression', LogisticRegression(max_iter=1000))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

One call¶

from_estimator mirrors scikit-learn's RocCurveDisplay.from_estimator: it calls predict_proba (falling back to decision_function), selects the positive-class column, and builds the band on held-out data.

In [2]:

from rocci import from_estimator

band = from_estimator(clf, X_test, y_test, random_state=0)
band.plot(show_vacuous=True)

from rocci import from_estimator band = from_estimator(clf, X_test, y_test, random_state=0) band.plot(show_vacuous=True)

Out[2]:

<Axes: xlabel='False positive rate', ylabel='True positive rate'>

The shaded region contains the entire true population ROC curve with 95% confidence — not each point separately, the whole curve at once. The hatched sliver at the far left is the region where no distribution-free lower bound exists (see below).

Reading the summary¶

In [3]:

print(band.summary())

print(band.summary())

rocci confidence band (envelope)
  samples: n_neg=111, n_pos=110
  coverage: 95% simultaneous
  AUC: 0.7552  (CI: 0.6925, 0.8151)
  band area (mean width): 0.3287
  backend: rust, n_boot=2000
  no distribution-free lower bound exists below FPR ~= 0.0603; increase the number of negatives to certify lower FPRs.
  floor jurisdictions: Beta floor: 39 pts, Wilson floor: 38 pts
  note: y_score looked like (n, 2) predict_proba output; used column 1 as the positive-class score.
  please cite rocci (see CITATION.cff).

Line by line:

• coverage: 95% simultaneous — the joint, whole-curve guarantee. You may read off as many operating points as you like without a multiplicity correction.
• AUC + CI — the exact Mann-Whitney AUC (identical to sklearn.roc_auc_score) with a bootstrap confidence interval.
• band area — mean vertical width; the tightness metric to compare settings or datasets.
• the vacuous-region line — below that FPR, certifying any lower bound is mathematically impossible without distributional assumptions; rocci says so instead of drawing one.

Operating points¶

Suppose the screening context tolerates at most 10% false positives. What TPR can we actually claim there?

In [4]:

lower, tpr, upper = band.at(0.10)
print(f"at FPR = 10%:  TPR = {float(tpr):.3f},  "
      f"simultaneous bounds [{float(lower):.3f}, {float(upper):.3f}]")

lower, tpr, upper = band.at(0.10) print(f"at FPR = 10%: TPR = {float(tpr):.3f}, " f"simultaneous bounds [{float(lower):.3f}, {float(upper):.3f}]")

at FPR = 10%:  TPR = 0.336,  simultaneous bounds [0.089, 0.645]

A 10% false-positive tolerance is 90% specificity, and TPR is sensitivity, so the same question reads more naturally as sens_at_spec — it returns exactly the values above:

In [5]:

lower, sens, upper = band.sens_at_spec(0.90)
print(f"at 90% specificity:  sensitivity = {float(sens):.3f},  "
      f"simultaneous bounds [{float(lower):.3f}, {float(upper):.3f}]")

lower, sens, upper = band.sens_at_spec(0.90) print(f"at 90% specificity: sensitivity = {float(sens):.3f}, " f"simultaneous bounds [{float(lower):.3f}, {float(upper):.3f}]")

at 90% specificity:  sensitivity = 0.336,  simultaneous bounds [0.089, 0.645]

You can also invert the question — at what specificity does the model reach a target sensitivity? — with spec_at_sens:

In [6]:

lower, spec, upper = band.spec_at_sens(0.80)
print(f"to reach 80% sensitivity:  specificity = {float(spec):.3f},  "
      f"simultaneous bounds [{float(lower):.3f}, {float(upper):.3f}]")

lower, spec, upper = band.spec_at_sens(0.80) print(f"to reach 80% sensitivity: specificity = {float(spec):.3f}, " f"simultaneous bounds [{float(lower):.3f}, {float(upper):.3f}]")

to reach 80% sensitivity:  specificity = 0.505,  simultaneous bounds [0.252, 0.766]

Because the band is simultaneous, querying ten operating points instead of one — in either direction — costs nothing in validity.

The band as data¶

In [7]:

band.to_dataframe().head()

band.to_dataframe().head()

Out[7]:

	fpr	lower	tpr	upper	attribution
0	0.000000	0.0	0.000000	0.218182	3
1	0.009009	0.0	0.136364	0.309091	1
2	0.018018	0.0	0.172727	0.327273	1
3	0.027027	0.0	0.172727	0.427273	1
4	0.036036	0.0	0.190909	0.427273	1

Everything on the result is a plain NumPy array (or scalar); to_dataframe() is a convenience for the pandas-inclined. The attribution column records which mechanism produced each point of the lower arm — the anatomy vignette explains how to read it.