Metadata-Version: 2.1
Name: ood-metrics
Version: 1.1.0
Summary: Calculate common OOD detection metrics
Home-page: https://github.com/tayden/ood-metrics
License: MIT
Keywords: OOD,out-of-distribution,anomaly,detection
Author: Taylor Denouden
Author-email: taylordenouden@gmail.com
Requires-Python: >=3.8,<3.11
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Requires-Dist: matplotlib (>=3.0,<4.0)
Requires-Dist: numpy (>=1.22,<2.0)
Requires-Dist: scikit-learn (>=1.0,<2.0)
Project-URL: Repository, https://github.com/tayden/ood-metrics
Description-Content-Type: text/markdown

# OOD Detection Metrics

Functions for computing metrics commonly used in the field of out-of-distribution (OOD) detection.

[![tests](https://github.com/tayden/ood-metrics/actions/workflows/tests.yml/badge.svg?branch=main)](https://github.com/tayden/ood-metrics/actions/workflows/tests.yml)

## Installation

`pip install ood-metrics`

## Metrics functions

### AUROC

Calculate and return the area under the ROC curve using unthresholded predictions on the data and a binary true label.

```python
from ood_metrics import auroc

labels = [0, 0, 0, 1, 0]
scores = [0.1, 0.3, 0.6, 0.9, 1.3]

assert auroc(scores, labels) == 0.75
```

### AUPR

Calculate and return the area under the Precision Recall curve using unthresholded predictions on the data and a binary true
label.

```python
from ood_metrics import aupr

labels = [0, 0, 0, 1, 0]
scores = [0.1, 0.3, 0.6, 0.9, 1.3]

assert aupr(scores, labels) == 0.25
```

### FPR @ 95% TPR

Return the FPR when TPR is at least 95%.

```python
from ood_metrics import fpr_at_95_tpr

labels = [0, 0, 0, 1, 0]
scores = [0.1, 0.3, 0.6, 0.9, 1.3]

assert fpr_at_95_tpr(scores, labels) == 0.25
```

### Detection Error

Return the misclassification probability when TPR is 95%.

```python
from ood_metrics import detection_error

labels = [0, 0, 0, 1, 0]
scores = [0.1, 0.3, 0.6, 0.9, 1.3]

assert detection_error(scores, labels) == 0.05
```

### Calculate all stats

Using predictions and labels, return a dictionary containing all novelty detection performance statistics.

```python
from ood_metrics import calc_metrics

labels = [0, 0, 0, 1, 0]
scores = [0.1, 0.3, 0.6, 0.9, 1.3]

assert calc_metrics(scores, labels) == {
    'fpr_at_95_tpr': 0.25,
    'detection_error': 0.05,
    'auroc': 0.75,
    'aupr_in': 0.25,
    'aupr_out': 0.94375
}
```

## Plotting functions

### Plot ROC

Plot an ROC curve based on unthresholded predictions and true binary labels.

```python

from ood_metrics import plot_roc

labels = [0, 0, 0, 1, 0]
scores = [0.1, 0.3, 0.6, 0.9, 1.3]

plot_roc(scores, labels)
# Generate Matplotlib AUROC plot
```

### Plot PR

Plot an Precision-Recall curve based on unthresholded predictions and true binary labels.

```python

from ood_metrics import plot_pr

labels = [0, 0, 0, 1, 0]
scores = [0.1, 0.3, 0.6, 0.9, 1.3]

plot_pr(scores, labels)
# Generate Matplotlib Precision-Recall plot
```

### Plot Barcode

Plot a visualization showing inliers and outliers sorted by their prediction of novelty.

```python

from ood_metrics import plot_barcode

labels = [0, 0, 0, 1, 0]
scores = [0.1, 0.3, 0.6, 0.9, 1.3]

plot_barcode(scores, labels)
# Shows visualization of sort order of labels occording to the scores.
```

