Metadata-Version: 2.1
Name: riverreliability
Version: 0.1.2
Summary: The river reliability diagram provides insight into classifier calibration with a visualization and metric based on the posterior balanaced accuracy.
Home-page: https://github.com/MaximLippeveld/riverreliability/tree/master/
Author: Maxim Lippeveld & Jonathan Peck
Author-email: maxim.lippeveld@ugent.be
License: Apache Software License 2.0
Keywords: reliability,confidence,classification,visualization,metric
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: matplotlib
Requires-Dist: numpy
Requires-Dist: scipy
Requires-Dist: pandas
Requires-Dist: sklearn

# River reliability



## Install

Install the package with:

`pip install riverreliability`

## How to use

Below, we show some basic funtionality of the package. Please look at the notebooks for more examples and documentation.

```python
np.random.seed(42)
```

We start of by generating a fake dataset for classification and splitting it in a train and test set.

```python
X, y = sklearn.datasets.make_classification(n_samples=5000, n_features=12, n_informative=3, n_classes=3)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, test_size=0.2, shuffle=True)
```

For this example we use an SVM. We fit it on the training data and generate probabilities for the test set.

```python
logreg = sklearn.svm.SVC(probability=True)
logreg.fit(X_train, y_train)
y_probs = logreg.predict_proba(X_test)
```

As a sanity check we compute some performance metrics.

```python
print(f"Accuracy: {sklearn.metrics.accuracy_score(y_test, y_probs.argmax(axis=1))}")
print(f"Balanced accuracy: {sklearn.metrics.balanced_accuracy_score(y_test, y_probs.argmax(axis=1))}")
```

    Accuracy: 0.808
    Balanced accuracy: 0.8084048918146675


To get an insight into calibration we can look at the posterior reliability diagrams and the PEACE metric.

We can plot the diagrams aggregated over all classes:

```python
ax = riverreliability.plots.river_reliability_diagram(y_probs.max(axis=1), y_probs.argmax(axis=1), y_test, bins="fd")

peace_metric = riverreliability.metrics.peace(y_probs.max(axis=1), y_probs.argmax(axis=1), y_test)
ax.set_title(f"PEACE: {peace_metric:.4f}")

_ = ax.legend()
```


![png](docs/images/output_11_0.png)


Or class-wise to spot miscalibrations for particular classes:

```python
import matplotlib.pyplot as plt
```

```python
axes = riverreliability.plots.class_wise_river_reliability_diagram(y_probs, y_probs.argmax(axis=1), y_test, bins=15)

peace_metric = riverreliability.metrics.class_wise_error(y_probs, y_probs.argmax(axis=1), y_test, base_error=riverreliability.metrics.peace)
_ = plt.suptitle(f"PEACE: {peace_metric:.4f}")
```


![png](docs/images/output_14_0.png)


In this particular example we can see that the classifier is well calibrated.

See the notebooks directory for more examples.


