Metadata-Version: 2.1
Name: lrbenchmark
Version: 0.1.1
Summary: Benchmarking Likelihood Ratio systems
Author: Netherlands Forensics Institute
Description-Content-Type: text/markdown
License-File: LICENSE.txt

LRBenchmark
=====

Repository for benchmarking Likelihood Ratio systems.

Prerequisites
-----------
- This repository is developed for Python 3.8.

Dependencies
-----------
All dependencies can either be installed by running `pip install -r requirements.txt` or `pip install .`.
  
Add new dependencies to the setup.py and always update the requirements by running: 
`pip-compile --output-file=requirements.txt setup.py`.

Usage
-----------
Running the benchmark can be done as follows:
1. Specify the parameters for the benchmark in the `lrbenchmark.yaml`
2. Run `python run.py`

The parameters for the benchmark must be provided in the following structure: 
```
experiment:
  repeats: 10
  scorer:
    - 'name scorer 1'
    - 'name scorer 2'
  calibrator: 
    - 'name calibrator'
  dataset:
    - 'name dataset'
  preprocessor:
    - 'name preprocessor 1'
    - 'name preprocessor 2'
```
At least 1 setting needs to be provided for each parameter, but more settings per parameter can be provided. The pipeline will
create the cartesian product over all parameter settings (except `repeats`) and will execute the experiments accordingly.

All possible settings can be found in `params.py`. The parameters that need to be set are:
- `repeats`: Number of repeats for each experiment.
- `scorer`: Scoring models for generating scores.
- `calibrator`: Models for calibrating scores. 
- `dataset`: Datasets on which the experiments can be executed.
- `preprocessor`: Data preprocessing steps. You can use the value `'dummy'` if no preprocessing is needed.


Example: Benchmark feature rank based LR system
-----------
This repository supports several data transformations, such as the possibility to transform X features from values to ranks. 
To benchmark these models against models without any transformations on X features, the following experiments (among others) could be 
defined in `lrbenchmark.yaml`. 
```
experiment:
  repeats: 10
  scorer:
    - 'LR'
    - 'XGB'
  calibrator:
    - 'logit'
  dataset:
    drugs_xtc:
      n_splits: 2
    glass:
      n_splits: 2
  preprocessor:
    - 'dummy'
    - 'rank_transformer'
```
When executing `python run.py` an experiment for all possible combination of parameters will be executed. The results for each experiment (metrics + plots)
will be stored in separate folders within the `output` folder.

Datasets
----------
There are currently two datasets implemented for this project:
- drugs_xtc: will be published on our github soon
- glass: LA-ICPMS measurements of elemental concentration from floatglass. The data will be downloaded automatically from https://github.com/NetherlandsForensicInstitute/elemental_composition_glass when used in the pipeline for the first time. 
