Metadata-Version: 2.1
Name: dragon_eval
Version: 0.2.1
Summary: Evaluation method for the DRAGON benchmark
Home-page: https://github.com/DIAGNijmegen/dragon_eval
Author: Joeran Bosma
Author-email: Joeran.Bosma@radboudumc.nl
Project-URL: Bug Tracker, https://github.com/DIAGNijmegen/dragon_eval/issues
Platform: unix
Platform: linux
Platform: osx
Platform: cygwin
Platform: win32
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: seqeval
Requires-Dist: evalutils==0.4.2
Requires-Dist: scikit-learn
Provides-Extra: testing
Requires-Dist: pytest>=6.0; extra == "testing"
Requires-Dist: pytest-cov>=2.0; extra == "testing"
Requires-Dist: mypy>=0.910; extra == "testing"
Requires-Dist: flake8>=3.9; extra == "testing"
Requires-Dist: tox>=3.24; extra == "testing"

# DRAGON Evaluation

Evaluation method for the DRAGON (Diagnostic Report Analysis: General Optimization of NLP) challenge. 

## Installation
A pre-built Docker container with the DRAGON evaluation method is available:

```
docker pull joeranbosma/dragon_eval
```

The DRAGON evaluation method can be pip-installed:

```
pip install dragon_eval
```

Or, alternatively, it can be installed from source:

```
pip install git+https://github.com/DIAGNijmegen/dragon_eval
```

The evaluation method was tested with Python 3.10. See [requirements.txt](requirements.txt) for a full list of exact package versions.

## Usage
The Docker container can be used to evaluate the synthetic datasets as specified in [evaluate.sh](evaluate.sh). To evaluate the synthetic tasks, place the predictions to evaluate in the `test-predictions` folder and run `./evaluate.sh`.

The DRAGON evaluation method can also be used from the command line (if installed with pip):

```
python -m dragon_eval --ground-truth-path="ground-truth" --predictions-path=test-predictions --output-file=metrics.json --folds 0 1 2 3 4 --tasks 000 001 002 003 004 005 006 007
```

The command above should work when executed from the `dragon_eval` folder, which needs to be cloned locally for the ground truth and prediction files to be present. Change the paths above when executing the command from a different place or storing the files in a different place. The tasks and folds to evaluate can be changed with the respective parameters.
