Metadata-Version: 2.1
Name: oc_preprocessing
Version: 0.0.5
Summary: This package is meant to preprocess OpenCitations source dumps so to make them easily usable in OpenCitations main processes, by deleting unused information, splitting big files, and validating identifiers
Home-page: https://github.com/opencitations/preprocess
Author: OpenCitations authors
Author-email: OpenCitations authors <contact@opencitations.net>
License: BSD
Project-URL: Homepage, https://github.com/opencitations/preprocess
Project-URL: Bug Tracker, https://github.com/opencitations/preprocess/issues
Keywords: preprocessing data dumps
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: ISC License (ISCL)
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE

[<img src="https://img.shields.io/badge/powered%20by-OpenCitations-%239931FC?labelColor=2D22DE" />](http://opencitations.net) [![Python package](https://github.com/opencitations/index/actions/workflows/python-package.yml/badge.svg?branch=farm_revision)](https://github.com/opencitations/index/actions/workflows/python-package.yml)
# OpenCitations: Preprocess

This software is meant to preprocess data dumps to be ingested in OpenCitations, provided by different data sources. 
The aim of the software is that of preprocessing data dumps in order to facilitate data parsing and extraction in OpenCitations Meta and OpenCitation Index processes. 
Note that preprocessing is not a mandatory step of data ingestion in OpenCitations. However, preprocessing is suggested when:
<ol>
<li>A consistent part of the bibliographic entities represented in the dump come without citation data</li>
<li>The dump content is redundant with respect to OpenCitations scopes (e.g.: duplicated citations retrievable both as addressed and received citations)</li>
<li>The dump consists of a unique big file, and it is too heavy to be processed all at once</li>
<li>A consistent part of the data provided is not relevant with respect to OpenCitations scopes (e.g.: discipline-specific and content-related metadata) </li>
</ol>


### Mandatory
- Python 3.8+

### Start the tests
```console
$ python -m unittest discover -s ./preprocessing/test -p "*.py"
```

## License
OpenCitations Index is released under the [ISC License](LICENSE).
