Metadata-Version: 2.1
Name: finetuning-scheduler
Version: 0.3.2
Summary: A PyTorch Lightning extension that enhances model experimentation with flexible fine-tuning schedules.
Home-page: https://github.com/speediedan/finetuning-scheduler
Download-URL: https://github.com/speediedan/finetuning-scheduler
Author: Dan Dale
Author-email: danny.dale@gmail.com
License: Apache-2.0
Project-URL: Bug Tracker, https://github.com/speediedan/finetuning-scheduler/issues
Project-URL: Documentation, https://finetuning-scheduler.readthedocs.io/en/latest/
Project-URL: Source Code, https://github.com/speediedan/finetuning-scheduler
Keywords: deep learning,pytorch,AI,machine learning,pytorch lightning,fine-tuning,finetuning
Classifier: Environment :: Console
Classifier: Natural Language :: English
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Image Recognition
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Requires-Python: >=3.7
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pytorch-lightning (<=1.8.2,>=1.8.0)
Requires-Dist: torch (>=1.9.*)
Provides-Extra: all
Requires-Dist: rich (!=10.15.0.a,>=10.2.2) ; extra == 'all'
Requires-Dist: fairscale (>=0.4.5) ; extra == 'all'
Requires-Dist: jsonargparse[signatures] (>=4.15.0) ; extra == 'all'
Requires-Dist: omegaconf (>=2.1.0) ; extra == 'all'
Requires-Dist: hydra-core (>=1.1.0) ; extra == 'all'
Requires-Dist: coverage (>=6.4) ; extra == 'all'
Requires-Dist: codecov (>=2.1) ; extra == 'all'
Requires-Dist: pytest (>=6.0) ; extra == 'all'
Requires-Dist: pytest-rerunfailures (>=10.2) ; extra == 'all'
Requires-Dist: twine (==3.2) ; extra == 'all'
Requires-Dist: mypy (>=0.920) ; extra == 'all'
Requires-Dist: flake8 (>=3.9.2) ; extra == 'all'
Requires-Dist: pre-commit (>=1.0) ; extra == 'all'
Requires-Dist: ipython[notebook] ; extra == 'all'
Requires-Dist: jupytext (>=1.10) ; extra == 'all'
Requires-Dist: nbval (>=0.9.6) ; extra == 'all'
Requires-Dist: datasets ; extra == 'all'
Requires-Dist: evaluate ; extra == 'all'
Requires-Dist: transformers (>=4.18.0) ; extra == 'all'
Requires-Dist: scikit-learn ; extra == 'all'
Requires-Dist: sentencepiece ; extra == 'all'
Provides-Extra: cli
Requires-Dist: jsonargparse[signatures] (>=4.15.0) ; extra == 'cli'
Requires-Dist: omegaconf (>=2.1.0) ; extra == 'cli'
Requires-Dist: hydra-core (>=1.1.0) ; extra == 'cli'
Provides-Extra: cpu
Requires-Dist: rich (!=10.15.0.a,>=10.2.2) ; extra == 'cpu'
Requires-Dist: jsonargparse[signatures] (>=4.15.0) ; extra == 'cpu'
Requires-Dist: omegaconf (>=2.1.0) ; extra == 'cpu'
Requires-Dist: hydra-core (>=1.1.0) ; extra == 'cpu'
Requires-Dist: coverage (>=6.4) ; extra == 'cpu'
Requires-Dist: codecov (>=2.1) ; extra == 'cpu'
Requires-Dist: pytest (>=6.0) ; extra == 'cpu'
Requires-Dist: pytest-rerunfailures (>=10.2) ; extra == 'cpu'
Requires-Dist: twine (==3.2) ; extra == 'cpu'
Requires-Dist: mypy (>=0.920) ; extra == 'cpu'
Requires-Dist: flake8 (>=3.9.2) ; extra == 'cpu'
Requires-Dist: pre-commit (>=1.0) ; extra == 'cpu'
Requires-Dist: ipython[notebook] ; extra == 'cpu'
Requires-Dist: jupytext (>=1.10) ; extra == 'cpu'
Requires-Dist: nbval (>=0.9.6) ; extra == 'cpu'
Requires-Dist: datasets ; extra == 'cpu'
Requires-Dist: evaluate ; extra == 'cpu'
Requires-Dist: transformers (>=4.18.0) ; extra == 'cpu'
Requires-Dist: scikit-learn ; extra == 'cpu'
Requires-Dist: sentencepiece ; extra == 'cpu'
Provides-Extra: cpu-extra
Requires-Dist: rich (!=10.15.0.a,>=10.2.2) ; extra == 'cpu-extra'
Requires-Dist: jsonargparse[signatures] (>=4.15.0) ; extra == 'cpu-extra'
Requires-Dist: omegaconf (>=2.1.0) ; extra == 'cpu-extra'
Requires-Dist: hydra-core (>=1.1.0) ; extra == 'cpu-extra'
Provides-Extra: dev
Requires-Dist: rich (!=10.15.0.a,>=10.2.2) ; extra == 'dev'
Requires-Dist: fairscale (>=0.4.5) ; extra == 'dev'
Requires-Dist: jsonargparse[signatures] (>=4.15.0) ; extra == 'dev'
Requires-Dist: omegaconf (>=2.1.0) ; extra == 'dev'
Requires-Dist: hydra-core (>=1.1.0) ; extra == 'dev'
Requires-Dist: coverage (>=6.4) ; extra == 'dev'
Requires-Dist: codecov (>=2.1) ; extra == 'dev'
Requires-Dist: pytest (>=6.0) ; extra == 'dev'
Requires-Dist: pytest-rerunfailures (>=10.2) ; extra == 'dev'
Requires-Dist: twine (==3.2) ; extra == 'dev'
Requires-Dist: mypy (>=0.920) ; extra == 'dev'
Requires-Dist: flake8 (>=3.9.2) ; extra == 'dev'
Requires-Dist: pre-commit (>=1.0) ; extra == 'dev'
Requires-Dist: ipython[notebook] ; extra == 'dev'
Requires-Dist: jupytext (>=1.10) ; extra == 'dev'
Requires-Dist: nbval (>=0.9.6) ; extra == 'dev'
Provides-Extra: examples
Requires-Dist: datasets ; extra == 'examples'
Requires-Dist: evaluate ; extra == 'examples'
Requires-Dist: transformers (>=4.18.0) ; extra == 'examples'
Requires-Dist: scikit-learn ; extra == 'examples'
Requires-Dist: sentencepiece ; extra == 'examples'
Requires-Dist: jsonargparse[signatures] (>=4.15.0) ; extra == 'examples'
Requires-Dist: omegaconf (>=2.1.0) ; extra == 'examples'
Requires-Dist: hydra-core (>=1.1.0) ; extra == 'examples'
Provides-Extra: extra
Requires-Dist: rich (!=10.15.0.a,>=10.2.2) ; extra == 'extra'
Requires-Dist: fairscale (>=0.4.5) ; extra == 'extra'
Requires-Dist: jsonargparse[signatures] (>=4.15.0) ; extra == 'extra'
Requires-Dist: omegaconf (>=2.1.0) ; extra == 'extra'
Requires-Dist: hydra-core (>=1.1.0) ; extra == 'extra'
Provides-Extra: ipynb
Requires-Dist: ipython[notebook] ; extra == 'ipynb'
Requires-Dist: jupytext (>=1.10) ; extra == 'ipynb'
Requires-Dist: nbval (>=0.9.6) ; extra == 'ipynb'
Provides-Extra: test
Requires-Dist: coverage (>=6.4) ; extra == 'test'
Requires-Dist: codecov (>=2.1) ; extra == 'test'
Requires-Dist: pytest (>=6.0) ; extra == 'test'
Requires-Dist: pytest-rerunfailures (>=10.2) ; extra == 'test'
Requires-Dist: twine (==3.2) ; extra == 'test'
Requires-Dist: mypy (>=0.920) ; extra == 'test'
Requires-Dist: flake8 (>=3.9.2) ; extra == 'test'
Requires-Dist: pre-commit (>=1.0) ; extra == 'test'

<div align="center">

<img src="https://github.com/speediedan/finetuning-scheduler/raw/v0.3.2/docs/source/_static/images/logos/logo_fts.png" width="401px">

**A PyTorch Lightning extension that enhances model experimentation with flexible fine-tuning schedules.**

______________________________________________________________________

<p align="center">
  <a href="https://finetuning-scheduler.readthedocs.io/en/stable/">Docs</a> •
  <a href="#Setup">Setup</a> •
  <a href="#examples">Examples</a> •
  <a href="#community">Community</a>
</p>

[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/finetuning-scheduler)](https://pypi.org/project/finetuning-scheduler/)
[![PyPI Status](https://badge.fury.io/py/finetuning-scheduler.svg)](https://badge.fury.io/py/finetuning-scheduler)
![Conda (channel only)](https://img.shields.io/conda/vn/conda-forge/finetuning-scheduler?color=%23000080)\
[![codecov](https://codecov.io/gh/speediedan/finetuning-scheduler/release/0.3.2/graph/badge.svg?flag=gpu)](https://codecov.io/gh/speediedan/finetuning-scheduler)
[![ReadTheDocs](https://readthedocs.org/projects/finetuning-scheduler/badge/?version=latest)](https://finetuning-scheduler.readthedocs.io/en/stable/)
[![DOI](https://zenodo.org/badge/455666112.svg)](https://zenodo.org/badge/latestdoi/455666112)
[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/speediedan/finetuning-scheduler/blob/master/LICENSE)

</div>

______________________________________________________________________

<img width="300px" src="https://github.com/speediedan/finetuning-scheduler/raw/v0.3.2/docs/source/_static/images/fts/fts_explicit_loss_anim.gif" alt="FinetuningScheduler explicit loss animation" align="right"/>

[FinetuningScheduler](https://finetuning-scheduler.readthedocs.io/en/stable/api/finetuning_scheduler.fts.html#finetuning_scheduler.fts.FinetuningScheduler) is simple to use yet powerful, offering a number of features that facilitate model research and exploration:

- easy specification of flexible fine-tuning schedules with explicit or regex-based parameter selection
  - implicit schedules for initial/naive model exploration
  - explicit schedules for performance tuning, fine-grained behavioral experimentation and computational efficiency
- automatic restoration of best per-phase checkpoints driven by iterative application of early-stopping criteria to each fine-tuning phase
- composition of early-stopping and manually-set epoch-driven fine-tuning phase transitions

______________________________________________________________________

## Setup

### Step 0: Install from PyPI

```bash
pip install finetuning-scheduler
```

<!--  -->

### Step 1: Import the FinetuningScheduler callback and start fine-tuning!

```python
from pytorch_lightning import Trainer
from finetuning_scheduler import FinetuningScheduler

trainer = Trainer(callbacks=[FinetuningScheduler()])
```

Get started by following [the Fine-Tuning Scheduler introduction](https://finetuning-scheduler.readthedocs.io/en/stable/index.html) which includes a [CLI-based example](https://finetuning-scheduler.readthedocs.io/en/stable/index.html#example-scheduled-fine-tuning-for-superglue) or by following the [notebook-based](https://pytorch-lightning.readthedocs.io/en/stable/notebooks/lightning_examples/finetuning-scheduler.html) Fine-Tuning Scheduler tutorial.

______________________________________________________________________

## Examples

### Scheduled Fine-Tuning For SuperGLUE

- [Notebook-based Tutorial](https://pytorch-lightning.readthedocs.io/en/stable/notebooks/lightning_examples/finetuning-scheduler.html)
- [CLI-based Tutorial](https://finetuning-scheduler.readthedocs.io/en/stable/#example-scheduled-fine-tuning-for-superglue)

______________________________________________________________________

## Continuous Integration

Fine-Tuning Scheduler is rigorously tested across multiple CPUs, GPUs and against major Python and PyTorch versions. Each Fine-Tuning Scheduler minor release (major.minor.patch) is paired with a PyTorch Lightning minor release (e.g. Fine-Tuning Scheduler 0.3 depends upon PyTorch Lightning 1.8).

To ensure maximum stability, the latest PyTorch Lightning patch release fully tested with Fine-Tuning Scheduler is set as a maximum dependency in Fine-Tuning Scheduler's requirements.txt (e.g. \<= 1.7.1). If you'd like to test a specific PyTorch Lightning patch version greater than that currently in Fine-Tuning Scheduler's requirements.txt, it will likely work but you should install Fine-Tuning Scheduler from source and update the requirements.txt as desired.

<details>
  <summary>Current build statuses for Fine-Tuning Scheduler </summary>

| System / (PyTorch/Python ver) |                                                                                                         1.9/3.7                                                                                                          |                                                                                                               1.13/3.7, 1.13/3.10                                                                                                                |
| :---------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
|      Linux \[GPUs\*\*\]       |                                                                                                            -                                                                                                             | [![Build Status](https://dev.azure.com//speediedan/finetuning-scheduler/_apis/build/status/Multi-GPU%20&%20Example%20Tests?branchName=refs%2Ftags%2F0.3.2)](https://dev.azure.com/speediedan/finetuning-scheduler/_build/latest?definitionId=2&branchName=refs%2Ftags%2F0.3.2) |
|     Linux (Ubuntu 20.04)      | [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?tag=0.3.2)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml) |             [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?tag=0.3.2)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml)             |
|           OSX (11)            | [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?tag=0.3.2)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml) |             [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?tag=0.3.2)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml)             |
|        Windows (2022)         | [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?tag=0.3.2)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml) |             [![Test](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml/badge.svg?tag=0.3.2)](https://github.com/speediedan/finetuning-scheduler/actions/workflows/ci_test-full.yml)             |

- \*\* tests run on two RTX 2070s

</details>

## Community

Fine-Tuning Scheduler is developed and maintained by the community in close communication with the [PyTorch Lightning team](https://pytorch-lightning.readthedocs.io/en/stable/governance.html). Thanks to everyone in the community for their tireless effort building and improving the immensely useful core PyTorch Lightning project.

PR's welcome! Please see the [contributing guidelines](https://finetuning-scheduler.readthedocs.io/en/stable/generated/CONTRIBUTING.html) (which are essentially the same as PyTorch Lightning's).

______________________________________________________________________

## Citing Fine-Tuning Scheduler

Please cite:

```tex
@misc{Dan_Dale_2022_6463952,
    author       = {Dan Dale},
    title        = {{Fine-Tuning Scheduler}},
    month        = Feb,
    year         = 2022,
    doi          = {10.5281/zenodo.6463952},
    publisher    = {Zenodo},
    url          = {https://zenodo.org/record/6463952}
    }
```

Feel free to star the repo as well if you find it useful or interesting. Thanks 😊!
