Metadata-Version: 2.1
Name: subaligner
Version: 0.3.6
Summary: Automatically synchronize and translate subtitles, or create new ones by transcribing, using pre-trained DNNs, Forced Alignments and Transformers.
Home-page: https://subaligner.readthedocs.io/en/latest/
Author: Xi Bai
Author-email: xi.bai.ed@gmail.com
License: MIT
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Intended Audience :: Developers
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: zict ==0.1.3
Requires-Dist: Werkzeug >=0.15.3
Requires-Dist: wrapt ==1.14.0
Requires-Dist: urllib3 ~=1.26.5
Requires-Dist: tornado ==5.1.0
Requires-Dist: toolz ==0.9.0
Requires-Dist: toml ==0.10.0
Requires-Dist: termcolor ==1.1.0
Requires-Dist: tensorflow <2.13,>=1.15.5
Requires-Dist: tblib ==1.3.2
Requires-Dist: six ~=1.15.0
Requires-Dist: scikit-learn <1.2.0
Requires-Dist: scipy <1.11.0
Requires-Dist: rsa ==4.7
Requires-Dist: PyYAML >=4.2b1
Requires-Dist: pytz ==2018.4
Requires-Dist: pystack-debugger ==0.8.0
Requires-Dist: pysubs2 ~=1.6.1
Requires-Dist: pysrt ==1.1.2
Requires-Dist: pyprof2calltree ==1.4.3
Requires-Dist: pydotplus ==2.0.2
Requires-Dist: pydot-ng ==1.0.0
Requires-Dist: pydot ==1.2.4
Requires-Dist: pycountry ~=20.7.3
Requires-Dist: pybind11 ~=2.11.1
Requires-Dist: pyasn1-modules ==0.2.7
Requires-Dist: pyasn1 ==0.4.8
Requires-Dist: py ==1.10.0
Requires-Dist: pluggy ==0.13.1
Requires-Dist: pkgconfig ~=1.5.5
Requires-Dist: pbr ==4.0.2
Requires-Dist: oauthlib ==3.1.0
Requires-Dist: numpy <1.24.0
Requires-Dist: numba >=0.50.0
Requires-Dist: networkx >=2.5.1
Requires-Dist: mccabe ==0.6.1
Requires-Dist: Markdown ==2.6.11
Requires-Dist: locket ==0.2.0
Requires-Dist: librosa <0.10.0
Requires-Dist: le-pycaption ==2.2.0a1
Requires-Dist: Keras-Preprocessing >=1.0.9
Requires-Dist: Keras-Applications >=1.0.8
Requires-Dist: joblib >=1.2.0
Requires-Dist: isort ==4.3.4
Requires-Dist: idna ==2.8
Requires-Dist: hyperopt ==0.2.4
Requires-Dist: html5lib ==1.0b9
Requires-Dist: h5py <4.0.0
Requires-Dist: HeapDict ==1.0.0
Requires-Dist: graphviz ==0.8.3
Requires-Dist: google-pasta ~=0.2
Requires-Dist: filelock <4.0.0
Requires-Dist: distributed ==1.13.0
Requires-Dist: decorator ==4.3.0
Requires-Dist: cycler ==0.10.0
Requires-Dist: cloudpickle ~=1.6.0
Requires-Dist: click ==5.1
Requires-Dist: chardet ~=3.0.4
Requires-Dist: certifi ==2019.11.28
Requires-Dist: captionstransformer ~=1.2.1
Requires-Dist: cachetools ==3.1.1
Requires-Dist: bleach ==3.3.0
Requires-Dist: beautifulsoup4 <4.9.0
Requires-Dist: astor ==0.7.1
Provides-Extra: dev
Requires-Dist: pygments ==2.7.4 ; extra == 'dev'
Requires-Dist: pylint ~=2.17.2 ; extra == 'dev'
Requires-Dist: parameterized ==0.8.1 ; extra == 'dev'
Requires-Dist: typing-extensions ==4.5.0 ; extra == 'dev'
Requires-Dist: types-setuptools ==57.4.9 ; extra == 'dev'
Requires-Dist: types-requests ==2.27.9 ; extra == 'dev'
Requires-Dist: mypy ==1.3.0 ; extra == 'dev'
Requires-Dist: pex <=2.1.80 ; extra == 'dev'
Requires-Dist: radish-bdd ~=0.13.3 ; extra == 'dev'
Requires-Dist: scikit-build ==0.11.1 ; extra == 'dev'
Requires-Dist: line-profiler ~=4.1.2 ; extra == 'dev'
Requires-Dist: snakeviz ==2.1.0 ; extra == 'dev'
Requires-Dist: twine <4.0.0 ; extra == 'dev'
Requires-Dist: pycodestyle ==2.5.0 ; extra == 'dev'
Requires-Dist: tox ~=3.23.0 ; extra == 'dev'
Requires-Dist: coverage ==5.5 ; extra == 'dev'
Requires-Dist: mock ==4.0.3 ; extra == 'dev'
Requires-Dist: aeneas ~=1.7.3.0 ; extra == 'dev'
Requires-Dist: openai-whisper ==20231117 ; extra == 'dev'
Requires-Dist: transformers <4.27.0 ; extra == 'dev'
Requires-Dist: torch <2.2.0 ; extra == 'dev'
Requires-Dist: sentencepiece ~=0.1.95 ; extra == 'dev'
Requires-Dist: docutils ~=0.17.0 ; extra == 'dev'
Requires-Dist: sphinx-rtd-theme ==0.5.0 ; extra == 'dev'
Requires-Dist: sphinx ==3.3.1 ; extra == 'dev'
Provides-Extra: docs
Requires-Dist: docutils ~=0.17.0 ; extra == 'docs'
Requires-Dist: sphinx-rtd-theme ==0.5.0 ; extra == 'docs'
Requires-Dist: sphinx ==3.3.1 ; extra == 'docs'
Provides-Extra: harmony
Requires-Dist: aeneas ~=1.7.3.0 ; extra == 'harmony'
Requires-Dist: openai-whisper ==20231117 ; extra == 'harmony'
Requires-Dist: transformers <4.27.0 ; extra == 'harmony'
Requires-Dist: torch <2.2.0 ; extra == 'harmony'
Requires-Dist: sentencepiece ~=0.1.95 ; extra == 'harmony'
Provides-Extra: llm
Requires-Dist: openai-whisper ==20231117 ; extra == 'llm'
Requires-Dist: transformers <4.27.0 ; extra == 'llm'
Requires-Dist: torch <2.2.0 ; extra == 'llm'
Requires-Dist: sentencepiece ~=0.1.95 ; extra == 'llm'
Provides-Extra: stretch
Requires-Dist: aeneas ~=1.7.3.0 ; extra == 'stretch'
Provides-Extra: translation
Requires-Dist: openai-whisper ==20231117 ; extra == 'translation'
Requires-Dist: transformers <4.27.0 ; extra == 'translation'
Requires-Dist: torch <2.2.0 ; extra == 'translation'
Requires-Dist: sentencepiece ~=0.1.95 ; extra == 'translation'

<div align="center">
  <img src="https://raw.githubusercontent.com/baxtree/subaligner/master/figures/subaligner.png" alt="subaligner" width="300" />
</div>

[![Build Status](https://github.com/baxtree/subaligner/actions/workflows/ci-pipeline.yml/badge.svg?branch=master)](https://github.com/baxtree/subaligner/actions/workflows/ci-pipeline.yml?query=branch%3Amaster) ![Codecov](https://img.shields.io/codecov/c/github/baxtree/subaligner)
[![Python 3.11](https://img.shields.io/badge/python-3.11-blue.svg)](https://www.python.org/downloads/release/python-3110/) [![Python 3.10](https://img.shields.io/badge/python-3.10-blue.svg)](https://www.python.org/downloads/release/python-3100/) [![Python 3.9](https://img.shields.io/badge/python-3.9-blue.svg)](https://www.python.org/downloads/release/python-390/) [![Python 3.8](https://img.shields.io/badge/python-3.8-blue.svg)](https://www.python.org/downloads/release/python-380/)
[![Documentation Status](https://readthedocs.org/projects/subaligner/badge/?version=latest)](https://subaligner.readthedocs.io/en/latest/?badge=latest)
[![GitHub license](https://img.shields.io/github/license/baxtree/subaligner)](https://github.com/baxtree/subaligner/blob/master/LICENSE)
[![PyPI](https://badge.fury.io/py/subaligner.svg)](https://badge.fury.io/py/subaligner)
[![Docker Pulls](https://img.shields.io/docker/pulls/baxtree/subaligner)](https://hub.docker.com/r/baxtree/subaligner)
[![Citation](https://zenodo.org/badge/228440472.svg)](https://doi.org/10.5281/zenodo.5603083)

## Supported Formats
Subtitle: SubRip, TTML, WebVTT, (Advanced) SubStation Alpha, MicroDVD, MPL2, TMP, EBU STL, SAMI, SCC and SBV.

Video/Audio: MP4, WebM, Ogg, 3GP, FLV, MOV, Matroska, MPEG TS, WAV, MP3, AAC, FLAC, etc.

:information_source: <small style="line-height: 1.2;">Subaligner relies on file extensions as default hints to process a wide range of audiovisual or subtitle formats. It is recommended to use extensions widely acceppted by the community to ensure compatibility.</small>

## Dependencies
Required by basic: [FFmpeg](https://www.ffmpeg.org/)
```
$ apt-get install ffmpeg
```
or
```
$ brew install ffmpeg
```

## Basic Installation
```
$ pip install -U pip && pip install -U setuptools wheel
$ pip install subaligner
```
or install from source:
```
$ git clone git@github.com:baxtree/subaligner.git && cd subaligner
$ pip install -U pip && pip install -U setuptools
$ python setup.py install
```
:information_source: <small style="line-height: 1.2;">It is highly recommended creating a virtual environment prior to installation.</small>

## Installation with Optional Packages Supporting Additional Features
```
# Install dependencies for enabling translation and transcription

$ pip install 'subaligner[llm]'
```
```
# Install dependencies for enabling forced alignment

$ pip install 'subaligner[stretch]'
```
```
# Install dependencies for setting up the development environment

$ pip install 'subaligner[dev]'
```
Note that both `subaligner[stretch]` and `subaligner[dev]` require additional dependencies to be pre-installed:
```
$ apt-get install espeak libespeak1 libespeak-dev espeak-data
```
or
```
$ brew install espeak
```
To install all supported features:
```
$ pip install 'subaligner[harmony]'
```

## Container Support
If you prefer using a containerised environment over installing everything locally, run:

```
$ docker run -v `pwd`:`pwd` -w `pwd` -it baxtree/subaligner bash
```
For users on Windows 10: [Docker Desktop](https://docs.docker.com/docker-for-windows/install/) is the only option at present.
Assuming your media assets are stored under `d:\media`, open built-in command prompt, PowerShell, or Windows Terminal and run:
```
docker pull baxtree/subaligner
docker run -v "/d/media":/media -w "/media" -it baxtree/subaligner bash
```

## Usage
```
# Single-stage alignment (high-level shift with lower latency)

$ subaligner -m single -v video.mp4 -s subtitle.srt
$ subaligner -m single -v https://example.com/video.mp4 -s https://example.com/subtitle.srt -o subtitle_aligned.srt
```
```
# Dual-stage alignment (low-level shift with higher latency)

$ subaligner -m dual -v video.mp4 -s subtitle.srt
$ subaligner -m dual -v https://example.com/video.mp4 -s https://example.com/subtitle.srt -o subtitle_aligned.srt
```
```
# Generate subtitles by transcribing audiovisual files
$ subaligner -m transcribe -v video.mp4 -ml eng -mr whisper -mf small -o subtitle_aligned.srt
$ subaligner -m transcribe -v video.mp4 -ml zho -mr whisper -mf medium -o subtitle_aligned.srt
```
```
# Alignment on segmented plain texts (double newlines as the delimiter)

$ subaligner -m script -v test.mp4 -s subtitle.txt -o subtitle_aligned.srt
$ subaligner -m script -v https://example.com/video.mp4 -s https://example.com/subtitle.txt -o subtitle_aligned.srt
```
```
# Alignment on multiple subtitles against the single media file

$ subaligner -m script -v test.mp4 -s subtitle_lang_1.txt -s subtitle_lang_2.txt
$ subaligner -m script -v test.mp4 -s subtitle_lang_1.txt subtitle_lang_2.txt
```
```
# Alignment on embedded subtitles

$ subaligner -m single -v video.mkv -s embedded:stream_index=0 -o subtitle_aligned.srt
$ subaligner -m dual -v video.mkv -s embedded:stream_index=0 -o subtitle_aligned.srt
```
```
# Translative alignment with the ISO 639-3 language code pair (src,tgt)

$ subaligner --languages
$ subaligner -m single -v video.mp4 -s subtitle.srt -t src,tgt
$ subaligner -m dual -v video.mp4 -s subtitle.srt -t src,tgt
$ subaligner -m script -v test.mp4 -s subtitle.txt -o subtitle_aligned.srt -t src,tgt
$ subaligner -m dual -v video.mp4 -tr helsinki-nlp -o subtitle_aligned.srt -t src,tgt
$ subaligner -m dual -v video.mp4 -tr facebook-mbart -tf large -o subtitle_aligned.srt -t src,tgt
$ subaligner -m dual -v video.mp4 -tr whisper -tf small -o subtitle_aligned.srt -t src,eng
```
```
# Transcribe audiovisual files and generate translated subtitles

$ subaligner -m transcribe -v video.mp4 -ml src -mr whisper -mf small -tr helsinki-nlp -o subtitle_aligned.srt -t src,tgt
```
```
# Shift subtitle manually by offset in seconds

$ subaligner -m shift --subtitle_path subtitle.srt -os 5.5
$ subaligner -m shift --subtitle_path subtitle.srt -os -5.5 -o subtitle_shifted.srt
```
```
# Run batch alignment against directories

$ subaligner_batch -m single -vd videos/ -sd subtitles/ -od aligned_subtitles/
$ subaligner_batch -m dual -vd videos/ -sd subtitles/ -od aligned_subtitles/
$ subaligner_batch -m dual -vd videos/ -sd subtitles/ -od aligned_subtitles/ -of ttml
```
```
# Run alignments with pipx

$ pipx run subaligner -m single -v video.mp4 -s subtitle.srt
$ pipx run subaligner -m dual -v video.mp4 -s subtitle.srt
```
```
# Run the module as a script
$ python -m subaligner -m single -v video.mp4 -s subtitle.srt
$ python -m subaligner -m dual -v video.mp4 -s subtitle.srt
```
```
# Run alignments with the docker image

$ docker pull baxtree/subaligner
$ docker run -v `pwd`:`pwd` -w `pwd` -it baxtree/subaligner subaligner -m single -v video.mp4 -s subtitle.srt
$ docker run -v `pwd`:`pwd` -w `pwd` -it baxtree/subaligner subaligner -m dual -v video.mp4 -s subtitle.srt
$ docker run -it baxtree/subaligner subaligner -m single -v https://example.com/video.mp4 -s https://example.com/subtitle.srt -o subtitle_aligned.srt
$ docker run -it baxtree/subaligner subaligner -m dual -v https://example.com/video.mp4 -s https://example.com/subtitle.srt -o subtitle_aligned.srt
```
The aligned subtitle will be saved at `subtitle_aligned.srt`. For details on CLIs, run `subaligner -h` or `subaligner_batch -h`, `subaligner_convert -h`, `subaligner_train -h` and `subaligner_tune -h` for additional utilities. `subaligner_1pass` and `subaligner_2pass` are shortcuts for running `subaligner` with `-m single` and `-m dual` options, respectively.

![](figures/screencast.gif)

## Advanced Usage
You can train a new model with your own audiovisual files and subtitle files:
```
$ subaligner_train -vd VIDEO_DIRECTORY -sd SUBTITLE_DIRECTORY -tod TRAINING_OUTPUT_DIRECTORY
```
Then you can apply it to your subtitle synchronisation with the aforementioned commands. For more details on how to train and tune your own model, please refer to [Subaligner Docs](https://subaligner.readthedocs.io/en/latest/advanced_usage.html).

## Anatomy
Subtitles can be out of sync with their companion audiovisual media files for a variety of causes including latency introduced by Speech-To-Text on live streams or calibration and rectification involving human intervention during post-production.

A model has been trained with synchronised video and subtitle pairs and later used for predicating shifting offsets and directions under the guidance of a dual-stage aligning approach. 

First Stage (Global Alignment):
![](figures/1st_stage.png)

Second Stage (Parallelised Individual Alignment):
![](figures/2nd_stage.png)

## Acknowledgement
This tool wouldn't be possible without the following packages:
[librosa](https://librosa.github.io/librosa/)
[tensorflow](https://www.tensorflow.org/)
[scikit-learn](https://scikit-learn.org)
[pycaption](https://pycaption.readthedocs.io)
[pysrt](https://github.com/byroot/pysrt)
[pysubs2](https://github.com/tkarabela/pysubs2)
[aeneas](https://www.readbeyond.it/aeneas/)
[transformers](https://huggingface.co/transformers/)
[openai-whisper](https://github.com/openai/whisper).

Thanks to Alan Robinson and Nigel Megitt for their invaluable feedback.


