Metadata-Version: 2.1
Name: flytekit
Version: 0.13.0b8
Summary: Flyte SDK for Python
Home-page: https://github.com/lyft/flytekit
Maintainer: Lyft
Maintainer-email: flyte-eng@lyft.com
License: apache2
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.6
Description-Content-Type: text/markdown
Requires-Dist: flyteidl (<1.0.0,>=0.18.6)
Requires-Dist: click (<8.0,>=6.6)
Requires-Dist: croniter (<4.0.0,>=0.3.20)
Requires-Dist: deprecated (<2.0,>=1.0)
Requires-Dist: boto3 (<2.0,>=1.4.4)
Requires-Dist: python-dateutil (<=2.8.1,>=2.1)
Requires-Dist: grpcio (<2.0,>=1.3.0)
Requires-Dist: protobuf (<4,>=3.6.1)
Requires-Dist: pytimeparse (<2.0.0,>=1.1.8)
Requires-Dist: pytz (<2018.5,>=2017.2)
Requires-Dist: keyring (>=18.0.1)
Requires-Dist: requests (<3.0.0,>=2.18.4)
Requires-Dist: responses (>=0.10.7)
Requires-Dist: six (<2.0.0,>=1.9.0)
Requires-Dist: sortedcontainers (>=1.5.9<3.0.0)
Requires-Dist: statsd (<4.0.0,>=3.0.0)
Requires-Dist: urllib3 (<2.0.0,>=1.22)
Requires-Dist: wrapt (<2.0.0,>=1.0.0)
Requires-Dist: papermill (>=1.2.0)
Requires-Dist: ipykernel (>=5.0.0)
Requires-Dist: black (==19.10b0)
Requires-Dist: configparser (<4.0.0,>=3.0.0) ; python_version<"3"
Requires-Dist: futures (<4.0.0,>=3.2.0) ; python_version<"3"
Requires-Dist: pathlib2 (<3.0.0,>=2.3.2) ; python_version<"3"
Provides-Extra: all
Requires-Dist: pyspark (<3.0.0,>=2.4.0) ; extra == 'all'
Requires-Dist: k8s-proto (<1.0.0,>=0.0.3) ; extra == 'all'
Requires-Dist: numpy (<2.0.0,>=1.14.0) ; extra == 'all'
Requires-Dist: pandas (<2.0.0,>=0.22.0) ; extra == 'all'
Requires-Dist: pyarrow (<1.0.0,>=0.11.0) ; extra == 'all'
Requires-Dist: hmsclient (<1.0.0,>=0.0.1) ; extra == 'all'
Requires-Dist: torch (<2.0.0,>=1.0.0) ; extra == 'all'
Provides-Extra: all-spark2.4
Requires-Dist: pyspark (<3.0.0,>=2.4.0) ; extra == 'all-spark2.4'
Requires-Dist: k8s-proto (<1.0.0,>=0.0.3) ; extra == 'all-spark2.4'
Requires-Dist: numpy (<2.0.0,>=1.14.0) ; extra == 'all-spark2.4'
Requires-Dist: pandas (<2.0.0,>=0.22.0) ; extra == 'all-spark2.4'
Requires-Dist: pyarrow (<1.0.0,>=0.11.0) ; extra == 'all-spark2.4'
Requires-Dist: hmsclient (<1.0.0,>=0.0.1) ; extra == 'all-spark2.4'
Requires-Dist: torch (<2.0.0,>=1.0.0) ; extra == 'all-spark2.4'
Provides-Extra: all-spark3
Requires-Dist: pyspark (>=3.0.0) ; extra == 'all-spark3'
Requires-Dist: k8s-proto (<1.0.0,>=0.0.3) ; extra == 'all-spark3'
Requires-Dist: numpy (<2.0.0,>=1.14.0) ; extra == 'all-spark3'
Requires-Dist: pandas (<2.0.0,>=0.22.0) ; extra == 'all-spark3'
Requires-Dist: pyarrow (<1.0.0,>=0.11.0) ; extra == 'all-spark3'
Requires-Dist: hmsclient (<1.0.0,>=0.0.1) ; extra == 'all-spark3'
Requires-Dist: torch (<2.0.0,>=1.0.0) ; extra == 'all-spark3'
Provides-Extra: hive_sensor
Requires-Dist: hmsclient (<1.0.0,>=0.0.1) ; extra == 'hive_sensor'
Provides-Extra: pytorch
Requires-Dist: torch (<2.0.0,>=1.0.0) ; extra == 'pytorch'
Provides-Extra: schema
Requires-Dist: numpy (<2.0.0,>=1.14.0) ; extra == 'schema'
Requires-Dist: pandas (<2.0.0,>=0.22.0) ; extra == 'schema'
Requires-Dist: pyarrow (<1.0.0,>=0.11.0) ; extra == 'schema'
Provides-Extra: sidecar
Requires-Dist: k8s-proto (<1.0.0,>=0.0.3) ; extra == 'sidecar'
Provides-Extra: spark
Requires-Dist: pyspark (<3.0.0,>=2.4.0) ; extra == 'spark'
Provides-Extra: spark3
Requires-Dist: pyspark (>=3.0.0) ; extra == 'spark3'

# Flytekit

[![PyPI version fury.io](https://badge.fury.io/py/flytekit.svg)](https://pypi.python.org/pypi/flytekit/)
[![PyPI download day](https://img.shields.io/pypi/dd/flytekit.svg)](https://pypi.python.org/pypi/flytekit/)
[![PyPI download month](https://img.shields.io/pypi/dm/flytekit.svg)](https://pypi.python.org/pypi/flytekit/)
[![PyPI format](https://img.shields.io/pypi/format/flytekit.svg)](https://pypi.python.org/pypi/flytekit/)
[![PyPI implementation](https://img.shields.io/pypi/implementation/flytekit.svg)](https://pypi.python.org/pypi/flytekit/)
![Codecov](https://img.shields.io/codecov/c/github/lyft/flytekit?style=plastic)


Python Library for easily authoring, testing, deploying, and interacting with Flyte tasks, workflows, and launch plans. To understand more about flyte refer to,
 - [Flyte homepage](https://flyte.org)
 - [Flyte master repository](https://github.com/lyft/flyte)

## Installation

Flytekit is designed for minimal footprint, and thus some features must be installed as extras.

### Base Installation

This is the lightest-weight SDK install. This installation includes everything you need to interact with Flyte.

Modules include:
1. The full Flyte IDL and an additional model layer for easier extension of the data model.
2. gRPC client for communicating with the platform.
3. Implementations for authoring and extending all Flyte entities (including tasks, workflows, and launch plans).

Tools include:
1. flyte-cli (Command-Line Interface for Interacting with the Flyte Platform)
2. pyflyte (Command-Line tool for easing the registration of Flyte entities)

```bash
pip install flytekit
```

### Plugin Installation
#### Spark

If `@spark_task` is to be used, one should install the `spark` plugin.

```bash
pip install "flytekit[spark]" for Spark 2.4.x
pip install "flytekit[spark3]" for Spark 3.x
```

#### Schema 

If `Types.Schema()` is to be used for computations involving large dataframes, one should install the `schema` extension.

```bash
pip install "flytekit[schema]"
```

#### Sidecar

If `@sidecar_task` is to be used, one should install the `sidecar` plugin.

```bash
pip install "flytekit[sidecar]"
```

### Pytorch

If `@pytorch_task` is to be used, one should install the `pytorch` plugin.

```bash
pip install "flytekit[pytorch]"
```

### TensorFlow

If `@tensorflow_task` is to be used, one should install the `tensorflow` plugin.

```bash
pip install flytekit[tensorflow]
```

### Full Installation

To install all or multiple available plugins, one can specify them individually:

```bash
pip install "flytekit[sidecar,spark,schema]"
```

Or install them with the `all` or `all-spark2.4` or `all-spark3` directives which will install all the plugins and a specific Spark version.
 Please note that `all` currently defaults to Spark 2.4.x. In a future release (starting 0.15.x), `all` will be switched to use Spark 3.x.


```bash
pip install "flytekit[all]"
```

## Development

### Recipes

```
$ make
Available recipes:
  setup        Install requirements
  fmt          Format code with black and isort
  lint         Run linters
  test         Run tests
  requirements Compile requirements
```

### Setup (Do Once)

```bash
virtualenv ~/.virtualenvs/flytekit
source ~/.virtualenvs/flytekit/bin/activate
make setup
```

### Formatting

We use [black](https://github.com/psf/black) and [isort](https://github.com/timothycrosley/isort) to autoformat code. Run the following command to execute the formatters:

```bash
source ~/.virtualenvs/flytekit/bin/activate
make fmt
```

### Testing

#### Unit Testing

```bash
source ~/.virtualenvs/flytekit/bin/activate
make test
```

### Updating requirements

Update requirements in [`setup.py`](setup.py), or update requirements for development in [`dev-requirements.in`](dev-requirements.in). Then, validate, pin and freeze all requirements by running:

```bash
source ~/.virtualenvs/flytekit/bin/activate
make requirements
```

This will re-create the [`requirements.txt`](requirements.txt) and [`dev-requirements.txt`](dev-requirements.txt) files which will be used for testing. You will have also have to re-run `make setup` to update your local environment with the updated requirements.


