Metadata-Version: 2.1
Name: metaphor-connectors
Version: 0.13.61
Summary: A collection of Python-based 'connectors' that extract metadata from various sources to ingest into the Metaphor app.
Home-page: https://metaphor.io
License: Apache-2.0
Author: Metaphor
Author-email: dev@metaphor.io
Requires-Python: >=3.8.1,<4.0
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Provides-Extra: all
Provides-Extra: bigquery
Provides-Extra: datafactory
Provides-Extra: dbt
Provides-Extra: looker
Provides-Extra: metabase
Provides-Extra: monte-carlo
Provides-Extra: mssql
Provides-Extra: mysql
Provides-Extra: postgresql
Provides-Extra: power-bi
Provides-Extra: redshift
Provides-Extra: snowflake
Provides-Extra: synapse
Provides-Extra: tableau
Provides-Extra: throughtspot
Provides-Extra: unity-catalog
Requires-Dist: GitPython (>=3.1.37,<4.0.0); extra == "all" or extra == "looker"
Requires-Dist: PyYAML (>=6.0,<7.0)
Requires-Dist: SQLAlchemy (>=1.4.46,<2.0.0); extra == "all" or extra == "mysql"
Requires-Dist: asyncpg (>=0.27.0,<0.28.0); extra == "all" or extra == "postgresql" or extra == "redshift"
Requires-Dist: aws-assume-role-lib (>=2.10.0,<3.0.0)
Requires-Dist: azure-identity (>=1.14.0,<2.0.0); extra == "all" or extra == "datafactory"
Requires-Dist: azure-mgmt-datafactory (>=3.1.0,<4.0.0); extra == "all" or extra == "datafactory"
Requires-Dist: boto3 (>=1.28.57,<2.0.0)
Requires-Dist: botocore (>=1.31.57,<2.0.0)
Requires-Dist: canonicaljson (>=2.0.0,<3.0.0)
Requires-Dist: databricks-sdk (>=0.12.0,<0.13.0); extra == "all" or extra == "unity-catalog"
Requires-Dist: databricks-sql-connector (>=2.9.3,<3.0.0); extra == "all" or extra == "unity-catalog"
Requires-Dist: google-cloud-bigquery (>=3.1.0,<4.0.0); extra == "all" or extra == "bigquery"
Requires-Dist: google-cloud-logging (>=3.5.0,<4.0.0); extra == "all" or extra == "bigquery"
Requires-Dist: jsonschema (>=4.18.6,<5.0.0)
Requires-Dist: lkml (>=1.3.1,<2.0.0); extra == "all" or extra == "looker"
Requires-Dist: looker-sdk (>=23.6.0,<24.0.0); extra == "all" or extra == "looker"
Requires-Dist: metaphor-models (==0.30.5)
Requires-Dist: msal (>=1.20.0,<2.0.0); extra == "all" or extra == "power-bi"
Requires-Dist: msgraph-beta-sdk (==1.0.0); extra == "all" or extra == "power-bi"
Requires-Dist: pathvalidate (>=3.2.0,<4.0.0)
Requires-Dist: pyarrow[pandas] (>=10.0.1,<11.0.0)
Requires-Dist: pycarlo (>=0.8.1,<0.9.0); extra == "all" or extra == "monte-carlo"
Requires-Dist: pydantic[email] (==2.5.1)
Requires-Dist: pymssql (>=2.2.9,<3.0.0); extra == "all" or extra == "mssql" or extra == "synapse"
Requires-Dist: pymysql (>=1.0.2,<2.0.0); extra == "all" or extra == "mysql"
Requires-Dist: python-dateutil (>=2.8.1,<3.0.0)
Requires-Dist: requests (>=2.28.1,<3.0.0)
Requires-Dist: setuptools (>=68.0.0,<69.0.0)
Requires-Dist: smart-open (>=6.3.0,<7.0.0)
Requires-Dist: snowflake-connector-python (>=3.5.0,<4.0.0); extra == "all" or extra == "snowflake"
Requires-Dist: sql-metadata (>=2.8.0,<3.0.0); extra == "all" or extra == "bigquery" or extra == "looker" or extra == "metabase" or extra == "power-bi" or extra == "snowflake"
Requires-Dist: sqllineage (>=1.3.8,<1.4.0); extra == "all" or extra == "redshift" or extra == "tableau"
Requires-Dist: tableauserverclient (>=0.25,<0.26); extra == "all" or extra == "tableau"
Requires-Dist: thoughtspot_rest_api_v1 (==1.5.3); extra == "all" or extra == "throughtspot"
Project-URL: Repository, https://github.com/MetaphorData/connectors
Description-Content-Type: text/markdown

<a href="https://metaphor.io"><img src="https://github.com/MetaphorData/connectors/raw/main/logo.png" width="300" /></a>

# Metaphor Connectors

[![Codecov](https://img.shields.io/codecov/c/github/MetaphorData/connectors)](https://app.codecov.io/gh/MetaphorData/connectors/tree/main)
[![CodeQL](https://github.com/MetaphorData/connectors/workflows/CodeQL/badge.svg)](https://github.com/MetaphorData/connectors/actions/workflows/codeql-analysis.yml)
[![PyPI Version](https://img.shields.io/pypi/v/metaphor-connectors)](https://pypi.org/project/metaphor-connectors/)
![Python version 3.8+](https://img.shields.io/badge/python-3.8%2B-blue)
![PyPI Downloads](https://img.shields.io/pypi/dm/metaphor-connectors)
[![Docker Pulls](https://img.shields.io/docker/pulls/metaphordata/connectors)](https://hub.docker.com/r/metaphordata/connectors)
[![License](https://img.shields.io/github/license/MetaphorData/connectors)](https://github.com/MetaphorData/connectors/blob/master/LICENSE)

This repository contains a collection of Python-based "connectors" that extract metadata from various sources to ingest into the [Metaphor](https://metaphor.io) platform.

## Installation

This package requires Python 3.8+ installed. You can verify the version on your system by running the following command,

```shell
python -V  # or python3 on some systems
```

Once verified, you can install the package using [pip](https://docs.python.org/3/installing/index.html),

```shell
pip install "metaphor-connectors[all]"  # or pip3 on some systems
```

This will install all the connectors and required dependencies. You can also choose to install only a subset of the dependencies by installing the specific [extra](https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras), e.g.

```shell
pip install "metaphor-connectors[snowflake]"
```

Similarly, you can also install the package using `requirements.txt` or `pyproject.toml`.

## Docker

We automatically push a [docker image](https://hub.docker.com/r/metaphordata/connectors) to Docker Hub as par of the CI/CD. See [this page](./docs/docker.md) for more details.

## GitHub Action

You can also run the connectors in your CI/CD pipeline using the [Metaphor Connectors](https://github.com/marketplace/actions/metaphor-connectors-github-action) GitHub Action.

## Connectors

Each connector is placed under its own directory under [metaphor](./metaphor) and extends the `metaphor.common.BaseExtractor` class.

| Connector Name                                                    | Metadata                                 |
|-------------------------------------------------------------------|------------------------------------------|  
| [azure_data_factory](metaphor/azure_data_factory/README.md)       | Lineage, Pipeline                        |
| [bigquery](metaphor/bigquery/README.md)                           | Schema, description, statistics, queries |
| [bigquery.lineage](metaphor/bigquery/lineage/README.md)           | Lineage                                  |
| [bigquery.profile](metaphor/bigquery/profile/README.md)           | Data profile                             |
| [dbt](metaphor/dbt/README.md)                                     | dbt model, test, lineage                 |
| [dbt.cloud](metaphor/dbt/cloud/README.md)                         | dbt model, test, lineage                 |
| [fivetran](metaphor/fivetran/README.md)                           | Lineage, Pipeline                        |
| [glue](metaphor/glue/README.md)                                   | Schema, description                      |
| [looker](metaphor/looker/README.md)                               | Looker view, explore, dashboard, lineage |
| [custom.data_quality](metaphor/custom/data_quality/README.md)     | Data quality                             |
| [custom.governance](metaphor/custom/governance/README.md)         | Ownership, tags, description             |
| [custom.lineage](metaphor/custom/lineage/README.md)               | Lineage                                  |
| [custom.metadata](metaphor/custom/metadata/README.md)             | Custom metadata                          |
| [custom.query_attributions](metaphor/custom/metadata/README.md)   | Query attritutions                       |
| [metabase](metaphor/metabase/README.md)                           | Dashboard, lineage                       |
| [monte_carlo](metaphor/monte_carlo/README.md)                     | Data monitor                             |
| [mssql](metaphor/mssql/README.md)                                 | Schema                                   |
| [mysql](metaphor/mysql/README.md)                                 | Schema, description                      |
| [postgresql](metaphor/postgresql/README.md)                       | Schema, description, statistics          |
| [postgresql.profile](metaphor/postgresql/profile/README.md)       | Data profile                             |
| [postgresql.usage](metaphor/postgresql/usage/README.md)           | Usage                                    |
| [power_bi](metaphor/power_bi/README.md)                           | Dashboard, lineage                       |
| [redshift](metaphor/redshift/README.md)                           | Schema, description, statistics, queries |
| [redshift.lineage](metaphor/redshift/lineage/README.md)           | Lineage                                  |
| [redshift.profile](metaphor/redshift/profile/README.md)           | Data profile                             |
| [snowflake](metaphor/snowflake/README.md)                         | Schema, description, statistics, queries |
| [snowflake.lineage](metaphor/snowflake/lineage/README.md)         | Lineage                                  |
| [snowflake.profile](metaphor/snowflake/profile/README.md)         | Data profile                             |
| [synapse](metaphor/synapse//README.md)                            | Schema, queries                          |
| [tableau](metaphor/tableau/README.md)                             | Dashboard, lineage                       |
| [thought_spot](metaphor/thought_spot/README.md)                   | Dashboard, lineage                       |
| [unity_catalog](metaphor/unity_catalog/README.md)                 | Schema, description                      |
| [unity_catalog.profile](metaphor/unity_catalog/profile/README.md) | Statistics                               |

## Development

See [Development Environment](docs/develop.md) for more instructions on how to setup your local development environment.

## Custom Connectors

See [Adding a Custom Connector](docs/custom.md) for instructions and a full example on how to create your custom connectors.

