Metadata-Version: 2.1
Name: toucan-connectors
Version: 0.41.2
Summary: Toucan Toco Connectors
Home-page: https://github.com/ToucanToco/toucan-connectors
Author: Toucan Toco
Author-email: dev@toucantoco.com
License: BSD
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Description-Content-Type: text/markdown
Requires-Dist: aiohttp
Requires-Dist: cached-property
Requires-Dist: jinja2
Requires-Dist: pydantic
Requires-Dist: pyjq
Requires-Dist: requests (<2.24.0)
Requires-Dist: tenacity
Requires-Dist: toucan-data-sdk
Requires-Dist: typing-extensions ; python_version < "3.8"
Provides-Extra: rok
Requires-Dist: requests ; extra == 'rok'
Provides-Extra: adobe
Requires-Dist: adobe-analytics ; extra == 'adobe'
Provides-Extra: aircall
Requires-Dist: bearer ; extra == 'aircall'
Provides-Extra: all
Requires-Dist: PyMySQL (>=0.8.0) ; extra == 'all'
Requires-Dist: adobe-analytics ; extra == 'all'
Requires-Dist: bearer ; extra == 'all'
Requires-Dist: cx-Oracle (>=6.2.1) ; extra == 'all'
Requires-Dist: dataiku-api-client ; extra == 'all'
Requires-Dist: elasticsearch ; extra == 'all'
Requires-Dist: facebook-sdk ; extra == 'all'
Requires-Dist: google-api-python-client ; extra == 'all'
Requires-Dist: google-api-python-client (>=1.7.5) ; extra == 'all'
Requires-Dist: gspread (>=3) ; extra == 'all'
Requires-Dist: oauth2client ; extra == 'all'
Requires-Dist: oauthlib ; extra == 'all'
Requires-Dist: pandas-gbq ; extra == 'all'
Requires-Dist: psycopg2 (>=2.7.4) ; extra == 'all'
Requires-Dist: pyhdb (>=0.3.4) ; extra == 'all'
Requires-Dist: pyhive[hive] ; extra == 'all'
Requires-Dist: pymongo (>=3.6.1) ; extra == 'all'
Requires-Dist: pyodbc ; extra == 'all'
Requires-Dist: requests ; extra == 'all'
Requires-Dist: requests-oauthlib ; extra == 'all'
Requires-Dist: snowflake-connector-python (<=2.2.10) ; extra == 'all'
Requires-Dist: tctc-odata ; extra == 'all'
Requires-Dist: toucan-client ; extra == 'all'
Provides-Extra: azure_mssql
Requires-Dist: pyodbc ; extra == 'azure_mssql'
Provides-Extra: dataiku
Requires-Dist: dataiku-api-client ; extra == 'dataiku'
Provides-Extra: elasticsearch
Requires-Dist: elasticsearch ; extra == 'elasticsearch'
Provides-Extra: facebook
Requires-Dist: facebook-sdk ; extra == 'facebook'
Provides-Extra: google_analytics
Requires-Dist: google-api-python-client ; extra == 'google_analytics'
Requires-Dist: oauth2client ; extra == 'google_analytics'
Provides-Extra: google_big_query
Requires-Dist: pandas-gbq ; extra == 'google_big_query'
Provides-Extra: google_cloud_mysql
Requires-Dist: PyMySQL (>=0.8.0) ; extra == 'google_cloud_mysql'
Provides-Extra: google_my_business
Requires-Dist: google-api-python-client (>=1.7.5) ; extra == 'google_my_business'
Provides-Extra: google_sheets
Requires-Dist: bearer ; extra == 'google_sheets'
Provides-Extra: google_spreadsheet
Requires-Dist: gspread (>=3) ; extra == 'google_spreadsheet'
Requires-Dist: oauth2client ; extra == 'google_spreadsheet'
Provides-Extra: hive
Requires-Dist: pyhive[hive] ; extra == 'hive'
Provides-Extra: http_api
Requires-Dist: oauthlib ; extra == 'http_api'
Requires-Dist: requests-oauthlib ; extra == 'http_api'
Provides-Extra: lightspeed
Requires-Dist: bearer ; extra == 'lightspeed'
Provides-Extra: mongo
Requires-Dist: pymongo (>=3.6.1) ; extra == 'mongo'
Provides-Extra: mssql
Requires-Dist: pyodbc ; extra == 'mssql'
Provides-Extra: mysql
Requires-Dist: PyMySQL (>=0.8.0) ; extra == 'mysql'
Provides-Extra: odata
Requires-Dist: oauthlib ; extra == 'odata'
Requires-Dist: requests-oauthlib ; extra == 'odata'
Requires-Dist: tctc-odata ; extra == 'odata'
Provides-Extra: oracle_sql
Requires-Dist: cx-Oracle (>=6.2.1) ; extra == 'oracle_sql'
Provides-Extra: postgres
Requires-Dist: psycopg2 (>=2.7.4) ; extra == 'postgres'
Provides-Extra: sap_hana
Requires-Dist: pyhdb (>=0.3.4) ; extra == 'sap_hana'
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python (<=2.2.10) ; extra == 'snowflake'
Provides-Extra: toucan_toco
Requires-Dist: toucan-client ; extra == 'toucan_toco'

[![Pypi-v](https://img.shields.io/pypi/v/toucan-connectors.svg)](https://pypi.python.org/pypi/toucan-connectors)
[![Pypi-pyversions](https://img.shields.io/pypi/pyversions/toucan-connectors.svg)](https://pypi.python.org/pypi/toucan-connectors)
[![Pypi-l](https://img.shields.io/pypi/l/toucan-connectors.svg)](https://pypi.python.org/pypi/toucan-connectors)
[![Pypi-wheel](https://img.shields.io/pypi/wheel/toucan-connectors.svg)](https://pypi.python.org/pypi/toucan-connectors)
[![GitHub Actions](https://github.com/ToucanToco/toucan-connectors/workflows/CI/badge.svg)](https://github.com/ToucanToco/toucan-connectors/actions?query=workflow%3ACI)
[![Coverage](https://sonarcloud.io/api/project_badges/measure?project=ToucanToco_toucan-connectors&metric=coverage)](https://sonarcloud.io/dashboard?id=ToucanToco_toucan-connectors)

# Toucan Connectors
[Toucan Toco](https://toucantoco.com/fr/) data connectors are plugins to the Toucan Toco platform,
configured with dictionaries (cf. `DataSource` class) and returning
[Pandas DataFrames](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html).

## Setup
In order to work you need `make` and `Python 3.6`, `3.7` or `3.8` (consider
running `pip install -U pip setuptools` if needed)
You can then install:
- the main dependencies by typing `pip install -e .`
- the test requirements by typing `pip install -r requirements-testing.txt`

You should be able to run basic tests `pytest tests/test_connector.py`

:warning: To test and use `mssql` (and `azure_mssql`) you need to install the Microsoft ODBC driver for SQL Server for
[Linux](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15)
or [MacOS](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15)

:warning: On macOS, to test the `postgres` connector, you need to install `postgresql` by running for instance `brew install postgres`.
You can then install the library with `env LDFLAGS='-L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib' pip install psycopg2`

## Testing a connector
If you want to run the tests for another connector, you can install the extra dependencies  
(e.g to test MySQL just type `pip install -e ".[mysql]"`)  
Now `pytest tests/mysql` should run all the mysql tests properly.

If you want to run the tests for all the connectors you can add all the dependencies by typing  
`pip install -e ".[all]"` and `make test`.

## Adding a connector

To generate the connector and test modules from boilerplate, run:  

```
$ make new_connector type=mytype
```

`mytype` should be the name of a system we would like to build a connector for, 
such as `MySQL` or `Hive` or `Magento`.

#### Step 1 : Tests
Open the folder in `tests` for the new connector. You can start writing your tests
before implementing it.

Some connectors are tested with calls to the actual data systems that they target,
for example `elasticsearch`, `mongo`, `mssql`. Other are tested with mocks of the
classes or functions returning data that you are wrapping (see : `HttpAPI`, or
`microstrategy`).

If you have a container for your target system, please do not hesitate to add a docker image in
the `docker-compose.yml`. You can then use the fixture `service_container` to automatically
start the docker and shut it down for you!

:warning: _The test runner assumes you have all the docker images locally,
if not please run pytest with `--pull` to retrieve them_

#### Step 2 : New connector
Open the folder `mytype` in `toucan_connectors` for your new connector and
create your classes

```python
import pandas as pd

# Careful here you need to import ToucanConnector from the deep path, not the __init__ path.
from toucan_connectors.toucan_connector import ToucanConnector, ToucanDataSource


class MyTypeDataSource(ToucanDataSource):
    """Model of my datasource"""
    query: str


class MyTypeConnector(ToucanConnector):
    """Model of my connector"""
    data_source_model: MyTypeDataSource

    host: str
    port: int
    database: str

    def _retrieve_data(self, data_source: MyTypeDataSource) -> pd.DataFrame:
        """how to retrieve a dataframe"""
```

Please add your connector in `toucan_connectors/__init__.py`.
The key is what we call the `type` of the connector, which 
is basically like an id used to retrieve it.
```python
CONNECTORS_CATALOGUE = {
  ...,
  'MyType': 'mytype.mytype_connector.MyTypeConnector',
  ...
}
```

You can now generate and edit the documentation page for your connector:

```shell
PYTHONPATH=. python doc/generate.py MyTypeConnector > doc/mytypeconnector.md
```

#### Step 3 : Register your connector
Add the main requirements to the `setup.py` in the `extras_require` dictionary:
```ini
extras_require = {
    ...
    'mytype': ['my_dependency_pkg1==x.x.x', 'my_dependency_pkg2>=x.x.x']
}
```
If you need to add testing dependencies, add them to the `requirements-testing.txt` file.

### Step 4 : Create a pull request
Make sure your new code is properly formatted by typing `make lint`.
If it's not, please use `make format`!
You can now create a pull request!


