Metadata-Version: 2.1
Name: cloud2sql
Version: 0.6.0
Summary: Read infrastructure data from your cloud and export it to a SQL database.
Home-page: https://github.com/someengineering/cloud2sql
License: Apache Software License 2.0
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: SQLAlchemy (==1.4.45)
Requires-Dist: PyYAML (>=6.0)
Requires-Dist: rich (>=12.6.0)
Requires-Dist: resotoclient (>=1.2.1)
Requires-Dist: posthog (>=2.2.0)
Requires-Dist: requests (>=2.28.1)
Requires-Dist: resotolib (>=3.0.0)
Requires-Dist: resoto-plugin-aws (>=3.0.0)
Requires-Dist: resoto-plugin-digitalocean (>=3.0.0)
Requires-Dist: resoto-plugin-gcp (>=3.0.0)
Requires-Dist: resoto-plugin-k8s (>=3.0.0)
Provides-Extra: all
Requires-Dist: pymysql (>=1.0.2) ; extra == 'all'
Requires-Dist: psycopg2-binary (>=2.9.5) ; extra == 'all'
Requires-Dist: snowflake-sqlalchemy (>=1.4.5) ; extra == 'all'
Requires-Dist: pyarrow (==10.0.1) ; extra == 'all'
Provides-Extra: mariadb
Requires-Dist: pymysql (>=1.0.2) ; extra == 'mariadb'
Provides-Extra: mysql
Requires-Dist: pymysql (>=1.0.2) ; extra == 'mysql'
Provides-Extra: parquet
Requires-Dist: pyarrow (==10.0.1) ; extra == 'parquet'
Provides-Extra: postgresql
Requires-Dist: psycopg2-binary (>=2.9.5) ; extra == 'postgresql'
Provides-Extra: snowflake
Requires-Dist: snowflake-sqlalchemy (>=1.4.5) ; extra == 'snowflake'

# cloud2sql
Read infrastructure data from your cloud and export it to a SQL database.


## Installation

Python 3.9 or higher is required.
It is recommended to use a separate virtual environment for this project. You can create one with the following command:

```bash
python3 -m venv venv --prompt "cloud2sql"
source venv/bin/activate
```

Once the virtual environment is activated, you can install cloud2sql:

```bash
pip install cloud2sql[all]
```

If you only require support for a specific database, instead of `cloud2sql[all]` you can choose between `cloud2sql[snowflake]`, `cloud2sql[parquet]`, `cloud2sql[postgresql]`, `cloud2sql[mysql]`.

## Usage

The sources and destinations for `cloud2sql` are configured via a configuration file. Create your own configuration by adjusting the [config template file](./config-template.yaml).
You can safely delete the sections that are not relevant to you (e.g. if you do not use AWS, you can delete the `aws` section).
All sections refer to cloud providers and are enabled if a configuration section is provided.

The following databases are currently supported:

#### SQLite

```
destinations:
    sqlite:
        database: /path/to/database.db
```

#### PostgreSQL

```
destinations:
    postgresql:
        host: 127.0.0.1
        port: 5432
        user: cloud2sql
        password: changeme
        database: cloud2sql
        args:
            key: value
```

#### MySQL

```
destinations:
    mysql:
        host: 127.0.0.1
        port: 3306
        user: cloud2sql
        password: changeme
        database: cloud2sql
        args:
            key: value
```

#### MariaDB

```
destinations:
    mariadb:
        host: 127.0.0.1
        port: 3306
        user: cloud2sql
        password: changeme
        database: cloud2sql
        args:
            key: value
```

#### Snowflake

```
destinations:
    snowflake:
        host: myorg-myaccount
        user: cloud2sql
        password: changeme
        database: cloud2sql/public
        args:
            warehouse: compute_wh
            role: accountadmin
```

#### Apache Parquet

```
destinations:
    parquet:
        path: /path/to/parquet/files
        batch_size: 100_000
```

#### My database is not listed here

cloud2sql uses SQLAlchemy to connect to the database. If your database is not listed here, you can check if it is supported in [SQLAlchemy Dialects](https://docs.sqlalchemy.org/en/20/dialects/index.html).
Install the relevant driver and use the connection string from the documentation.

#### Example

We use a minimal configuration [example](./config-example.yaml) and export the data to a SQLite database.
The example uses our AWS default credentials and the default kubernetes config.

```bash
cloud2sql --config config-example.yaml
```

## Local Development

Create a local development environment with the following command:

```bash
make setup
source venv/bin/activate
```
