Metadata-Version: 2.1
Name: cloud-files
Version: 0.1.0
Summary: Fast access to cloud storage and local FS.
Home-page: https://github.com/seung-lab/cloud-files/
Author: William Silversmith
Author-email: ws9@princeton.edu
License: BSD
Platform: UNKNOWN
Classifier: Intended Audience :: Developers
Classifier: Development Status :: 4 - Beta
Classifier: License :: OSI Approved :: BSD License
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.5
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Description-Content-Type: text/markdown
Requires-Dist: boto3 (>=1.4.7)
Requires-Dist: chardet (>=3.0.4)
Requires-Dist: gevent
Requires-Dist: google-auth (>=1.10.0)
Requires-Dist: google-cloud-core (>=1.1.0)
Requires-Dist: google-cloud-storage (>=1.24.1)
Requires-Dist: protobuf (>=3.3.0)
Requires-Dist: requests (>=2.22.0)
Requires-Dist: six (>=1.10.0)
Requires-Dist: tenacity (>=4.10.0)
Requires-Dist: tqdm
Requires-Dist: urllib3[brotli,secure] (>=1.25.7)

CloudFiles: Fast access to cloud storage and local FS.
========

```python
from cloudfiles import CloudFiles

cf = CloudFiles('gs://bucket/') # google cloud storage
cf = CloudFiles('s3://bucket/') # Amazon S3
cf = CloudFiles('file:///home/coolguy/') # local filesystem
cf = CloudFiles('https://website.com/coolguy/') # arbitrary web server

# more options
cf = CloudFiles(
    's3://bucket/', 
    num_threads=20, 
    progress=True, # display progress bar
    secrets=credential_json, # provide your own secrets
    green=False, # whether to use green threads
)


cf.get('filename')
cf.get([ 'filename_1', 'filename_2' ]) # threaded automatically

cf.put('filename', content)
cf.put_json('filename', content)
cf.puts([{
    'path': 'filename',
    'content': content,
}, ... ]) # automatically threaded
cf.put_jsons(...) # same as puts

cf.list()
cf.delete('filename')
cf.delete([ 'filename_1', 'filename_2', ... ]) # threaded

cf.exists('filename')
cf.exists([ filename_1, ... ]) # threaded
```

CloudFiles is a pure python client for accessing cloud storage or the local file system in a threaded fashion without hassle. 

### Highlights

1. Fast file access due to transparent threading.
2. Supports Google Cloud Storage, Amazone S3, local filesystems, and arbitrary web servers with a similar file access structure making hybrid or multi-cloud easy.
3. Robust to flaky network connections. Retries using an exponential random window to avoid network collisions when working in a large cluster.
4. Supports gzip and brotli\* compression.
5. Supports HTTP Range reads.
6. Supports green threads, which are important for achieving maximum performance on virtualized servers.

\* Except on Google Cloud Storage.

## Installation 

```bash
pip install cloud-files
```

### Credits

CloudFiles is derived from the [CloudVolume.Storage](https://github.com/seung-lab/cloud-volume/tree/master/cloudvolume/storage) system.  

Storage was initially created by William Silversmith and Ignacio Tartavull. It was maintained and improved by William Silversmith and includes improvements by Nico Kemnitz (extremely fast exists) and Ben Falk (brotli).



