Metadata-Version: 2.1
Name: hadoop-fs-wrapper
Version: 0.4.6
Summary: Python Wrapper for Hadoop Java API
Home-page: UNKNOWN
Author: ECCO Sneaks & Data
Author-email: esdsupport@ecco.com
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE

# Hadoop FileSystem Java Class Wrapper 
Typed Python wrappers for [Hadoop FileSystem](https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/FileSystem.html) class family.

## Installation
You can install this package from `pypi` on any Hadoop or Spark runtime:
```commandline
pip install hadoop-fs-wrapper
```

Select a version that matches hadoop version you are using:

| Hadoop Version | Compatible hadoop-fs-wrapper version |
|----------------|:------------------------------------:|
| 3.2.x          |                0.4.x                 |
| 3.3.x          |                0.4.x                 |

## Usage
Common use case is accessing Hadoop FileSystem from Spark session object:
```python
from hadoop_fs_wrapper.wrappers.file_system import FileSystem

file_system = FileSystem.from_spark_session(spark=spark_session)
```

Then, for example, one can check if there are any files under specified path:
```python
def is_valid_source_path(file_system: FileSystem, path: str) -> bool:
    """
     Checks whether a regexp path refers to a valid set of paths
    :param file_system: pyHadooopWrapper FileSystem
    :param path: path e.g. (s3a|abfss|file|...)://hello@world.com/path/part*.csv
    :return: true if path resolves to existing paths, otherwise false
    """
    return len(file_system.glob_status(path)) > 0
```

## Contribution

Currently basic filesystem operations (listing, deleting, search, iterative listing etc.) are supported. If an operation you require is not yet wrapped,
please open an issue or create a PR.

All changes are tested against Spark 3.2 running in local mode.

