Metadata-Version: 2.1
Name: data-parser
Version: 0.0.3
Summary: Data parser to parse newline delimited logs into tabular format.
Home-page: UNKNOWN
Author: Lee Zhi Yong
Author-email: zhiyongengineering@email.com
License: UNKNOWN
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Requires-Python: >=3
Description-Content-Type: text/markdown
Requires-Dist: pandas
Provides-Extra: pyspark
Requires-Dist: pyspark ; extra == 'pyspark'

# Parse raw logs to tabular format
This package helps to parse new line delimited logs to tabular formats. The user provides the regex, file path and column names, and a dataframe will be returned.  
Depending on the supplied mode (local/spark), a pandas dataframe or a spark dataframe will be returned.

## Features
### Local mode
1. Regex matching is done using multiprocessing.
1. Glob searching for files.
1. Lazy evaluation of files. This allows larger than memory datasets to be parsed, but note that upon parsing, the resultant pandas dataframe must be able to fit in memory.

## Installation
### Purely for local usage (No pyspark)
pip install data-parser
### Both local and pyspark
pip install data-parser[pyspark]

## Usage - Local (Pandas)
```python
from data_parser import DataSource

# Bind 9: Feb  5 09:12:11 ns1 named[80090]: client 192.168.10.12#3261: query: www.server.example IN A
dns = DataSource(
    path='/path/to/dnsdir/*.txt',  # Glob patterns supported
    mode='local'
)

# Pandas dataframe is returned
dns_df = dns.parse(
    regex='^([A-Z][a-z]{2})\s+(\d+) (\d{2}\:\d{2}\:\d{2}) (\S+).+client ([^\s#]+)#(\d+)',
    col_names=['month', 'day', 'time', 'nameserver', 'query_ip', 'port'],
    on_error='raise'
)
```

## Usage - Spark (Pyspark)
```python
from data_parser import DataSource

# Bind 9: Feb  5 09:12:11 ns1 named[80090]: client 192.168.10.12#3261: query: www.server.example IN A
dns = DataSource(
    path='/path/to/dns/log',
    mode='spark'
)

# Spark dataframe is returned
dns_df = dns.parse(
    regex='^([A-Z][a-z]{2})\s+(\d+) (\d{2}\:\d{2}\:\d{2}) (\S+).+client ([^\s#]+)#(\d+)',
    col_names=['month', 'day', 'time', 'nameserver', 'query_ip', 'port'],
    on_error='raise'
)
```


