### Navigation

- index
- modules |
- next |
- previous |
- pywws 14.03.dev1177 documentation »
- Python programs and modules »

# pywws.DataStore

DataStore.py - stores readings in easy to access files

## Introduction

This module is at the core of pywws. It stores data on disc, but without the overhead of a full scale database system. I have designed it to run on a small memory machine such as my Asus router. To minimise memory usage it only loads one day’s worth of raw data at a time into memory.

From a “user” point of view, the data is accessed as a cross between a list and a dictionary. Each data record is indexed by a `datetime.datetime` object (dictionary behaviour), but records are stored in order and can be accessed as slices (list behaviour).

For example, to access the hourly data for Christmas day 2009, one might do the following:

    from datetime import datetime
    from pywws import DataStore
    hourly = DataStore.hourly_store('weather_data')
    for data in hourly[datetime(2009, 12, 25):datetime(2009, 12, 26)]:
        print data['idx'], data['temp_out']

Some more examples of data access:

    # get value nearest 9:30 on Christmas day 2008
    data[data.nearest(datetime(2008, 12, 25, 9, 30))]
    # get entire array, equivalent to data[:]
    data[datetime.min:datetime.max]
    # get last 12 hours worth of data
    data[datetime.utcnow() - timedelta(hours=12):]

Note that the `datetime.datetime` index is in UTC. You may need to apply an offset to convert to local time.

The module provides five classes to store different data. `data_store` takes “raw” data from the weather station; `calib_store`, `hourly_store`, `daily_store` and `monthly_store` store processed data (see `pywws.Process`). All three are derived from the same `core_store` class, they only differ in the keys and types of data stored in each record.

## Detailed API

Functions

`safestrptime`(date_string[, format]) |

Classes

`ParamStore`(root_dir, file_name) |
`RawConfigParser`([defaults, dict_type, ...]) |
`calib_store`(root_dir) | Stores ‘calibrated’ weather station data.
`core_store`(root_dir) |
`daily_store`(root_dir) | Stores daily summary weather station data.
`data_store`(root_dir) | Stores raw weather station data.
`date` | date(year, month, day) –> date object
`datetime` | datetime(year, month, day[, hour[, minute[, second[, microsecond[,tzinfo]]]]])
`hourly_store`(root_dir) | Stores hourly summary weather station data.
`monthly_store`(root_dir) | Stores monthly summary weather station data.
`params`(root_dir) | Parameters are stored in a file “weather.ini” in root_dir.
`status`(root_dir) | Status is stored in a file “status.ini” in root_dir.
`timedelta` | Difference between two datetime values.

`pywws.DataStore.``safestrptime`(date_string, format=None)

class `pywws.DataStore.``ParamStore`(root_dir, file_name)
  `flush`()
  `get`(section, option, default=None)

    Get a parameter value and return a string.

    If default is specified and section or option are not defined in the file, they are created and set to default, which is then the return value.

  `get_datetime`(section, option, default=None)
  `set`(section, option, value)

    Set option in section to string value.

  `unset`(section, option)

    Remove option from section.

class `pywws.DataStore.``params`(root_dir)

  Parameters are stored in a file “weather.ini” in root_dir.

class `pywws.DataStore.``status`(root_dir)

  Status is stored in a file “status.ini” in root_dir.

class `pywws.DataStore.``core_store`(root_dir)
  `before`(idx)

    Return datetime of newest existing data record whose datetime is < idx.

    Might not even be in the same year! If no such record exists, return None.

  `after`(idx)

    Return datetime of oldest existing data record whose datetime is >= idx.

    Might not even be in the same year! If no such record exists, return None.

  `nearest`(idx)

    Return datetime of record whose datetime is nearest idx.

  `flush`()

class `pywws.DataStore.``data_store`(root_dir)

  Stores raw weather station data.

  `key_list` = ['idx', 'delay', 'hum_in', 'temp_in', 'hum_out', 'temp_out', 'abs_pressure', 'wind_ave', 'wind_gust', 'wind_dir', 'rain', 'status', 'illuminance', 'uv']
  `conv` = {'status': <type 'int'>, 'wind_ave': <type 'float'>, 'rain': <type 'float'>, 'hum_in': <type 'int'>, 'temp_out': <type 'float'>, 'wind_dir': <type 'int'>, 'hum_out': <type 'int'>, 'illuminance': <type 'float'>, 'wind_gust': <type 'float'>, 'idx': <function safestrptime at 0x27e2a28>, 'uv': <type 'int'>, 'temp_in': <type 'float'>, 'delay': <type 'int'>, 'abs_pressure': <type 'float'>}

class `pywws.DataStore.``calib_store`(root_dir)

  Stores ‘calibrated’ weather station data.

  `key_list` = ['idx', 'delay', 'hum_in', 'temp_in', 'hum_out', 'temp_out', 'abs_pressure', 'rel_pressure', 'wind_ave', 'wind_gust', 'wind_dir', 'rain', 'status', 'illuminance', 'uv']
  `conv` = {'status': <type 'int'>, 'wind_ave': <type 'float'>, 'rain': <type 'float'>, 'rel_pressure': <type 'float'>, 'hum_in': <type 'int'>, 'temp_out': <type 'float'>, 'wind_dir': <type 'int'>, 'hum_out': <type 'int'>, 'illuminance': <type 'float'>, 'wind_gust': <type 'float'>, 'idx': <function safestrptime at 0x27e2a28>, 'uv': <type 'int'>, 'temp_in': <type 'float'>, 'delay': <type 'int'>, 'abs_pressure': <type 'float'>}

class `pywws.DataStore.``hourly_store`(root_dir)

  Stores hourly summary weather station data.

  `key_list` = ['idx', 'hum_in', 'temp_in', 'hum_out', 'temp_out', 'abs_pressure', 'rel_pressure', 'pressure_trend', 'wind_ave', 'wind_gust', 'wind_dir', 'rain', 'illuminance', 'uv']
  `conv` = {'pressure_trend': <type 'float'>, 'wind_ave': <type 'float'>, 'rain': <type 'float'>, 'rel_pressure': <type 'float'>, 'hum_in': <type 'int'>, 'temp_out': <type 'float'>, 'wind_dir': <type 'int'>, 'hum_out': <type 'int'>, 'wind_gust': <type 'float'>, 'idx': <function safestrptime at 0x27e2a28>, 'uv': <type 'int'>, 'temp_in': <type 'float'>, 'illuminance': <type 'float'>, 'abs_pressure': <type 'float'>}

class `pywws.DataStore.``daily_store`(root_dir)

  Stores daily summary weather station data.

  `key_list` = ['idx', 'start', 'hum_out_ave', 'hum_out_min', 'hum_out_min_t', 'hum_out_max', 'hum_out_max_t', 'temp_out_ave', 'temp_out_min', 'temp_out_min_t', 'temp_out_max', 'temp_out_max_t', 'hum_in_ave', 'hum_in_min', 'hum_in_min_t', 'hum_in_max', 'hum_in_max_t', 'temp_in_ave', 'temp_in_min', 'temp_in_min_t', 'temp_in_max', 'temp_in_max_t', 'abs_pressure_ave', 'abs_pressure_min', 'abs_pressure_min_t', 'abs_pressure_max', 'abs_pressure_max_t', 'rel_pressure_ave', 'rel_pressure_min', 'rel_pressure_min_t', 'rel_pressure_max', 'rel_pressure_max_t', 'wind_ave', 'wind_gust', 'wind_gust_t', 'wind_dir', 'rain', 'illuminance_ave', 'illuminance_max', 'illuminance_max_t', 'uv_ave', 'uv_max', 'uv_max_t']
  `conv` = {'temp_in_min': <type 'float'>, 'temp_in_max': <type 'float'>, 'uv_ave': <type 'float'>, 'temp_out_max': <type 'float'>, 'temp_out_min': <type 'float'>, 'abs_pressure_max': <type 'float'>, 'start': <function safestrptime at 0x27e2a28>, 'hum_in_max_t': <function safestrptime at 0x27e2a28>, 'uv_max_t': <function safestrptime at 0x27e2a28>, 'temp_in_max_t': <function safestrptime at 0x27e2a28>, 'abs_pressure_min_t': <function safestrptime at 0x27e2a28>, 'illuminance_ave': <type 'float'>, 'rel_pressure_ave': <type 'float'>, 'rain': <type 'float'>, 'rel_pressure_min_t': <function safestrptime at 0x27e2a28>, 'wind_gust': <type 'float'>, 'temp_out_min_t': <function safestrptime at 0x27e2a28>, 'wind_gust_t': <function safestrptime at 0x27e2a28>, 'hum_out_ave': <type 'float'>, 'rel_pressure_min': <type 'float'>, 'hum_out_min_t': <function safestrptime at 0x27e2a28>, 'temp_out_max_t': <function safestrptime at 0x27e2a28>, 'uv_max': <type 'int'>, 'illuminance_max_t': <function safestrptime at 0x27e2a28>, 'rel_pressure_max': <type 'float'>, 'temp_in_ave': <type 'float'>, 'hum_in_min_t': <function safestrptime at 0x27e2a28>, 'hum_out_max': <type 'int'>, 'illuminance_max': <type 'float'>, 'wind_ave': <type 'float'>, 'abs_pressure_max_t': <function safestrptime at 0x27e2a28>, 'temp_in_min_t': <function safestrptime at 0x27e2a28>, 'hum_out_max_t': <function safestrptime at 0x27e2a28>, 'hum_out_min': <type 'int'>, 'rel_pressure_max_t': <function safestrptime at 0x27e2a28>, 'wind_dir': <type 'int'>, 'temp_out_ave': <type 'float'>, 'abs_pressure_ave': <type 'float'>, 'idx': <function safestrptime at 0x27e2a28>, 'hum_in_max': <type 'int'>, 'abs_pressure_min': <type 'float'>, 'hum_in_ave': <type 'float'>, 'hum_in_min': <type 'int'>}

class `pywws.DataStore.``monthly_store`(root_dir)

  Stores monthly summary weather station data.

  `key_list` = ['idx', 'start', 'hum_out_ave', 'hum_out_min', 'hum_out_min_t', 'hum_out_max', 'hum_out_max_t', 'temp_out_ave', 'temp_out_min_lo', 'temp_out_min_lo_t', 'temp_out_min_hi', 'temp_out_min_hi_t', 'temp_out_min_ave', 'temp_out_max_lo', 'temp_out_max_lo_t', 'temp_out_max_hi', 'temp_out_max_hi_t', 'temp_out_max_ave', 'hum_in_ave', 'hum_in_min', 'hum_in_min_t', 'hum_in_max', 'hum_in_max_t', 'temp_in_ave', 'temp_in_min_lo', 'temp_in_min_lo_t', 'temp_in_min_hi', 'temp_in_min_hi_t', 'temp_in_min_ave', 'temp_in_max_lo', 'temp_in_max_lo_t', 'temp_in_max_hi', 'temp_in_max_hi_t', 'temp_in_max_ave', 'abs_pressure_ave', 'abs_pressure_min', 'abs_pressure_min_t', 'abs_pressure_max', 'abs_pressure_max_t', 'rel_pressure_ave', 'rel_pressure_min', 'rel_pressure_min_t', 'rel_pressure_max', 'rel_pressure_max_t', 'wind_ave', 'wind_gust', 'wind_gust_t', 'wind_dir', 'rain', 'rain_days', 'illuminance_ave', 'illuminance_max_lo', 'illuminance_max_lo_t', 'illuminance_max_hi', 'illuminance_max_hi_t', 'illuminance_max_ave', 'uv_ave', 'uv_max_lo', 'uv_max_lo_t', 'uv_max_hi', 'uv_max_hi_t', 'uv_max_ave']
  `conv` = {'uv_ave': <type 'float'>, 'illuminance_max_hi_t': <function safestrptime at 0x27e2a28>, 'uv_max_lo_t': <function safestrptime at 0x27e2a28>, 'temp_out_max_hi_t': <function safestrptime at 0x27e2a28>, 'temp_out_max_lo': <type 'float'>, 'abs_pressure_max': <type 'float'>, 'rel_pressure_min_t': <function safestrptime at 0x27e2a28>, 'temp_in_min_hi': <type 'float'>, 'temp_in_min_lo': <type 'float'>, 'start': <function safestrptime at 0x27e2a28>, 'hum_in_max_t': <function safestrptime at 0x27e2a28>, 'rain_days': <type 'int'>, 'abs_pressure_min_t': <function safestrptime at 0x27e2a28>, 'temp_out_max_hi': <type 'float'>, 'illuminance_ave': <type 'float'>, 'temp_out_min_lo': <type 'float'>, 'illuminance_max_ave': <type 'float'>, 'uv_max_hi': <type 'int'>, 'rel_pressure_ave': <type 'float'>, 'rain': <type 'float'>, 'temp_out_min_hi': <type 'float'>, 'wind_gust': <type 'float'>, 'temp_in_max_hi_t': <function safestrptime at 0x27e2a28>, 'temp_in_max_lo': <type 'float'>, 'temp_in_min_ave': <type 'float'>, 'wind_gust_t': <function safestrptime at 0x27e2a28>, 'hum_out_ave': <type 'float'>, 'rel_pressure_min': <type 'float'>, 'hum_out_min_t': <function safestrptime at 0x27e2a28>, 'illuminance_max_lo': <type 'float'>, 'temp_in_min_lo_t': <function safestrptime at 0x27e2a28>, 'illuminance_max_hi': <type 'float'>, 'temp_out_max_lo_t': <function safestrptime at 0x27e2a28>, 'temp_in_max_hi': <type 'float'>, 'rel_pressure_max': <type 'float'>, 'uv_max_hi_t': <function safestrptime at 0x27e2a28>, 'temp_in_ave': <type 'float'>, 'uv_max_ave': <type 'float'>, 'temp_in_max_ave': <type 'float'>, 'hum_in_min_t': <function safestrptime at 0x27e2a28>, 'hum_out_max': <type 'int'>, 'temp_in_min_hi_t': <function safestrptime at 0x27e2a28>, 'temp_out_min_ave': <type 'float'>, 'temp_out_min_hi_t': <function safestrptime at 0x27e2a28>, 'wind_ave': <type 'float'>, 'abs_pressure_max_t': <function safestrptime at 0x27e2a28>, 'hum_out_max_t': <function safestrptime at 0x27e2a28>, 'hum_out_min': <type 'int'>, 'rel_pressure_max_t': <function safestrptime at 0x27e2a28>, 'uv_max_lo': <type 'int'>, 'wind_dir': <type 'int'>, 'temp_out_ave': <type 'float'>, 'abs_pressure_ave': <type 'float'>, 'idx': <function safestrptime at 0x27e2a28>, 'temp_out_min_lo_t': <function safestrptime at 0x27e2a28>, 'illuminance_max_lo_t': <function safestrptime at 0x27e2a28>, 'hum_in_max': <type 'int'>, 'abs_pressure_min': <type 'float'>, 'temp_in_max_lo_t': <function safestrptime at 0x27e2a28>, 'hum_in_ave': <type 'float'>, 'temp_out_max_ave': <type 'float'>, 'hum_in_min': <type 'int'>}

### Navigation

- index
- modules |
- next |
- previous |
- pywws 14.03.dev1177 documentation »
- Python programs and modules »