Metadata-Version: 2.0
Name: redis-helper
Version: 0.3.18
Summary: Easily store, index, and modify Python dicts in Redis (with flexible searching)
Home-page: https://github.com/kenjyco/redis-helper
Author: Ken
Author-email: kenjyco@gmail.com
License: MIT
Download-URL: https://github.com/kenjyco/redis-helper/tarball/v0.3.18
Keywords: redis,dictionary,secondary index,model,log,prototype,helper
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.5
Classifier: Topic :: Software Development :: Libraries
Classifier: Intended Audience :: Developers
Requires-Dist: click (>=6.0)
Requires-Dist: hiredis (==0.2.0)
Requires-Dist: input-helper
Requires-Dist: pytz
Requires-Dist: redis (==2.10.5)
Requires-Dist: ujson (==1.35)

About
-----

Install redis-helper, create an instance of ``redis_helper.Collection``
(**the args/kwargs define the model**) and use the ``add``, ``get``,
``update``, ``delete``, and ``find`` methods to:

-  quickly store/retrieve/modify Python dicts in Redis
-  filter through indexed fields with simple/flexible find arguments
-  power real-time dashboards with metrics at a variety of time ranges
-  super-charge event logging and system debugging
-  build FAST prototypes and simulators
-  greatly simplify data access patterns throughout application

See the `request logging demo <https://asciinema.org/a/101422?t=1:10>`__
and `urls
demo <https://asciinema.org/a/75kl95ty9vg2jl93pfz9fbs9q?t=1:00>`__ (with
``unique_field`` defined). The
`examples <https://github.com/kenjyco/redis-helper/tree/master/examples>`__
they reference are **short** and **easy to read**.

The `redis-helper project <https://github.com/kenjyco/redis-helper>`__
evolved from a `reference Python
project <https://github.com/kenjyco/beu/tree/4aea6146fc5f01df3e344b9fadddf28b795dac89>`__
that would be **easy to teach** and follow many practical best practices
and useful patterns. Main purpose was to have something that was super
**easy to configure** (a single ``~/.config/redis-helper/settings.ini``
file for multiple application environments) that did cool things with
`Redis <http://redis.io/topics/data-types-intro>`__.

The `redis-helper package <https://pypi.python.org/pypi/redis-helper>`__
provides a ``Collection`` class that was designed to be **easy to
interact with** in the shell (for exploration, experimentation, and
debugging). Most methods on a ``Collection`` help **minimize typing**
(passing multiple arguments in a single delimited string when
appropriate) and do "the most reasonable thing" whenever possible.

The first time that ``redis_helper`` is imported, the sample
`settings.ini <https://github.com/kenjyco/redis-helper/blob/master/redis_helper/settings.ini>`__
file will be copied to the ``~/.config/redis-helper`` directory.

Install latest tag/release of `redis-helper package <https://pypi.python.org/pypi/redis-helper>`__
--------------------------------------------------------------------------------------------------

::

    % pip install redis-helper

Install latest commit on master of `redis-helper project <https://github.com/kenjyco/redis-helper>`__
-----------------------------------------------------------------------------------------------------

::

    % pip install git+git://github.com/kenjyco/redis-helper

Intro
-----

`Redis <http://redis.io/topics/data-types-intro>`__ is a fast in-memory
**data structure server**, where each stored object is referenced by a
key name. Objects in Redis correspond to one of several basic types,
each having their own set of specialized commands to perform operations.
The `redis Python package <https://github.com/andymccurdy/redis-py>`__
provides the
`StrictRedis <https://redis-py.readthedocs.org/en/latest/#redis.StrictRedis>`__
class, which contains methods that correspond to all of the Redis server
commands.

When initializing Collection objects, you must specify the "namespace"
and "name" of the collection (which are used to create the internally
used ``_base_key`` property). All Redis keys associated with a
Collection will have a name pattern that starts with the ``_base_key``.

.. code:: python

    import redis_helper as rh


    request_logs = rh.Collection(
        'log',
        'request',
        index_fields='status,uri,host',
        json_fields='request,response,headers'
    )

    urls = rh.Collection(
        'web',
        'url',
        unique_field='name',
        index_fields='domain,_type'
    )

    notes = rh.Collection(
        'input',
        'note',
        index_fields='topic,tag',
        insert_ts=True
    )

-  a ``unique_field`` can be specified on a collection if items in the
   collection should not contain duplicate values for that particular
   field

   -  the ``unique_field`` cannot also be included in ``json_fields`` or
      ``pickle_fields``
   -  if you specify a ``unique_field``, that field must exist on each
      item you add to the collection

-  use ``index_fields`` to specify which fields you will want to filter
   on when using the ``find`` method

   -  the values for data fields being indexed MUST be simple strings or
      numbers
   -  the values for data fields being indexed SHOULD NOT be long
      strings, as the values themselves are part of the index keys

-  use ``json_fields`` to specify which fields should be JSON encoded
   before insertion to Redis (using the very fast
   `ujson <https://pypi.python.org/pypi/ujson>`__ library)
-  use ``pickle_fields`` to specify which fields should be pickled
   before insertion to Redis
-  set ``insert_ts=True`` to create an additional index to store insert
   times

   -  only do this if you are storing items that you are likely to
      update and also likely to want to know the original insert time

      -  each time an object is updated, the score associated with the
         ``hash_id`` (at the ``_ts_zset_key``) is updated to the current
         timestamp
      -  the score associated with the ``hash_id`` (at the
         ``_in_zset_key``) is never updated

Essentially, you can store a Python
`dict <https://docs.python.org/3/tutorial/datastructures.html#dictionaries>`__
in a Redis `hash <https://redis.io/topics/data-types#hashes>`__ and
index some of the fields in Redis
`sets <https://redis.io/topics/data-types#sets>`__. The collection's
``_ts_zset_key`` is the Redis key name for the `sorted
set <https://redis.io/topics/data-types#sorted-sets>`__ containing the
``hash_id`` of every hash in the collection (with the ``score`` being a
``utc_float`` corresponding to the UTC time the ``hash_id`` was added or
modified).

-  if ``insert_ts=True`` was passed in when initializing the
   ``Collection`` (or sub-class), then the collection will also define
   ``self.in_zset_key`` to be the Redis key name for the sorted set (for
   ``hash_id`` and ``utc_float`` of insert time)

.. code:: python

    request_logs.add(
        method='get',
        status=400,
        host='blah.net',
        uri='/info',
        request={'x': 50, 'y': 100},
        response={'error': 'bad request'},
    )

    urls.add(
        name='redis-helper github',
        url='https://github.com/kenjyco/redis-helper',
        domain='github.com',
        _type='repo',
    )

The ``get`` method is a wrapper to `hash
commands <http://redis.io/commands#hash>`__ ``hget``, ``hmget``, or
``hgetall``. The actual hash command that gets called is determined by
the number of fields requested.

-  a Python dict is typically returned from ``get``
-  if ``item_format`` is specified, a string will be returned matching
   that format instead

.. code:: python

    request_logs.get('log:request:1')
    request_logs.get('log:request:1', 'host,status')
    request_logs.get('log:request:1', item_format='{status} for {host}{uri}')
    request_logs.get_by_position(0, item_format='{status} for {host}{uri}')
    urls.get_by_position(-1, 'domain,url')
    urls.get_by_unique_value('redis-helper github', item_format='{url} points to a {_type}')

-  the ``get_by_position`` and ``get_by_unique_value`` methods are
   wrappers to ``get``

   -  the ``get_by_unique_value`` method is only useful if a
      ``unique_field`` was set on the Collection

The ``find`` method allows you to return data for items in the
collection that match some set of search criteria. Multiple search terms
(i.e. ``index_field:value`` pairs) maybe be passed in the ``terms``
parameter, as long as they are separated by one of ``,`` ``;`` ``|``.
Any fields specified in the ``get_fields`` parameter are passed along to
the ``get`` method (when the actual fetching takes place).

-  when using ``terms``, all terms that include the same field will be
   treatead like an "or" (union of related sets), then the intersection
   of different sets will be computed
-  see the Redis `set commands <https://redis.io/commands#set>`__ and
   `sorted set commands <https://redis.io/commands#sorted_set>`__

There are many options for specifying time ranges in the ``find`` method
including:

-  ``since`` and ``until`` when specifying ``num:unit`` strings (i.e.
   15:seconds, 1.5:weeks, etc)
-  ``start_ts`` and ``end_ts`` when specifying timestamps with a form
   between ``YYYY`` and ``YYYY-MM-DD HH:MM:SS.f``
-  ``start`` and ``end`` when specifying a ``utc_float``
-  for ``since``, ``until``, ``start_ts``, and ``end_ts``, multiple
   values may be passed in the string, as long as they are separated by
   one of ``,`` ``;`` ``|``.

   -  when multiple time ranges are specified, the ``find`` method will
      determine all reasonable combinations and return a result-set per
      combination (instead of returning a list of items, returns a dict
      of list of items)

If ``count=True`` is specified, the number of results matching the
search criteria are returned instead of the actual results

-  if there are multiple time ranges specified, counts will be returned
   for each combination

.. code:: python

    request_logs.find('status:400, host:blah.net', get_fields='uri,error')
    request_logs.find(since='1:hr, 30:min', until='15:min, 5:min')
    request_logs.find(count=True, since='1:hr, 30:min', until='15:min, 5:min')
    urls.find(count=True, since='1:hr, 30:min, 10:min, 5:min, 1:min')
    urls.find(start_ts='2017-02-03', end_ts='2017-02-03 7:15:00')
    urls.find(start_ts='2017-02-03', item_format='{_ts} -> {_id}')

The ``update`` method allows you to change values for some fields
(modifying the ``unique_field``, when it is specified, is not allowed).

-  every time a field is modified for a particular ``hash_id``, the
   previous value and score (timestamp) are stored in a Redis hash
-  the ``old_data_for_hash_id`` or ``old_data_for_unique_value`` methods
   can be used to retrieve the history of all changes for a ``hash_id``

.. code:: python

    urls.update('web:url:1', _type='fancy', notes='this is a fancy url')
    urls.old_data_for_hash_id('web:url:1')
    urls.old_data_for_unique_value('redis-helper github'

Local development setup
-----------------------

::

    % git clone https://github.com/kenjyco/redis-helper
    % cd redis-helper
    % ./dev-setup.bash

The
`dev-setup.bash <https://github.com/kenjyco/redis-helper/blob/master/dev-setup.bash>`__
script will create a virtual environment in the ``./venv`` directory
with extra dependencies (ipython, pdbpp, pytest), then copy
``settings.ini`` to the ``~/.config/redis-helper`` directory.

Running tests in development setup
----------------------------------

The
`setup.cfg <https://github.com/kenjyco/redis-helper/blob/master/setup.cfg>`__
file contains the options for ``py.test``, currently ``-vsx -rs --pdb``.

The ``-vsx -rs --pdb`` options will run tests in a verbose manner and
output the reason why tests were skipped (if any were skipped). If there
are any failing tests, ``py.test`` will stop on the first failure and
drop you into a `pdb++ <https://pypi.python.org/pypi/pdbpp/>`__ debugger
session.

See the `debugging
section <https://github.com/kenjyco/redis-helper#settings-environments-testing-and-debugging>`__
of the README for tips on using the debugger and setting breakpoints (in
the actual `project
code <https://github.com/kenjyco/redis-helper/tree/master/redis_helper>`__,
or in the `test
code <https://github.com/kenjyco/redis-helper/tree/master/tests>`__).

::

    % venv/bin/py.test

or

::

    % venv/bin/python3 setup.py test

    Note: This option requires ``setuptools`` to be installed.

Usage
-----

.. code:: python

    >>> import redis_helper as rh
    >>> collection = rh.Collection(..., index_fields='field1,field3')
    >>> hash_id = collection.add(field1='', field2='', field3='', ...)
    >>> collection.add(...)
    >>> collection.add(...)
    >>> collection.update(hash_id, field1='', field4='', ...)
    >>> change_history = collection.old_data_for_hash_id(hash_id)
    >>> data = collection.get(hash_id)
    >>> some_data = collection.get(hash_id, 'field1,field3')
    >>> results = collection.find(...)
    >>> results2 = collection.find('field1:val,field3:val', ...)
    >>> results3 = collection.find(..., get_fields='field2,field4')
    >>> counts = collection.find(count=True, ...)
    >>> top_indexed = collection.index_field_info()
    >>> collection.delete(hash_id, ...)

Basics - Part 1 (request logging demo)
--------------------------------------

`Demo <https://asciinema.org/a/101422?t=1:10>`__ bookmarks:

-  `1:10 <https://asciinema.org/a/101422?t=1:10>`__ is when the
   ``ipython`` session is started with
   ``venv/bin/ipython -i request_logs.py``
-  `3:14 <https://asciinema.org/a/101422?t=3:14>`__ is when a second
   ``ipython`` session is started (in a separate tmux pane) to simulate
   a steady stream of requests with
   ``slow_trickle_requests(randomsleep=True, show=True)``
-  `4:22 <https://asciinema.org/a/101422?t=4:22>`__ is when the
   ``index_field_info`` method is used to get the latest counts of top
   indexed items
-  `6:11 <https://asciinema.org/a/101422?t=6:11>`__ is when
   ``slow_trickle_requests(.001)`` is run to simulate a large quick
   burst in traffic
-  `7:00 <https://asciinema.org/a/101422?t=7:00>`__ is when multiple
   values are passed in the ``since`` argument of ``find``...
   ``request_logs.find(count=True, since='5:min, 1:min,   30:sec')``
-  `8:37 <https://asciinema.org/a/101422?t=8:37>`__ is when ``get`` and
   ``get_by_position`` methods are used with a variety of arguments to
   change the structure of what's returned
-  `10:33 <https://asciinema.org/a/101422?t=10:33>`__ is when the
   ``redis_helper.ADMIN_TIMEZONE`` is changed at run time from
   ``America/Chicago`` to ``Europe/London``
-  `11:27 <https://asciinema.org/a/101422?t=11:27>`__ is when ``find``
   is used with a variety of arguments to change the structure of what's
   returned
-  `14:30 <https://asciinema.org/a/101422?t=14:30>`__ is when ``find``
   is used with multiple search terms and multiple ``since`` values...
   ``request_logs.find('host:dogs.com,   uri:/breeds', count=True, since='5:min, 1:min, 10:sec')``
-  `15:54 <https://asciinema.org/a/101422?t=15:54>`__ is when the
   ``update`` method is used to modify data and change history is
   retrieved with the ``old_data_for_hash_id`` method

The first demo walks through the following:

-  creating a virtual environment, installing redis-helper, and
   downloading example files

   ::

       $ python3 -m venv venv
       $ venv/bin/pip3 install redis-helper ipython
       $ venv/bin/rh-download-examples
       $ cat ~/.config/redis-helper/settings.ini
       $ venv/bin/ipython -i request_logs.py

-  using the sample ``Collection`` defined in
   `request\_logs.py <https://github.com/kenjyco/redis-helper/blob/master/examples/request_logs.py>`__
   to

   -  show values of some properties on a ``Collection``

      -  ``redis_helper.Collection._base_key``
      -  ``redis_helper.Collection.now_pretty``
      -  ``redis_helper.Collection.now_utc_float``
      -  ``redis_helper.Collection.keyspace``
      -  ``redis_helper.Collection.size``
      -  ``redis_helper.Collection.first``
      -  ``redis_helper.Collection.last``

   -  show values of some settings from ``redis_helper``

      -  ``redis_helper.APP_ENV``
      -  ``redis_helper.REDIS_URL``
      -  ``redis_helper.REDIS``
      -  ``redis_helper.SETTINGS_FILE``
      -  ``redis_helper.ADMIN_TIMEZONE``

   -  show output from some methods on a ``Collection``

      -  ``redis_helper.Collection.index_field_info()``
      -  ``redis_helper.Collection.find()``
      -  ``redis_helper.Collection.find(count=True)``
      -  ``redis_helper.Collection.find(count=True, since='30:sec')``
      -  ``redis_helper.Collection.find(since='30:sec')``
      -  ``redis_helper.Collection.find(since='30:sec', admin_fmt=True)``
      -  ``redis_helper.Collection.find(count=True, since='5:min, 1:min, 30:sec')``
      -  ``redis_helper.Collection.find('index_field:value')``
      -  ``redis_helper.Collection.find('index_field:value', all_fields=True, limit=2)``
      -  ``redis_helper.Collection.find('index_field:value', all_fields=True, limit=2, admin_fmt=True, item_format='{_ts} -> {_id}')``
      -  ``redis_helper.Collection.find('index_field:value', get_fields='field1,field2', include_meta=False)``
      -  ``redis_helper.Collection.find('index_field1:value1, index_field2:value2', count=True)``
      -  ``redis_helper.Collection.find('index_field1:value1, index_field2:value2', count=True, since='5:min, 1:min, 10:sec')``
      -  ``redis_helper.Collection.get(hash_id)``
      -  ``redis_helper.Collection.get(hash_id, 'field1,field2,field3')``
      -  ``redis_helper.Collection.get(hash_id, include_meta=True)``
      -  ``redis_helper.Collection.get(hash_id, include_meta=True, fields='field1,field2')``
      -  ``redis_helper.Collection.get(hash_id, include_meta=True, item_format='{_ts} -> {_id}')``
      -  ``redis_helper.Collection.get_by_position(0)``
      -  ``redis_helper.Collection.get_by_position(0, include_meta=True, admin_fmt=True)``
      -  ``redis_helper.Collection.update(hash_id, field1='value1', field2='value2')``
      -  ``redis_helper.Collection.old_data_for_hash_id(hash_id)``

Basics - Part 2 (urls demo, with unique field)
----------------------------------------------

`Demo <https://asciinema.org/a/75kl95ty9vg2jl93pfz9fbs9q?t=1:00>`__
bookmarks:

-  ``TODO``

The second demo walks through the following:

-  using the sample ``Collection`` defined in
   `urls.py <https://github.com/kenjyco/redis-helper/blob/master/examples/urls.py>`__
   to

   -  ``TODO``

Settings, environments, testing, and debugging
----------------------------------------------

To trigger a debugger session at a specific place in the `project
code <https://github.com/kenjyco/redis-helper/tree/master/redis_helper>`__,
insert the following, one line above where you want to inspect

::

    import pdb; pdb.set_trace()

To start the debugger inside `test
code <https://github.com/kenjyco/redis-helper/tree/master/tests>`__, use

::

    pytest.set_trace()

-  use ``(l)ist`` to list context lines
-  use ``(n)ext`` to move on to the next statement
-  use ``(s)tep`` to step into a function
-  use ``(c)ontinue`` to continue to next break point (i.e.
   ``set_trace()`` lines in your code)
-  use ``sticky`` to toggle sticky mode (to constantly show the
   currently executing code as you move through with the debugger)
-  use ``pp`` to pretty print a variable or statement

If the redis server at ``redis_url`` (in the **test section** of
``~/.config/redis-server/settings.ini``) is not running or is not empty,
redis server tests will be skipped.

Use the ``APP_ENV`` environment variable to specify which section of the
``settings.ini`` file your settings will be loaded from. Any settings in
the ``default`` section can be overwritten if explicity set in another
section.

-  if no ``APP_ENV`` is explicitly set, ``dev`` is assumed
-  the ``APP_ENV`` setting is overwritten to be ``test`` no matter what
   was set when calling ``py.test`` tests


