Metadata-Version: 1.0
Name: winchester
Version: 0.54.dev33
Summary: An OpenStack notification event processing library.
Home-page: https://github.com/stackforge/stacktach-winchester
Author: Monsyne Dragon
Author-email: mdragon@rackspace.com
License: Apache-2
Description: winchester
        ==========
        
        An OpenStack notification event processing library based on persistant streams.
        
        Winchester is designed to process event streams, such as those produced from
        OpenStack notifications. Events are represented as simple python dictionaries.
        They should be flat dictionaries (not nested), with a minimum of three keys:
        
            "message_id":   A unique identifier for this event, such as a uuid.
            "event_type":   A string identifying the event's type. Usually a hierarchical dotted name like "foo.bar.baz"
            "timestamp":    Time the event occurred (a python datetime, in UTC)
        
        The individual keys of the event dictionary are called *traits* and can be
        strings, integers, floats or datetimes. For processing of the (often large)
        notifications that come out of OpenStack, winchester uses the
        [StackDistiller library](https://github.com/StackTach/stackdistiller) to
        extract flattened events from the notifications, that only contain the data
        you actually need for processing.
        
        Winchester's processing is done through *triggers* and *pipelines*.
        
        A *trigger* is composed of a *match_criteria* which is like a
        persistant query, collecting events you want to process into a
        persistant *stream* (stored in a sql database), a set of distinguishing
        traits, which can separate your list of events into distinct streams,
        similar to a **GROUP BY** clause in an SQL query, and a *fire_criteria*,
        which specifies the conditions a given *stream* has to match for the
        trigger to fire. When it does, the events in the *stream* are sent to
        a *pipeline* listed as the *fire_pipeline* for processing as a batch.
        Also listed is an *expire_timestamp*. If a given stream does not meet
        the *fire_criteria* by that time, it is expired, and can be sent to
        an *expire_pipeline* for alternate processing. Both *fire_pipeline*
        and *expire_pipeline* are optional, but at least one of them must
        be specified.
        
        A *pipeline* is simply a list of simple *handlers*. Each *handler*
        in the pipeline receives the list of events in a given stream,
        sorted by timestamp, in turn. *Handlers* can filter events from the list,
        or add new events to it. These changes will be seen by *handlers* further
        down the pipeline. *Handlers* should avoid operations with side-effects,
        other than modifying the list of events, as pipeline processing can be
        re-tried later if there is an error. Instead, if all handlers process the
        list of events without raising an exception, a *commit* call is made on
        each handler, giving it the chance to perform actions, like sending data
        to external systems. *Handlers* are simple to write, as pretty much any
        object that implements the appropriate *handle_events*, *commit* and
        *rollback* methods can be a *handler*.
        
        ## Installing and running.
        
        Winchster is installable as a simple python package.
        Once installed, and the appropriate database url is specified in the
        *winchester.yaml* config file (example included in the *etc* directory),
        you can create the appropriate database schema with:
        
            winchester_db -c <path_to_your_config_files>/winchester.yaml upgrade head
        
        If you need to run the SQL by hand, or just want to look at the schema, the
        following will print out the appropriate table creation SQL:
        
            winchester_db -c <path_to_your_config_files>/winchester.yaml upgrade --sql head
        
        Once you have done that, and configured the appropriate *triggers.yaml*,
        *pipelines.yaml*, and, if using StackDistiller, *event_definitions.yaml* configs
        (again, examples are in *etc* in the winchester codebase), you can add events
        into the system by calling the *add_event* method of Winchester's TriggerManager.
        If you are processing OpenStack notifications, you can call *add_notification*,
        which will pare down the notification into an event with StackDistiller, and
        then call *add_event* with that. If you are reading OpenStack notifications off
        of a RabbitMQ queue, there is a plugin for the
        [Yagi](https://github.com/rackerlabs/yagi) notification processor included with
        Winchester. Simply add "winchester.yagi\_handler.WinchesterHandler" to the "apps"
        line in your *yagi.conf* section for the queues you want to listen to, and add a:
        
            [winchester]
            config_file = <path_to_your_config_files>/winchester.yaml
        
        section to the *yagi.conf*.
        
        To run the actual pipeline processing, which is run as a separate daemon, run:
        
            pipeline_worker -c <path_to_your_config_files>/winchester.yaml
        
        You can pass the *-d* flag to the *pipeline_worker* to tell it to run as a background
        daemon.
        
        Winchester uses an optimistic locking scheme in the database to coordinate firing,
        expiring, and processing of streams, so you can run as many processes (like
        Yagi's *yagi-event* daemon) feeding TriggerManagers as you need to handle the
        incoming events, and as many *pipeline_worker*s as you need to handle the resulting
        processing load, scaling the system horizontally.
        
        
Keywords: stacktach
event_processing
pipeline
events
notification
openstack
triggers
Platform: UNKNOWN
