Metadata-Version: 1.1
Name: distlog
Version: 0.1.dev0
Summary: Logging for distributed systems.
Home-page: https://github.com/lnoor/distlog
Author: Leo Noordergraaf
Author-email: leo@noordergraaf.net
License: GPLv3
Description: Distlog
        =======
        
        Is a Python library and a set of applications to facilitate logging in a
        distributed application.
        
        The reason to create yet another logging tool arose while developing an
        application based on (micro) services. Even though each component produces a
        liberal amount of logging data making sense of it is challenging at times.
        
        This is because each log file only tells part of the story.
        The challenge is to find those parts of the log entries that are relevant to a
        particular problem request. I tried to solve this by using correlation IDs.
        These IDs are assigned when a request first enters the system and is passed on
        to each service that is invoked to satisfy that request. It sort of works but it
        takes a lot of time piecing all the bits together.
        
        Hence the desire to let the system sort it out itself.
        
        Requirements
        ============
        
        My preferred solution should:
        
        Interoperate
        ------------
        
        Interoperate with Python's native logging library.
            This will automatically include logging activity from incorporated libraries
            which nearly always use Python's native logging support.
        
        Distributed
        -----------
        
        Allow multiple clients to produce log entries.
            It is after all meant as a solution for a distributed application.
        
        Centralize all logs in a single location.
            This will make it possible to combine entries from different sources
            relating to the same request.
        
        Allow multiple clients to process the log entries.
            Some will store the log entries on permanent storage (file, database).
            Others may filter the entries and display them as if tailing a logfile.
        
        Structured
        ----------
        
        Offer additional structure to the log entries.
            Normal logs are a sequential but otherwise unstructured list of hopefully
            significant facts. What I'd like to see is that the log entries tell me what
            is being achieved and wheter that succeeded or not. Each of these goals may
            itself consist of subgoals creating a tree of goals which together reach a
            conclusion.
        
            For many years now we modularize and structure our code, yet none of that is
            visible in the log entries that tell us how the program is progressing.
        
        The structure in the log entries should transcent process bounderies.
            When one process invokes the services of another process the structure of
            the log entries should reflect this. A process switch should not break the
            tree structure described above.
        
        Dataset oriented
        ----------------
        
        Support record logging.
            Traditionally log entries are supposed to be a single text string. I want to
            see all of Python's LogRecord fields in the log entry. And probably quite a
            few more fields as well. We can assume the LogRecord fields as a stable
            basis set of properties that are expanded by custom properties depending on
            the application and function.
        
        Oh and the whole thing should be modular off course.
            - Structured logging
            - Multi process logging
            - Record logging
        
            Should be separate functional modules that can be combined to provide all of
            the above.
        
        There is just one little thing: it doesn't exist. Yet.
        
        The distlog package
        ====================
        The distlog package exports a couple classes and functions:
        
        * :class:`ZmqHandler`
            This is a handler for the standard Python logging library that transports
            :class:`LogRecords`. For it to work it needs a serializer that prepares the
            :class:`LogRecord` instances for transportation. This serializer should be
            derived from the :class:`~distlog.formatters.Serializer` class. Choose one
            of the predefined serializers :class:`JSONFormatter` or
            :class:`PickleFormatter` and set it as the formatter using
            :method:`ZmqHandler.setFormatter`.
        
        * :class:`JSONFormatter`
            This is one of the predefined serializers, it converts a :class:`LogRecord`
            instances to JSON strings that can be sent via 0MQ.
        
        * :class:`PickleFormatter`
            This is the other predefined serializer. it converts a :class:`LogRecord` to
            the binary Python specific pickle format. Note that the serialization method
            is part of the message so that the receiver knows how to deserialize the
            message.
        
        * :function:`task`
            This function returns a :class:`Task` object that implements as a context
            handler and acts as the top level logging context for Pythons normal logger
            functions.
        
        * :function:`to`
            This function returns a :class:`Task` instance that is a child of another
            logging context thus constructing a hierarchy of logging contexts.
        
        * :function:`import_task`
            Creates a top level logging context that is subordinate to the logging
            context in a different process. Using :method:`Task.get_foreign_task` to
            obtain a task/context id, transferring this id from one process to another
            and passing it as parameter to :function:`import_task` it is possible to
            relate the activities across processes as belonging to one and the same
            activity.
        
        Logging infrastructure
        ======================
        The process(es) forming the application that is being logged send out their log
        messages using :class:`ZmqHandler`. This class sends the message out using a
        0MQ PUSH socket that binds to the well-known address of the corresponding PULL
        socket of the `distlogd` process.
        
        The `distlogd` process receives all log messages and filters them using a
        configurable rule set. Based on the rule(s) which triggered, the message is
        forwarded to one or more processors.
        
        The configuration file, probably a YAML file, may instruct `distlogd` to load
        processors as plugins. A processor will apply one or more filters on a message
        and uses the outcome to decide to handle a message or not.
        
Keywords: scoped structured distributed logging
Platform: any
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Environment :: Other Environment
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: GNU General Public License v3 (GPLv3)
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3.4
Classifier: Topic :: System :: Logging
Classifier: Topic :: Utilities
