Metadata-Version: 1.1
Name: json-logging
Version: 0.0.1
Summary: JSON Python Logging
Home-page: https://github.com/thangbn/json-logging-python
Author: Bui Nguyen Thang (Bob)
Author-email: bob.bui@outlook.com
License: Apache License 2.0
Description: json-logging
        ============
        
        Python logging library to emit JSON log that can be easily indexed and
        searchable by logging infrastructure such as
        `ELK <https://www.elastic.co/webinars/introduction-elk-stack>`_.
        
        | If you’re using Cloud Foundry, it worth to check out the library
          `SAP/cf-python-logging-support <https://github.com/SAP/cf-python-logging-support>`_
          which I’m also original author and contributor. 
        
        1. Features
        ===========
        
        1. Emit JSON logs (see format
           detail in reference)
        2. Support **correlation-id**
           `[1] <https://stackoverflow.com/questions/25433258/what-is-the-x-request-id-http-header>`_
        3. Lightweight, no dependencies, minimal configuration needed (1 LoC to
           get it working)
        4. Fully compatible with Python **logging** module. Support both Python
           2.7.x and 3.x
        5. Support HTTP request instrumentation. Built in support for
           `Flask <http://flask.pocoo.org/>`_ &
           `Sanic <https://github.com/channelcat/sanic>`_. Extensible to support other web
           frameworks. PR welcome :smiley: .
        6. Support inject arbitrary extra properties to JSON log message.
        
        2. Usage
        ========
        
        Install by running this command: 
        
        .. code:: python
        
            pip install json-logging
        
        By default log will be emitted in normal format to ease the local
        development. To enable it on production set either
        **json_logging.ENABLE_JSON_LOGGING** or **ENABLE_JSON_LOGGING
        environment variable** to true.
        
        | To configure, call **json_logging.init(framework_name)**. Once
          configured library will try to configure all loggers (existing and
          newly created) to emit log in JSON format.
        | See following use cases for more detail.
        
        | TODO: update guide on how to use ELK stack to view log
        
        
        2.1 Non-web application log
        ---------------------------
        
        This mode don’t support **correlation-id**.
        
        .. code:: python
        
            import json_logging, logging, sys
        
            # log is initialized without a web framework name
            json_logging.ENABLE_JSON_LOGGING = True
            json_logging.init()
        
            logger = logging.getLogger("test-logger")
            logger.setLevel(logging.DEBUG)
            logger.addHandler(logging.StreamHandler(sys.stdout))
        
            logger.info("test logging statement")
        
        
        2.2 Web application log
        -----------------------
        
        
        Flask
        ~~~~~
        
        .. code:: python
        
            import datetime, logging, sys, json_logging, flask
        
            app = flask.Flask(__name__)
            json_logging.ENABLE_JSON_LOGGING = True
            json_logging.init(framework_name='flask')
            json_logging.init_request_instrument(app)
        
            # init the logger as usual
            logger = logging.getLogger("test-logger")
            logger.setLevel(logging.DEBUG)
            logger.addHandler(logging.StreamHandler(sys.stdout))
        
            @app.route('/')
            def home():
                logger.info("test log statement")
                return "Hello world : " + str(datetime.datetime.now())
        
            if __name__ == "__main__":
                app.run(host='0.0.0.0', port=int(5000), use_reloader=False)
        
        
        Sanic
        ~~~~~
        
        .. code:: python
        
            import logging, sys, json_logging, sanic
        
            app = sanic.Sanic()
            json_logging.ENABLE_JSON_LOGGING = True
            json_logging.init(framework_name='sanic')
            json_logging.init_request_instrument(app)
        
            # init the logger as usual
            logger = logging.getLogger("sanic-integration-test-app")
            logger.setLevel(logging.DEBUG)
            logger.addHandler(logging.StreamHandler(sys.stdout))
        
            @app.route("/")
            async def home(request):
                logger.info("test log statement")
                return sanic.response.text("hello world")
        
            if __name__ == "__main__":
                app.run(host="0.0.0.0", port=8000)
        
        
        2.3 Get current correlation-id
        ------------------------------
        
        Current request correlation-id can be retrieved and pass to downstream
        services call as follow:
        
        .. code:: python
        
            correlation_id = json_logging.get_correlation_id()
            # use correlation id for downstream service calls here
        
        In request context, if one is not present, a new one might be generated
        depends on CREATE_CORRELATION_ID_IF_NOT_EXISTS setting value.
        
        
        2.4 Log extra properties
        ------------------------
        
        Extra property can be added to logging statement as follow:
        
        .. code:: python
        
            logger.info("test log statement", extra = {'props' : {'extra_property' : 'extra_value'}})
        
        
        2.5 Root logger
        ---------------
        
        If you want to use root logger as main logger to emit log. Made sure you
        call **config_root_logger()** after initialize root logger (by
        logging.basicConfig() or logging.getLogger(‘root’))
        `[2] <#2-python-logging-propagate>`_
        
        .. code:: python
        
            logging.basicConfig()
            json_logging.config_root_logger()
        
        
        3. Configuration
        ================
        
        logging library can be configured by setting the value in json_logging
        
        +-------------------------------------+----------------------+----------------------+
        | Name                                | Description          | Default value        |
        +=====================================+======================+======================+
        | ENABLE_JSON_LOGGING                 | Whether to enable    | false                |
        |                                     | JSON logging         |                      |
        |                                     | mode.Can be set as   |                      |
        |                                     | an environment       |                      |
        |                                     | variable, enable     |                      |
        |                                     | when set to to       |                      |
        |                                     | either one in        |                      |
        |                                     | following list       |                      |
        |                                     | (case-insensitive)   |                      |
        |                                     | **[‘true’, ‘1’, ‘y’, |                      |
        |                                     | ‘yes’]**             |                      |
        +-------------------------------------+----------------------+----------------------+
        | ENABLE_JSON_LOGGING_DEBUG           | Whether to enable    | true                 |
        |                                     | debug logging for    |                      |
        |                                     | this library for     |                      |
        |                                     | development purpose. |                      |
        +-------------------------------------+----------------------+----------------------+
        | CORRELATION_ID_HEADERS              | List of HTTP headers | [‘X-Correlation-ID’, |
        |                                     | that will be used to | ‘X-Request-ID’]      |
        |                                     | look for             |                      |
        |                                     | correlation-id       |                      |
        |                                     | value. HTTP headers  |                      |
        |                                     | will be searched one |                      |
        |                                     | by one according to  |                      |
        |                                     | list order           |                      |
        +-------------------------------------+----------------------+----------------------+
        | EMPTY_VALUE                         | Default value when a | ‘-’                  |
        |                                     | logging record       |                      |
        |                                     | property is None     |                      |
        +-------------------------------------+----------------------+----------------------+
        | CORRELATION_ID_GENERATOR            | function to generate | uuid.uuid1           |
        |                                     | unique               |                      |
        |                                     | correlation-id       |                      |
        +-------------------------------------+----------------------+----------------------+
        | JSON_SERIALIZER                     | function to encode   | json.dumps           |
        |                                     | object to JSON       |                      |
        +-------------------------------------+----------------------+----------------------+
        | COMPONENT_ID                        | Uniquely identifies  | EMPTY_VALUE          |
        |                                     | the software         |                      |
        |                                     | component that has   |                      |
        |                                     | processed the        |                      |
        |                                     | current request      |                      |
        +-------------------------------------+----------------------+----------------------+
        | COMPONENT_NAME                      | A human-friendly     | EMPTY_VALUE          |
        |                                     | name representing    |                      |
        |                                     | the software         |                      |
        |                                     | component            |                      |
        +-------------------------------------+----------------------+----------------------+
        | COMPONENT_INSTANCE_INDEX            | Instance’s index of  | 0                    |
        |                                     | horizontally scaled  |                      |
        |                                     | service              |                      |
        +-------------------------------------+----------------------+----------------------+
        | CREATE_CORRELATION_ID_IF_NOT_EXISTS | Whether to generate  | True                 |
        |                                     | a new correlation-id |                      |
        |                                     | in case one is not   |                      |
        |                                     | present              |                      |
        +-------------------------------------+----------------------+----------------------+
        
        
        4. Python References
        ====================
        
        TODO: update Python API docs on Github page
        
        
        5. Framework support plugin development
        =======================================
        
        To add support for a new web framework, you need to extend following
        classes in
        `**framework_base** <https://github.com/thangbn/json-logging-python/blob/master/json_logging/framework_base.py>`_ and
        register support using
        `**json_logging.register_framework_support** <https://github.com/thangbn/json-logging-python/blob/master/json_logging/__init__.py#L38>`_
        method:
        
        +---------------------------------------+----------------------+----------------------+
        | Class                                 | Description          | Mandatory            |
        +=======================================+======================+======================+
        | RequestAdapter                        | Helper class help to | no                   |
        |                                       | extract              |                      |
        |                                       | logging-relevant     |                      |
        |                                       | information from     |                      |
        |                                       | HTTP request object  |                      |
        +---------------------------------------+----------------------+----------------------+
        | ResponseAdapter                       | Helper class help to | yes                  |
        |                                       | extract              |                      |
        |                                       | logging-relevant     |                      |
        |                                       | information from     |                      |
        |                                       | HTTP response object |                      |
        +---------------------------------------+----------------------+----------------------+
        | FrameworkConfigurator                 | Class to perform     | no                   |
        |                                       | logging              |                      |
        |                                       | configuration for    |                      |
        |                                       | given framework as   |                      |
        |                                       | needed               |                      |
        +---------------------------------------+----------------------+----------------------+
        | AppRequestInstrumentationConfigurator | Class to perform     | no                   |
        |                                       | request              |                      |
        |                                       | instrumentation      |                      |
        |                                       | logging              |                      |
        |                                       | configuration        |                      |
        +---------------------------------------+----------------------+----------------------+
        
        Take a look at
        `**json_logging/base_framework.py** <https://github.com/thangbn/json-logging-python/blob/master/json_logging/framework_base.py>`_,
        `**json_logging.flask** <https://github.com/thangbn/json-logging-python/tree/master/json_logging/framework/flask>`_
        and
        `**json_logging.sanic** <https://github.com/thangbn/json-logging-python/tree/master/json_logging/framework/sanic>`_
        packages for reference implementations.
        
        
        6. FAQ & Troubleshooting
        ========================
        
        1. I configured everything, but no logs are printed out?
        
           -  Forgot to add handlers to your logger?
           -  Check whether logger is disabled.
        
        2. Same log statement is printed out multiple times.
        
           -  Check whether the same handler is added to both parent and child
              loggers [2]
           -  If you using flask, by default option **use_reloader** is set to
              **True** which will start 2 instances of web application. change
              it to False to disable this behaviour
              `[3] <#3-more-on-flask-use-reloader>`_
        
        3. Can not install Sanic on Windows?
        
        you can install Sanic on windows by running these commands:
        
        ::
        
            git clone --branch 0.7.0 https://github.com/channelcat/sanic.git
            set SANIC_NO_UVLOOP=true
            set SANIC_NO_UJSON=true
            pip3 install .
        
        
        7. References
        =============
        
        [0] Full logging format references
        ----------------------------------
        
        2 types of logging statement will be emitted by this library: 
        - Application log: normal logging statement e.g.:
        
        ::
        
            {
                "type": "log",
                "written_at": "2017-12-23T16:55:37.280Z",
                "written_ts": 1514048137280721000,
                "component_id": "1d930c0xd-19-s3213",
                "component_name": "ny-component_name",
                "component_instance": 0,
                "logger": "test logger",
                "thread": "MainThread",
                "level": "INFO",
                "line_no": 22,
                "correlation_id": "1975a02e-e802-11e7-8971-28b2bd90b19a",
                "extra_property": "extra_value"
            }
        
        -  Request log: request instrumentation logging statement which recorded
           request information such as response time, request size, etc.
        
        ::
        
            {
                "type": "request",
                "written_at": "2017-12-23T16:55:37.280Z",
                "written_ts": 1514048137280721000,
                "component_id": "-",
                "component_name": "-",
                "component_instance": 0,
                "correlation_id": "1975a02e-e802-11e7-8971-28b2bd90b19a",
                "remote_user": "user_a",
                "request": "/index.html",
                "referer": "-",
                "x_forwarded_for": "-",
                "protocol": "HTTP/1.1",
                "method": "GET",
                "remote_ip": "127.0.0.1",
                "request_size_b": 1234,
                "remote_host": "127.0.0.1",
                "remote_port": 50160,
                "request_received_at": "2017-12-23T16:55:37.280Z",
                "response_time_ms": 0,
                "response_status": 200,
                "response_size_b": "122",
                "response_content_type": "text/html; charset=utf-8",
                "response_sent_at": "2017-12-23T16:55:37.280Z"
            }
        
        See following tables for detail format explanation: 
        
         - Common field
        
        +-------------------+-----------------+-----------------+-----------------+
        | Field             | Description     | Format          | Example         |
        +===================+=================+=================+=================+
        | written_at        | The date when   | ISO 8601        | 2017-12-23T15:1 |
        |                   | this log        | YYYY-MM-DDTHH:M | 4:02.208Z       |
        |                   | message was     | M:SS.milliZ     |                 |
        |                   | written.        |                 |                 |
        +-------------------+-----------------+-----------------+-----------------+
        | written_ts        | The timestamp   | long number     | 145682055381684 |
        |                   | in nano-second  |                 | 9408            |
        |                   | precision when  |                 |                 |
        |                   | this request    |                 |                 |
        |                   | metric message  |                 |                 |
        |                   | was written.    |                 |                 |
        +-------------------+-----------------+-----------------+-----------------+
        | correlation_id    | The timestamp   | string          | db2d002e-2702-4 |
        |                   | in nano-second  |                 | 1ec-66f5-c002a8 |
        |                   | precision when  |                 | 0a3d3f          |
        |                   | this request    |                 |                 |
        |                   | metric message  |                 |                 |
        |                   | was written.    |                 |                 |
        +-------------------+-----------------+-----------------+-----------------+
        | type              | Type of         | string          |                 |
        |                   | logging. “logs” |                 |                 |
        |                   | or “request”    |                 |                 |
        +-------------------+-----------------+-----------------+-----------------+
        | component_id      | Uniquely        | string          | 9e6f3ecf-def0-4 |
        |                   | identifies the  |                 | baf-8fac-9339e6 |
        |                   | software        |                 | 1d5645          |
        |                   | component that  |                 |                 |
        |                   | has processed   |                 |                 |
        |                   | the current     |                 |                 |
        |                   | request         |                 |                 |
        +-------------------+-----------------+-----------------+-----------------+
        | component_name    | A               | string          | my-component    |
        |                   | human-friendly  |                 |                 |
        |                   | name            |                 |                 |
        |                   | representing    |                 |                 |
        |                   | the software    |                 |                 |
        |                   | component       |                 |                 |
        +-------------------+-----------------+-----------------+-----------------+
        | component_instance| Instance’s      | string          | 0               |
        |                   | index of        |                 |                 |
        |                   | horizontally    |                 |                 |
        |                   | scaled service  |                 |                 |
        +-------------------+-----------------+-----------------+-----------------+
        
         -  application logs
        
        +-----------------+-----------------+-----------------+-----------------+
        | Field           | Description     | Format          | Example         |
        +=================+=================+=================+=================+
        | msg             | The actual      | string          | This is a log   |
        |                 | message string  |                 | message         |
        |                 | passed to the   |                 |                 |
        |                 | logger.         |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | level           | The log “level” | string          | INFO            |
        |                 | indicating the  |                 |                 |
        |                 | severity of the |                 |                 |
        |                 | log message.    |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | thread          | Identifies the  | string          | http-nio-4655   |
        |                 | execution       |                 |                 |
        |                 | thread in which |                 |                 |
        |                 | this log        |                 |                 |
        |                 | message has     |                 |                 |
        |                 | been written.   |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | logger          | The logger name |                 |                 |
        |                 | that emits the  |  string         |requests-logger  |
        |                 | log message.    |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        
         -  request logs:
        
        +-----------------+-----------------+-----------------+-----------------+
        | Field           | Description     | Format          | Example         |
        +=================+=================+=================+=================+
        | request         | request path    | string          | /get/api/v2     |
        |                 | that has been   |                 |                 |
        |                 | processed.      |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | request_receive | The date when   | ISO 8601        | 2015-01-24      |
        | d_at            | an incoming     | YYYY-MM-DDTHH:M | 14:06:05.071Z   |
        |                 | request was     | M:SS.milliZ     |                 |
        |                 | received by the | The precision   |                 |
        |                 | producer.       | is in           |                 |
        |                 |                 | milliseconds.   |                 |
        |                 |                 | The timezone is |                 |
        |                 |                 | UTC.            |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | response_sent_a | The date when   | ditto           | 2015-01-24      |
        | t               | the response to |                 | 14:06:05.071Z   |
        |                 | an incoming     |                 |                 |
        |                 | request was     |                 |                 |
        |                 | sent to the     |                 |                 |
        |                 | consumer.       |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | response_time_m | How many        | float           | 43.476          |
        | s               | milliseconds it |                 |                 |
        |                 | took the        |                 |                 |
        |                 | producer to     |                 |                 |
        |                 | prepare the     |                 |                 |
        |                 | response.       |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | protocol        | Which protocol  | string          | HTTP/1.1        |
        |                 | was used to     |                 |                 |
        |                 | issue a request |                 |                 |
        |                 | to a producer.  |                 |                 |
        |                 | In most cases,  |                 |                 |
        |                 | this will be    |                 |                 |
        |                 | HTTP (including |                 |                 |
        |                 | a version       |                 |                 |
        |                 | specifier), but |                 |                 |
        |                 | for outgoing    |                 |                 |
        |                 | requests        |                 |                 |
        |                 | reported by a   |                 |                 |
        |                 | producer, it    |                 |                 |
        |                 | may contain     |                 |                 |
        |                 | other values.   |                 |                 |
        |                 | E.g. a database |                 |                 |
        |                 | call via JDBC   |                 |                 |
        |                 | may report,     |                 |                 |
        |                 | e.g. “JDBC/1.2” |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | method          | The             | string          | GET             |
        |                 | corresponding   |                 |                 |
        |                 | protocol        |                 |                 |
        |                 | method.         |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | remote_ip       | IP address of   | string          | 192.168.0.1     |
        |                 | the consumer    |                 |                 |
        |                 | (might be a     |                 |                 |
        |                 | proxy, might be |                 |                 |
        |                 | the actual      |                 |                 |
        |                 | client)         |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | remote_host     | host name of    | string          | my.happy.host   |
        |                 | the consumer    |                 |                 |
        |                 | (might be a     |                 |                 |
        |                 | proxy, might be |                 |                 |
        |                 | the actual      |                 |                 |
        |                 | client)         |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | remote_port     | Which TCP port  | string          | 1234            |
        |                 | is used by the  |                 |                 |
        |                 | consumer to     |                 |                 |
        |                 | establish a     |                 |                 |
        |                 | connection to   |                 |                 |
        |                 | the remote      |                 |                 |
        |                 | producer.       |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | remote_user     | The username    | string          | user_name       |
        |                 | associated with |                 |                 |
        |                 | the request     |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | request_size_b  | The size in     | long            | 1234            |
        |                 | bytes of the    |                 |                 |
        |                 | requesting      |                 |                 |
        |                 | entity or       |                 |                 |
        |                 | “body” (e.g.,   |                 |                 |
        |                 | in case of POST |                 |                 |
        |                 | requests).      |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | response_size_b | The size in     | long            | 1234            |
        |                 | bytes of the    |                 |                 |
        |                 | response entity |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | response_status | The status code | long            | 200             |
        |                 | of the          |                 |                 |
        |                 | response.       |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | response_conten | The MIME type   | long            | application/jso |
        | t_type          | associated with |                 | n               |
        |                 | the entity of   |                 |                 |
        |                 | the response if |                 |                 |
        |                 | available or    |                 |                 |
        |                 | specified       |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | referer         | For HTTP        | string          | /index.html     |
        |                 | requests,       |                 |                 |
        |                 | identifies the  |                 |                 |
        |                 | address of the  |                 |                 |
        |                 | webpage         |                 |                 |
        |                 | (i.e. the URI   |                 |                 |
        |                 | or IRI) that    |                 |                 |
        |                 | linked to the   |                 |                 |
        |                 | resource being  |                 |                 |
        |                 | requested.      |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        | x_forwarded_for | Comma-separated | string          | 192.0.2.60,10.1 |
        |                 | list of IP      |                 | 2.9.23          |
        |                 | addresses, the  |                 |                 |
        |                 | left-most being |                 |                 |
        |                 | the original    |                 |                 |
        |                 | client,         |                 |                 |
        |                 | followed by     |                 |                 |
        |                 | proxy server    |                 |                 |
        |                 | addresses that  |                 |                 |
        |                 | forwarded the   |                 |                 |
        |                 | client request. |                 |                 |
        +-----------------+-----------------+-----------------+-----------------+
        
        [1] What is correlation-id/request id
        -------------------------------------
        
        https://stackoverflow.com/questions/25433258/what-is-the-x-request-id-http-header
        
        [2] Python logging propagate
        ----------------------------
        https://docs.python.org/3/library/logging.html#logging.Logger.propagate
        https://docs.python.org/2/library/logging.html#logging.Logger.propagate
        
        [3] more on flask use_reloader
        ------------------------------
        
        http://flask.pocoo.org/docs/0.12/errorhandling/#working-with-debuggers
Keywords: json,elastic,python,python3,python2,logging,logging-library,json,elasticsearch,elk,elk-stack,logstash,kibana
Platform: any
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: System :: Logging
Classifier: Framework :: Flask
