.. index:: message enrichment, SOW; message enrichment,

.. _enrichment:

State of the World Message Enrichment
=====================================

Topics recorded to the State of the World (SOW) can provide inline message
enrichment for messages published to the topic. This capability is
especially useful for applications that do consistent, simple
transformations on incoming data. For example, you can use this
capability to automatically add a calculated price to an incoming order,
to map abbreviated data such as status codes to easier-to-understand
values, or even to compute the value of a field used for a SOW key.

AMPS provides two distinct stages of message enrichment: *preprocessing* and 
*enrichment*. The *preprocessing* stage occurs before AMPS calculates the SOW 
key for the message. Fields that are added or updated in the preprocessing 
stage can be used as the SOW key for the message. Given that this stage occurs 
before the SOW key is generated, this stage does not have access to the
previous state of the message in the SOW. The enrichment stage occurs
after AMPS calculates the SOW key. *Enrichment* performed at this stage
has access to the previous state of the SOW.

If entitlement for the instance uses content filters for publish
entitlements, these filters are applied to the incoming message *before*
either enrichment stage runs. For more details on the steps involved in
enrichment, see :ref:`SOW Update and Enrichment<#ug-sow-update-sequence>`.

Message enrichment only affects the message data, not the metadata on
the message. In other words, while enrichment can change any field in
the data, you cannot change metadata properties such as the topic the
message was published to, the acknowledgments requested on the message,
or the authenticated username for the publish command.

Message enrichment rewrites the message before the messages are stored
in AMPS or delivered to publishers. AMPS also provides the ability to
aggregate or analyze messages while preserving the original state of the
message, as described in the chapter on :ref:`Aggregating and Analyzing Data in AMPS <#ug-views>`.
If a subscriber only needs a subset of data in a message, AMPS provides the
ability for that subscriber to provide a *select list* to retrieve only the
needed data.

Preprocessing Messages
-----------------------

The preprocessing stage of AMPS enrichment allows you to alter a message
before the SOW key is calculated. This gives you the ability to easily
add or transform fields that are used in the SOW key. Use this stage to
enrich messages when the enriched field should be used as part of the
SOW key. To specify preprocessing for a topic, you add a
``Preprocessing`` directive to the ``Topic`` configuration for the SOW
topic.


.. tip::

   Use ``Preprocessing`` when you need to change the value of
   a field that is part of the ``Key`` for the topic. Otherwise,
   use ``Enrichment``.

Preprocessing field directives operate on a single message and construct
fields based on that message. The results of the preprocessing
expression are merged into the incoming message. Any field in the source
message that is not changed or removed during preprocessing is left
unchanged, so it is not necessary to include all fields in the message
in the ``Preprocessing`` block.

Since preprocessing fields apply to a specific message, preprocessing
fields cannot specify the topic or message type in an XPath identifier.

By default, AMPS serializes fields with a NULL value in the preprocessing 
result. Preprocessing fields can include a directive that specifies that 
if a field contains a NULL value, it should be removed from the set of 
fields rather than serialized. The directive ``HINT OPTIONAL`` applied 
to the XPath identifier specifies that if the result of the source 
expression is ``NULL``, AMPS does not provide the value for the message 
type to serialize. For example, use the following directive to remove a 
``/source`` field if the value provided is not in a specific list of values:

.. code-block:: xml

    <Field>IF(/source IN ('a','e','f'), /source, NULL)
           AS /source HINT OPTIONAL</Field>

For more information on constructing preprocessing fields, see
:ref:`#construct-preprocessing-fields`.

Enriching Messages
--------------------

AMPS enrichment operates on a message after the SOW key is computed, but
before an incoming delta publish is merged to an existing message, or
the incoming message is written to the transaction log, stored to the
SOW, used to update views, or delivered to subscribers. Use this
enrichment stage when the enrichment process depends on the previous
values of the message, or when the updated fields will not be used in
the SOW key. To specify enrichment for a topic, you add an
``Enrichment`` directive to the configuration for the SOW topic.

Enrichment field directives operate on a single message and construct
fields based on that message. Enrichment expressions operate on the
current message and change the current message. The results of the
enrichment directives are merged into the incoming message. Any field in
the source message that is not changed or removed during enrichment
is left unchanged, so it is not necessary to include all fields in the
message in the ``Enrichment`` directive.

Since enrichment fields apply to a specific message, enrichment fields
cannot specify the topic or message type in an XPath identifier.

Within an enrichment expression, AMPS provides two special modifiers for
XPath identifiers that specify whether an XPath identifier refers to the
current incoming message or the previous state of the message. These
modifiers apply only to the source expression, and cannot be used in
special modifiers. They are:

.. list-table:: *XPath Identifier Modifiers for Enrichment*
     :header-rows: 1

     * - Modifier

       - Description

     * - ``OF CURRENT``
         

       - Specify that the XPath identifier refers to the
         incoming message.

     * - ``OF PREVIOUS``
         

       - Specify that the XPath identifier refers to the
         previous state of the message in the SOW. If there
         is no record in the SOW for this message, all
         identifiers that specify ``OF PREVIOUS`` return
         ``NULL``.

By default, AMPS serializes fields with a NULL value during enrichment.
Enrichment fields can include a directive that specifies that if a field 
contains a NULL value, it should be removed from the set of fields
rather than serialized. The directive ``HINT OPTIONAL`` applied to the
XPath identifier specifies that if the result of the source expression
is ``NULL``, AMPS does not include the value in the set of XPath
identifiers for the message type to serialize. For example, use the
following directive to use remove a ``/source`` field if the value
provided is not in a specific list of values:

.. code-block:: xml

    <Field>IF(/source IN ('a','e','f'), /source, NULL)
           AS /source HINT OPTIONAL</Field>

For more information on constructing enrichment fields, see
:ref:`#construct-enrichment-fields` 
and :ref:`#construct-preprocessing-fields`.

.. _#ug-sow-update-sequence:

SOW Update and Enrichment Processing
-------------------------------------

The following diagram presents a simplified, high-level view of the
update process for an individual message. For the purposes of this
diagram, views and conflated topics can be considered listeners on the
SOW topic, while applications that connect to AMPS and the
``on-publish`` and ``on-deliver`` actions can be considered subscribers.

.. image:: ../../common/chapters/sow/svg/sow_update_sequence.svg
   :width: 30.0%

It's important to keep in mind the following aspects of the SOW update
sequence:

-  If the publish is disallowed due to topic-based entitlements or the
   publish filter specified for entitlements, there is no change to the
   state of the SOW. The entitlement filter (if one exists), is applied
   to the incoming message *before* preprocessing, enrichment, or delta
   merge occurs.

-  AMPS records the enriched message in the transaction log and SOW
   file. When AMPS is configured for enrichment or your application
   performs a delta publish, the transaction log and SOW do *not*
   preserve a record of the original message received by AMPS. Instead,
   they record the enriched and merged message.

-  Content filtering for subscriptions, views, and so forth is done on
   the final enriched and merged message, not on the original message as
   published.

