.. index:: delta, incremental message update, Command; delta publish,

.. _#ug-delta:

11. Delta Messaging
====================

AMPS delta messaging allows applications to work with only the changed
parts of a message in the SOW. In high-performance messaging, it's
important that applications not waste time or bandwidth for messages
that they aren't going to use.

Delta messaging has two distinct aspects:

-  *delta subscribe* - allows subscribers to receive just the fields that
   are updated within a message.

-  *delta publish* - allows publishers to update and add fields within a
   message by publishing only the updates into the SOW.

While these features are often used together, the features are
independent. For example, a subscriber can request a regular
subscription even if a publisher is publishing deltas. Likewise, a
subscriber can request a delta subscription even if a publisher is
publishing full messages.

To be able to use delta messages, the message type for the subscription
must support delta messages. All of the included AMPS message types,
except for ``binary`` and ``struct``, support delta messages, with the
limitations described in each section below. For custom message types,
contact the message type implementer to determine whether delta support is
implemented.

These features can often improve performance in environments where
bandwidth is at a premium. Because these features require AMPS to parse,
compare, and create messages, these features can consume somewhat more
CPU on the AMPS server than a simple publish or subscribe, particularly
for large messages with complex structure (such as deeply-nested XML).

Delta Subscribe
----------------

Delta subscribe allows applications to receive only the changed parts of
a message when an update is made to a record in the SOW. When a delta
subscription is active, AMPS compares the new state of the message to
the old state of the message, creates a message for the difference, and
sends the difference message to subscribers. Using this approach can
simplify processing on the client side, and can improve performance when
network bandwidth is the most important constraint.

For example, consider a SOW that contains the following messages, with
the ``order`` field as the key of the SOW topic:

.. image:: ../png/delta_original_messages.png

Now, consider an update that changes the status of order number 3:

.. code-block:: javascript

    {
        "order":3,
        "customer":"Patrick",
        "status":"pending",
        "qty":1000,
        "ticker":"MSFT"
    }

For a regular subscription, subscribers receive the entire message. With
a delta subscription, subscribers receive just the key of the SOW topic
and any changed fields:

.. image:: ../png/delta_update_message.png

This can significantly reduce the amount of network traffic,
and can also simplify processing for subscribers, since the only
information sent is the information needed by the subscriber to take
action on the message.

When the ``oof`` option is specified on a delta subscription, AMPS will deliver
the full message when a previously out-of-focus message comes into focus,
even if only some of the fields in the message have changed.

Using Delta Subscribe
^^^^^^^^^^^^^^^^^^^^^^

Because a client must process delta subscriptions using substantially
different logic than regular subscriptions, delta subscription is
implemented as a separate set of AMPS commands rather than simply as an
option on subscribe commands. AMPS supports two different ways to
request a delta subscription:

+--------------------------------------+--------------------------------------+
| Command                              | Result                               |
+======================================+======================================+
| ``delta_subscribe``                  | Register a delta subscription,       |
|                                      | starting with newly received         |
|                                      | messages.                            |
+--------------------------------------+--------------------------------------+
| ``sow_and_delta_subscribe``          | Replay the state of the SOW and      |
|                                      | atomically register a delta          |
|                                      | subscription.                        |
+--------------------------------------+--------------------------------------+

**Table 11.1:** *Delta subscribe commands*

Applications most commonly use ``sow_and_delta_subscribe`` to receive
the current state of messages in the SOW before they begin receiving
deltas.

Options for Delta Subscribe
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The delta subscribe command accepts several options that are unique to
delta subscriptions. These options control the precise behavior of delta
messages:

+------------------------------+---------------------------------------------+
| Option                       | Result                                      |
+==============================+=============================================+
| ``no_empties``               | Do not send messages if no data fields      |
|                              | have been updated. By default, AMPS will    |
|                              | publish a delta for every publish to the    |
|                              | record, even if the data has not changed.   |
|                              | By specifying this option, AMPS will only   |
|                              | send messages when there is changed data.   |
+------------------------------+---------------------------------------------+
| ``no_sowkey``                | Do not include the AMPS generated SowKey    |
|                              | with messages. By default, AMPS includes    |
|                              | this key to help you identify unique        |
|                              | records within the SOW.                     |
+------------------------------+---------------------------------------------+
| ``send_keys``                | Include the SOW key fields in the message.  |
|                              | Because the SOW key fields indicate which   |
|                              | message to update, without this option,     |
|                              | updates to delta messages will never        |
|                              | contain the SOW key fields. For views, the  |
|                              | SOW key fields are the fields specified in  |
|                              | the ``Grouping`` element.                   |
|                              |                                             |
|                              | *AMPS accepts this option for backward      |
|                              | compatibility. As of AMPS 4.0, this option  |
|                              | is included on delta subscriptions by       |
|                              | default.*                                   |
+------------------------------+---------------------------------------------+
| ``oof``                      | AMPS will deliver out of focus messages on  |
|                              | this subscription. When focus tracking is   |
|                              | enabled, AMPS will also deliver the full    |
|                              | message to a subscription when a previously |
|                              | out-of-focus message comes into focus.      |
+------------------------------+---------------------------------------------+

**Table 11.2:** *Options for delta subscriptions*

Delta subscriptions also support the options provided for regular
subscriptions, including the timestamp option and the conflation options
described in :ref:`Chapter 3 Conflation<#ug-pub-sub-conflation>`.


Identifying Changed Records
~~~~~~~~~~~~~~~~~~~~~~~~~~~

When an application that uses delta subscriptions receives a message,
that message can either be a new record or an update to an existing
record. AMPS offers two strategies for an application to tell whether
the record is a new record or an existing record, and identify which
record has changed if the message is an update to an existing record.

The two basic approaches are as follows:

1. By default, each message delivered through a delta subscription
   contains a SowKey header field. This field is the identifier that
   AMPS assigns to track a distinct record in the SOW. If the
   application has previously received a SowKey with that value, then
   the new message is an update to the record with that SowKey value. If
   the application has not previously received a SowKey with that value,
   then the new message contains a new record.

2. Delta messages can also contain the key fields from the SOW in the
   body of the message. This is controlled by the ``send_keys`` option
   on the subscription, which is always enabled as of AMPS 4.0. With
   this approach, the application parses the body of the message to find
   the key. If the application has previously received the key, then the
   message is an update to that existing record. Otherwise, the message
   contains a new record.

In either case, AMPS delivers the information the application needs to
determine if the record is new or changed. The application chooses how
to interpret that information, and what actions to take based on the
changes to the record.

AMPS also supports out-of-focus notification for delta subscriptions, as
described in :ref:`Chapter 9<#ug-oof>`. If your application needs to know when a
record is deleted, expires, or no longer matches a subscription, you can
use out-of-focus messages to be notified.

Conflated Subscriptions and Delta Subscribe
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

AMPS provides subscription conflation on delta subscriptions. When
conflation is enabled, each delta message during the conflation interval
is merged into the conflated message. The message that is delivered is
the merge of all of the deltas that arrived during the conflation
interval.

Because AMPS combines successive delta messages into a single update, a
delta subscription that uses conflation may receive values that are
identical to the previous values. For example, consider the following
record in a SOW that uses ``/id`` as the key:

.. code-block:: javascript

    {
        "id": 99,
        "status":"open",
        "notes":"none",
        "xref":82
    }

Assume that the following updates to the record are published during the
conflation interval:

.. code-block:: javascript

    {"id": 99, "status":"questioned", "notes":"none", "xref":82}
    {"id": 99, "status":"questioned", "notes":"jcarlo hold", "xref":82}
    {"id": 99, "status":"cleared", "notes":"none", "xref":82}
    {"id": 99, "status":"open", "notes":"none", "xref":82}

At the end of the conflation interval, the subscription will receive the
delta message

.. code-block:: javascript

    {
        "id": 99,
        "status":"open",
        "notes":"none"
    }

The ``/id`` field is included because that field is the key of the SOW,
and all of the delta messages produced during the conflation interval
contained that key. The ``/status`` and ``/notes`` fields are included
because there were changes to these values during the conflation
interval. The delta messages produced during the conflation interval
contained changed values, so the merged update contains those fields and
the state of the values at the end of the conflation interval. The
``/xref`` field is not included, because none of the delta messages
produced during the conflation interval contained that field.

Select Lists and Delta Subscribe
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When a ``delta_subscribe`` or ``sow_and_delta_subscribe`` command provides a
select list and the ``no_empties`` option, only changes to fields included in the
select list will prompt a publish to the client. Changes to fields that the subscriber does not receive
do not produce a publish.

Delta Subscribe Support
~~~~~~~~~~~~~~~~~~~~~~~

To produce delta messages, the message type and the topic must both
support delta subscribe. When this is not the case, AMPS accepts the
subscription, but provides full messages rather than delta messages.

All of the basic message types provided with AMPS support delta
subscribe with the exception of the ``binary`` message type, ``struct`` message
types, and protobuf message types that use protobuf version 3. Composite message
types support delta subscribe if they use the ``composite-local`` definition,
as described in the section on composite message types.

AMPS queues do not support delta subscribe. AMPS accepts a delta
subscription for a queue, but produces full messages from the queue.

Bookmark subscriptions do not support delta subscribe. AMPS does not accept 
a delta bookmark subscription.

All other AMPS topic types that are based on a SOW support delta
subscribe. AMPS topics that do not use a SOW do not support delta
subscribe, and instead produce full messages.

Multiple Subscriptions and Delta Subscribe
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When a single connection to AMPS has multiple subscriptions, AMPS sends
the message to that client once, with information on the set of
subscriptions that match. AMPS sends a message that will include the
requested data for all of the matching subscriptions. For example, if a
message matches one subscription that requests full messages and another
subscription from the same connection that requests deltas, both
subscriptions will receive a full message. If your application depends
on receiving deltas, take care that the application does not issue
non-delta subscriptions for the same set of messages on the same
connection.

Delta Publish
----------------

Delta publish allows publishers to update a message in the SOW by
providing just the key fields for the SOW and the data to update. When
AMPS receives a delta publish, AMPS parses the incoming message and the
existing messages, identifies changed fields, and creates an updated
message that merges changed fields from the delta publish into the
existing message. If there is no existing message, the result of a delta
publish is the same as if the update were merged into an empty message:
the fields in the delta publish, and only those fields, are present
in the result.

This can be particularly useful in cases where more than one worker acts
on a record. For example, an order fulfillment application may need to
check inventory, to ensure that the order is available, and check credit
to be sure that the customer is approved for the order. These checks may
be run in parallel, by different worker processes. With delta publish,
each worker process updates the part of the record that the worker is
responsible for, without affecting any other part of the record. Delta
publish saves the worker from having to query the record and construct a
full update, and eliminates the possibility of incorrect updates when
two workers try to update the record at the same time.

For example, consider an order published to the SOW:

.. code-block:: javascript

    {
        "id":735,
        "customer":"Patrick",
        "item":90123,
        "qty":1000,
        "state":"new"
    }

Using delta publishing, two independent workers can operate on the
record in parallel, safely making updates and preparing the record for a
final fulfillment process.

The inventory worker process is responsible for checking inventory. This
worker subscribes to messages where
the ``/state = 'new' AND /inventory IS NULL AND /credit IS NULL``. This
process receives the new message and verifies that the inventory system
contains 1000 of the item # 90123. When it verifies this, it uses delta
publish to publish the following update:

.. code-block:: javascript

    {
        "id":735,
        "inventory":"available"
    }

The credit worker process verifies that the customer is permitted to
bill for the total amount. Like the inventory worker, this worker
subscribes to messages where the
``/state = 'new' and /inventory IS NULL and /credit IS NULL``. This
process receives the new message and verifies that the customer is
allowed to bill the total value of the order. When the check is
complete, the credit worker publishes this message:

.. code-block:: javascript

    {
        "id":735,
        "credit":"approved"
    }

After both of these processes run, the SOW contains the following
record:

.. code-block:: javascript

    {
        "id": 735,
        "credit": "approved",
        "inventory": "available",
        "customer": "Patrick",
        "item": 90123,
        "qty": 1000,
        "state": "new"
    }

The fulfillment worker would subscribe to messages where
``/state = 'new' AND /inventory IS NOT NULL AND /credit IS NOT NULL``.

Using Delta Publish
^^^^^^^^^^^^^^^^^^^

Because delta messages must be processed and merged into the existing
SOW record, AMPS provides a distinct command for delta publish.

+-----------------------+-------------------------------------------------------+
| Command               | Result                                                |
+=======================+=======================================================+
| ``delta_publish``     | Publish a delta message. If no record exists in the   |
|                       | SOW, add the message to the SOW. If a record exists   |
|                       | in the SOW, merge the data from this record into the  |
|                       | existing record.                                      |
+-----------------------+-------------------------------------------------------+

**Table 11.3:** *Delta publish command*

Understanding Delta Publish
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When AMPS receives a delta publish request, AMPS first performs any preprocessing
or enrichment specified for the incoming message. AMPS then fully parses both
the incoming message and the destination message, and then merges the contents
of the incoming update into the existing message. The mechanism that
merges the messages is message-type independent, and does not rely on
the syntax or semantics of any particular message type.

The message is merged before the message is persisted to the State of the World,
recorded in the transaction log, replicated, or delivered to subscribers.

Because delta publish uses a very simple, compact syntax  -- a partial message
-- for updates, AMPS makes certain assumptions about the intent of an update:

1. AMPS replaces the contents of the existing message based on the
   parsed identifiers in the update. An update to an anonymous element
   or the root level of a subdocument is a *full replacement* of that
   value.

   For example, consider the following JSON document::

       {"id":42, "contents":{"packages":[{"box":"chocolates"},
                                         {"bowl":"noodles"}]}}

   An update to the ``packages`` field of the document *replaces* the
   subdocument, so providing the following ``delta_publish``::

        {"id":42, "contents":{"packages":[{"basket":"eggs"}]}}

   Replaces the previous value of ``packages``, and produces the message::

        {"id":42, "contents":{"packages":[{"basket":"eggs"}]}}

2. A missing field in a delta publish means that the content
   of that field is unchanged.

   Because AMPS treats the absence of a field in a delta update to
   mean that the previous value is unchanged, a ``delta_publish``
   cannot be used to remove fields from a document.

   For example, consider the following JSON document::

      {"id":42", "flowers":"roses"}

   A ``delta_publish`` that attempts to remove the ``flowers`` field
   by simply removing it will be treated as though there is no update
   to that field. In other words, a ``delta_publish`` of::

      {"id":42}

   Will produce no change to the ``flowers`` field, and result in a
   document of::

      {"id":42, "flowers":"roses"}

   To remove a field from a message, republish the full message.

3. AMPS does not evaluate whether a field has a different value than
   the previous message or has a republish of the previous value
   when processing a ``delta_publish`` -- any field present in the
   delta publish will replace the corresponding field in the existing
   message.

   For example, consider an application that advertises whether
   a given system is accepting requests with messages along the
   lines of::

      {"systemId":"A34-Astro", "status":"available",
                               "commandAccepted":["print","scan","fax"]}

   Now, the system sends a delta publish update that does not change the status::

     {"systemId":"A34-Astr", "status":"available"}

   Even though no value in the message has changed, the publish is still
   processed, and the publish will still be delivered to subscribers.
   The fact that it replaces an existing value with the same value
   does not matter for AMPS. This is still considered an update. (An individual
   subscriber can choose not to receive updates that do not change the value of
   a field by using the ``no_empties`` option with a delta subscription.)


Delta Publish Support
~~~~~~~~~~~~~~~~~~~~~~

To accept delta publishes, the message type and the topic must both
support delta publish. When this is not the case, AMPS accepts the
publish, but may not produce the expected results.

All of the basic message types provided with AMPS support delta publish
with the exception of the ``binary`` message type and protobuf message types
that use protobuf version 3. Composite message types support delta publish
if they use the ``composite-local`` definition, as described in the section
on composite message types. Types that do not support delta publish, produce
the full, literal message provided with a delta publish command (rather than
merging the publish into the previous state of the message).

When a topic uses the ``composite-local`` message type, parts of the
composite that are provided as empty (that is, zero-length) are
considered to be unchanged, and the merged message contains the existing
contents of that part. This provides a convenient way to update only one
part of a composite message, without having to republish data that has
not changed. For example, a ``composite-local`` type contains a JSON
part and a binary part can modify the JSON part without having to
republish the full binary part.

AMPS queues support delta publish to an underlying topic, if that
underlying topic maintains a SOW. The merged message is provided to the
AMPS queue.

All other AMPS topic types that are based on a SOW and accept publish
commands support delta publish. AMPS topics that do not use a SOW do not
support delta publish, so publishing a delta message to those topics
produces the full, literal message from the publish command rather than
a merged message. Without a SOW configured for the topic, AMPS does not
track the current value of a message, and therefore does not have a way
to merge the publish into an existing message.


