# 12. Conflated Topics

AMPS provides the ability for the server to conflate updates to a SOW topic by defining a conflated topic for that SOW topic.

A conflated topic will retain messages published to the underlying topic for a certain period of time, and provide the latest update for each distinct message in the underlying topic at the end of that period of time. In effect, AMPS guarantees that a subscriber will receive no more than one update for a given message per conflation interval.

A conflated topic provides a way to reduce the bandwidth and processing for subscribers in cases where the subscriber needs periodic updates with the current state of the message rather than a rapid set of updates with each individual change to the message.

For example, an application that presents a user interface to display rapidly changing data often uses conflation, since the value of a record may change more rapidly than the user interface is able to be refreshed. For this application, creating a conflated topic and then having the application subscribe to that topic can reduce network traffic, reduce processing load, and provide a more responsive user interface than if the application were to subscribe to the underlying topic and try to process every update.

AMPS provides the ability to conflate messages for an individual subscription, as described in Conflated Subscriptions. When a single subscriber requires conflation, requesting conflation for that subscription is a reasonable approach to take. In cases where all instances of an application can benefit from conflation, conflated topics are a more efficient and scalable approach. A conflated topic is a copy of one SOW topic into another with the ability to control the update interval. In this case, AMPS maintains conflation for the entire topic. There is no need for subscribers to independently request conflation, and AMPS does not need to spend resources processing conflation for each subscriber individually.

The underlying topic for a conflated topic can be a `Topic`, a `View`, or another `ConflatedTopic`.

To better see the value in a conflated topic, imagine a SOW topic called `ORDER_STATE` exists in an AMPS instance. `ORDER_STATE` messages are published frequently to the topic. Meanwhile, there are several subscribing clients that are watching updates to this topic and displaying the latest state in a GUI front-end.

If this GUI front-end only needs updates in five second intervals from the `ORDER_STATE` topic, then more frequent updates would be wasteful of network and client-side processing resources. To reduce network congestion, a conflated topic for the `ORDER_STATE` topic can be created which will contain a copy of `ORDER_STATE` updated in five second intervals. Only the changed records from `ORDER_STATE` will be copied to the conflated topic and then sent to the subscribing clients. Those records with multiple updates within the time interval will have their latest updated values copied to the conflated topic, and only those conflated values are sent to the clients. This results in substantial savings in bandwidth for records with high update rates. This can also result in substantial savings in processing overhead for a client.

AMPS treats the conflated topic as a conflated version of the underlying topic. Applications cannot publish directly to the conflated topic. Likewise, AMPS does not recalculate the SOW key for messages delivered from the conflated topic: these messages have the same SOW key as the corresponding message in the underlying topic.

AMPS indexes conflated topics in the same way that it indexes the underlying topic in the SOW. When a query uses a given field, AMPS will automatically create a memo index for that field. A configuration can also declare one or more `HashIndex` indexes for a conflated topic.

AMPS uses the following conflation strategy:

- When a message arrives for a given key, if no message is already pending for that key, begin the conflation interval.
- If an update arrives during the conflation interval for a given key, replace the pending message with the update for a subscription, or merge the update into the pending message for a delta subscription. All of the metadata on the message is fully replaced.
- If the update is an `oof` notification, and the key has not previously been delivered to the subscription, note that the message should not be delivered.
- At the end of the conflation interval, deliver the current version of the message unless the current message is an `oof` that should not be delivered. Remove the message from the set of messages being conflated.

This means that, when conflation is used, the first update for a given key will arrive after the conflation interval. Further, the application must expect that any given update may be delayed by up to the conflation interval.

## SOW/ConflatedTopic

AMPS provides the ability to create ongoing snapshots of a SOW topic, called conflated topics (also called topic replicas in previous releases of AMPS). Conflated topics are updated on an interval, and store a snapshot of the current state of the world at each interval. This helps to manage bandwidth to clients that do not act on each update, such as a client UI that refreshes every second rather than with every update.

For compatibility with previous AMPS versions, AMPS allows you to use `ReplicaDefinition` as a synonym for `ConflatedTopic`.

Element | Description
--- | ---
Name | String used to define the name of the conflated topic. While AMPS doesn’t enforce naming conventions, it can be convenient to name the conflated topic based on the underlying topic name. For example, if the underlying topic is `orders`, it can be convenient to name the conflated topic `orders-C`. If no `Name` is provided, AMPS accepts `Topic` as a synonym for `Name` to provide compatibility with versions of AMPS previous to 5.0.
UnderlyingTopic | String used to define the SOW topic which provides updates to the conflated topic. This must exactly match the name of a SOW topic. When the underlying topic is a regular expression topic, this must match the `Name` of the topic rather than the `Pattern` of the topic. In this case, the conflated topic will conflate all of the topics in the underlying regular expression topic into individual topics. The names of the individual topics are set using the `TopicFormat` element in this case.
MessageType | The message format of the underlying topic. This `MessageType` must be the `MessageType` of the provided `UnderlyingTopic`.
`Interval` | The frequency at which AMPS updates the data in the conflated topic. Default: `5s` Minimum: `100ms`
`Filter` | Content filter that is applied to the underlying topic. Only messages that match the content filter are stored in the conflated topic.
`HashIndex` | AMPS provides the ability to do fast lookup for SOW records based on specific fields. When one or more `HashIndex` elements are provided, AMPS creates a hash index for the fields specified in the element. These indexes are created on startup, and are kept up to date as records are added, removed, and updated. The `HashIndex` element contains a `Key` element for each field in the hash index. AMPS uses a hash index when a query uses exact matching for all of the fields in the index. AMPS does not use hash indexes for range queries or regular expressions. AMPS automatically creates a hash index for the set of fields specified in the set of `Key` fields for the `UnderlyingTopic`, if that topic provides `Key` fields.

SOW/ConflatedTopic Parameters

If the underlying topic of a `ConflatedTopic` is a regular expression topic, an additional configuration parameter is required to specify the topic name that AMPS will use for each conflated topic.

Element | Description
--- | ---
TopicFormat | String used to format the name of the conflated topics when the underlying topic is a regular expression topic. When the underlying topic is a regular expression topic, AMPS will create a conflated topic for each topic within the regular expression topic. This element specifies how AMPS will construct the name of the conflated topics produced. The `TopicFormat` must contain the string `%n`, which is replaced with the name of the topic in the underlying regular expression topic. For example, if the `TopicFormat` is `%n-C`, when the underlying regular expression topic contains a topic named `/my/orders`, AMPS will produce a conflated topic with the name `/my/orders-C`. There is no default for this element. This element cannot contain characters that are significant for regular expressions (such as `.`, `^`, and so on).

SOW/ConflatedTopic Parameters when the UnderlyingTopic is a regular expression topic

    <ConflatedTopic>
        <Topic>FastPublishTopic-C</Topic>
        <MessageType>nvfix</MessageType>
        <UnderlyingTopic>FastPublishTopic</UnderlyingTopic>
        <Interval>5s</Interval>
        <Filter>/region = 'A'</Filter>
    </ConflatedTopic>
    <ConflatedTopic>
        <Topic>LongIntervalTopic-C</Topic>
        <MessageType>json</MessageType>
        <UnderlyingTopic>FastPublishTopic</UnderlyingTopic>
        <Interval>120s</Interval>
        <HashIndex>
           <Key>/order/status</Key>
        </HashIndex>
    </ConflatedTopic>
    <ConflatedTopic>
       <Topic>ConflateUnderlyingRegex</Topic>
       <MessageType>bflat</MessageType>
       <UnderlyingTopic>TheRegexTopic</UnderlyingTopic>
       <Interval>20s</Interval>
    </ConflatedTopic>