# 27. Operation and Deployment

This chapter contains guidelines and best practices to help plan and prepare an environment, to ensure optimal performance and stability for AMPS deployments.

See the Troubleshooting chapter for information on troubleshooting problems with AMPS, including information on using the utilities that come with AMPS.

## Capacity Planning

Sizing an AMPS deployment can be a complicated process that includes many factors, such as: configuration parameters used for AMPS, the data used within the deployment and how the deployment will be used. This section presents guidelines that you can use in sizing your host environment for an AMPS deployment given the following components that need to be taken into account: Memory, Storage, CPU and Network.

Capacity planning is one of the most important aspects of ensuring that an AMPS deployment can meet the needs of the application.

Important

The capacity planning formulas in this section are intended to help you size a system to run an instance of AMPS. The actual resource consumption will vary based on usage and configuration.

### System Goals and Requirements

When planning the capacity for a system, the most important questions to understand are: the purpose of the instance and the Service Level Agreement (SLA) offered by the instance. For example, is this a server for use by a development team for early exploration of ideas, or will this instance be core infrastructure for a major application? Is it important that the instance has the absolute minimum latency possible, or is the most important aspect of the system query response time for a 1TB topic in the SOW?

Since AMPS efficiently uses the system hardware, the limits of an AMPS instance are typically a result of the limitations of the underlying host system. Proper capacity planning (and operating system tuning) can mean the difference between an instance that performs well and handles increased traffic without incident and an instance that constantly pushes the hardware to the limit and becomes less responsive when traffic increases.

#### Single-Tenant or Multi-Tenant

AMPS performs well in both single-tenant and multi-tenant installations. When choosing whether to host more than one AMPS instance on a given system, it is important to plan for the highest level of traffic expected on all instances simultaneously. In a business setting, it is common for a sudden increase in traffic to affect a number of systems in the business, rather than being isolated to just one system. When planning capacity for a multi-tenant system, provision a host that exceeds the total maximum capacity required for all AMPS instances on the system at peak load.

For multi-tenant installations, disable AMPS-level NUMA tuning in the configuration file.

#### Physical Server or Virtual Machines

Although AMPS is designed to be highly adaptive to hardware on which it runs, AMPS does not require a dedicated physical server. AMPS can be successfully deployed on either physical hardware or virtual machines. In either deployment model, 60East recommends tuning Linux for best performance rather than accepting the distribution defaults (which are typically tuned for interactive use rather than for a high performance server).

Typically, installations that require the highest level of performance and lowest levels of latency deploy on physical hardware (with a single AMPS instance per server). Installations that are willing to trade predictable performance for ease and flexibility of deployment often use virtual machines.

When deployed on a virtual machine, disable AMPS-level NUMA tuning in the configuration file.

60East does not recommend over-committing the underlying hardware. When deploying on a virtual machine, it is important to consider the capacity of both the virtual machine itself and the underlying host hardware. In other words, the total memory needed by all virtual machines – with all applications hosted by those machines running at peak traffic simultaneously – should not exceed the physical memory of the hardware. Likewise, the total number of CPUs specified in all of the virtual machines on the host should not exceed the number of CPUs on the host hardware, the network bandwidth needed should not exceed the bandwidth allocated to the host, the traffic to the storage device should not exceed the throughput that the storage device is capable of, and so on. In an enterprise environment, it is not unusual for a wide variety of applications to all see peak loads at the same time, so the system should be provisioned to provide enough capacity that every hosted application can meet peak throughput requirements at the same time.

Develop a plan for monitoring the physical hardware as well as the virtualized host environment. If possible, the monitoring plan should include a method for correlating the activity on the virtual machine to the activity on the physical host (for example, it would be important to be able to correlate CPU saturation on the virtual machine to CPU saturation on the physical host).

60East does not recommend using virtualization systems that dynamically move running virtual machines for load-balancing purposes in an application that requires low latency or predictable response times. Although these systems work well for their intended purpose, a machine migration typically takes an extended period of time (for example, a target maximum time of 1s) to finalize the migration. During that time, the virtual machine (and therefore, AMPS) is temporarily paused. A pause this long is typically orders of magnitude longer than the typical low-latency system can tolerate for service interruption, and is effectively a temporary service outage during the migration.

Although this sort of migration is typically unworkable for load-balancing, the migration is much less downtime than would be required to stop and restart AMPS. These systems could be a good choice for reducing downtime for hardware-level or network-level maintenance.

### Memory

AMPS is designed for high performance. It is designed to use memory, as needed, to improve performance and reduce latency. One of the most important aspects of managing an AMPS instance is, being sure that the instance has enough physical memory available to perform well.

This section contains general guidelines for creating an approximate sizing estimate for the AMPS process itself. An estimate on total memory capacity for a server would include the estimate for the AMPS process itself and estimates for any other processes running on the system (including monitoring software, security software, other applications, update and maintenance tasks, and so on). Notice that it is possible for AMPS to maintain quantities of data much larger than physical memory (for example, terabytes of SOW data). For instances that have this requirement, contact 60East support for tuning and sizing guidance.

#### Estimating AMPS Instance Memory Usage

The best way to estimate the memory usage for an AMPS instance is to simulate, as closely as possible, the traffic and usage pattern for the instance and collect statistics that show the amount of memory that the instance uses.

If actual numbers aren’t available, you can use the formulas in this section to come up with a working approximation of the amount of memory to make available to AMPS for a given amount of data, number of clients, and so on. The AMPS server will use memory as necessary for performance, so the formulas here offer general estimates for system sizing purposes rather than precise predictions.

AMPS needs less than 1GB for its own binary image and initial start up state for most configurations. For production instances, we estimate 5GB as a typical working memory footprint for an active installation.

As a general estimate, because of indexing for queries, AMPS may need up to twice the size of messages stored in a topic in the SOW to fully index that topic (the same sizing applies to messages in views and conflated topics). AMPS maintains a copy of the latest journal file in memory for quick access, and maintains a small amount of metadata for each message in an AMPS queue. The `MessageMemoryLimit` configured for the instance (or the total of all `MessageMemoryLimit` settings for each `Transport` in the instance) specifies the total amount of memory devoted to buffering messages for clients, including conflated subscriptions, aggregated subscriptions, and paginated subscriptions.

This puts a general estimate of the amount of memory to be available for the AMPS server itself at:

    5GB + ``SowSizeEstimate``
        + ( C * 4096 bytes)
        + ( TMemLimit ) + (J * 2) + (Q * 250 bytes) [ + (QA * 20 bytes) ]

Memory capacity estimate equation

where:

> `SowSizeEstimate = Estimates for SOW topic size, as described below (in bytes)`
> `C = Number of Clients`
> `TMemLimit = Total of all MessageMemoryLimit settings in the instance`
> `J = JournalSize setting`
> `Q = Total number of active unacknowledged messages in the queues for the instance`
> `QA = Total number of acknowledgments received for messages that are not yet in the queue`

By default, all unacknowledged messages in the instance will be active in the queue. When a queue specifies a `TargetQueueDepth`, the total number of active unacknowledged messages for the queue will, in most cases, be limited to the `TargetQueueDepth`.

When acknowledgment messages are received for messages that are not currently active in the queue, AMPS must track those acknowledgments to be able to efficiently prevent those messages from entering the queue. Not every application consumption pattern can produce this situation; however, if it arises, this calculation can help estimate the amount of memory required to maintain information about these acknowledgments until the message enters the queue.

To calculate the `SowSizeEstimate`, the memory footprint required for topics, views and conflated topics in the SOW, use the following formula to calculate for each (`Topic`, `View` and `ConflatedTopic`):

    ( 2 * (S + 128 bytes) * M ) + ((16 bytes * M) * H)

Sow Topic memory capacity estimate equation

where:

> `S = Average message size for the Topic, View or ConflatedTopic in the SOW (in bytes)`
> `M = Maximum expected number of messages for the Topic, View or ConflatedTopic`
> `H = Number of hash indexes for the Topic, View or ConflatedTopic`

Estimating topic-by-topic generally gives a more precise estimate. However, if that data is not available, you can also use overall message sizes and message count for the instance.

If more configuration detail is available, it may be possible to create a more precise estimate. For example, if the `SlabSize` configured for a SOW topic is not an exact fit for the message size + header, it is possible to estimate the amount of free space remaining in each slab.

As a simple example, a general estimate of the amount of memory that should be left available to run an instance of AMPS might be:

    5GB + [ ( 2 * (1024+128) * 1,000,000 ) + (16 * (1,000,000 * 2) ) +
            ( 2 * ( 512+128) * 1,000,000 ) + ( 0 ) +
            ( 2 * (1024+128) * 8,000,000 ) + (16 * (8,000,000 * 4) ]
        + ( 200 * 4096) + ( 10,000,000,000) +
          ( 1,000,000,000 * 2 )  + ( 750,000 * 250)

Example memory estimation equation

where:

> For the `SowSizeEstimate`, the instance will have two
> Topics and a View.
> For the first topic:
> `S = 1024`
> `M = 4,750,000`
> `H = 2`
> For a view over the first topic (the view uses no HashIndexes):
> `S = 512`
> `M = 3,000,000`
> `H = 0`
> For the second topic:
> `S = 1024`
> `M = 8,000,000`
> `H = 4`
> For the overall AMPS estimate:
> `C = 200`
> `TMemLimit = 10,000,000,000 (10GB)`
> `J = 1,000,000,000 (1GB)`
> `Q = 750,000`

This shows sizing for an AMPS deployment with the following characteristics:

- Three topics in the SOW (including one view):
  - One topic has a message size of 1024 bytes, will hold 5 million messages and configure 2 hash indexes.
  - One view has a message size of 512 bytes, will hold 3 million messages and does not configure a hash index.
  - One topic has a message size of 1024 bytes, will hold 8 million messages and will configure 4 hash indexes.
- A maximum of 10GB of memory for in-flight messages and working state (for aggregated subscriptions, pagination sets and so on)
- A journal size setting configured to 1GB
- A maximum of 750,000 total unacknowledged messages at a time across all message queues
- No more than 200 clients connected simultaneously
- No external modules loaded

This estimate suggests that no less than 52GB of physical memory on the server should be available for the AMPS instance itself while AMPS is processing the expected volume of messages. When AMPS first starts, or if traffic is light, AMPS may consume less than the estimated amount. AMPS may also consume more than this amount of memory during memory-intensive operations in some cases.

Important

The formula in this section is a general estimate designed to produce a recommended minimum amount of physical memory to have available for AMPS. It is intended as a guideline when actual measurements are not available. For more accurate estimates, use measurements of the expected workload. A given instance of AMPS may not match these estimates at any particular time, based on usage, precise configuration, traffic, client activity, and so forth.

#### Estimating Overall System Capacity

The AMPS instance memory usage is one component of estimating the needs of the overall system. In addition to this, there is also: operating system tasks, management and maintenance (including monitoring, security and management software), and any other applications running on the system must also be taken into consideration.

Further, Linux memory management is most efficient when the operating system has 10-20% headroom.

For best performance and a lower risk of problems related to an unexpected spike in message volume, 60East recommends factoring in all of the components that will consume memory on the system, and then sizing the overall physical memory to handle 200% of the capacity estimated while still retaining 10-20% physical RAM. Note that these are rough guidelines. An especially critical system, or a system that has in the past seen larger volumes might size memory to 350% or more, while a less critical system might allocate less than 200% of the estimated capacity. A VM on a developer desktop might be sized at or below the capacity estimate, since the system is completely under the control of a single user and is not intended to handle production loads.

For example, in the estimate above, the system should reserve a minimum of 52GB of free RAM for the AMPS process itself. Suppose that the monitoring, access control and server management software are very lightweight and only consume 3GB of memory under production load. The following estimates would be reasonable:

- Production server with strict SLA and tolerance for usage variation - 128GB

  This estimate accounts for 200% of the estimated required capacity while still allowing 16GB (between 10 and 20% of the physical memory) free for efficient memory management. (Calculation: 52GB for AMPS, 3GB for the monitoring software = 55GB. Multiply by 2 = 110GB.) For a server that needs to be available during periods of heavy activity, this sizing could be a good option.

- Production server with stable usage or variable SLA - 96GB

  This estimate covers the expected capacity of the AMPS server and monitoring software and leaves enough headroom for the operating system, but does not allow for large growth in volume or changes in usage patterns. For a server with predictable usage and volumes, or a server where some performance impact is acceptable if volume or usage increases and where the general estimates for AMPS capacity are known to be very precise, this server size could be a good option.

- Shared development server with minimal performance SLA - 64GB

  This minimal estimate covers the expected capacity of the AMPS server and monitoring system, but does not leave enough excess capacity for the server to absorb unexpected traffic. This server would be expected to have periodic performance degradation and to possibly exit due to out of memory conditions if the volume of traffic increases or the usage pattern changes. This could be a good sizing for an instance that is used only for development, where the instance is not guaranteed to provide any particular performance guarantees and it is acceptable for the server to be temporarily unavailable if there was an unexpected increase in load.

### Storage

AMPS needs enough space to store its own binary images, configuration files, SOW persistence files, log files, transaction log journals, and slow client offline storage, if any. Not every deployment configures a SOW or transaction log, so the storage requirements are largely driven by the configuration.

#### AMPS Log Files

Log file sizes vary depending on the log level and how the engine is used. For example, in the worst-case, `trace` level logging. AMPS will need at least enough storage for every message published into AMPS and every message sent out of AMPS plus 20%.

For `info` level logging, a good estimate of AMPS log file sizes would be 2MB per 10 million messages published.

Logging space overhead can be capped by implementing a log rotation strategy which uses the same file name for each rotation. This strategy effectively truncates the file when it reaches the log rotation threshold to prevent it from growing larger.

#### SOW Topics

When calculating the amount of storage to reserve for topics in the SOW, there are a couple of factors to keep in mind. The first is the average size of messages stored in the SOW, the number of messages stored in the SOW and the `SlabSize` defined in the configuration file for each `Topic`. Using these values, it is possible to estimate the minimum and maximum storage requirements for the SOW.

A rough estimate of the minimum size for a SOW topic is as follows:

    Min = ( MsgSize * MsgCount ) + ( Cores * SlabSize )

Minimum SOW Size

where:

> `Min = Minimum SOW size`
> `MsgSize = Average SOW message size`
> `MsgCount = Number of SOW messages`
> `Cores = Number of processor cores in the system`
> `SlabSize = Slab Size for the SOW`

A rough estimate of the maximum size for a SOW topic is as follows:

    Max =  ( ( MsgCount / ( ( SlabSize / MsgSize) / 2) ) * SlabSize) + (Cores * SlabSize)

Maximum SOW Size

where:

> `Max = Maximum SOW size`
> `MsgCount = Number of SOW messages`
> `SlabSize = Slab size for the SOW`
> `MsgSize = Average SOW message size`
> `Cores = Number of CPU cores in the system`

The storage requirements should be between the two values above, however it is still possible for the SOW to consume additional storage based on the unused capacity configured for each SOW topic.

Notice that, as suggested in this calculation, AMPS reserves the configured `SlabSize` for each processor core in the system the first time a thread running on that core writes to the SOW.

For example, in an AMPS configuration file with the `SlabSize` set to 1MB, the SOW for this topic will consume 1MB per processor core with no messages stored in the SOW. Pre-allocating SOW capacity in chunks, as a chunk is needed, is more efficient for the operating system and storage devices, and helps amortize the SOW extension costs over more messages.

It is also important to be aware of the maximum message size that AMPS guarantees the SOW can hold. The maximum message size is calculated in the following manner:

    Max = SlabSize - 64 bytes

Maximum Message Size allowed in SOW

where:

> `Max = Maximum message size that can be stored in the SOW`
> `SlabSize = The configured SlabSize for the SOW`

This calculation says that the maximum message size that can be stored in the SOW in a single message is the `SlabSize` minus 64 bytes for the record header information.

#### Transaction Logs

Transaction logs are used for message replay, replication and to ensure consistency in environments where each message is critical. Transaction logs are optional in AMPS (though some features require them), and transaction logs can be configured to record specific topics.

When planning for transaction logs, there are three main considerations:

1. The total size needed for the transaction log, including in disaster recovery scenarios
2. The size to allow for each file that makes up the transaction log
3. How many files to preallocate

You can calculate the approximate total size of the transaction log once the system reaches steady state as follows:

    Capacity = ( S + 512 bytes ) * N  + ( J ) + ( TI )

Transaction Log Sizing Approximation

where:

> `Capacity = Estimated storage capacity required for transaction log`
> `S = Average message size`
> `N = Number of messages to retain`
> `J = Journal file size`
> `TI = Topic Index (if configured), larger of 200MB or (64 * number of messages indexed)`

Size your files to match the aging policy for the transaction log data. To remove data from the transaction log, use AMPS actions to remove the journal files that are no longer needed. You can size your files to make this easier. For example, if your application typically generates 100GB a day of transaction log, you could size your files in 25GB units to make it easier to remove 100GB increments.

AMPS allows you to preallocate files for the transaction log. For applications that are very latency-sensitive, preallocation can help provide consistent latency. We recommend that those applications preallocate files, if storage capacity and retention policy permit. For example, an application that sees heavy throughput during a working day might preallocate enough files so that there is no need for additional allocation within the working day.

Notice that, if your application uses replication, the AMPS transaction log maintenance actions will not delete unreplicated messages that this instance is responsible for replicating. This means that, when calculating the maximum storage space required, the recovery window for a failure is also important. For example, many systems have a policy of not restarting a failed system until a scheduled maintenance window: if one server in a replicated set of servers could, potentially, be offline for up to 8 hours, then the other servers must be able to store a minimum of 8 hours of journals, even in cases where the normal retention period would be shorter.

#### File-Backed Queue Metadata

AMPS can, optionally, persist queue metadata to the filesystem to allow metadata to be paged out of memory and potentially improve recovery time.

For any queue that uses the `FileBackedMetadata` option, the following formula can be used to estimate the storage space that AMPS may use for each queue:

    GREATER OF ( (MaxMsgCount * 250 bytes ) or 4MB )

Queue Metadata Size

where:

> `MaxMsgCount = Maximum number of messages active in the queue`

The maximum number of messages active is the largest number of unacknowledged messages in the queue since the instance was started, as measured by the maximum value of the `queue_depth` metric for the queue. Notice that if the queue also uses the `TargetQueueDepth` option, the active message count will typically be the `TargetQueueDepth` unless AMPS has temporarily expanded this depth to avoid halting queue delivery.

AMPS preallocates files of approximately 4MB for the metadata cache, then grows the file if needed to maintain metadata. The size of the file does not shrink while the instance is running. AMPS may reduce the size of the file during recovery. As with all capacity estimates, this formula is intended to provide a working approximation of the amount of disk space needed, and does not mean that the file will be precisely the size the formula indicates.

#### Choosing Storage Devices

The previous sections discuss the scope of sizing the storage, however scenarios exist where the performance of the storage devices must also be taken into consideration.

In cases where messages are persisted (to the transaction log, to a topic in the SOW, or both), overall throughput of the instance can be limited by the performance of the storage device. It is important that the storage device be able to keep up with the peak rate at which the instance will receive messages.

Different aspects of the AMPS server have different patterns of access to storage, as shown below:

Access patterns for AMPS features
Feature | Storage Usage
--- | ---
State of the World | Random access read/write/update
Transaction Log | Sequential write for recording messages / sequential read for replay, replication and queue message distribution
Error and Event Log | Sequential write only
Statistics Database | Random access read/write/update

For each of these uses, ensure that the underlying system has enough I/O bandwidth to meet the needs of an instance. For example, publishing to a topic that is in the SOW and also recorded in the transaction log will write the message to both the SOW and the transaction log, and may (depending on logging settings) also generate a write to the error and event log.

Consider a case where an instance is recording messages in the transaction log at a high incoming message rate. If performance greater than 50MB/second is required for the AMPS transaction log, experience has demonstrated that flash storage (or better) would be recommended. Magnetic hard disks lack the performance to produce results greater than this with a consistent latency profile.

For applications that require high performance and persist state, 60East recommends separate storage for the SOW, the transaction log and the error and event log where practical.

### CPU

SOW queries with content filtering make heavy use of CPU-based operations and, as such, CPU performance directly impacts the content filtering performance and rates at which AMPS processes messages. The number of cores within a CPU largely determines how quickly SOW queries execute.

AMPS contains optimizations which are only enabled on recent 64-bit x86 CPUs. To achieve the highest level performance, consider deploying on a CPU which includes support for the SSE 4.2 instruction set.

To give an idea of AMPS performance, repeated testing has demonstrated that a moderate query filter with 5 predicates can be executed against 1KB messages at more than 1,000,000 messages per second, per core on an Intel i7 3GHz CPU. This applies to both subscription based content filtering and SOW queries. Actual messaging rates will vary based on matching ratios and network utilization.

### Network

When capacity planning a network for AMPS, the requirements for messaging traffic are largely dependent on the following factors:

- Average message size
- The rate at which publishers will publish messages to AMPS
- The number of publishers and the number of subscribers

AMPS requires sufficient network capacity to service inbound publishing as well as outbound messaging requirements. In most deployments, outbound messaging to subscribers and query clients has the highest bandwidth requirements due to the increased likeliness for a “one to many” relationship of a single published message matching subscriptions/queries for many clients.

Estimating network capacity requires knowledge about several factors, including but not limited to: the average message size published to the AMPS instance, the number of messages published per second, the average expected match ratio per subscription, the number of subscriptions, and the background query load. Once these key metrics are known, then the necessary network capacity can be calculated:

    R * Sz ( 1 + M * Sb ) + Q

Network capacity formula

where:

> `R = Rate`
> `Sz = Average Message Size`
> `M = Match Ratio`
> `Sb = Number of Subscribers`
> `Q = Query Load`

where “Query Load” is defined as:

    Mq * S * Qs

where:

> `Mq = Messages per Query`
> `S = Average Message Size`
> `Qs = Queries per Second`

In a deployment required to process published messages at a rate of 5000 messages per second, with each message having an average message size of 600 bytes, the expected match rate per subscription is 2% (or 0.02) with 100 subscriptions. The deployment is also expected to process 5 queries per 1 minute (or 12 queries per second), with each query expected to return 1000 messages.

    5000 * 600 B * ( 1 + 0.02 * 100 ) + ( 1000 * 600 B * 1/12 ) ~ 9 MB / s ~ 72 Mb / s

Based on these requirements, this deployment would need at least 72Mb/s of network capacity to achieve the desired goals. This analysis demonstrates AMPS by itself would fall into a 100Mb/s class network. It is important to note, this analysis does not examine any other network based activity which may exist on the host and as such, a larger capacity networking infrastructure than 100Mb/s would likely be required.

#### Replication Network Bandwidth

For replication connections, the general recommendation is to estimate bandwidth needs as though each outgoing replication destination is a subscriber that subscribes to all of the replicated topics, and each incoming destination is a publisher that fully publishes the replicated topics. Although AMPS replication connections support compression, the general recommendation is to provision enough network capacity to support the full replication stream, and then to use compression to save capacity.

#### Additional Network Considerations

When calculating the available bandwidth for an instance, it is also important to take into account any other use of the network. For example, if the system uses network attached storage, and the traffic to that storage is not isolated from messaging traffic, bandwidth to the storage device should also be taken into account when planning the network capacity available to the instance.

Likewise, any other process that consumes bandwidth, such as monitoring applications or log collection processes, should be considered when planning the overall bandwidth capacity available.

### NUMA Considerations

AMPS is designed to take advantage of non-uniform memory access (NUMA). For the lowest latency in networking, we recommend that you install your NIC in the slot closest to NUMA node 0. When AMPS NUMA tuning is enabled, AMPS runs critical threads on node 0, so positioning the NIC closest to that node provides the shortest path from processor to NIC.

When a single instance of AMPS is deployed on the system (physical host), as is the case with most critical production systems, 60East recommends leaving AMPS NUMA tuning enabled (this is the default).

If more than one instance of AMPS is running on the same physical host, or if other CPU-intensive processes are running on the same physical host, 60East recommends disabling AMPS NUMA tuning in the AMPS configuration file and relying on the operating system NUMA management. Likewise, if a mechanism is used to restrict AMPS to specific processors, AMPS NUMA tuning should be disabled.

Important

When AMPS is deployed on a virtual machine, 60East recommends disabling the AMPS level NUMA tuning in the configuration file.

## Linux Operating System Configuration

This section covers some settings which are specific to running AMPS on a Linux Operating System.

### ulimit

The `ulimit` command is used by a Linux administrator to get and set user limits on various system resources.

`ulimit -c`

It is common for an AMPS instance to be configured to consume gigabytes of memory for large SOW caches. If a failure were to occur in a large deployment it could take seconds (maybe even hours, depending on storage performance and process size) to dump the core file. AMPS has a minidump reporting mechanism built in that collects information important to debugging an instance before exiting. This minidump is much faster than dumping a core file to disk. For this reason, it is recommended that the per user core file size limit is set to 0 to prevent a large process image from being dumped to storage.

`ulimit -n`

The number of file descriptors allowed for a user running AMPS needs to be at least double the sum of counts for the following: connected clients, SOW topics and pre-allocated journal files.

- Minimum: 4096
- Recommended: 32768, or the value recommended by AMPS in any diagnostic messages, whichever is greater

### Transparent Huge Pages

Transparent huge pages is enabled by default for most linux distributions, which can add significant overhead to memory management. For this reason 60East recommends changing the setting to `madvise`, which requires applications to explicitly request use of transparent huge pages. Previously the recommendation was to use `never`, which is still acceptable, but is less flexible as it disables transparent huge pages for all applications running on the system.

To change the setting until the operating system is rebooted, the following command can be used:

    echo never > /sys/kernel/mm/transparent_hugepage/enabled

To make a permanent change to this setting, add the above command to the startup scripts, or add the `transparent_hugepage=madvise` option to the kernel startup flags (see the documentation for your Linux distribution for details).

Recommended: madvise

### /proc/sys/fs/aio-max-nr

Each AMPS instance requires AIO in the kernel to support at least 16384 plus 8192 for each SOW topic in simultaneous I/O operations. The setting `aio-max-nr` is global to the host and impacts all applications. As such this value needs to be set high enough to service all applications using AIO on the host.

- Minimum: 65536
- Recommended: 1048576

To view the value of this setting, as root you can enter the following command:

    cat /proc/sys/fs/aio-max-nr

To edit this value, as root you can enter the following command:

    sysctl -w fs.aio-max-nr = 1048576

This command will update the value for `/proc/sys/fs/aio-max-nr` and allow 1,048,576 simultaneous I/O operations, but will only do so until the next time the machine is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting:

    fs.aio-max-nr = 1048576

### /proc/sys/fs/file-max

Each AMPS instance needs file descriptors to service connections and maintain file handles for open files. This number needs to be at least double the sum of counts for the following: connected clients, SOW topics and pre-allocated journal files. This file-max setting is global to the host and impacts all applications, so this needs to be set high enough to service all applications on the host.

- Minimum: 262144
- Recommended: 6815744

To view the value of this setting, as root you can enter the following command:

    cat /proc/sys/fs/file-max

To edit this value, as root you can enter the following command:

    sysctl -w fs.file-max = 6815744

This command will update the value for `/proc/sys/fs/file-max` and allow 6,815,744 concurrent files to be opened, but will only do so until the next time the machine is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting:

    fs.file-max = 6815744

### /proc/sys/vm/min_free_kbytes

This parameter sets the minimum amount of memory to keep free in the system. Setting this value properly can help the operating system function more effectively in low-memory situations. If this value is set too low, the operating system can have difficulty reclaiming memory, which can lead to unnecessary out-of-memory events. If this value is set too high, overall system efficiency decreases as the operating system can spend more time than necessary reclaiming memory.

60East recommends setting this parameter to 1% of the physical memory on the system, rounding up to the nearest GB.

Notice that the units of this parameter are in kilobytes. For example, to set this value for a system that has 128GB of memory, you would calculate 1% of the physical memory (1.28 GB), round up to the nearest GB (2 GB) and then allocate 2000000 KB as the min_free_kbytes.

- Minimum: 1000000 (1GB)
- Recommended: 1% of physical memory, rounded up to the nearest GB

To edit this value, as root you can enter the following command:

    sysctl -w vm.min_free_kbytes=2000000

Notice that this tuning recommendation is designed for a server-class machine with a reasonable amount of memory. For a small development machine or blade (for example, a system with less than 32GB of memory), leaving this parameter at the operating system default may be more appropriate.

### /proc/sys/vm/max_map_count

AMPS makes extensive use of memory mapped files, and frequently modifies the maps. The `/proc/sys/vm/max_map_count` parameter sets the maximum number of maps that the Linux kernel will allow for a process. If the number of requested maps exceeds the number of maps in this parameter, memory allocation operations can fail even when there is sufficient memory available.

This setting is global to the host and applies to all applications, so this needs to be set high enough for the most map-intensive application on the host.

- Minimum: 65530
- Recommended: 500000

To edit this value, as root you can enter the following command:

    sysctl -w vm.max_map_count=500000

This command will update the value for `/proc/sys/vm/max_map_count` and allow 500,000 maps to be created, but will only do so until the next time the machine is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting:

    vm.max_map_count=500000

### /proc/sys/vm/swappiness

AMPS performs best when the data that it needs to retain is in memory. If the operating system needs to use swap because the system requires more memory than is available, performance degrades substantially.

60East recommends that, for systems that host performance-critical instances of AMPS, the `vm.swappiness` setting is set to `1`. This will minimize swapping on this system, which will improve performance with the tradeoff of making it more likely for processes to be killed by the operating system in low-memory situations.

This setting is global to the host and applies to all applications.

- Recommended: 1

To edit this value, as root you can enter the following command:

    sysctl -w vm.swappiness=1

This command will update the value for `/proc/sys/vm/swappiness` and direct the operating system to avoid using swap space until the system is under severe memory pressure.

Using the command above will change the swappiness setting until the operating system is rebooted.To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting:

    vm.swappiness=1

### /proc/sys/net/ipv4/tcp_frto

This option controls whether Forward RTO-Recovery (FRTO) is enabled for the TCP network. Enabling FRTO can be beneficial for overall network performance if a system is sending packets over wireless networks with substantial interference (for example, public WiFi in an urban area). However, this recovery algorithm can reduce performance in wired networks. While this option is enabled on most current Linux distributions by default, disabling the option can improve network performance.

60East recommends disabling this option unless the server is directly delivering traffic over a congested WiFi network.

This setting is global to the host and applies to all applications.

- Recommended: 0

To edit this value, as root you can enter the following command:

    sysctl -w net.ipv4.tcp_frto=0

This command will update the value for `/proc/sys/net/ipv4/tcp_frto` and direct the operating system to disable FRTO.

Using the command above will change the setting until the operating system is rebooted. To make a permanent change to this setting, as a root user, edit the `/etc/sysctl.conf` file and either edit or append the following setting:

    net.ipv4.tcp_frto=0

## Upgrading an AMPS Installation

This chapter describes how to upgrade an existing installation of AMPS. The steps presented here focus on upgrading the installation itself, and should be the only steps you need for upgrades that change the HOTFIX version number or the PREVIEW version number (as described in AMPS Version Number Components).

For changes that update the MAJOR or MINOR version number, AMPS may add features, change file or network formats, or change behavior. For these upgrades, you may need to make changes to the AMPS configuration file or update applications to adapt to new features or changes in behavior.

60East recommends maintaining a test environment that you can use to test upgrades, particularly when an upgrade changes MAJOR or MINOR versions and you are taking advantage of new features or changed behavior.

When the AMPS instance participates in replication, you must coordinate the instance upgrades when upgrading across AMPS versions.

AMPS supports replication to and from versions 5.2.0.0 and later for the purposes of rolling upgrade. For long-term deployment, 60East recommends that all AMPS instances that replicate to each other have the same MAJOR and MINOR version number, and preferably run the same release of AMPS.

### Upgrade Steps

Upgrading an AMPS installation involves the following steps:

1. Stop the running instance
2. Install the new AMPS binaries
3. If you are upgrading from an AMPS version prior to 5.0.0.0, upgrade any data files or configuration files that you want to retain
4. If necessary, update the configuration file for the instance
5. If necessary, update any applications that will use new features
6. Restart the service

AMPS supports replication from version 5.2.0.0 and later to this version of AMPS for the purposes of rolling upgrade with no (or minimal) downtime. 60East recommends that production installations of AMPS have the same MAJOR and MINOR version number at a minimum, and preferably run identical versions of AMPS.

### Upgrading AMPS Data Files

AMPS may change the format and content of data files when upgrading across versions, as specified by the MAJOR and MINOR version number. This most commonly occurs when new features are added to AMPS that require different or additional information in the persisted files. The HISTORY file for the AMPS release lists when changes have been made that require data file changes.

When upgrading to AMPS 5.0 or later, from a release prior to 5.0, you must upgrade the data files. For versions of AMPS 5.0 and later, backward compatibility is maintained and therefore there have been no changes to the data file formats.

The AMPS distribution includes the `amps_upgrade` utility to process and upgrade data files. Unless you are upgrading from a version of AMPS prior to 5.0, there is no need to use this utility when upgrading AMPS.

### Downgrading AMPS Data Files

The contents of AMPS data files, including SOW topic files, transaction log journals, and the statistics database are not guaranteed to be backward compatible for versions that change the major, minor, or preview version numbers.

Even when the file format is compatible, newer versions of AMPS may include new options or use metadata in a way that older versions of AMPS are not aware of. Downgrading an instance of AMPS while preserving data files may produce unexpected or incorrect behavior.

## Best Practices

This section covers a selection of best practices for deploying AMPS.

### Monitoring

AMPS exposes the statistics available for monitoring via a RESTful interface, known as the Monitoring Interface, which is configured as the administration port. This interface allows developers and administrators to easily inspect various aspects of AMPS performance and resource consumption using standard monitoring tools.

At times, AMPS will emit log messages notifying that a thread has encountered a deadlock or stressful operation. These messages will repeat with the word “stuck” in them. AMPS will attempt to resolve these issues, however after 60 seconds of a single thread being stuck, AMPS will automatically emit a minidump to the previously configured minidump directory. This minidump can be used by 60East support to assist in troubleshooting the location of the stuck thread or the stressful process.

Monitor the contents of `dmesg` on the instance for errors that affect the AMPS process. For example, if the operating system runs low on memory and begins shutting down processes, this information will be recorded in `dmesg`. Likewise, system events such as hardware failures that can affect AMPS are most likely to be recorded in the `dmesg` output.

Another area to examine when monitoring AMPS is the `last_active` monitor for the processors. This can be found in the `/amps/instance/processors/all/last_active` url in the monitoring interface. If the `last_active` value continually increases for more than one minute and there is a noticeable decline in the quality of service, then it may be best to fail-over and restart the AMPS instance.

### Logging

60East recommends that an instance of AMPS used for production log at `info` level (at a minimum). This provides a basic record of the operations requested in AMPS, and is the minimum level of logging needed to troubleshoot most issues. Further, a production instance should have the capacity available to log at a more verbose level if necessary, for troubleshooting and diagnostic purposes.

An instance used for development or UAT purposes should typically log at `trace` level so that the interaction between an application and AMPS is captured.

60East also recommends capturing stdout and stderr for the AMPS process. This can provide information about operating system or runtime errors in the event that a problem occurs outside of the control of AMPS (and, therefore, cannot be recorded in the AMPS event log).

### Stopping AMPS

To stop AMPS, ensure that AMPS runs the `amps-action-do-shutdown` action. By default, this action is run when AMPS receives `SIGHUP`, `SIGINT`, or `SIGTERM`. However, you can also configure an action to shut down AMPS in response to other conditions. For example, if your company policy is to reboot servers every Saturday night, and AMPS is not running as a system service (or daemon), you could schedule an AMPS shutdown every Saturday before the system reboot.

When AMPS is installed to run as a system service (or daemon), AMPS installs shutdown scripts that will cleanly stop AMPS during a system shutdown or reboot.

### SOW Parameters

Choosing the ideal `SlabSize` for your SOW topic is a balance between the frequency of SOW expansion and storage space efficiency. A large `SlabSize` will preallocate space for records when AMPS begins writing to the SOW.

If detailed tuning is not necessary, 60East recommends leaving the `SlabSize` at the default size if your messages are smaller than the default `SlabSize`. If your messages are larger than the default SlabSize, a good starting point for the `SlabSize` is to set it to several times the maximum message size you expect to store in the SOW.

There are three considerations when setting the optimum `SlabSize`:

1. Frequency of allocations
2. Overall size of the SOW
3. Efficient use of space

A `SlabSize` that is small results in frequent extensions of your SOW topic to occur. These frequent extensions can reduce throughput in a heavily loaded system, and in extreme cases can exhaust the kernel limit on the number of regions that a process can map. Increasing the `SlabSize` will reduce the number of allocations.

When the `SlabSize` is large, then the risk of the SOW resize affecting performance is reduced. Since each slab is larger, however, there will be more space consumed if you are only storing a small number of messages: this cost will amortize as the number of messages in the SOW exceeds the number of cores in the system * the number of messages that fit into a slab.

To most efficiently use space, set a `SlabSize` that minimizes the amount of unused space in a slab. For example, if your message sizes are average 512 bytes but can reach a maximum of 1.2 MB, one approach would be to set a `SlabSize` of 2.5MB to hold approximately 5 average-sized messages and two of the larger-sized messages. Looking at the actual distribution of message sizes in the SOW (which can be done with the `amps_sow_dump` utility) can help you determine how best to size slabs for maximum space efficiency.

For optimizing the `SlabSize`, determine how important each aspect of SOW tuning is for your application, and adjust the configuration to balance allocation frequency, overall SOW size, and space to meet the needs of your application.

Given AMPS is highly-parallelized, AMPS operates more efficiently when it is able to run tasks in parallel. When considering options for SlabSize, be sure that the value you choose will result in a number of slabs that is at least equal to the number of cores in the system. A SlabSize setting that results in only a few slabs could cause reduced query performance. For example, a system with a single publisher and a SlabSize large enough to hold all of the records produced by that publisher, doesn’t allow a query to be parallelized since all of the records will be in a single slab.

### Slow Clients

As described in Slow Client Management, AMPS provides capacity limits for slow clients to reduce the memory resources consumed by slow clients. This section discusses tuning slow client handling to achieve your availability goals.

#### Slow Client Offlining for Large Result Sets

The default settings for AMPS work well in a wide variety of applications with minimal tuning.

If you have particularly large SOW topics and your application is disconnecting clients due to exceeding the offlining threshold when the clients are retrieving large SOW query result sets, 60East recommends the following settings as a baseline for further tuning:

Client Offline Settings for Large Result Sets
Parameter | Recommendation
--- | ---
`MessageMemoryLimit` | This controls the maximum memory consumed by AMPS for client messages. You can increase this parameter to allow AMPS to use more memory for records. Notice, however, that memory devoted to client messages is unavailable for other purposes. Recommended starting point for tuning large result sets: 10% of the system memory (for example, on a server with 128GB of memory, start with a 13GB limit). 60East recommends tuning the `MessageDiskLimit` first. If necessary, increase this parameter by 1-2% at a time. Use caution with settings over 20%: devoting large amounts of memory to client messages may cause swapping and reduce, rather than increase, overall performance.
`MessageDiskLimit` | The maximum amount of space to consume for offline messages. Recommended starting point for tuning large result sets: average record size * number of expected records * number of simultaneous clients, or `MessageMemoryLimit`, whichever is greater.
`MessageDiskPath` | The path in which to store offline message files. 60East recommends that the message disk path be hosted on fast, high-capacity storage such as a PCIe-attached flash drive. The available storage capacity of the disk must be greater than the configured `MessageDiskLimit`. Pay attention to the performance characteristics of the device: for example, some devices suffer reduced performance when they run low on free space, so for those devices you would want to make sure that there is space available on the device even when AMPS is close to the `MessageDiskLimit`.

60East recommends that you use these settings as a baseline for further tuning, bearing in mind the needs and expected messaging patterns of your application.

#### WAN Traffic and Slow Client Settings

In some installations, a single AMPS instance will serve both applications that are local to the instance and applications that retrieve data over a higher-latency network. For example, applications in a small regional office may use a server in another region over a WAN.

In these situations, consider either adjusting the slow client settings so that those clients can complete operations such as large SOW queries successfully, or consider creating a separate transport with higher capacity settings that will be used only by the small number of clients that require these settings due to network limitations. In particular, if you set a `ClientMessageAgeLimit` for an instance or transport, ensure that this limit is large enough that the network can consume the results of the SOW queries that clients are expected to make within the allotted time.

### Minidump

AMPS includes the ability to generate a minidump file, which can be used by 60East support, to help troubleshoot a problematic instance.

The minidump captures thread state information: a snapshot of where in the source code each thread is, the call stack for each thread, and the register information for each frame of the call stack. A minidump also contains basic information about the system that AMPS was running on, such as the processor type and number of sockets. Minidumps do not contain other internal state of AMPS or the contents of application memory. Minidumps do not contain detailed information about the host system, and have no information about the state of the host or operating system. Instead, minidumps identify the point of failure to help 60East quickly narrow down the issue without generating large files or potentially compromising sensitive data.

Minidumps can be produced much faster than a standard core dump, and use significantly less space since the minidump contains only a small subset of the information a core dump would contain (see ulimit for more configuration options). Because minidumps are relatively inexpensive, the AMPS server may produce minidumps for temporary conditions that the server subsequently recovers from. AMPS also allows creation of a minidump on demand.

Generation of a minidump file occurs in the following ways:

1. When AMPS detects a crash internally, a minidump file will automatically be generated. This includes cases where an AMPS thread or critical internal component has not reported progress for an extended period of time (typically 300 seconds).
2. When a user clicks on the `minidump` link in the `amps/instance/administrator` link from the administrator console (see the AMPS Monitoring Reference for more information).
3. By sending the running AMPS process the `SIGQUIT` signal.
4. In response to a configured action.
5. If a thread fails to report progress with the AMPS thread monitor for approximately 60 seconds, a minidump will automatically be generated. This should be sent to AMPS support for evaluation along with a description of the operations taking place at the time (typically, info level or more verbose logging).

By default the minidump is configured to write to `/tmp`, but this can be changed in the AMPS configuration by modifying the `MiniDumpDirectory`. 60East recommends monitoring the minidump directory.

If minidumps occur, contact 60East support for diagnosis and troubleshooting. Bear in mind that minidumps are often a symptom of a slowdown in the server due to resource constraints rather than an indication that the server has exited.

Once a minidump is submitted to 60East (and acknowledged as received), there is no further need to retain that minidump. 60East recommends removing minidumps when they are no longer needed.

### Deployment and Upgrade Plan

60East offers a deployment checklist for use when planning or upgrading an installation of AMPS. The checklist covers recommendations for operations considerations such as:

- Capacity Planning
- Operating System Configuration
- AMPS Configuration
- Developing and Configuring Maintenance Plans
- Creating a Monitoring Strategy
- Creating a Patching and Upgrade Plan
- Creating a Support Plan and Verifying the Support Process

The checklist may not cover all aspects of deployment in a particular environment, but can be used to create a checklist and deployment plan for your environment.

## Accessing AMPS Through a Proxy

For some installations, it is important to be able to configure access to an AMPS instance through a proxy server.

AMPS client connections and AMPS replication connections are TCP connections that use a custom protocol to communicate. Any proxy that does not alter the content of packets will work as expected for both client connections and replication connections.

Using a proxy with AMPS is most straightforward when there is a one-to-one relationship between a port on the proxy and a port on the AMPS instance. In many cases, though, it is useful to have a single port on the proxy and then to route to a different backend instance of AMPS.

### Websocket Connections

Current versions of the AMPS javascript client support the ability to insert arbitrary paths between the hostname and port and the protocol and message type components of the AMPS connection URI.

For example, a URI of `ws://proxyhost:8080/amps-prod-system/amps/json` could be used by the proxy to route to the system designated `amps-prod-system`.

### Galvanometer Connections

The Galvanometer relies on the path provided to determine which resource to return. If it’s necessary to proxy multiple systems behind a single port, it is typically most useful to have the proxy rewrite the URI when forwarding the request. For example, the proxy could rewrite a request for `http://proxyhost:8085/prod-system/` to `http://amps-prod-system:8085/`.

For example, an NGINX proxy configuration could be implemented along the lines of:

    server {
        listen 8085 default;
        listen [::]:8085;

      server_name AMPS;

      # server context
      location /amps-prod-system/ {
          proxy_set_header X-Forwarded-Host $host;
          proxy_set_header X-Forwarded-Server $host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_pass http://amps-prod-system:8085/; #hostname and port of AMPS host for first AMPS instance
      }
      location /amps-dev-system/ {
          proxy_set_header X-Forwarded-Host $host;
          proxy_set_header X-Forwarded-Server $host;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_pass http://amps-dev-system:8085/; #hostname and port of AMPS host for second AMPS instance
      }
    }

Notice that Galvanometer may not be able to provide information on replicated instances when Galvanometer is accessed through a proxy. Galvanometer relies on the hostnames provided by the AMPS configuration to locate replicated instances and retrieve information from their admin interfaces. If Galvanometer does not have access to the replicated instances at the hostname or IP provided in the AMPS configuration file, it will not be able to display information about replicated instances. This does not indicate an issue with replication on those instances, it just means that the Galvanometer view (which is constructed by Galvanometer in the browser) cannot provide the information.

### Load-Balancing Considerations

Some proxy systems are also intended to implement load-balancing. The approach that a given installation uses to determine how to distribute connections depends on the resource that the load-balancing system is managing, and the needs of the application. For example, an application where all subscribers produce roughly the same amount of traffic could use a simple round-robin load balancing system to distribute network load. On the other hand, a proxy for an application where subscribers execute complex aggregated subscriptions might monitor CPU load or memory consumption on a set of servers that host AMPS and attempt to route a new connection to the server with the lowest current load.

Regardless of the approach used, it is important to keep in mind that, for a publisher or queue consumer, the same considerations for the proxy apply as would apply for specifying failover equivalents to a client application directly. In particular, a publisher or a subscriber that processes a queue should not fail over across a replication connection that uses asynchronous acknowledgment (or a connection that has been downgraded to be asynchronous). If this happens, there is a possibility of message loss and/or inconsistent transaction log contents. See the discussion of synchronous and asynchronous acknowledgment in Sync vs Async Acknowledgment Types for more details.