:orphan:

====================
Inventory Management
====================

.. default-domain:: mongodb

.. contents:: On this page
   :local:
   :backlinks: none
   :depth: 1
   :class: singlecol

Overview
--------

This case study provides an overview of practices and patterns for
designing and developing the inventory management portions of an
E-commerce application.

.. seealso:: ":doc:`/use-cases/product-catalog`."

Problem
~~~~~~~

Customers in e-commerce stores regularly add and remove items from
their "shopping cart," change quantities multiple times, abandon the
cart at any point, and sometimes have problems during and after
checkout that require a hold or canceled order. These activities make
it difficult to maintain inventory systems and counts and ensure that
customers cannot "buy" items that are unavailable while they shop in
your store.

Solution
~~~~~~~~

This solution keeps the traditional metaphor of the shopping cart, but
the shopping cart will *age*. After a shopping cart has been inactive
for a certain period of time, all items in the cart re-enter the
available inventory and the cart is empty. The state transition
diagram for a shopping cart is below:

.. figure:: /figures/use-cases-inventory1.png
   :align: center
   :alt: The state transition diagram for a shopping cart is below.

Schema
~~~~~~

Inventory collections must maintain counts of the current available
inventory of each stock-keeping unit (SKU; or item) as well as a list
of items in carts that may return to the available inventory if they
are in a shopping cart that times out. In the following example, the
``_id`` field stores the SKU:

.. code-block:: javascript

   {
       _id: '00e8da9b',
       qty: 16,
       carted: [
           { qty: 1, cart_id: 42,
             timestamp: ISODate("2012-03-09T20:55:36Z"), },
           { qty: 2, cart_id: 43,
             timestamp: ISODate("2012-03-09T21:55:36Z"), },
       ]
   }

.. note::

   These examples use a simplified schema.  In a production
   implementation, you may choose to merge this schema with the
   product catalog schema described in the
   ":doc:`/use-cases/product-catalog`" document.

The SKU above has 16 items in stock, 1 item a cart, and 2 items in a
second cart.  This leaves a total of 19 unsold items of merchandise.

To model the shopping cart objects, you need to maintain ``sku``,
``quantity``, fields embedded in a shopping cart :term:`document`:

.. code-block:: javascript

   {
       _id: 42,
       last_modified: ISODate("2012-03-09T20:55:36Z"),
       status: 'active',
       items: [
           { sku: '00e8da9b', qty: 1, item_details: {...} },
           { sku: '0ab42f88', qty: 4, item_details: {...} }
       ]
   }

.. note::

   The ``item_details`` field in each line item allows your
   application to display the cart contents to the user without
   requiring a second query to fetch details from the catalog
   collection.

Operations
----------

This section introduces operations that you may use to support an
e-commerce site. All examples in this document use the Python
programming language and the :api:`PyMongo <python/current>`
:term:`driver` for MongoDB, but you can implement this system using
any language you choose.

Add an Item to a Shopping Cart
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Moving an item from the available inventory to a cart is a fundamental
requirement for a shopping cart system. The most important requirement
is to ensure that your application will never move an unavailable item
from the inventory to the cart.

Ensure that inventory is only updated if there is sufficient inventory
to satisfy the request with the following ``add_item_to_cart``
function operation.

.. code-block:: python

   def add_item_to_cart(cart_id, sku, qty, details):
       now = datetime.utcnow()

       # Make sure the cart is still active and add the line item
       result = db.cart.update(
           {'_id': cart_id, 'status': 'active' },
           { '$set': { 'last_modified': now },
             '$push': {
                 'items': {'sku': sku, 'qty':qty, 'details': details } } },
           w=1)
       if not result['updatedExisting']:
           raise CartInactive()

       # Update the inventory
       result = db.inventory.update(
           {'_id':sku, 'qty': {'$gte': qty}},
           {'$inc': {'qty': -qty},
            '$push': {
                'carted': { 'qty': qty, 'cart_id':cart_id,
                            'timestamp': now } } },
           w=1)
       if not result['updatedExisting']:
           # Roll back our cart update
           db.cart.update(
               {'_id': cart_id },
               { '$pull': { 'items': {'sku': sku } } })
           raise InadequateInventory()

.. admonition:: The system does not trust that the available inventory can satisfy a request

   First this operation checks to make sure that the cart is
   "active" before adding an item. Then, it verifies that the available
   inventory to satisfy the request before decrementing inventory.

   If there is not adequate inventory, the system removes the cart
   update: by specifying ``w=1`` and checking the result allows
   the application to report an error if the cart is inactive or
   available quantity is insufficient to satisfy the request.

.. note::

   This operation requires no :term:`indexes <index>` beyond the
   default index on the ``_id`` field.

Modifying the Quantity in the Cart
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The following process underlies adjusting the quantity of items in a
users cart. The application must ensure that when a user increases the
quantity of an item, in addition to updating the ``carted`` entry for
the user's cart, that the inventory exists to cover the modification.

.. code-block:: python

   def update_quantity(cart_id, sku, old_qty, new_qty):
       now = datetime.utcnow()
       delta_qty = new_qty - old_qty

       # Make sure the cart is still active and add the line item
       result = db.cart.update(
           {'_id': cart_id, 'status': 'active', 'items.sku': sku },
           {'$set': {
                'last_modified': now,
                'items.$.qty': new_qty },
           },
           w=1)
       if not result['updatedExisting']:
           raise CartInactive()

       # Update the inventory
       result = db.inventory.update(
           {'_id':sku,
            'carted.cart_id': cart_id,
            'qty': {'$gte': delta_qty} },
           {'$inc': {'qty': -delta_qty },
            '$set': { 'carted.$.qty': new_qty, 'timestamp': now } },
           w=1)
       if not result['updatedExisting']:
           # Roll back our cart update
           db.cart.update(
               {'_id': cart_id, 'items.sku': sku },
               {'$set': { 'items.$.qty': old_qty } })
           raise InadequateInventory()

.. note::

   That the positional operator ``$`` updates the particular
   ``carted`` entry and item that matched the query.

   This allows the application to update the inventory and keep track
   of the data needed to "rollback" the cart in a single atomic
   operation. The code also ensures that the cart is active.

.. note::

   This operation requires no :term:`indexes <index>` beyond the
   default index on the ``_id`` field.

Checking Out
~~~~~~~~~~~~

The checkout operation must: validate the method of payment and remove
the ``carted`` items after the transaction succeeds. Consider the
following procedure:

.. code-block:: python

   def checkout(cart_id):
       now = datetime.utcnow()

       # Make sure the cart is still active and set to 'pending'. Also
       #     fetch the cart details so we can calculate the checkout price
       cart = db.cart.find_and_modify(
           {'_id': cart_id, 'status': 'active' },
           update={'$set': { 'status': 'pending','last_modified': now } } )
       if cart is None:
           raise CartInactive()

       # Validate payment details; collect payment
       try:
           collect_payment(cart)
           db.cart.update(
               {'_id': cart_id },
               {'$set': { 'status': 'complete' } } )
           db.inventory.update(
               {'carted.cart_id': cart_id},
               {'$pull': {'cart_id': cart_id} },
               multi=True)
       except:
           db.cart.update(
               {'_id': cart_id },
               {'$set': { 'status': 'active' } } )
           raise

Begin by "locking" the cart by setting its status to "pending" Then
the system will verify that the cart is still active and collect
payment data. Then, the :dbcommand:`findAndModify` :term:`command
<database command>` makes it possible to update the cart atomically
and return its details to capture payment information. Then:

- If the payment is successful, then the application will remove the
  ``carted`` items from the inventory documents and set the cart to
  ``complete``.

- If payment is unsuccessful, the application will unlock the cart by
  setting its status to ``active`` and report a payment error.

.. note::

   This operation requires no :term:`indexes <index>` beyond the
   default index on the ``_id`` field.

Returning Inventory from Timed-Out Carts
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Process
```````

Periodically, your application must "expire" inactive carts and return
their items to available inventory. In the example that follows the
variable ``timeout`` controls the length of time before a cart
expires:

.. code-block:: python

   def expire_carts(timeout):
       now = datetime.utcnow()
       threshold = now - timedelta(seconds=timeout)

       # Lock and find all the expiring carts
       db.cart.update(
           {'status': 'active', 'last_modified': { '$lt': threshold } },
           {'$set': { 'status': 'expiring' } },
           multi=True )

       # Actually expire each cart
       for cart in db.cart.find({'status': 'expiring'}):

           # Return all line items to inventory
           for item in cart['items']:
               db.inventory.update(
                   { '_id': item['sku'],
                     'carted.cart_id': cart['id'],
                     'carted.qty': item['qty']
                   },
                   {'$inc': { 'qty': item['qty'] },
                    '$pull': { 'carted': { 'cart_id': cart['id'] } } })

           db.cart.update(
               {'_id': cart['id'] },
               {'$set': { 'status': 'expired' })

This procedure:

#. finds all carts that are older than the ``threshold`` and are due
   for expiration.

#. for each "expiring" cart, return all items to the available
   inventory.

#. once the items return to the available inventory, set the
   ``status`` field to ``expired``.

Indexing
````````

To support returning inventory from timed-out cart, create an index to
support queries on
their ``status`` and ``last_modified`` fields. Use the following
operations in the Python/PyMongo shell:

.. code-block:: python

   db.cart.ensure_index([('status', 1), ('last_modified', 1)])

.. note

   The order of the fields in the index definition. In order to
   support range queries such as :operator:`$lt`, field with the range
   must be the last item in the index.

   Also, there is no need to create a single-field index on the
   ``status`` field: any queries for ``status`` can use this compound
   index.

Error Handling
~~~~~~~~~~~~~~

The above operations do not account for one possible failure
situation: if an exception occurs after updating the shopping cart but
before updating the inventory collection. This would result in a
shopping cart that may be absent or expired but items have not
returned to available inventory.

To account for this case, your application will need a periodic
cleanup operation that finds inventory items that have ``carted``
items and check that to ensure that they exist in a user's cart, and
return them to available inventory if they do not.

.. code-block:: python

   def cleanup_inventory(timeout):
       now = datetime.utcnow()
       threshold = now - timedelta(seconds=timeout)

       # Find all the expiring carted items
       for item in db.inventory.find(
           {'carted.timestamp': {'$lt': threshold }}):

           # Find all the carted items that matched
           carted = dict(
                     (carted_item['cart_id'], carted_item)
                     for carted_item in item['carted']
                     if carted_item['timestamp'] < threshold)

           # First Pass: Find any carts that are active and refresh the carted items
           for cart in db.cart.find(
               { '_id': {'$in': carted.keys() },
               'status':'active'}):
               cart = carted[cart['_id']]

               db.inventory.update(
                   { '_id': item['_id'],
                     'carted.cart_id': cart['_id'] },
                   { '$set': {'carted.$.timestamp': now } })
               del carted[cart['_id']]

           # Second Pass: All the carted items left in the dict need to now be
           #    returned to inventory
           for cart_id, carted_item in carted.items():
               db.inventory.update(
                   { '_id': item['_id'],
                     'carted.cart_id': cart_id,
                     'carted.qty': carted_item['qty'] },
                   { '$inc': { 'qty': carted_item['qty'] },
                     '$pull': { 'carted': { 'cart_id': cart_id } } })

To summarize: This operation finds all "carted" items that have time
stamps older than the threshold. Then, the process makes two passes
over these items:

#. Of the items with time stamps older than the threshold, if the cart
   is still active, it resets the time stamp to maintain the carts.

#. Of the stale items that remain in inactive carts, the operation
   returns these items to the inventory.

.. note::

   The function above is safe for use because it checks to ensure that
   the cart has expired before returning items from the cart to
   inventory. However, it could be long-running and slow other updates
   and queries.

   Use judiciously.

Sharding
--------

If you need to :term:`shard` the data for this system, the ``_id``
field is an ideal :term:`shard key` for both carts and products
because most update operations use the ``_id`` field. This allows
:program:`mongos` to route all updates that select on ``_id`` to a
single :program:`mongod` process.

There are two drawbacks for using ``_id`` as a shard key:

- If the cart collection's ``_id`` is an incrementing value, all new
  carts end up on a single shard.

  You can mitigate this effect by choosing a random value upon the
  creation of a cart, such as a hash (i.e. MD5 or SHA-1) of an
  ObjectID, as the ``_id``. The process for this operation would
  resemble the following:

  .. code-block:: python

     import hashlib
     import bson

     cart_id = bson.ObjectId()
     cart_id_hash = hashlib.md5(str(cart_id)).hexdigest()

     cart = { "_id": cart_id, "cart_hash": cart_id_hash }
     db.cart.insert(cart)

- Cart expiration and inventory adjustment requires update operations
  and queries to broadcast to all shards when using ``_id`` as a shard
  key.

  This may be less relevant as the expiration functions run relatively
  infrequently and you can queue them or artificially slow them down
  (as with judicious use of :py:meth:`sleep() <python:time.sleep()>`)
  to minimize server load.

Use the following commands in the Python/PyMongo console to shard the
cart and inventory collections:

.. code-block:: pycon

   >>> db.command('shardCollection', 'inventory'
   ...            'key': { '_id': 1 } )
   { "collectionsharded" : "inventory", "ok" : 1 }
   >>> db.command('shardCollection', 'cart')
   ...            'key': { '_id': 1 } )
   { "collectionsharded" : "cart", "ok" : 1 }

Additional Resources
--------------------

`Retail Reference Architecture Part 1 (Blog Post) <http://www.mongodb.com/blog/post/retail-reference-architecture-part-1-building-flexible-searchable-low-latency-product?jmp=docs>`_
