Metadata-Version: 2.1
Name: taskgraph
Version: 0.11.1
Summary: Parallel task graph framework
Home-page: https://github.com/natcap/taskgraph
Maintainer: Natural Capital Project Software Team
Maintainer-email: jamesdouglassusa@gmail.com
License: In this license, "Natural Capital Project" is defined as the parties of
        Stanford University, The Nature Conservancy, World Wildlife Fund Inc.,
        and University of Minnesota.
        
        This tool has an open license. All people are invited to use the tool
        under the following conditions and terms:
        
        Copyright (c) 2020, Natural Capital Project
        
        All rights reserved.
        
        Redistribution and use in source and binary forms, with or without
        modification, are permitted provided that the following conditions are
        met:
        
         * Redistributions of source code must retain the above copyright
           notice, this list of conditions and the following disclaimer.
        
         * Redistributions in binary form must reproduce the above copyright
           notice, this list of conditions and the following disclaimer in the
           documentation and/or other materials provided with the
           distribution.
        
         * Neither the name of Natural Capital Project nor the names of
           its contributors may be used to endorse or promote products derived
           from this software without specific prior written permission.
        
        THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
        "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
        LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
        A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
        OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
        SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
        LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
        DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
        THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
        (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
        OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
        
Keywords: parallel,multiprocessing,distributed,computing
Classifier: Intended Audience :: Developers
Classifier: Topic :: System :: Distributed Computing
Classifier: Development Status :: 5 - Production/Stable
Classifier: Natural Language :: English
Classifier: Operating System :: MacOS :: MacOS X
Classifier: Operating System :: Microsoft
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: License :: OSI Approved :: BSD License
Requires-Python: >=3.6
Description-Content-Type: text/x-rst
License-File: LICENSE.txt
Requires-Dist: retrying >=1.3.0
Provides-Extra: niced_processes
Requires-Dist: psutil ; extra == 'niced_processes'

===============
About TaskGraph
===============

``TaskGraph`` is a library that was developed to help manage complicated
computational software pipelines consisting of long running individual tasks.
Many of these tasks could be executed in parallel, almost all of them wrote
results to disk, and many times results could be reused from part of the
pipeline. TaskGraph manages all of this for you. With it you can schedule
tasks with dependencies, avoid recomputing results that have already been
computed, and allot multiple CPU cores to execute tasks in parallel if
desired.

TaskGraph Dependencies
----------------------

Task Graph is written in pure Python, but if the ``psutils`` package is
installed the distributed multiprocessing processes will be ``nice``\d.

Example Use
-----------

Install ``TaskGraph`` with

``pip install taskgraph``

Then

.. code-block:: python

  import os
  import pickle
  import logging

  import taskgraph

  logging.basicConfig(level=logging.DEBUG)

  def _create_list_on_disk(value, length, target_path):
      """Create a numpy array on disk filled with value of `size`."""
      target_list = [value] * length
      pickle.dump(target_list, open(target_path, 'wb'))


  def _sum_lists_from_disk(list_a_path, list_b_path, target_path):
      """Read two lists, add them and save result."""
      list_a = pickle.load(open(list_a_path, 'rb'))
      list_b = pickle.load(open(list_b_path, 'rb'))
      target_list = []
      for a, b in zip(list_a, list_b):
          target_list.append(a+b)
      pickle.dump(target_list, open(target_path, 'wb'))

  # create a taskgraph that uses 4 multiprocessing subprocesses when possible
  if __name__ == '__main__':
      workspace_dir = 'workspace'
      task_graph = taskgraph.TaskGraph(workspace_dir, 4)
      target_a_path = os.path.join(workspace_dir, 'a.dat')
      target_b_path = os.path.join(workspace_dir, 'b.dat')
      result_path = os.path.join(workspace_dir, 'result.dat')
      result_2_path = os.path.join(workspace_dir, 'result2.dat')
      value_a = 5
      value_b = 10
      list_len = 10
      task_a = task_graph.add_task(
          func=_create_list_on_disk,
          args=(value_a, list_len, target_a_path),
          target_path_list=[target_a_path])
      task_b = task_graph.add_task(
          func=_create_list_on_disk,
          args=(value_b, list_len, target_b_path),
          target_path_list=[target_b_path])
      sum_task = task_graph.add_task(
          func=_sum_lists_from_disk,
          args=(target_a_path, target_b_path, result_path),
          target_path_list=[result_path],
          dependent_task_list=[task_a, task_b])

      task_graph.close()
      task_graph.join()

      # expect that result is a list `list_len` long with `value_a+value_b` in it
      result = pickle.load(open(result_path, 'rb'))


Caveats
-------

* Taskgraph's default method of checking whether a file has changed
  (``hash_algorithm='sizetimestamp'``) uses the filesystem's modification
  timestamp, interpreted in integer nanoseconds.  This check is only as
  accurate as the filesystem's timestamp.  For example:

  * FAT and FAT32 timestamps have a 2-second modification timestamp resolution
  * exFAT has a 10 millisecond timestamp resolution
  * NTFS has a 100 nanosecond timestamp resolution
  * HFS+ has a 1 second timestamp resolution
  * APFS has a 1 nanosecond timestamp resolution
  * ext3 has a 1 second timestamp resolution
  * ext4 has a 1 nanosecond timestamp resolution

  If you suspect timestamp resolution to be an issue on your filesystem, you
  may wish to store your files on a filesystem with more accurate timestamps or
  else consider using a different ``hash_algorithm``.


Running Tests
-------------

Taskgraph includes a ``tox`` configuration for automating builds across
multiple python versions and whether ``psutil`` is installed.  To execute all
tests on all platforms, run:

    $ tox

Alternatively, if you're only trying to run tests on a single configuration
(say, python 3.7 without ``psutil``), you'd run::

    $ tox -e py37

Or if you'd like to run the tests for the combination of Python 3.7 with
``psutil``, you'd run::

    $ tox -e py37-psutil

If you don't have multiple python installations already available on your system,
an easy way to accomplish this is to use ``tox-conda``
(https://github.com/tox-dev/tox-conda) which will use conda environments to manage
the versions of python available::

    $ pip install tox-conda
    $ tox
