Bugs and improvements:

- in failing a task, jobs are not in agent:
  <zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``> failed with traceback:
  Failure: zc.async.interfaces.AbortedError: 

  <zc.async.job.Job (oid 33532777, db '') ``zc.async.job.Job (oid 33530083, db '') :fail()``> failed with traceback:
  *--- Failure #7 (pickled) ---
  /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.py:299: _call_with_retry(...)
   [ Locals ]
    res : 'None'
    self : "<zc.async.job.Job (oid 33532777, db '') ``zc.async.job.Job (oid 33530083, db '') :fail()``>"
    tm : '<transaction._manager.ThreadTransactionManager object at 0xb7a5ea4c>'
    call : '<function <lambda> at 0xb686fd4c>'
    ct : '0'
   ( Globals )
    success_or_failure : '<function success_or_failure at 0xb6cb917c>'
    __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.pyc'"
    persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
    zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
    pytz : "<module 'pytz' from '/opt/z4m/eggs/pytz-2008a-py2.4.egg/pytz/__init__.py'>"
    __name__ : "'zc.async.job'"
    datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
    types : "<module 'types' from '/opt/cleanpython24/lib/python2.4/types.pyc'>"
    transaction : "<module 'transaction' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/transaction/__init__.pyc'>"
    completeStartedJobArguments : '<function completeStartedJobArguments at 0xb6cb91b4>'
    twisted : "<module 'twisted' from '/opt/z4m/eggs/zope.app.twisted-3.4.0-py2.4.egg/twisted/__init__.pyc'>"
    _repr : '<function _repr at 0xb6cb9144>'
    Job : "<class 'zc.async.job.Job'>"
    ZODB : "<module 'ZODB' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/ZODB/__init__.pyc'>"
    rwproperty : "<module 'rwproperty' from '/opt/z4m/eggs/rwproperty-1.0-py2.4.egg/rwproperty.pyc'>"
    zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
    BTrees : "<module 'BTrees' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/BTrees/__init__.pyc'>"
    __doc__ : 'None'
  /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.py:291: <lambda>(...)
   [ Locals ]
    self : "<zc.async.job.Job (oid 33532777, db '') ``zc.async.job.Job (oid 33530083, db '') :fail()``>"
    effective_args : '[]'
    effective_kwargs : '{}'
   ( Globals )
    success_or_failure : '<function success_or_failure at 0xb6cb917c>'
    __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.pyc'"
    persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
    zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
    pytz : "<module 'pytz' from '/opt/z4m/eggs/pytz-2008a-py2.4.egg/pytz/__init__.py'>"
    __name__ : "'zc.async.job'"
    datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
    types : "<module 'types' from '/opt/cleanpython24/lib/python2.4/types.pyc'>"
    transaction : "<module 'transaction' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/transaction/__init__.pyc'>"
    completeStartedJobArguments : '<function completeStartedJobArguments at 0xb6cb91b4>'
    twisted : "<module 'twisted' from '/opt/z4m/eggs/zope.app.twisted-3.4.0-py2.4.egg/twisted/__init__.pyc'>"
    _repr : '<function _repr at 0xb6cb9144>'
    Job : "<class 'zc.async.job.Job'>"
    ZODB : "<module 'ZODB' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/ZODB/__init__.pyc'>"
    rwproperty : "<module 'rwproperty' from '/opt/z4m/eggs/rwproperty-1.0-py2.4.egg/rwproperty.pyc'>"
    zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
    BTrees : "<module 'BTrees' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/BTrees/__init__.pyc'>"
    __doc__ : 'None'
  /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.py:362: fail(...)
   [ Locals ]
    self : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
    e : '<zc.async.interfaces.AbortedError instance at 0xb07fa7ac>'
   ( Globals )
    success_or_failure : '<function success_or_failure at 0xb6cb917c>'
    __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.pyc'"
    persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
    zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
    pytz : "<module 'pytz' from '/opt/z4m/eggs/pytz-2008a-py2.4.egg/pytz/__init__.py'>"
    __name__ : "'zc.async.job'"
    datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
    types : "<module 'types' from '/opt/cleanpython24/lib/python2.4/types.pyc'>"
    transaction : "<module 'transaction' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/transaction/__init__.pyc'>"
    completeStartedJobArguments : '<function completeStartedJobArguments at 0xb6cb91b4>'
    twisted : "<module 'twisted' from '/opt/z4m/eggs/zope.app.twisted-3.4.0-py2.4.egg/twisted/__init__.pyc'>"
    _repr : '<function _repr at 0xb6cb9144>'
    Job : "<class 'zc.async.job.Job'>"
    ZODB : "<module 'ZODB' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/ZODB/__init__.pyc'>"
    rwproperty : "<module 'rwproperty' from '/opt/z4m/eggs/rwproperty-1.0-py2.4.egg/rwproperty.pyc'>"
    zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
    BTrees : "<module 'BTrees' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/BTrees/__init__.pyc'>"
    __doc__ : 'None'
  /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.py:396: resumeCallbacks(...)
   [ Locals ]
    callbacks : '[]'
    length : '1'
    tm : '<transaction._manager.ThreadTransactionManager object at 0xb7a5ea4c>'
    self : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
    j : "<zc.async.job.Job (oid 33530088, db '') ``zc.z4m.content.query.catalog.addProcessingTask(zc.async.queue.Queue (oid 33148381, db ''), delay=datetime.timedelta(0, 600))``>"
   ( Globals )
    success_or_failure : '<function success_or_failure at 0xb6cb917c>'
    __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/job.pyc'"
    persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
    zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
    pytz : "<module 'pytz' from '/opt/z4m/eggs/pytz-2008a-py2.4.egg/pytz/__init__.py'>"
    __name__ : "'zc.async.job'"
    datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
    types : "<module 'types' from '/opt/cleanpython24/lib/python2.4/types.pyc'>"
    transaction : "<module 'transaction' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/transaction/__init__.pyc'>"
    completeStartedJobArguments : '<function completeStartedJobArguments at 0xb6cb91b4>'
    twisted : "<module 'twisted' from '/opt/z4m/eggs/zope.app.twisted-3.4.0-py2.4.egg/twisted/__init__.pyc'>"
    _repr : '<function _repr at 0xb6cb9144>'
    Job : "<class 'zc.async.job.Job'>"
    ZODB : "<module 'ZODB' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/ZODB/__init__.pyc'>"
    rwproperty : "<module 'rwproperty' from '/opt/z4m/eggs/rwproperty-1.0-py2.4.egg/rwproperty.pyc'>"
    zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
    BTrees : "<module 'BTrees' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/BTrees/__init__.pyc'>"
    __doc__ : 'None'
  /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.py:77: jobCompleted(...)
   [ Locals ]
    job : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
    self : '<zc.async.agent.Agent object at 0xb07e0d2c>'
   ( Globals )
    __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.pyc'"
    persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
    Agent : "<class 'zc.async.agent.Agent'>"
    datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
    chooseFirst : '<function chooseFirst at 0xb6700d4c>'
    addMainAgentActivationHandler : '<function addMainAgentActivationHandler at 0xb6700dbc>'
    zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
    __name__ : "'zc.async.agent'"
    zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
    __doc__ : 'None'
  /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.py:61: remove(...)
   [ Locals ]
    item : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
    self : '<zc.async.agent.Agent object at 0xb07e0d2c>'
   ( Globals )
    __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.pyc'"
    persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
    Agent : "<class 'zc.async.agent.Agent'>"
    datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
    chooseFirst : '<function chooseFirst at 0xb6700d4c>'
    addMainAgentActivationHandler : '<function addMainAgentActivationHandler at 0xb6700dbc>'
    zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
    __name__ : "'zc.async.agent'"
    zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
    __doc__ : 'None'
  /opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.py:58: index(...)
   [ Locals ]
    i : "<zc.async.job.Job (oid 33532777, db '') ``zc.async.job.Job (oid 33530083, db '') :fail()``>"
    ix : '0'
    self : '<zc.async.agent.Agent object at 0xb07e0d2c>'
    item : "<zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``>"
   ( Globals )
    __file__ : "'/opt/z4m/eggs/zc.async-1.1-py2.4.egg/zc/async/agent.pyc'"
    persistent : "<module 'persistent' from '/opt/z4m/eggs/ZODB3-3.8.0-py2.4-linux-i686.egg/persistent/__init__.pyc'>"
    Agent : "<class 'zc.async.agent.Agent'>"
    datetime : "<module 'datetime' from '/opt/cleanpython24/lib/python2.4/lib-dynload/datetime.so'>"
    chooseFirst : '<function chooseFirst at 0xb6700d4c>'
    addMainAgentActivationHandler : '<function addMainAgentActivationHandler at 0xb6700dbc>'
    zope : "<module 'zope' from '/opt/z4m/eggs/zope.app.xmlrpcintrospection-3.4.0a1-py2.4.egg/zope/__init__.pyc'>"
    __name__ : "'zc.async.agent'"
    zc : "<module 'zc' from '/opt/z4m/eggs/zc.buildout-1.0.0-py2.4.egg/zc/__init__.pyc'>"
    __doc__ : 'None'
  exceptions.ValueError: <zc.async.job.Job (oid 33530083, db '') ``zc.z4m.content.query.catalog.processCatalogQueues()``> not in Agent
  *--- End of Failure #7 ---

- try to make this look less frightening:

    2008-05-08T13:05:02 ERROR zc.async.events UUID ecbad1cc-1a89-11dd-8f17-0015c5e8367a already activated in queue  (oid ???): another process?  To stop poll attempts in this process, set ``zc.async.dispatcher.get().activated = False``.  To stop polls permanently, don't start a zc.async.dispatcher!

- need retry tasks, particularly for callbacks

- need CRITICAL logs for callbacks

For future versions:

- Write the z3monitor tests.
- queues should be pluggable like agent with filter
- show how to broadcast, maybe add conveniences
- show how to use with collapsing jobs (hint to future self: use external queue
  to put in work, and have job(s) just pull what they can see from queue)
- write tips and tricks
  * avoid long transactions if possible.  really avoid long transactions
    involving frequently written objects.  Discuss ramifications and
    strategies.
  * in zope.app.testing.functional tests, zc.async doesn't do well being
    started in a layer's setup because then it is associated with the
    wrapped layer DB, and the test is associated with the DemoStorage wrapper,
    so that the test can see what zc.async does, but zc.async can't see what
    the test does.  The current workaround is to start the dispatcher in the
    test or the test set up (but, again, *not* The layer set up).
  * In tests, don't check to see if poll is activated until after the first
    poll. Try ``zc.async.testing.get_poll(zc.async.dispatcher.get(), 0)``, for
    instance.
  * In tests, be aware that DemoStorage does not support mvcc and does not
    support conflict resolution, so you may experience ConflictError (write and
    particularly read) problems with it that you will not experience as much,
    or at all, with a storage that supports those features such as FileStorage.
    Notice that all of the tests in this package use FileStorage.
  * callbacks should be very, very quick, and very reliable.  If you want to do
    something that might take a while, put another job in the queue
  

For some other package, maybe:

- TTW Management and logging views, as in zasync (see goals in the "History"
  section of the README).
  