Prev Up Next

# processing package reference

The `processing` package mostly replicates the API of the `threading` module.

# Classes and exceptions

> class `Process(group=None, target=None, name=None, args=(), kwargs={})`

>   An analogue of `threading.Thread`.

>   See Process objects.

> exception `BufferTooShort`

>   Exception raised by the `recvBytesInto()` method of a connection object when the supplied buffer object is too small for the message read.

>   If `e` is an instance of `BufferTooShort` then `e.args[0]` will give the message as a byte string.

# Pipes and Queues

When using multiple processes one generally uses message passing for communication between processes and avoids having to use any synchronization primitives like locks.

For passing messages one can use a pipe (for a connection between two processes) or a queue (which allows multiple producers and consumers).

Note that one can also create a shared queue by using a manager object -- see Managers.

For an example of the usage of queues for interprocess communication see ex_workers.py.

> `Pipe(duplex=True)`

>   Returns a pair `(conn1, conn2)` of connection objects representing the ends of a pipe.

>   If `duplex` is true then the pipe is two way; otherwise `conn1` can only be used for receiving messages and `conn2` can only be used for sending messages.

>   See Connection objects.

> `Queue(maxsize=0)`

>   Returns a process shared queue object. The usual `Empty` and `Full` exceptions from the standard library's `Queue` module are raised to signal timeouts.

>   See Queue objects.

# Synchronization primitives

Generally synchronization primitives are not as necessary in a multiprocess program as they are in a mulithreaded program. See the documentation for the standard library's `threading` module.

Note that one can also create synchronization primitves by using a manager object -- see Managers.

> `BoundedSemaphore(value=1)`

>   Returns a bounded semaphore object: a clone of `threading.BoundedSemaphore`.

>   (On Mac OSX this is indistiguishable from `Semaphore()` because `sem_getvalue()` is not implemented on that platform).

> `Condition(lock=None)`

>   Returns a condition variable: a clone of `threading.Condition`.

>   If `lock` is specified then it should be a `Lock` or `RLock` object from `processing`.

> `Event()`
>   Returns an event object: a clone of `threading.Event`.
> `Lock()`
>   Returns a non-recursive lock object: a clone of `threading.Lock`.
> `RLock()`
>   Returns a recursive lock object: a clone of `threading.RLock`.
> `Semaphore(value=1)`
>   Returns a bounded semaphore object: a clone of `threading.Semaphore`.

Acquiring with a timeout

The `acquire()` method of `BoundedSemaphore`, `Lock`, `RLock` and `Semaphore` has a timeout parameter not supported by the equivalents in `threading`. The signature is `acquire(block=True, timeout=None)` with keyword parameters being acceptable. If `block` is true and `timeout` is not `None` then it specifies a timeout in seconds. If `block` is false then `timeout` is ignored.

Interrupting the main thread

If the SIGINT signal generated by Ctrl-C arrives while the main thread is blocked by a call to `BoundedSemaphore.acquire()`, `Lock.acquire()`, `RLock.acquire()`, `Semaphore.acquire()`, `Condition.acquire()` or `Condition.wait()` then the call will be immediately interrupted and `KeyboardInterrupt` will be raised.

This differs from the behaviour of `threading` where SIGINT will be ignored while the equivalent blocking calls are in progress.

# Shared Objects

It is possible to create shared objects using shared memory which can be inherited by child processes.

> `Value(typecode_or_type, *args, **, lock=True)`

>   Returns a ctypes object allocated from shared memory. By default the return value is actually a synchronized wrapper for the object.

>   `typecode_or_type` determines the type of the returned object: it is either a ctypes type or a one character typecode of the kind used by the `array` module. `*args` is passed on to the constructor for the type.

>   If `lock` is true (the default) then a new lock object is created to synchronize access to the value. If `lock` is a `Lock` or `RLock` object then that will be used to synchronize access to the value. If `lock` is false then access to the returned object will not be automatically protected by a lock, so it will not necessarily be "process-safe".

>   Note that `lock` is a keyword only argument.

> `Array(typecode_or_type, size_or_initializer, **, lock=True)`

>   Returns a ctypes array allocated from shared memory. By default the return value is actually a synchronized wrapper for the array.

>   `typecode_or_type` determines the type of the elements of the returned array: it is either a ctypes type or a one character typecode of the kind used by the `array` module. If `size_or_initializer` is an integer then it determines the length of the array, and the array will be initially zeroed. Otherwise `size_or_initializer` is a sequence which is used to initialize the array and whose length determines the length of the array.

>   If `lock` is true (the default) then a new lock object is created to synchronize access to the value. If `lock` is a `Lock` or `RLock` object then that will be used to synchronize access to the value. If `lock` is false then access to the returned object will not be automatically protected by a lock, so it will not necessarily be "process-safe".

>   Note that `lock` is a keyword only argument.

>   Note that an array of `ctypes.c_char` has `value` and `rawvalue` attributes which allow one to use it to store and retrieve strings -- see the documentation for ctypes in the standard library.

See also sharedctypes.

# Managers

Managers provide a way to create data which can be shared between different processes.

> `Manager()`

>   Returns a started `SyncManager` object which can be used for sharing objects between processes. The returned manager object corresponds to a spawned child process and has methods which will create shared objects and return corresponding proxies.

>   The methods for creating shared objects are

>   > `list()`, `dict()`, `Namespace()`, `Value()`, `Array()`, `Lock()`, `RLock()`, `Semaphore()`, `BoundedSemaphore()`, `Condition()`, `Event()`, `Queue()`.

>   See SyncManager.

It is possible to create managers which support other types -- see Customized managers.

# Process Pools

One can create a pool of processes which will carry out tasks submitted to it.

> `Pool(processes=None, initializer=None, initargs=())`

>   Returns a process pool object which controls a pool of worker processes to which jobs can be submitted.

>   It supports asynchronous results with timeouts and callbacks and has a parallel map implementation.

>   `processes` is the number of worker processes to use. If `processes` is `None` then the number returned by `cpuCount()` is used. If `initializer` is not `None` then each worker process will call `initializer(*initargs)` when it starts.

>   See Pool objects.

# Logging

Some support for logging is available. Note, however, that the `logging` package does not use process shared locks so it is possible (depending on the handler type) for messages from different processes to get mixed up.

> `enableLogging(level, HandlerType=None, handlerArgs=(), format=None)`

>   Enables logging and sets the debug level used by the package's logger to `level`. See documentation for the `logging` module in the standard library.

>   If `HandlerType` is specified then a handler is created using `HandlerType(*handlerArgs)` and this will be used by the logger -- any previous handlers will be discarded. If `format` is specified then this will be used for the handler; otherwise `format` defaults to `'[%(levelname)s/%(processName)s] %(message)s'`. (The logger used by `processing` allows use of the non-standard `'%(processName)s'` format.)

>   If `HandlerType` is not specified and the logger has no handlers then a default one is created which prints to `sys.stderr`.

>   Note: on Windows a child process does not directly inherit its parent's logger; instead it will automatically call `enableLogging()` with the same arguments which were used when its parent process last called `enableLogging()` (if it ever did).

> `getLogger()`
>   Returns the logger used by `processing`. If `enableLogging()` has not yet been called then `None` is returned.

Below is an example session with logging turned on:

    >>> import processing, logging
    >>> processing.enableLogging(level=logging.INFO)
    >>> processing.getLogger().warning('doomed')
    [WARNING/MainProcess] doomed
    >>> m = processing.Manager()
    [INFO/SyncManager-1] child process calling self.run()
    [INFO/SyncManager-1] manager bound to '\\\\.\\pipe\\pyc-2776-0-lj0tfa'
    >>> del m
    [INFO/MainProcess] sending shutdown message to manager
    [INFO/SyncManager-1] manager received shutdown message
    [INFO/SyncManager-1] manager exiting with exitcode 0

# Miscellaneous

> `activeChildren()`

>   Return list of all live children of the current process.

>   Calling this has the side affect of "joining" any processes which have already finished.

> `cpuCount()`
>   Returns the number of CPUs in the system. May raise `NotImplementedError`.
> `currentProcess()`

>   An analogue of `threading.currentThread()`.

>   Returns the object corresponding to the current process.

> `freezeSupport()`

>   Adds support for when a program which uses the `processing` package has been frozen to produce a Windows executable. (Has been tested with `py2exe`, `PyInstaller` and `cx_Freeze`.)

>   One needs to call this function straight after the `if __name__ == '__main__'` line of the main module. For example

>       from processing import Process, freezeSupport

>       def f():
>           print 'hello world!'

>       if __name__ == '__main__':
>           freezeSupport()
>           Process(target=f).start()

>   If the `freezeSupport()` line is missed out then trying to run the frozen executable will raise `RuntimeError`.

>   If the module is being run normally by the python interpreter then `freezeSupport()` has no effect.

Note

- The `processing.dummy` package replicates the API of `processing` but is no more than a wrapper around the `threading` module.
- `processing` contains no analogues of `activeCount`, `enumerate`, `settrace`, `setprofile`, `Timer`, or `local` from the `threading` module.

# Subsections

- Process objects
- Queue objects
- Connection objects
- Manager objects
- Proxy objects
- Pool objects
- Shared ctypes object
- Listeners and Clients