Metadata-Version: 2.1
Name: iterlib
Version: 1.1.6
Summary: Parallel and concurrent iterators
Home-page: https://github.com/rohits2/iterlib
Author: Rohit Singh
Author-email: singhrohit2@hotmail.com
License: BSD
Description: # iterlib
        
        A library for easy iterator-based concurrency and parallelism.
        
        | Build Status |
        |--------------|
        |[![CircleCI](https://circleci.com/gh/rohits2/iterlib.svg?style=svg)](https://circleci.com/gh/rohits2/iterlib)|
        
        ## What is this?
        
        Have you ever been working with `map` or a generator and gotten annoyed with how slow lazy evaluation made some tasks?
        Have you ever wondered "could I run this generator in the background?"
        
        This library exists as the answer to that question.
        It implements asynchronous preloading generators and parallel `map`.
        Both the preloaders and the parallel `map` implementations support `multiprocessing` and `threading` as backends.
        
        ## How do I use it?
        ### Preload
        Here's a simple example of preloading a generator:
        
        ```python3
        from iterlib import thread_preload
        
        gen = (x**2 for x in range(100000))
        preloaded_gen = thread_preload(gen, buffer_size=100)
        ```
        
        That's it! The generator will now preload up to 100 items in the background.  When you call `next(preloaded_gen)`, either directly or indirectly through a `for` statement, it will return values from the preloaded queue.
        
        ### Parallel Map
        Preloading generators has a significant limitation: it's impossible to use more than one background executor because access to iterators requires synchronization.  However, most generators tend to be `map`s over other iterators, which opens an opportunity.  We can't parallelize reads from an iterator, but we can parallelize function calls.
        
        Use one of `threaded_map` or `process_map` when you know your generator is a map:
        
        ```python3
        from iterlib import thread_map
        
        gen = [x for x in range(100000)]
        mapped_gen = thread_map(lambda x: x**2, buffer_size=100, num_workers=4)
        ```
        
        This will create an `ItemizedMap` named `mapped_gen`.  When you call `iter(mapped_gen)`, a generator will be created in the background that will preload up to 100 samples per worker (so 400 total in this example).
        
        > **Careful**: These functions has different semantics than the regular Python `map`!  If you `map` over an indexable collection (like a list or numpy array) the returned `ItemizedMap` will also be an indexable collection that lazily evaluates the `map` for each element you access!  Only when `iter` is called (in a for loop or directly) will it return an asynchronous generator.
        
        
Keywords: iterators,generators,threading,multiprocessing,map
Platform: UNKNOWN
Classifier: Programming Language :: Python :: 3
Description-Content-Type: text/markdown
Provides-Extra: test
Provides-Extra: dev
