Metadata-Version: 2.1
Name: thoth-pybench
Version: 0.0.5
Summary: Adopted PyBench tool to benchmark Python interpreter by the Thoth team.
Home-page: https://openbenchmarking.org/test/pts/pybench
Author: Marc-Andre Lemburg
Author-email: mal@lemburg.com
Maintainer: Francesco Murdaca
Maintainer-email: fmurdaca@redhat.com
License: UNKNOWN
Download-URL: https://github.com/thoth-station/thoth-pybench
Description: This repository contains an adopted version of "PyBench"
        from https://openbenchmarking.org/test/pts/pybench.
        
        Additional changes have been made by the Thoth team to make this great tool
        work on Python3+. This package is also available on PyPI:
        
          https://pypi.org/project/thoth-pybench
        
        ________________________________________________________________________
        
        PYBENCH - A Python Benchmark Suite
        ________________________________________________________________________
        
             Extendable suite of of low-level benchmarks for measuring
                  the performance of the Python implementation 
                         (interpreter, compiler or VM).
        
        pybench is a collection of tests that provides a standardized way to
        measure the performance of Python implementations. It takes a very
        close look at different aspects of Python programs and let's you
        decide which factors are more important to you than others, rather
        than wrapping everything up in one number, like the other performance
        tests do (e.g. pystone which is included in the Python Standard
        Library).
        
        pybench has been used in the past by several Python developers to
        track down performance bottlenecks or to demonstrate the impact of
        optimizations and new features in Python.
        
        The command line interface for pybench is the file pybench.py. Run
        this script with option '--help' to get a listing of the possible
        options. Without options, pybench will simply execute the benchmark
        and then print out a report to stdout.
        
        
        Micro-Manual
        ------------
        
        Run 'pybench.py -h' to see the help screen.  Run 'pybench.py' to run
        the benchmark suite using default settings and 'pybench.py -f <file>'
        to have it store the results in a file too.
        
        It is usually a good idea to run pybench.py multiple times to see
        whether the environment, timers and benchmark run-times are suitable
        for doing benchmark tests. 
        
        You can use the comparison feature of pybench.py ('pybench.py -c
        <file>') to check how well the system behaves in comparison to a
        reference run. 
        
        If the differences are well below 10% for each test, then you have a
        system that is good for doing benchmark testings.  Of you get random
        differences of more than 10% or significant differences between the
        values for minimum and average time, then you likely have some
        background processes running which cause the readings to become
        inconsistent. Examples include: web-browsers, email clients, RSS
        readers, music players, backup programs, etc.
        
        If you are only interested in a few tests of the whole suite, you can
        use the filtering option, e.g. 'pybench.py -t string' will only
        run/show the tests that have 'string' in their name.
        
        This is the current output of pybench.py --help:
        
        """
        ------------------------------------------------------------------------
        PYBENCH - a benchmark test suite for Python interpreters/compilers.
        ------------------------------------------------------------------------
        
        Synopsis:
         pybench.py [option] files...
        
        Options and default settings:
          -n arg           number of rounds (10)
          -f arg           save benchmark to file arg ()
          -c arg           compare benchmark with the one in file arg ()
          -s arg           show benchmark in file arg, then exit ()
          -w arg           set warp factor to arg (10)
          -t arg           run only tests with names matching arg ()
          -C arg           set the number of calibration runs to arg (20)
          -d               hide noise in comparisons (0)
          -v               verbose output (not recommended) (0)
          --with-gc        enable garbage collection (0)
          --with-syscheck  use default sys check interval (0)
          --timer arg      use given timer (time.time)
          -h               show this help text
          --help           show this help text
          --debug          enable debugging
          --copyright      show copyright
          --examples       show examples of usage
        
        Version:
         2.0
        
        The normal operation is to run the suite and display the
        results. Use -f to save them for later reuse or comparisons.
        
        Available timers:
        
           time.time
           time.clock
           systimes.processtime
        
        Examples:
        
        python2.1 pybench.py -f p21.pybench
        python2.5 pybench.py -f p25.pybench
        python pybench.py -s p25.pybench -c p21.pybench
        """
        
        License
        -------
        
        See LICENSE file.
        
        
        Sample output
        -------------
        
        """
        -------------------------------------------------------------------------------
        PYBENCH 2.0
        -------------------------------------------------------------------------------
        * using Python 2.4.2
        * disabled garbage collection
        * system check interval set to maximum: 2147483647
        * using timer: time.time
        
        Calibrating tests. Please wait...
        
        Running 10 round(s) of the suite at warp factor 10:
        
        * Round 1 done in 6.388 seconds.
        * Round 2 done in 6.485 seconds.
        * Round 3 done in 6.786 seconds.
        ...
        * Round 10 done in 6.546 seconds.
        
        -------------------------------------------------------------------------------
        Benchmark: 2006-06-12 12:09:25
        -------------------------------------------------------------------------------
        
            Rounds: 10
            Warp:   10
            Timer:  time.time
        
            Machine Details:
               Platform ID:  Linux-2.6.8-24.19-default-x86_64-with-SuSE-9.2-x86-64
               Processor:    x86_64
        
            Python:
               Executable:   /usr/local/bin/python
               Version:      2.4.2
               Compiler:     GCC 3.3.4 (pre 3.3.5 20040809)
               Bits:         64bit
               Build:        Oct  1 2005 15:24:35 (#1)
               Unicode:      UCS2
        
        
        Test                             minimum  average  operation  overhead
        -------------------------------------------------------------------------------
                  BuiltinFunctionCalls:    126ms    145ms    0.28us    0.274ms
                   BuiltinMethodLookup:    124ms    130ms    0.12us    0.316ms
                         CompareFloats:    109ms    110ms    0.09us    0.361ms
                 CompareFloatsIntegers:    100ms    104ms    0.12us    0.271ms
                       CompareIntegers:    137ms    138ms    0.08us    0.542ms
                CompareInternedStrings:    124ms    127ms    0.08us    1.367ms
                          CompareLongs:    100ms    104ms    0.10us    0.316ms
                        CompareStrings:    111ms    115ms    0.12us    0.929ms
                        CompareUnicode:    108ms    128ms    0.17us    0.693ms
                         ConcatStrings:    142ms    155ms    0.31us    0.562ms
                         ConcatUnicode:    119ms    127ms    0.42us    0.384ms
                       CreateInstances:    123ms    128ms    1.14us    0.367ms
                    CreateNewInstances:    121ms    126ms    1.49us    0.335ms
               CreateStringsWithConcat:    130ms    135ms    0.14us    0.916ms
               CreateUnicodeWithConcat:    130ms    135ms    0.34us    0.361ms
                          DictCreation:    108ms    109ms    0.27us    0.361ms
                     DictWithFloatKeys:    149ms    153ms    0.17us    0.678ms
                   DictWithIntegerKeys:    124ms    126ms    0.11us    0.915ms
                    DictWithStringKeys:    114ms    117ms    0.10us    0.905ms
                              ForLoops:    110ms    111ms    4.46us    0.063ms
                            IfThenElse:    118ms    119ms    0.09us    0.685ms
                           ListSlicing:    116ms    120ms    8.59us    0.103ms
                        NestedForLoops:    125ms    137ms    0.09us    0.019ms
                  NormalClassAttribute:    124ms    136ms    0.11us    0.457ms
               NormalInstanceAttribute:    110ms    117ms    0.10us    0.454ms
                   PythonFunctionCalls:    107ms    113ms    0.34us    0.271ms
                     PythonMethodCalls:    140ms    149ms    0.66us    0.141ms
                             Recursion:    156ms    166ms    3.32us    0.452ms
                          SecondImport:    112ms    118ms    1.18us    0.180ms
                   SecondPackageImport:    118ms    127ms    1.27us    0.180ms
                 SecondSubmoduleImport:    140ms    151ms    1.51us    0.180ms
               SimpleComplexArithmetic:    128ms    139ms    0.16us    0.361ms
                SimpleDictManipulation:    134ms    136ms    0.11us    0.452ms
                 SimpleFloatArithmetic:    110ms    113ms    0.09us    0.571ms
              SimpleIntFloatArithmetic:    106ms    111ms    0.08us    0.548ms
               SimpleIntegerArithmetic:    106ms    109ms    0.08us    0.544ms
                SimpleListManipulation:    103ms    113ms    0.10us    0.587ms
                  SimpleLongArithmetic:    112ms    118ms    0.18us    0.271ms
                            SmallLists:    105ms    116ms    0.17us    0.366ms
                           SmallTuples:    108ms    128ms    0.24us    0.406ms
                 SpecialClassAttribute:    119ms    136ms    0.11us    0.453ms
              SpecialInstanceAttribute:    143ms    155ms    0.13us    0.454ms
                        StringMappings:    115ms    121ms    0.48us    0.405ms
                      StringPredicates:    120ms    129ms    0.18us    2.064ms
                         StringSlicing:    111ms    127ms    0.23us    0.781ms
                             TryExcept:    125ms    126ms    0.06us    0.681ms
                        TryRaiseExcept:    133ms    137ms    2.14us    0.361ms
                          TupleSlicing:    117ms    120ms    0.46us    0.066ms
                       UnicodeMappings:    156ms    160ms    4.44us    0.429ms
                     UnicodePredicates:    117ms    121ms    0.22us    2.487ms
                     UnicodeProperties:    115ms    153ms    0.38us    2.070ms
                        UnicodeSlicing:    126ms    129ms    0.26us    0.689ms
        -------------------------------------------------------------------------------
        Totals:                           6283ms   6673ms
        """
        ________________________________________________________________________
        
        Writing New Tests
        ________________________________________________________________________
        
        pybench tests are simple modules defining one or more pybench.Test
        subclasses.
        
        Writing a test essentially boils down to providing two methods:
        .test() which runs .rounds number of .operations test operations each
        and .calibrate() which does the same except that it doesn't actually
        execute the operations.
        
        
        Here's an example:
        ------------------
        
        from pybench import Test
        
        class IntegerCounting(Test):
        
            # Version number of the test as float (x.yy); this is important
            # for comparisons of benchmark runs - tests with unequal version
            # number will not get compared.
            version = 1.0
            
            # The number of abstract operations done in each round of the
            # test. An operation is the basic unit of what you want to
            # measure. The benchmark will output the amount of run-time per
            # operation. Note that in order to raise the measured timings
            # significantly above noise level, it is often required to repeat
            # sets of operations more than once per test round. The measured
            # overhead per test round should be less than 1 second.
            operations = 20
        
            # Number of rounds to execute per test run. This should be
            # adjusted to a figure that results in a test run-time of between
            # 1-2 seconds (at warp 1).
            rounds = 100000
        
            def test(self):
        
        	""" Run the test.
        
        	    The test needs to run self.rounds executing
        	    self.operations number of operations each.
        
                """
                # Init the test
                a = 1
        
                # Run test rounds
        	#
                # NOTE: Use xrange() for all test loops unless you want to face
        	# a 20MB process !
        	#
                for i in xrange(self.rounds):
        
                    # Repeat the operations per round to raise the run-time
                    # per operation significantly above the noise level of the
                    # for-loop overhead. 
        
        	    # Execute 20 operations (a += 1):
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
                    a += 1
        
            def calibrate(self):
        
        	""" Calibrate the test.
        
        	    This method should execute everything that is needed to
        	    setup and run the test - except for the actual operations
        	    that you intend to measure. pybench uses this method to
                    measure the test implementation overhead.
        
                """
                # Init the test
                a = 1
        
                # Run test rounds (without actually doing any operation)
                for i in xrange(self.rounds):
        
        	    # Skip the actual execution of the operations, since we
        	    # only want to measure the test's administration overhead.
                    pass
        
        Registering a new test module
        -----------------------------
        
        To register a test module with pybench, the classes need to be
        imported into the pybench.Setup module. pybench will then scan all the
        symbols defined in that module for subclasses of pybench.Test and
        automatically add them to the benchmark suite.
        
        
        Breaking Comparability
        ----------------------
        
        If a change is made to any individual test that means it is no
        longer strictly comparable with previous runs, the '.version' class
        variable should be updated. Therefafter, comparisons with previous
        versions of the test will list as "n/a" to reflect the change.
        
        
        Version History
        ---------------
        
          2.0: rewrote parts of pybench which resulted in more repeatable
               timings:
                - made timer a parameter
                - changed the platform default timer to use high-resolution
                  timers rather than process timers (which have a much lower
                  resolution)
                - added option to select timer
                - added process time timer (using systimes.py)
                - changed to use min() as timing estimator (average
                  is still taken as well to provide an idea of the difference)
                - garbage collection is turned off per default
                - sys check interval is set to the highest possible value
                - calibration is now a separate step and done using
                  a different strategy that allows measuring the test
                  overhead more accurately
                - modified the tests to each give a run-time of between
                  100-200ms using warp 10
                - changed default warp factor to 10 (from 20)
                - compared results with timeit.py and confirmed measurements
                - bumped all test versions to 2.0
                - updated platform.py to the latest version
                - changed the output format a bit to make it look
                  nicer
                - refactored the APIs somewhat
          1.3+: Steve Holden added the NewInstances test and the filtering 
               option during the NeedForSpeed sprint; this also triggered a long 
               discussion on how to improve benchmark timing and finally
               resulted in the release of 2.0
          1.3: initial checkin into the Python SVN repository
        
        
        Have fun,
        --
        Marc-Andre Lemburg
        mal@lemburg.com
        
Platform: UNKNOWN
Description-Content-Type: text/plain
