Metadata-Version: 2.1
Name: pyspark-stubs
Version: 2.3.0.post4
Summary: A collection of the Apache Spark stub files
Home-page: UNKNOWN
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Platform: UNKNOWN
Requires-Dist: pyspark (<2.4.0,>=2.3.0)

PySpark Stubs
=============

|Build Status| |PyPI version|

A collection of the Apache Spark `stub
files <https://www.python.org/dev/peps/pep-0484/#stub-files>`__. These
files were generated by
`stubgen <https://github.com/python/mypy/blob/master/mypy/stubgen.py>`__
and manually edited to include accurate type hints.

Tests and configuration files have been originally contributed to the
`Typeshed project <https://github.com/python/typeshed/>`__. Please refer
to its `contributors
list <https://github.com/python/typeshed/graphs/contributors>`__ and
`license <https://github.com/python/typeshed/blob/master/LICENSE>`__ for
details.

Motivation
----------

-  Static error detection (see
   `SPARK-20631 <https://issues.apache.org/jira/browse/SPARK-20631>`__)

   |SPARK-20631|



-  Improved completion for chained method calls.

Usage
-----

Please note that the guidelines for distribution of type information is
still work in progress (`PEP 561 - Distributing and Packaging Type
Information <https://www.python.org/dev/peps/pep-0561/>`__). Currently
installation script overlays existing Spark installations (``pyi`` stub
files are copied next to their ``py`` counterparts in the PySpark
installation directory). If this approach is not acceptable you can stub
files to the search path manually.

According to `PEP
484 <https://www.python.org/dev/peps/pep-0484/#storing-and-distributing-stub-files>`__:

    Third-party stub packages can use any location for stub storage.
    Type checkers should search for them using PYTHONPATH.

Moreover:

    Third-party stub packages can use any location for stub storage.
    Type checkers should search for them using PYTHONPATH. A default
    fallback directory that is always checked is
    shared/typehints/python3.5/ (or 3.6, etc.)

You'll also need a type checker for example:

-  `Mypy <https://github.com/python/mypy>`__
-  `Pytype <https://github.com/google/pytype/>`__

Installation
------------

Please check usage before proceeding.

The package is available on PYPI:

.. code:: bash

    pip install pyspark-stubs

In general package is useful only in development, and the shouldn't be
installed in a production environment.

Version Compatibility
---------------------

Package versions follow PySpark versions with exception to maintenance releases - i.e. `pyspark-stubs==2.3.0` should be compatible with `pyspark>=2.3.0,<2.4.0`.
Maintenance releases (`post1`, `post2`, ..., `postN`) are reserved for internal annotations updates.

API Coverage
------------

+------------------------------------------------+---------------------+--------------------+------------+
| Module                                         | Dynamically typed   | Statically typed   | Notes      |
+================================================+=====================+====================+============+
| pyspark                                        | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.accumulators                           | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.broadcast                              | ✔                   | ✔                  | Mixed      |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.cloudpickle                            | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.conf                                   | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.context                                | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.daemon                                 | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.files                                  | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.find\_spark\_home                      | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.heapq3                                 | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.java\_gateway                          | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.join                                   | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml                                     | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.base                                | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.classification                      | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.clustering                          | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.common                              | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.evaluation                          | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.feature                             | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.linalg                              | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.param                               | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.param.\_shared\_params\_code\_gen   | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.param.shared                        | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.pipeline                            | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.recommendation                      | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.regression                          | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.tests                               | ✘                   | ✘                  | Tests      |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.tuning                              | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.util                                | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.ml.wrapper                             | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib                                  | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.classification                   | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.clustering                       | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.common                           | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.evaluation                       | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.feature                          | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.fpm                              | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.linalg                           | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.linalg.distributed               | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.random                           | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.recommendation                   | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.regression                       | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.stat                             | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.stat.KernelDensity               | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.stat.\_statistics                | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.stat.distribution                | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.stat.test                        | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.tests                            | ✘                   | ✘                  | Tests      |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.tree                             | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.mllib.util                             | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.profiler                               | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.rdd                                    | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.rddsampler                             | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.resultiterable                         | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.serializers                            | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.shell                                  | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.shuffle                                | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql                                    | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.catalog                            | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.column                             | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.conf                               | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.context                            | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.dataframe                          | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.functions                          | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.group                              | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.readwriter                         | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.session                            | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.streaming                          | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.tests                              | ✘                   | ✘                  | Tests      |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.types                              | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.utils                              | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.sql.window                             | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.statcounter                            | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.status                                 | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.storagelevel                           | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.streaming                              | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.streaming.context                      | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.streaming.dstream                      | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.streaming.flume                        | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.streaming.kafka                        | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.streaming.kinesis                      | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.streaming.listener                     | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.streaming.tests                        | ✘                   | ✘                  | Tests      |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.streaming.util                         | ✔                   | ✘                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.tests                                  | ✘                   | ✘                  | Tests      |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.traceback\_utils                       | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.version                                | ✘                   | ✔                  |            |
+------------------------------------------------+---------------------+--------------------+------------+
| pyspark.worker                                 | ✘                   | ✘                  | Internal   |
+------------------------------------------------+---------------------+--------------------+------------+

Disclaimer
----------


Apache Spark, Spark, PySpark, Apache, and the Spark logo are `trademarks <https://www.apache.org/foundation/marks/>`__ of `The
Apache Software Foundation <http://www.apache.org/>`__. This project is not owned, endorsed, or
sponsored by The Apache Software Foundation.

.. |Build Status| image:: https://travis-ci.org/zero323/pyspark-stubs.svg?branch=master
   :target: https://travis-ci.org/zero323/pyspark-stubs
.. |PyPI version| image:: https://badge.fury.io/py/pyspark-stubs.svg
   :target: https://badge.fury.io/py/pyspark-stubs
.. |SPARK-20631| image:: https://i.imgur.com/GfDCGjv.gif
     :alt: SPARK-20631


