Metadata-Version: 2.1
Name: palaestrai
Version: 3.0.1
Summary: A Training Ground for Autonomous Agents
Home-page: http://palaestr.ai/
Author: The ARL Developers
Author-email: eric.veith@offis.de
License: LGPLv2
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: GNU Lesser General Public License v2 (LGPLv2)
Classifier: Natural Language :: English
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.8.0
Requires-Dist: click
Requires-Dist: appdirs
Requires-Dist: ruamel.yaml
Requires-Dist: aiomultiprocess
Requires-Dist: setproctitle
Requires-Dist: pyarrow
Requires-Dist: zmq
Requires-Dist: alembic
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: psycopg2-binary
Requires-Dist: six
Requires-Dist: jsonpickle
Requires-Dist: SQLalchemy (<1.4.0)
Requires-Dist: sqlalchemy-utils
Provides-Extra: dev
Requires-Dist: tox ; extra == 'dev'
Requires-Dist: robot ; extra == 'dev'
Requires-Dist: robotframework ; extra == 'dev'
Requires-Dist: pytest ; extra == 'dev'
Requires-Dist: pytest-asyncio ; extra == 'dev'
Requires-Dist: pytest-cov ; extra == 'dev'
Requires-Dist: coverage ; extra == 'dev'
Requires-Dist: mypy ; extra == 'dev'
Requires-Dist: black ; extra == 'dev'
Requires-Dist: lxml ; extra == 'dev'

palastrAI: A Training Ground for Autonomous Agents
==================================================

About
-----

palaestrAI is a distributed framework to train and test all kinds of 
autonomous agents. It provides interfaces to any environment, be it
OpenAI Gym or co-simulation environments via mosaik. palaestrAI can
train and test any kind of autonomous agent in these environments:
From Deep Reinforcement Learning (DRL) algorithms over model-based to 
simple rule-based agents, all can train and test with or against
each other in a shared environment.

In short, palaestrAI can...

* ...train and test one or more agent of any algorithm
* ...place the agents on one or several environments at once, 
  depending on the agents' algorithm
* ...provides facilities to define and reproducibly run experiments

palaestrAI is the core framework of a whole ecosystem:

* hARL provides implementations of several DRL algorithms and 
  interfaces to existing DRL libraries.
* arsenAI provides all facilities needed for proper design
  of experiments.
* palaestrai-mosaik is a interface to the mosaik co-simulation
  software
* palaestrai-environments provides a number of simple, 
  easy to use environments for playing with palaestrAI

Use Cases
---------

palaestrAI is the framework for the Adversarial Resilience Learning
(ARL) reference implementation. The ARL core concept consists of two
agents, attacker and defender agents, working an a common model of a
cyber-phyiscal system (CPS). The attacker's goal is to de-stabilize the CPS,
whereas the defender works to keep the system in a stable and operational
state. Both agents do not perceive their opponent's actions directly, but only
the state of the CPS itself. This imples that none of the agents knows whether
anything they perceive through their sensors is the result of the dynamics of
the CPS itself or of another agent's action.  Also, none of the agents has an
internal model of the CPS. Attacker and defender alike have to explore the CPS
given their sensors and actuators independently and adapt to it. ARL is, in
that sense, suitable to a reinforcement learning approach.  Combined with the
fact the both agents do not simply learn the CPS, but also its respective
opponent, ARL implements system-of-systems deep reinforcement learning.

Installation
------------

Palaestrai is mainly written in Python, with the usual third-party library
here and there. It provides a `setup.py` file just as any well-behaved Python
program. Use::

   pip install .

or, for development::

   pip install -e .[dev]

(With zsh, you need to escape the parenthesis like this::

   pip install -e .\[dev\]

)

However, to run Palaestrai, some additional setup is currently required. First,
you need to install Docker. You find docker in your preferred distributions
package manager or you can use the following commands to start a convience
script::

   curl -fsSL https://get.docker.com -o get-docker.sh
   sudo sh get-docker.sh

Afterwards, you should add yourself to the Docker group::

   sudo usermod -aG docker $USER

Try if your Docker installation works by typing::

   docker images

(or any other valid docker command). If thats not the case, the easiest way
is to restart your system (or use some magic commands to start the docker
daemon). The next step is to start a Docker container for the store::

   docker run -d --name timescaledb -p 5432:5432 -e POSTGRES_PASSWORD=password timescale/timescaledb:1.7.4-pg12

You only need to run this command once. If you want to start the container
after a system reboot, the following command is sufficient::

   docker start timescaledb

Next, you need to setup the database. Type the following three commands::

   docker exec -it timescaledb psql -U postgres
   CREATE DATABASE Palaestrai;
   exit

Now, you need to add a *store_uri* in your *Palaestrai-runtime.conf.yaml*. If you
followed this guide step-by-step, the following store_uri should do::

   store_uri: postgresql://postgres:password@172.17.0.1:5432/Palaestrai

Finally, you can enable some loggers by changing their value from ERROR to
DEBUG or keep palaestrai silent.


Usage
-----

After installing (and, probably, configuring your logging.conf.yaml),
type::

   palaestrai database-create

to create the data store. Afterwards you can start the dummy experiment with::

   palaestrai experiment-start tests/fixtures/dummy_experiment.yml


Scientific Documentation
------------------------

Palaestrai uses techniques from numerous domains. The folder ``doc/sci`` contains a
number of rst files that give an overview of these topics. Each file contains
a short summary (about 1/2 page) about a given topic, e.g., DDQN. The file
answers the following questions:

1. What is it, and where do we use it?
2. What is its goal, and how does it achive it?
3. What are the interfaces and possible implementations?
4. What are the ideosyncrasies?

Development
-----------

Handling a Bug
``````````````

Palaestrai provides a `Makefile` as convience wrapper around many
functions usually required, e.g., to run tests, create a docker image, and the
like.

- A prose description of the bug: what did you intend to do, what happened
  instead?
- The error message, if there is any.
- The command line parameters and configuration
- Your version of Python you are using, and the version of all modules
  (``pip freeze`` gives you that).

Contributing
````````````

The typical work flow is as such:

1. File a bug/feature/support request in the issue tracker
2. Create a feature branch to work on your issue. Name it
   ``bug-<num>-<shortname>`` for bugs, ``feature-<num>-<shortname>`` for new
   features, etc.
3. Provide a unit test for the bug/feature you have been working on.
4. Fix the bug/work on the feature.
5. Run ``black -l 79 ./src/palaestrai ./tests`` to auto-format the code
6. Run ``tox`` and clean up all errors.
7. Request a merge. The merge will happen after a code review;
   work-in-progress code gets first merged into ``development``
8. Once the current development branch has ripened enough, it is merged to
   ``master``. The master branch must contain code that is stable. New
   releases are only tagged on master branch commits.

Coding Style
````````````

Have a look at our architecture document and diagrams in
``doc/architecture.rst``.

We adhere to PEP8_ or black with line length of 79.

Try hard to find fitting names for new modules and subdirectories. If you are
importing your own module as ``import X as Y`` and ``Y`` is
differing semantically from ``X``, it might be the right time to change
the name of ``X``. Please refrain from abbreviated names if it is not
absolutely clear (in two years from now) what the abbreviation signifies.
Specifically, use short variable names in functions, if you like to, but stick
to commonly known and accepted abbreviations, such as ``for i in list`` or
similar things. Avoid confusion with function names from the Python base
library, e.g., do not use ``exp`` as a shorthand for "experiment" (cf.
``math.exp(x, y)``.

Supply docstrings for every class and public function. Otherwise, when you
find yourself writing comments, consider writing better, self-explaining code
instead. When adding "TODO" or "FIXME" comments, make sure somebody else can
understand and begin to work on them. Use type hinting wherever possible.

Functions should not span more than one screen length.

Copyright & Authors
-------------------

All source code, except where otherwise mentioned, is Copyright (C) 2018, 2019
OFFIS e.V. Contributing authors are listed in order of their appeareance in
the file AUTHORS.

The dynamic loader used in the command-line utility relies more or less
verbatim on code from the Python project. See the file ``doc/python-license``.

The code in ``palaestrai.types`` comes from the OpenAI Gym_ project.  See the file
``doc/gym-license.md``.

.. _mosaik: http://mosaik.offis.de/
.. _PEP8: https://www.python.org/dev/peps/pep-0008/
.. _Gym: https://github.com/openai/gym

Running an example with MIDAS
-----------------------------

MIDAS is a collection of simulators for enery system simulation. To use all
features of MIDAS, you need access to two repositories on eprojects.offis.de.
First, you need to install pysimmods::

   pip install midas-mosaik


The next step is to get the MIDAS source::

   https://gitlab.com/midas-mosaik/midas

If you don't have access to either of these repos, write an email to
stephan.balduin@offis.de. **First, make sure you checked out the
development branch**. Otherwise, type::

    git checkout development

MIDAS should then be installed via pip.

Usage
`````

Examples how to use MIDAS together with Palaestrai can be found either in
Palaestrai: tests/system/extended_mosaik_experiment.py. The first one is probably easier
to understand since it uses the standard  cli, but the
second one shows the recommended usage, since it uses the special Palaestrai
interface of midas, which will be explained in the following in more
detail.

The experiment loader (EL) is kind of an intermediate prototype for the
steps between CPS-Abstract Ontology and CPS-Experiment Generator.
The EL loads an experiment file, creates a governor, agents,
environments, and a simulator. An exampe experiment file is located
in Palaestrai: tests/fixtures/example_experiment.yml

Any questions regarding MIDAS can be send to stephan.balduin@offis.de.

Note: MIDAS is still WIP. The API may change in the near future. This short
guide will be updated as soon as possible


