Metadata-Version: 2.1
Name: paramount
Version: 0.3.5
Summary: Business Evals for LLMs
Home-page: https://github.com/ask-fini/paramount
Author: Hakim K
Author-email: 5586611+hakimkhalafi@users.noreply.github.com
Project-URL: Source Code, https://github.com/ask-fini/paramount
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: requests
Requires-Dist: pandas
Requires-Dist: pytz
Requires-Dist: flask
Requires-Dist: python-dotenv
Requires-Dist: scikit-learn
Requires-Dist: sqlalchemy
Requires-Dist: psycopg2-binary
Requires-Dist: gunicorn
Requires-Dist: toml

## paramount

Business Evaluations for LLM Chats - let your agents easily evaluate AI chat accuracy.

### Getting Started
1) Install the package: 

```
pip install paramount
```

2) Decorate your AI function:

```py
@paramount.record()
def my_ai_function(message_history, question,  ...): #  Inputs
    # LLM invocations happen here
    return llm_answer, llm_references, llm_message_history,  ...  # Outputs
```

3) After `my_ai_function(...)` has run several times, launch the Paramount UI to evaluate results: 

```shell
paramount
```

Your SMEs can now evaluate recordings and track accuracy improvements over time.

Paramount runs completely offline in your private environment.

### Configuration

In order to set up successfully, define which input and output parameters represent the chat list used in the LLM.

This is done via the `paramount.toml` configuration file that you add in your project root dir. 

It will be autogenerated for you with defaults if this file doesn't already exist on first run.

```toml
[record]
enabled = true
function_url = "http://localhost:9000"  # The url to the flask app, for replay

[db]
type = "csv" # postgres also available
	[db.postgres] 
	connection_string = ""

[api]
endpoint = "http://localhost" # url and port for paramount UI/API
port = 9001
split_by_id = false # In case you have several bots and want to split them by ID
identifier_colname = "" 

[ui]  # These are display elements for the UI

# For the table display - define which columns should be shown
meta_cols = ['recorded_at']
input_cols = ['args__message_history', 'args__question']  # Matches my_ai_function() example
output_cols = ['1', '2']  # 1 and 2 are indexes for llm_answer and llm_references in example above

# For the chat display - describe how your chat structure is set up. This example uses OpenAI format.
chat_list = "output__3"  # Matches llm_message_history. Must be a list of dicts to display chat format
chat_list_role_param = "role"  # Key in list of dicts describing the role in the chat
chat_list_content_param = "content"  # Key in list of dicts describing the content
```
It is also possible to describe references via config but is not shown here for simplicity. 

See `paramount.toml.example` for more info.  
### Docker

By using `Dockerfile.server`, you can containerize and deploy the whole package (including the client).

With Docker, you will need to mount the `paramount.toml` file dynamically into the container for it to work. 

```shell
docker build -t paramount-server -f Dockerfile.server . # or make docker-build-server
docker run -dp 9001:9001 paramount-server # or make docker-run-server
```
