Metadata-Version: 2.1
Name: fiddler-auditor
Version: 0.0.1rc0
Summary: Auditing large language models made easy.
Author-email: Fiddler Labs <support@fiddler.ai>
License: Elastic License 2.0 (ELv2)
Project-URL: repository, https://github.com/fiddler-labs/fiddler-auditor
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: OS Independent
Requires-Python: <4.0,>=3.7.12
Description-Content-Type: text/markdown
Provides-Extra: test
License-File: LICENSE

# ![alt text](https://global-uploads.webflow.com/5e067beb4c88a64e31622d4b/6030124ca93f9ce13a57aa79_favicon.png "Fiddler Auditor") Fiddler Auditor 
Auditing Large Language Models made easy! 🍰

[![lint](https://github.com/fiddler-labs/fiddler-auditor/actions/workflows/codelint.yml/badge.svg)](https://github.com/fiddler-labs/fiddler-auditor/actions/workflows/codelint.yml)
[![test](https://github.com/fiddler-labs/fiddler-auditor/actions/workflows/test.yml/badge.svg)](https://github.com/fiddler-labs/fiddler-auditor/actions/workflows/test.yml)


## 🔬 What is Fiddler Auditor

<div align="left">
    <img src="docs/source/images/monitoring-generative-ai-models_fiddler-auditor.png"
         alt="Fiddler Auditor Capabilities"/>
</div>

Language models enable companies to build and launch innovative applications to make their workforce smarter and increase customer satisfaction. 
However, it’s been known that LLMs can hallucinate, generate adversarial responses that can harm users, and even expose private information that they were trained on when prompted or unprompted. It's more critical than ever for ML and software application teams to minimize these risks and weaknesses before launching LLMs and NLP models. As a result, it’s important for you to include a process to audit language models thoroughly before production.
The Fiddler Auditor enables you to test LLMs and NLP models, identify weaknesses in the models, and mitigate potential adversarial outcomes before deploying them to production.

## 📰 Features and Capabilities
Fiddler Auditor supports

- Red-teaming LLMs for your use-case with prompt perturbation
- Integration with LangChain
- Custom evaluation metrics
- Generative and Discriminative NLP models
- Comparison of LLMs (Upcoming)

<div align="left">
    <img src="docs/source/images/fiddler-auditor-prompt-evaluation.png"
         alt="Example Report"/>
</div>

## 📦 Installation

`pip install fiddler-auditor`

## 🚀 Quick-start guides
- [Evaluate LLM Correctness and Robustness](examples/LLM_Evaluation.ipynb)
- [Evaluate LLMs with custom metrics](examples/Custom_Evaluation.ipynb)


## 📖 Documentation
- [Website](https://docs.fiddler.ai)


## 🙋‍♀ Contribution
We are continuously updating this library to support language models as they evolve. 

- Contributions in the form of suggestions and PRs to Fiddler Auditor are welcome!
- If you have feedback regarding this library, feel free to raise issues in this repository.

## Community
- Join the [Fiddler Community](https://www.fiddler.ai/slackinvite)
- Visit the [Fiddler Resources Library](https://www.fiddler.ai/resources)
- Enjoy using Fiddler Auditor? Leave us a star on [GitHub](https://github.com/fiddler-labs/fiddler-auditor)
- Follow us on [Twitter](https://twitter.com/fiddlerlabs)
- Subscribe to our [blog](https://www.fiddler.ai/blog#subscribe)
- [Sign-up to try Fiddler](https://www.fiddler.ai/trial)
