Metadata-Version: 2.1
Name: fiddlecube
Version: 0.1.3
Summary: Generate ideal question-answer dataset for testing your LLM.
Author: Kaushik Srinivasan
Author-email: kaushik@fiddlecube.ai
Requires-Python: >=3.9,<3.13
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: requests (>=2.32.3,<3.0.0)
Description-Content-Type: text/markdown

# FiddleCube - Generate ideal question-answers for testing RAG

FiddleCube generates an ideal question-answer dataset for testing your LLM. Run tests on this dataset before pushing any prompt or RAG upgrades.

## Quickstart

### Install FiddleCube

```bash
pip3 install fiddlecube
```

### Register

Run

```bash
fiddlecube register
```

Follow the steps to get an API key.

### Usage

```python
from fiddlecube import FiddleCube

fc = FiddleCube(api_key="<api-key>")
dataset = fc.generate(
    [
        "The cat did not want to be petted.",
        "The cat was not happy with the owner's behavior.",
    ],
    10,
)
dataset
```

```json
{
  "results": [
    {
      "query": "Question: Why did the cat not want to be petted?",
      "contexts": ["The cat did not want to be petted."],
      "answer": "The cat did not want to be petted because it was not in the mood for physical affection at that moment.",
      "score": 0.8,
      "question_type": "SIMPLE"
    },
    {
      "query": "Was the cat pleased with the owner's actions?",
      "contexts": ["The cat was not happy with the owner's behavior."],
      "answer": "No, the cat was not pleased with the owner's actions.",
      "score": 0.8,
      "question_type": "NEGATIVE"
    }
  ],
  "status": "COMPLETED",
  "num_tokens_generated": 44,
  "rate_limited": false
}
```

## Ensuring diversity and correctness

- The questions are spread across the vector embeddings to ensure completeness of testing.
- The queries and responses are evaluated for correctness and context relevancy.
- Citations to the database context are maintained for ease of testing and auditing.

