Metadata-Version: 2.1
Name: llama-cpp-guidance
Version: 0.1.1
Summary: A guidance compatibility layer for llama-cpp-python
Author: Nicholas Yager
Author-email: yager@nicholasyager.com
Requires-Python: >=3.9,<3.12
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: guidance (>=0.0.64,<0.0.65)
Requires-Dist: llama_cpp_python (>=0.1.83)
Requires-Dist: loguru (>=0.7.1,<0.8.0)
Description-Content-Type: text/markdown

# llama.cpp Guidance

[![pypi version shield](https://img.shields.io/pypi/v/llama-cpp-guidance)](https://img.shields.io/pypi/v/llama-cpp-guidance)

The `llama-cpp-guidance` package provides an LLM client compatibility layer between [llama-cpp-python] and [guidance].

## Installation

The `llama-cpp-guidance` package can be installed using pip.

```console
pip install llama-cpp-guidance
```

> [!INFO]
> It is highly recommended that you follow the installation instructions for [llama-cpp-python] after installing `llama-cpp-guidance` to ensure that you have hardware acceleration setup appropriately.

## Basic Usage

Once installed, you can use the `LlamaCpp` class like any other guidance-compatible LLM class.

```python
from pathlib import Path
from llama_cpp_guidance.llm import LlamaCpp
import guidance

guidance.llm = LlamaCpp(
    model_path=Path("../path/to/llamacpp/model.gguf"),
    n_gpu_layers=1,
    n_threads=8
)

program = guidance(
    "The best thing about the beach is {{~gen 'best' temperature=0.7 max_tokens=10}}"
)
output = program()
print(output)
```

```console
The best thing about the beach is that there’s always something to do.
```

[llama-cpp-python]: https://github.com/abetlen/llama-cpp-python/
[guidance]: https://github.com/guidance-ai/guidance/

