Metadata-Version: 2.3
Name: icalfa
Version: 0.3.4
Summary: A fork of the InterCode benchmark used to evaluate natural language to Bash command translation.
Project-URL: Repository, https://github.com/westenfelder/InterCode-ALFA
Author-email: Finn Westenfelder <finnw@mit.edu>
License: Copyright 2024 MIT-ALFA
        
        Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
License-File: LICENSE.txt
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Requires-Dist: docker
Requires-Dist: gymnasium
Requires-Dist: openai
Requires-Dist: pandas
Requires-Dist: rich
Requires-Dist: scikit-learn
Description-Content-Type: text/markdown

# InterCode-ALFA

## Description
A fork of the InterCode benchmark used to evaluate natural language to Bash command translation.  
[Dataset](https://huggingface.co/datasets/westenfelder/InterCode-ALFA-Data)  
[PyPI Package](https://pypi.org/project/icalfa/)  

![InterCode-ALFA Diagram](https://raw.githubusercontent.com/westenfelder/InterCode-ALFA/main/icalfa.png)


## Installation
- Install Docker Engine [Instructions](https://docs.docker.com/engine/install/)
- Configure Docker for non-sudo users [Instructions](https://docs.docker.com/engine/install/linux-postinstall/)
- Create python virtual environment
```bash
apt install python3.12-venv
python3 -m venv icalfa-venv
source icalfa-venv/bin/activate
```
- Install InterCode-ALFA
```bash
pip install icalfa
```
- [Optional] If you want to use a local LLM, install Ollama
```bash
curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:70b
```
- [Optional] If you want to use the embedding comparison method, install mxbai-embed-large
```bash
ollama pull mxbai-embed-large
```


## Usage
- Run the benchmark
```python
import os
from icalfa import submit_command
from datasets import load_dataset

# Store OpenAI key as environment variable 
os.environ['ICALFA_OPENAI_API_KEY'] = '...'

# Load dataset
dataset = load_dataset("westenfelder/InterCode-ALFA-Data")['train']

# Iterate through the dataset
score = 0
for index, row in enumerate(dataset):

    # Retrieve natural language prompt
    prompt = row['query']

    # Convert natural language prompt to Bash command here

    # Submit Bash command for benchmark scoring. 0 = incorrect, 1 = correct
    score += submit_command(index=index, command="...")

    # Retrieve ground truth commands
    ground_truth_command = row['gold']
    ground_truth_command2 = row['gold2']

# Print the benchmark result
print(score/len(dataset))
```

- submit_command parameters
```python
# By default icalfa uses OpenAI's GPT-4 model and expects an API key
submit_command(index, command, eval_mode="openai", eval_param="gpt-4-0613")

# A local model can be used via Ollama
submit_command(index, command, eval_mode="ollama", eval_param="llama3.1:70b")

# You can also test the original method used in Princeton's InterCode benchmark
submit_command(index, command, eval_mode="tfidf")

# An embedding based comparison method is also available
# This uses the mxbai-embed-large model via Ollama, with the eval_param specifying the similarity threshold
submit_command(index, command, eval_mode="embed", eval_param=0.75)
```

- Manage Docker containers
```bash
# Stop containers
docker stop $(docker ps -a --filter "name=intercode*" -q)

# Delete containers
docker rm $(docker ps -a --filter "name=intercode*" -q)
```


## Building
```bash
# update version in pyproject.toml and __init__.py
rm -rf dist
python3 -m build
python3 -m twine upload --repository pypi dist/*
pip install --upgrade icalfa
```


## Credits
InterCode-ALFA is a fork of the InterCode benchmark developed by the Princeton NLP group.  
[InterCode Website](https://intercode-benchmark.github.io/)  
[InterCode PyPI Package](https://pypi.org/project/intercode-bench/#description)  
