Open-source lLM training, gateways & evals

Transformers

huggingface/transformers

The standard Python library for defining, running and fine-tuning open models across text, vision and audio.

pip install "transformers[torch]"

LLaMA-Factory

hiyouga/LlamaFactory

No-code web UI and CLI to fine-tune 100+ open LLMs and VLMs (LoRA, QLoRA, full).

docker run -it --rm --gpus=all --ipc=host hiyouga/llamafactory:latest

LiteLLM

BerriAI/litellm

Gateway and SDK to call 100+ LLM APIs in OpenAI format with cost tracking, fallbacks, load balancing and logging.

uv tool install 'litellm[proxy]'

promptfoo

promptfoo/promptfoo

CLI and CI tool to test prompts, agents and RAG, compare models, and red-team LLM apps.

npm install -g promptfoo

DeepEval

confident-ai/deepeval

Pytest-style framework for evaluating LLM outputs with ready-made metrics.

pip install -U deepeval

Ragas

vibrantlabsai/ragas

Evaluation toolkit for RAG and LLM apps (faithfulness, relevance, etc.) with test-set generation.

pip install ragas

Axolotl

axolotl-ai-cloud/axolotl

Config-driven tool for fine-tuning and post-training open LLMs.

docker run --gpus '"all"' --ipc=host --rm -it axolotlai/axolotl:main-latest

Laminar

lmnr-ai/lmnr

Open-source tracing, evals and observability platform built for AI agents.

pip install --upgrade 'lmnr[all]'