Evaluation & Testing

The LLM Evaluation Framework

Ragas — a leading open-source project in the AI/LLM ecosystem.

the LLM vulnerability scanner

ReLE评测：中文AI大模型能力评测（持续更新）：目前已囊括335个大模型，覆盖chatgpt、gpt-5.2、o4-mini、谷歌gemini-3-pro、Claude-4.5、文心ERNIE-X1.1、ERNIE-5.0-Thinking、qwen3-max、百川、讯飞星火、商汤senseChat等商用模型， 以及kimi-k2、ernie4.5、minimax-M2、deepseek-v3.2、qwen3-2507、llama4、智谱GLM-4.6、gemma3、mistral等开源大模型。不仅提供排行榜，也提供规模超200万的大模型缺陷库！方便广大社区研究分析、改进大模型。

The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices

The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.

One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks

A full-stack AI Red Teaming platform securing AI ecosystems via OpenClaw Security Scan, Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.

Evaluation and Tracking for LLM Experiments and AI Agents

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

Build, enrich, and transform datasets using AI models with no code

A powerful tool for automated LLM fuzzing. It is designed to help developers and security researchers identify and mitigate potential jailbreaks in their LLM APIs.

Prompty makes it easy to create, manage, debug, and evaluate LLM prompts for your AI applications. Prompty is an asset class and format for LLM prompts designed to enhance observability, understandability, and portability for developers.

UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection

The open source post-building layer for agents. Our environment data and evals power agent post-training (RL, SFT) and monitoring.

Agentic testing for agentic codebases

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表，主要面向基础大模型评测，旨在探求生成式AI的技术边界.

Awesome papers involving LLMs in Social Science.

Deliver safe & effective language models

Deliver safe & effective language models

Data-Driven Evaluation for LLM-Powered Applications

A comprehensive set of LLM benchmark scores and provider prices. (deprecated, read more in README)

Mock everything your AI app talks to — LLM APIs, MCP, A2A, vector DBs, search. One package, one port, zero dependencies.

Build, Improve Performance, and Productionize your LLM Application with an Integrated Framework