Inference Engines

llama.cpp — a leading open-source project in the AI/LLM ecosystem.

vLLM — a leading open-source project in the AI/LLM ecosystem.

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

Find secrets with Gitleaks 🔑

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

✍🏻 Source Code Deep Dives, System Design & Engineering Blogs | Halfrost-Field 冰霜之地：源码解析、系统设计与工程实践笔记

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

Official inference library for Mistral models

High-speed Large Language Model Serving for Local Deployment

A 2.78-trillion-parameter Kimi K3 running inference on a single CPU in 8.24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

The easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!

A Datacenter Scale Distributed Inference Serving Framework

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Open-source implementation of AlphaEvolve

Plano is an AI-native proxy server and data plane for agentic apps - centralizing orchestration, safety, observability, and smart LLM routing so you can deliver agents faster.

Gemma 4 26B-A4B inference in ~2 GB of RAM on any M-series MacBook

FlashInfer: Kernel Library for LLM Serving

Standardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on Kubernetes

⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.

Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg/5xXzkMu8Zk

Performance-optimized AI inference on your GPUs. Unlock superior throughput by selecting and tuning engines like vLLM or SGLang.