Inference Engines

RuVector is a High Performance, Real-Time, Self-Learning, Vector Graph Neural Network, and Database built in Rust.

RuVector is a High Performance, Real-Time, Self-Learning, Vector Graph Neural Network, and Database built in Rust.

Optimizing inference proxy for LLMs

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

Sparsity-aware deep learning inference runtime for CPUs

A portable accelerated SQL query, search, and LLM-inference engine, written in Rust, for data-grounded AI apps and agents.

Distributed LLM inference. Connect home devices into a powerful cluster to accelerate LLM inference. More devices means faster inference.

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.

MLX Studio - Home of JANG_Q - Image Gen/Edit + Chat/Code All in one - + OpenClaw (Anthropic API)

A highly optimized LLM inference acceleration engine for Llama and its variants.

Yet Another Language Model: LLM inference in C++/CUDA, no libraries except for I/O

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

校招、秋招、春招、实习好项目，带你从零动手实现支持LLama2/3和Qwen2.5的大模型推理框架。

Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2

A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).