Model Serving

Serverless LLM Serving for Everyone.

Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference.

FastAPI Skeleton App to serve machine learning models production-ready.

Python + Inference - Model Deployment library in Python. Simplest model inference server ever.

Open Model Engine (OME) — Kubernetes operator for LLM serving, GPU scheduling, and model lifecycle management. Works with SGLang, vLLM, TensorRT-LLM, and Triton

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

xFasterTransformer — open-source AI/LLM project.

A REST API for Caffe using Docker and Go

Learn to serve Stable Diffusion models on cloud infrastructure at scale. This Lightning App shows load-balancing, orchestrating, pre-provisioning, dynamic batching, GPU-inference, micro-services working together via the Lightning Apps framework.

PMetal: high-performance Apple Silicon framework for local LLM inference, LoRA/QLoRA fine-tuning, serving, quantization, and MLX/Metal acceleration.

Work with LLMs on a local environment using containers

Fast GPU OCR server. 270 img/s on FUNSD. TensorRT FP16, PP-OCRv5, HTTP + gRPC.

This is a repository for an nocode object detection inference API using the Yolov3 and Yolov4 Darknet framework.

This is a repository for an nocode object detection inference API using the Yolov4 and Yolov3 Opencv.