Model Serving

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

TensorRT-LLM — a leading open-source project in the AI/LLM ecosystem.

A framework for efficient model inference with omni-modality models

Olares: An Open-Source Personal Cloud to Reclaim Your Data

In this repository, I will share some useful notes and references about deploying deep learning-based models in production.

🚀 Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice. ⚡️ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). 🍻 OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. 🗃️ Llama3, Mistral, etc. 🧑‍💻 Video Tutorials.

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

High-performance inference framework for large language models, focusing on efficiency, flexibility, and availability.

RamaLama is an open-source developer tool that simplifies the local serving of AI models from any source and facilitates their use for inference in production, all through the familiar language of containers.

Community maintained hardware plugin for vLLM on Ascend

Turn any computer or edge device into a command center for your computer vision projects.

Superlinked Inference Engine is an Open-source inference server and production cluster for embeddings, reranking, and extraction.

🏕️ Reproducible development environment for humans and agents

AICI: Prompts as (Wasm) Programs

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

An open source DevOps tool from the CNCF for packaging and versioning AI/ML models, datasets, code, and configuration into an OCI Artifact.

Hopsworks - Data-Intensive AI platform with a Feature Store

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

The simplest way to serve AI/ML models in production

A throughput-oriented high-performance serving framework for LLMs

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

A scalable inference server for models optimized with OpenVINO™

An open-source computer vision framework to build and deploy apps in minutes

Model Deployment at Scale on Kubernetes 🦄️