Deployment and Serving

AirLLM is an open-source tool designed to optimize memory usage during large language model (LLM) inference, enabling mo...

AITemplate is a Python framework that converts deep neural networks into optimized CUDA or HIP C++ code for high-perform...

BentoML is an open-source framework designed for packaging, deploying, and serving machine learning models in production...

BISHENG is an open-source LLM application devops platform designed to streamline the deployment and management of large ...

DeepDetect is a machine learning server designed for deploying and serving models in production environments. It support...

NVIDIA Dynamo is a high-performance inference framework optimized for deploying and serving generative AI and reasoning ...

exo is an open-source tool designed to simplify the deployment and management of AI clusters using everyday consumer dev...

Genkit is an open-source framework designed to streamline the development and deployment of AI-powered applications by p...

Inference is a high-performance inference server designed for deploying computer vision models, supporting popular archi...

IPEX-LLM is a PyTorch-based library optimized for running large language models (LLMs) efficiently on Intel CPUs and GPU...

Jina-serve is an open-source framework for building and deploying AI services with support for multiple communication pr...

KServe is a Kubernetes Custom Resource Definition for serving predictive and generative machine learning models. It prov...

KTransformers is an open-source framework designed to optimize and streamline large language model (LLM) inference, offe...

Langtrace is an open-source observability tool designed for LLM applications, leveraging Open Telemetry to provide real-...

Lepton AI is a Python library designed to simplify the deployment and serving of AI models by converting Python code int...

LightLLM is a Python-based framework designed for efficient inference and serving of large language models (LLMs), optim...

LiteLLM is a Python SDK and proxy server that standardizes interactions with over 100 LLM APIs by converting them into O...

LitServe is an open-source AI model serving engine built on FastAPI, designed to deploy and manage custom inference engi...

Llmfit is a terminal tool that optimizes LLM models for a system's hardware configuration. It detects the system's RAM, ...

LocalAI is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/mudler/LocalAI.sv...

MindsDB is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/mindsdb/mindsdb.s...

mini-sglang is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/sgl-project/m...

MLRun is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/mlrun/mlrun.svg?cac...

MLServer is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/SeldonIO/mlserve...

Mosec is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/mosecorg/mosec.svg?...

nano-vllm is a lightweight, optimized implementation of vLLM designed for fast offline inference. It leverages technique...

nndeploy is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/nndeploy/nndeplo...

Nuclio is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/nuclio/nuclio.svg?...

OpenVINO is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/openvinotoolkit/...

OptiLLM is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/algorithmicsuperi...

PowerInfer is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/SJTU-IPADS/Pow...

Prompt2Model is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/neulab/promp...

RamaLama is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/containers/ramal...

RunAnywhere is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/RunanywhereAI...

Seldon Core is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/SeldonIO/seld...

text-generation-inference is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars...

TorchServe is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/pytorch/serve....

torchtune is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/meta-pytorch/to...

Transformer Lab is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/transform...

Triton Inference Server is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/t...

Vercel AI is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/vercel/ai.svg?c...

Vespa is an AI agent in the Deployment and Serving category. ![](https://img.shields.io/github/stars/vespa-engine/vespa....