Best LLM Inference AI Agents (2026)

LLM Inference

DeepSpeed-Mii is an open-source library designed to optimize large language model (LLM) inference by enabling low-latenc...

LLM Inference

deploy-llms-with-ansible is an open-source Ansible playbook that automates the deployment of large language models (LLMs...

LLM Inference

exllama is a highly optimized implementation of the Llama language model for inference, specifically designed to work ef...

LLM Inference

FastChat is an open-source platform for serving and interacting with multiple large language models (LLMs) efficiently. ...

LLM Inference

FasterTransformer is an open-source library developed by NVIDIA for accelerating transformer-based large language model ...

LLM Inference

Infinity is a Python-based tool designed for efficient text-embedding inference, enabling users to generate embeddings l...

LLM Inference

Liger-Kernel provides optimized Triton kernels for efficient large language model (LLM) training, focusing on performanc...

LLM Inference

LMDeploy is a high-performance inference and serving framework designed for large language models (LLMs) and vision-lang...

LLM Inference

MInference is an open-source tool designed to optimize the inference process of long-context large language models (LLMs...

LLM Inference

mistral.rs is a Rust-based implementation for efficient inference of Mistral language models, focusing on high performan...

LLM Inference

prima.cpp is a distributed implementation of llama.cpp designed to enable efficient inference of large language models (...

LLM Inference

SGLang is an open-source framework designed for efficient serving of large language models (LLMs) and vision language mo...

LLM Inference

SkyPilot is an open-source framework for running large language models (LLMs) and batch jobs across multiple cloud provi...

LLM Inference

TensorRT-LLM is an open-source library by NVIDIA designed to optimize and accelerate inference for large language models...

LLM Inference

Text-Embeddings-Inference is a Rust-based tool for efficiently generating text embeddings using pre-trained models. It p...

LLM Inference

TGI (Text Generation Inference) is a toolkit designed for deploying and serving Large Language Models (LLMs) efficiently...

LLM Inference

vLLM is a high-performance inference engine designed for large language models (LLMs), optimized for throughput and memo...