LLM Inference

DeepSpeed-Mii is an open-source library designed to optimize large language model (LLM) inference by enabling low-latenc...

deploy-llms-with-ansible is an open-source Ansible playbook that automates the deployment of large language models (LLMs...

exllama is a highly optimized implementation of the Llama language model for inference, specifically designed to work ef...

FastChat is an open-source platform for serving and interacting with multiple large language models (LLMs) efficiently. ...

FasterTransformer is an open-source library developed by NVIDIA for accelerating transformer-based large language model ...

Infinity is a Python-based tool designed for efficient text-embedding inference, enabling users to generate embeddings l...

Liger-Kernel provides optimized Triton kernels for efficient large language model (LLM) training, focusing on performanc...

LMDeploy is a high-performance inference and serving framework designed for large language models (LLMs) and vision-lang...

MInference is an open-source tool designed to optimize the inference process of long-context large language models (LLMs...

mistral.rs is a Rust-based implementation for efficient inference of Mistral language models, focusing on high performan...

prima.cpp is a distributed implementation of llama.cpp designed to enable efficient inference of large language models (...

SGLang is an open-source framework designed for efficient serving of large language models (LLMs) and vision language mo...

SkyPilot is an open-source framework for running large language models (LLMs) and batch jobs across multiple cloud provi...

TensorRT-LLM is an open-source library by NVIDIA designed to optimize and accelerate inference for large language models...

Text-Embeddings-Inference is a Rust-based tool for efficiently generating text embeddings using pre-trained models. It p...

TGI (Text Generation Inference) is a toolkit designed for deploying and serving Large Language Models (LLMs) efficiently...

vLLM is a high-performance inference engine designed for large language models (LLMs), optimized for throughput and memo...