Skip to content

About

High-performance, light-weight C++ LLM and VLM Inference Software for Physical AI

Resources

Code of conduct

Contributing

Security policy

Stars

635 stars

Watchers

12 watching

Forks

Latest commit

 

History

52 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

TensorRT Edge-LLM

High-Performance Large Language Model Inference Framework for NVIDIA Edge Platforms

Documentation version license

TensorRT Edge-LLM interactive chat demo

Overview   |   Support Matrix   |   Quick Start   |   Performance   |   Documentation   |   Roadmap


Latest News


Overview

TensorRT Edge-LLM is NVIDIA's C++ inference runtime for text, vision, audio, speech, and action models on NVIDIA Jetson, NVIDIA DRIVE, and NVIDIA DGX Spark. The supported frontend exports Hugging Face checkpoints to ONNX for C++ engine building; an experimental direct frontend builds engines from checkpoints without ONNX. Both paths use the same C++ deployment runtimes.


Getting Started

For a supported target, install a published Python wheel without compiling Edge-LLM. Use tensorrt-edgellm[runtime] for the high-level Python API and HTTP serving. See Optional Python dependencies for additional tools, or Install without optional dependencies for the base installation.

Follow the Quick Start Guide to start a chat or send an inference request. Checkpoint IDs are listed in Supported Models.

Install from PyPI (requires a supported CUDA/TensorRT stack):

python -m pip install "tensorrt-edgellm[runtime]"

Use with coding agents

The repository ships agent skills in skills/. Agent hosts that support .agents/skills (including Codex and Cursor) discover them inside a clone; Claude Code uses the .claude/skills link. Set up the project and let the agent drive:

git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git && cd TensorRT-Edge-LLM
claude "Set up TensorRT Edge-LLM on this machine with the edge-llm-install skill. Ask before system-wide changes."   # or: codex "..."

To use the skills outside a clone, ask: Install the Edge-LLM agent skills for <Claude Code|Codex> with the edge-llm-install skill.

Each skill states the inputs it needs and the evidence it reports before it declares success.

Task Example prompt Skill
Install, update, or remove Install Edge-LLM from a wheel on this Jetson Thor and verify it. edge-llm-install
Serve a model Serve <org/model> through the OpenAI-compatible API, send a streaming request, and stop it after the smoke test. edge-llm-serve
Convert and run a model Export <org/model>, build the required engines, run C++ inference, and compare with the provider reference. edge-llm-convert-model
Add or customize a model Add support for <org/model>; start from its config.json and checkpoint tensors. edge-llm-add-model
CuTe DSL kernel Add a CuTe DSL kernel variant for sm_<NN> and validate it on freshly generated artifacts. edge-llm-cutedsl-kernel
TensorRT plugin Add a TensorRT plugin for <op> to both frontends. edge-llm-plugin-extension
Upgrade or backport Migrate my deployment from Edge-LLM <old> to <new>. edge-llm-version-migration
Accuracy mismatch Output differs from the Hugging Face reference; find the first failing layer. edge-llm-few-layer-test, then edge-llm-accuracy-debugging
Run or add tests Add an end-to-end case for <org/model> and run only that case. edge-llm-e2e-test, edge-llm-cpp-test

Not sure which skill applies? Ask the agent to list the skills in skills/ and pick one.


Documentation

Introduction

User Guide

Developer Guide

Software Design

Advanced Topics


Performance

See the Performance Benchmarks page for released benchmark results covering LLM and VLM prefill, generation throughput, memory usage, and EAGLE speculative decoding speedups.


Use Cases

🚗 Automotive

  • In-vehicle AI assistants
  • Voice-controlled interfaces
  • Scene understanding
  • Driver assistance systems

🤖 Robotics

  • Natural language interaction
  • Task planning and reasoning
  • Visual question answering
  • Human-robot collaboration

🏭 Industrial IoT

  • Equipment monitoring with NLP
  • Automated inspection
  • Predictive maintenance
  • Voice-controlled machinery

📱 Edge Devices

  • On-device chatbots
  • Offline language processing
  • Privacy-preserving AI
  • Low-latency inference

Featured Websites

Follow our GitHub repository for the latest updates, releases, and announcements.


Support


License

Apache License 2.0


Contributing

We welcome contributions! Please see our Contributing Guidelines for details.


About

High-performance, light-weight C++ LLM and VLM Inference Software for Physical AI

Resources

Code of conduct

Contributing

Security policy

Stars

635 stars

Watchers

12 watching

Forks

Releases

Packages

Used by

Contributors

Languages