NVIDIA NIM 推理微服务,用于使用兼容 OpenAI 的 API(自托管或云端)部署 AI 模型
📖 中文导读:NVIDIA NIM 推理微服务,用于使用兼容 OpenAI 的 API(自托管或云端)部署 AI 模型 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。
name: nvidia-nim
description: NVIDIA NIM inference microservices for deploying AI models with OpenAI-compatible APIs, self-hosted or cloud
NVIDIA NIM Expert Skill
You are an expert in NVIDIA NIM (NVIDIA Inference Microservices) - a set of accelerated inference microservices for deploying foundation models on any cloud, data center, workstation, or PC.
Overview
NVIDIA NIM provides:
OpenAI-compatible APIs for seamless integration with existing tools
Optimized inference using TensorRT-LLM, vLLM, and Triton Inference Server
Flexible deployment - self-hosted containers or NVIDIA's cloud API
Enterprise-ready - part of NVIDIA AI Enterprise with security updates
Quick Start
Cloud API (integrate.api.nvidia.com)
from openai import OpenAI
client = OpenAI(
base_url="https://integrate.api.nvidia.com/v1",
api_key="nvapi-YOUR_API_KEY" # Get from build.nvidia.com
)
response = client.chat.completions.create(
model="meta/llama-3.1-70b-instruct",
messages=[{"role": "user", "content": "Hello, world!"}],
temperature=0.7,
max_tokens=1024
)
print(response.choices[0].message.content)
Self-Hosted Container
# Pull and run NIM container
docker run -d --gpus all \
-e NGC_API_KEY=$NGC_API_KEY \
-p 8000:8000 \
nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
# Use with OpenAI client
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
Supported Models
LLM Models
Model
Size
Best For
meta/llama-3.1-405b-instruct
405B
Complex reasoning, enterprise
meta/llama-3.1-70b-instruct
70B
General purpose, balanced
meta/llama-3.1-8b-instruct
8B
Fast inference, cost-effective
mistralai/mixtral-8x22b-instruct-v0.1
141B
Multi-expert reasoning
nvidia/nemotron-4-340b-instruct
340B
Enterprise, high accuracy
google/gemma-2-27b-it
27B
Efficient, open weights
Vision Models (VLM)
Model
Capabilities
microsoft/phi-3-vision-128k-instruct
Image understanding
nvidia/vila-1.5-40b
Video/image analysis
google/paligemma-3b-mix-224
Multimodal tasks
Embedding Models
Model
Dimensions
Use Case
nvidia/nv-embedqa-e5-v5
1024
RAG, semantic search
nvidia/nv-embed-v2
4096
High-quality embeddings
nvidia/llama-3.2-nv-embedqa-1b-v2
2048
Balanced performance
Reranking Models
Model
Use Case
nvidia/nv-rerankqa-mistral-4b-v3
Document reranking
nvidia/llama-3.2-nv-rerankqa-1b-v2
Fast reranking
API Reference
Chat Completions
response = client.chat.completions.create(
model="meta/llama-3.1-70b-instruct",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain quantum computing."}
],
temperature=0.7,
top_p=0.95,
max_tokens=2048,
stream=True, # Enable streaming
frequency_penalty=0.0,
presence_penalty=0.0
)
# Handle streaming
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="")
Embeddings
response = client.embeddings.create(
model="nvidia/nv-embedqa-e5-v5",
input=["Your text to embed"],
encoding_format="float" # or "base64"
)
embeddings = response.data[0].embedding