llm-integration

anhvth's avatarfrom anhvth

Guide for using LLM utilities in speedy_utils, including memoized OpenAI clients and chat format transformations.

⭐3stars🔀0forks📁View on GitHub🕐Updated Jan 6, 2026

When & Why to Use This Skill

This Claude skill streamlines LLM integration by providing memoized OpenAI clients and robust chat format transformation utilities. It helps developers significantly reduce API costs and latency through automatic caching while ensuring seamless compatibility between various chat formats like ChatML, ShareGPT, and raw text, making it an essential tool for efficient AI application development.

Use Cases

  • Cost and Latency Optimization: Use memoized OpenAI clients to cache API responses during iterative prompt engineering, preventing redundant billing and speeding up the development cycle.
  • Data Format Interoperability: Seamlessly transform conversation data between formats such as ShareGPT, ChatML, and plain text to ensure compatibility across different LLM providers and platforms.
  • LLM Fine-tuning Preparation: Convert raw chat logs or datasets into standardized formats required for training or fine-tuning local language models using built-in transformation logic.
  • Asynchronous Workflow Integration: Implement high-performance, non-blocking LLM calls with persistent caching for scalable Python applications and agentic workflows.
name'llm-integration'
description'Guide for using LLM utilities in speedy_utils, including memoized OpenAI clients and chat format transformations.'

LLM Integration Guide

This skill provides comprehensive guidance for using the LLM utilities in speedy_utils.

When to Use This Skill

Use this skill when you need to:

  • Make OpenAI API calls with automatic caching (memoization) to save costs and time.
  • Transform chat messages between different formats (ChatML, ShareGPT, Text).
  • Prepare prompts for local LLM inference.

Prerequisites

  • speedy_utils installed.
  • openai package installed for API clients.

Core Capabilities

Memoized OpenAI Clients (MOpenAI, MAsyncOpenAI)

  • Drop-in replacements for OpenAI and AsyncOpenAI.
  • Automatically caches post (chat completion) requests.
  • Uses speedy_utils caching backend (disk/memory).
  • Configurable per-instance caching.

Chat Format Transformation (transform_messages)

  • Converts between:
    • chatml: List of {"role": "...", "content": "..."} dicts.
    • sharegpt: Dict with {"conversations": [{"from": "...", "value": "..."}]}.
    • text: String with <|im_start|> tokens.
    • simulated_chat: Human/AI transcript format.
  • Supports applying tokenizer templates.

Usage Examples

Example 1: Memoized OpenAI Call

Make repeated calls without hitting the API twice.

from llm_utils.lm.openai_memoize import MOpenAI

# Initialize just like OpenAI client
client = MOpenAI(api_key="sk-...")

# First call hits the API
response1 = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello"}]
)

# Second call returns cached result instantly
response2 = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello"}]
)

Example 2: Async Memoized Call

Same as above but for async workflows.

from llm_utils.lm.openai_memoize import MAsyncOpenAI
import asyncio

async def main():
    client = MAsyncOpenAI(api_key="sk-...")
    response = await client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": "Hi"}]
    )

Example 3: Transforming Chat Formats

Convert ShareGPT format to ChatML.

from llm_utils.chat_format.transform import transform_messages

sharegpt_data = {
    "conversations": [
        {"from": "human", "value": "Hi"},
        {"from": "gpt", "value": "Hello there"}
    ]
}

# Convert to ChatML list
chatml_data = transform_messages(sharegpt_data, frm="sharegpt", to="chatml")
# Result: [{'role': 'user', 'content': 'Hi'}, {'role': 'assistant', 'content': 'Hello there'}]

# Convert to Text string
text_data = transform_messages(chatml_data, frm="chatml", to="text")
# Result: "<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\nHello there<|im_end|>\n<|im_start|>assistant\n"

Guidelines

  1. Caching Behavior:

    • The cache key is generated from the arguments passed to create.
    • If you change any parameter (e.g., temperature, model), it counts as a new request.
    • Cache is persistent if configured (default behavior of memoize).
  2. Format Detection:

    • transform_messages tries to auto-detect input format, but it's safer to specify frm explicitly.
  3. Tokenizer Support:

    • You can pass a HuggingFace tokenizer to transform_messages to use its specific chat template.

Limitations

  • Streaming: Memoization does NOT work with streaming responses (stream=True).
  • Side Effects: If your LLM calls rely on randomness (high temperature) and you want different results each time, disable caching or change the seed/input.