ai-multimodal

nb150301 的头像作者 nb150301

使用Google Gemini API处理和生成多媒体内容。功能包括分析音频文件(带时间戳的转录、摘要、语音理解、音乐/声音分析最长9.5小时)、理解图像(标题、物体检测、OCR、视觉问答、分割)、处理视频(场景检测、问答、时间分析、YouTube网址最长6小时)、从文档中提取(PDF表格、表单、图表、图表、多页)、生成图像(使用Imagen 4的文本到图像、编辑、合成、优化)、生成视频(使用Veo 3的文本到视频、8秒片段带原生音频)。适用于处理音频/视频文件、分析图像或截图、处理PDF文档、从媒体中提取结构化数据、从文本提示创建图像/视频,或实现多模态AI功能。支持Gemini 3/2.5、Imagen 4和Veo 3模型,上下文窗口最大2M tokens。

📖 中文导读:使用Google Gemini API处理和生成多媒体内容。功能包括分析音频文件(带时间戳的转录、摘要、语音理解、音乐/声音分析最长9.5小时)、理解图像(标题、物体检测、OCR、视觉问答、分割)、处理视频(场景检测、问答、时间分析、YouTube网址最长6小时)、从文档中提取(PDF表格、表单、图表、图表、多页)、生成图像(使用Imagen 4的文本到图像、编辑、合成、优化)、生成视频(使用Veo 3的文本到视频、8秒片段带原生音频)。适用于处理音频/视频文件、分析图像或截图、处理PDF文档、从媒体中提取结构化数据、从文本提示创建图像/视频,或实现多模态AI功能。支持Gemini 3/2.5、Imagen 4和Veo 3模型,上下文窗口最大2M tokens。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: ai-multimodal description: Process and generate multimedia content using Google Gemini API. Capabilities include analyze audio files (transcription with timestamps, summarization, speech understanding, music/sound analysis up to 9.5 hours), understand images (captioning, object detection, OCR, visual Q&A, segmentation), process videos (scene detection, Q&A, temporal analysis, YouTube URLs, up to 6 hours), extract from documents (PDF tables, forms, charts, diagrams, multi-page), generate images (text-to-image with Imagen 4, editing, composition, refinement), generate videos (text-to-video with Veo 3, 8-second clips with native audio). Use when working with audio/video files, analyzing images or screenshots, processing PDF documents, extracting structured data from media, creating images/videos from text prompts, or implementing multimodal AI features. Supports Gemini 3/2.5, Imagen 4, and Veo 3 models with context windows up to 2M tokens. license: MIT allowed-tools: - Bash - Read - Write - Edit

AI Multimodal

Process audio, images, videos, documents, and generate images/videos using Google Gemini's multimodal API.

Setup

export GEMINI_API_KEY="your-key"  # Get from https://aistudio.google.com/apikey
pip install google-genai python-dotenv pillow

Quick Start

Analyze media: python scripts/gemini_batch_process.py --files <file> --task <analyze|transcribe|extract>

Generate content: python scripts/gemini_batch_process.py --task <generate|generate-video> --prompt "description"

Models

Image generation: imagen-4.0-generate-001 (standard), imagen-4.0-ultra-generate-001 (quality), imagen-4.0-fast-generate-001 (speed) Video generation: veo-3.1-generate-preview (8s clips with audio) Analysis: gemini-2.5-flash (recommended), gemini-2.5-pro (advanced)

Scripts

gemini_batch_process.py: Process audio/image/video/PDF, generate images/videos media_optimizer.py: Compress/resize/convert media for API limits document_converter.py: Convert DOCX/XLSX/PPTX to PDF

Use --help for options.

References

Load for detailed guidance:

Topic File
Audio references/audio-processing.md
Images references/vision-understanding.md
Image Gen references/image-generation.md
Video references/video-analysis.md
Video Gen references/video-generation.md

Limits

Formats: Audio (WAV/MP3/AAC, 9.5h), Images (PNG/JPEG/WEBP, 3.6k), Video (MP4/MOV, 6h), PDF (1k pages) Size: 20MB inline, 2GB File API

Resources: API Docs | Pricing