test-mcp-connector

majiayu000 的头像作者 majiayu000

仅当用户明确请求基于 MCP 的测试时才触发此技能:**必需的触发器(全部必须明确提及“MCP”):**-“使用 mcp 测试连接器”-“测试 mcp 连接器”-“使用 mcp 测试 [提供程序]”-“使用 mcp 测试 [提供程序]”-“运行 mcp 连接器测试”-“针对 [提供程序]”进行 mcp 测试“**不要触发:**- 通用“测试连接器”请求(使用 stackone 运行) / test_actions 相反) - 没有明确提及 MCP 的“测试 [provider]” - 定期验证或测试请求 - 任何未明确提及 MCP 的测试此技能使用 Claude Agent SDK 构建一个 REAL 代理,发送自然语言提示来评估操作描述是否对代理友好。它比常规测试更加密集,并且仅应在明确要求时使用。

📖 中文导读:仅当用户明确请求基于 MCP 的测试时才触发此技能:**必需的触发器(全部必须明确提及“MCP”):**-“使用 mcp 测试连接器”-“测试 mcp 连接器”-“使用 mcp 测试 [提供程序]”-“使用 mcp 测试 [提供程序]”-“运行 mcp 连接器测试”-“针对 [提供程序]”进行 mcp 测试“**不要触发:**- 通用“测试连接器”请求(使用 stackone 运行) / test_actions 相反) - 没有明确提及 MCP 的“测试 [provider]” - 定期验证或测试请求 - 任何未明确提及 MCP 的测试此技能使用 Claude Agent SDK 构建一个 REAL 代理,发送自然语言提示来评估操作描述是否对代理友好。它比常规测试更加密集,并且仅应在明确要求时使用。 以下正文为该技能的原始 SKILL.md 内容(英文),上方安装命令可直接使用。

name: test-mcp-connector description: | ONLY trigger this skill when the user EXPLICITLY asks for MCP-based testing:

Required triggers (ALL must mention "MCP" explicitly):

  • "test connector with mcp"
  • "test mcp connector"
  • "test [provider] with mcp"
  • "use mcp to test [provider]"
  • "run mcp connector test"
  • "mcp test for [provider]"

DO NOT trigger for:

  • Generic "test the connector" requests (use stackone run / test_actions instead)
  • "test [provider]" without explicit MCP mention
  • Regular validation or testing requests
  • Any testing that doesn't explicitly mention MCP

This skill builds a REAL agent with Claude Agent SDK that sends natural language prompts to evaluate if action descriptions are agent-friendly. It's more intensive than regular testing and should only be used when explicitly requested.

MCP Connector Testing

Setup (REQUIRED - Collect ALL upfront)

Required Description
Account ID StackOne account ID (linked provider account)
StackOne API Key StackOne API key (credentials:read scope)
Anthropic API Key For the test agent (check ANTHROPIC_API_KEY env, ask if not set)
Profile CLI profile for stackone push
Provider Provider name (e.g., datadog, intercom)

⛔ Account ID = Complete Authentication

If the user provides an Account ID, you have EVERYTHING needed for authentication.

The Account ID is a linked account where provider credentials (API keys, tokens, subdomains, secrets) are ALREADY stored in StackOne.

When Account ID is provided, DO NOT ask for:

  • ❌ Provider API keys
  • ❌ Provider tokens or secrets
  • ❌ Any provider-specific credentials

Authentication = Account ID + StackOne API Key. Nothing else needed.


Phase 1: Quick Connectivity Check (Optional)

# Test single action
stackone run src/configs/<provider>/<provider>.connector.s1.yaml \
  --account-id <ACCOUNT_ID> --action-id <key>_<action> --profile=<profile>

# Push after changes
stackone push src/configs/<provider>/<provider>.connector.s1.yaml --profile=<profile>

Phase 2: Agent Simulation (PRIMARY)

Core Principle

Test via agent conversations, not direct tool calls. The goal: can an agent understand action descriptions and use them correctly?

// ❌ WRONG - bypasses agent understanding
await callMCPTool("provider_list_items", { path: {}, query: {} });

// ✅ CORRECT - agent decides what to call
await agent.chat("Show me all items in the system");

Test Agent Setup

All test files: test-agent/<provider>/ (gitignored)

mkdir -p test-agent/<provider> && cd test-agent/<provider>
npm init -y && npm install @anthropic-ai/claude-agent-sdk tsx

Agent code:

import { query } from "@anthropic-ai/claude-agent-sdk";

const STACKONE_API_KEY = process.env.STACKONE_API_KEY;
const ACCOUNT_ID = process.env.ACCOUNT_ID;

async function testPrompt(prompt: string) {
  const toolCalls: any[] = [];
  let result: any;

  for await (const message of query({
    prompt,
    options: {
      model: "claude-haiku-4-5",
      mcpServers: {
        "stackone": {
          type: "http",
          url: "https://api.stackone.com/mcp",
          headers: {
            "Authorization": `Basic ${Buffer.from(`${STACKONE_API_KEY}:`).toString("base64")}`,
            "x-account-id": ACCOUNT_ID
          }
        }
      },
      allowedTools: ["mcp__stackone__*"]
    }
  })) {
    if (message.type === "assistant" && message.message?.content) {
      toolCalls.push(...message.message.content.filter((b: any) => b.type === "tool_use"));
    }
    if (message.type === "result" && message.subtype === "success") {
      result = message.result;
    }
  }
  return { prompt, toolCalls, result };
}

Run: cd test-agent/<provider> && npx tsx test.ts

Testing Process

For EACH action in the connector:

  1. Generate realistic prompt (natural language, no action names)
  2. Send to agent, capture: tool called, params, interpretation
  3. Evaluate: ✅ Right tool + params = PASS | ❌ Wrong = FAIL → fix

Include multi-turn workflows:

User: "Show me items in category X"     → list_items
User: "Get details on the first one"    → get_item (uses ID from previous)
User: "Update its status to active"     → update_item (same ID)

Coverage Requirements

  • Test ALL actions - if connector has 50 actions, test 50 actions
  • Track progress: [15/50] action_name... ✅ PASS
  • Include single-turn AND multi-turn tests
  • Don't skip write operations

Fix Loop

  1. Diagnose failure (description unclear? missing param? wrong tool?)
  2. Fix connector YAML
  3. Push: stackone push <path> --profile=<profile>
  4. Retry same prompt → verify fix
  5. Continue to next action

Output

## Test Results: <provider> connector

### Summary
- Total actions: 50 | Tested: 50 (100%)
- Initial pass rate: 85% (42/50)
- Final pass rate: 100% (50/50)
- Fixes applied: 8

### Changes Made
| Action | Issue | Fix |
|--------|-------|-----|
| list_items | Agent confused with search | Added "primary listing endpoint" to description |
| create_item | Missing required field | Added `category_id` to required params |

### Issue Patterns
- Description clarity: 5 fixes
- Missing params: 2 fixes
- Naming conflicts: 1 fix

### Files Modified
- src/configs/<provider>/<provider>.connector.s1.yaml

Required in report: Total vs tested (must equal), before/after pass rates, every fix, files modified.