writing-ast-check

jack-michaud's avatarfrom jack-michaud

Create AST-based checks for evaluating generated Python code quality

⭐3stars🔀0forks📁View on GitHub🕐Updated Dec 29, 2025
[Evaluation and Benchmarks]

When & Why to Use This Skill

This Claude skill provides a specialized framework for creating AST-based (Abstract Syntax Tree) checks to programmatically evaluate the quality, syntax, and architectural integrity of AI-generated Python code. It enables developers to move beyond simple string matching by analyzing the actual structure of the code, ensuring that generated outputs adhere to modern standards and specific project requirements within automated evaluation pipelines.

Use Cases

  • Automated Syntax Enforcement: Verify that AI-generated code utilizes modern Python features, such as PEP 604 union types (str | None), rather than legacy typing conventions.
  • LLM Output Quality Assurance: Implement rigorous programmatic checks within evaluation frameworks to score the reliability, maintainability, and stylistic consistency of code produced by Large Language Models.
  • Custom Benchmarking for Agents: Develop specialized evaluation suites that measure an agent's ability to follow complex coding instructions and avoid specific anti-patterns in a production-like environment.
  • Regression Testing for Code Generation: Integrate AST helpers into CI/CD pipelines to ensure that updates to AI prompts or models do not degrade the structural quality of the generated code.
namewriting-ast-check
descriptionCreate AST-based checks for evaluating generated Python code quality

Writing AST-Based Checks for Evals

This process documents how to create robust, AST-based checks for evaluating the quality of AI-generated Python code.

Process

1. Define the Check Criteria

Clearly specify what you're checking for:

Example: We want to ensure generated code uses modern Python union syntax (str | None) instead of legacy Optional[str] from the typing module.

Acceptance criteria:

  • ✅ Files using | None syntax
  • ✅ Files with no optional types at all
  • ❌ Files using Optional[T] syntax

2. Add the check function to the AST Helper Module

Create a dedicated function in AST helpers: evals/<eval_name>/ast_helpers.py

3. Add Check to EvalResult

Update evals/<eval_name>/checks.py:

class EvalResult:
    # Used the writing-services skill
    used_service_skill = Check(default=False, passed=True)

    # Used | None instead of Optional
    used_none_instead_of_optional = Check(default=False, passed=True)

    def to_dict(self) -> dict:
        """Convert eval results to a dictionary for logging."""
        return {
            "used_service_skill": self.used_service_skill.did_pass(),
            "used_none_instead_of_optional": self.used_none_instead_of_optional.did_pass(),
        }

4. Integrate Check in Eval

Update __main__.py to run the check after the agent completes:

from .ast_helpers import check_uses_union_none_syntax

async def main(gym_project_directory: Path) -> None:
    # ... agent execution code ...

    # Check if the generated logger.py uses | None instead of Optional
    logger_file_path = gym_project_directory / "jack-software/evals/logger.py"
    if check_uses_union_none_syntax(logger_file_path):
        eval_result.used_none_instead_of_optional.mark(True)

    # ... logging code ...

5. Write Comprehensive Tests

Add a class to the test_ast_helpers.py with test cases.

Testing Your Check

Run the tests:

make test-services-eval