Files

David Culbreth 3b14c65998 re-uploading work

2026-02-04 17:46:30 -06:00

8.7 KiB

Raw Blame History

Pack Installation with Testing Integration

Overview

Pack installation and registration now includes automatic test execution to validate that packs work correctly in the target environment. This provides fail-fast validation and ensures quality across the pack ecosystem.

Features

Automatic Test Execution: Tests run automatically during pack installation/registration
Fail-Fast Validation: Installation fails if tests don't pass (unless forced)
Test Result Storage: All test results are stored in the database for audit trails
Flexible Control: Skip tests or force installation with command-line flags
Test Result Display: CLI shows test results with pass/fail status

Installation Methods

1. Register Pack (Local Directory)

# Basic registration with automatic testing
attune pack register /path/to/pack

# Force registration even if pack already exists
attune pack register /path/to/pack --force

# Skip tests during registration
attune pack register /path/to/pack --skip-tests

# Combine flags
attune pack register /path/to/pack --force --skip-tests

How it works:

Reads pack.yaml from the directory
Creates pack record in database
Syncs workflows from pack directory
Runs pack tests (if not skipped)
Fails registration if tests fail (unless --force is used)
Stores test results in database

2. Install Pack (Remote Source)

Status: Not yet implemented

Install a pack from a git repository or remote source:

# Install from git repository (future)
attune pack install https://github.com/attune/pack-slack.git

# Install specific version (future)
attune pack install https://github.com/attune/pack-slack.git --ref v1.0.0

This feature will be implemented in a future release.

API Endpoints

Register Pack

Endpoint: POST /api/v1/packs/register

Request Body:

{
  "path": "/path/to/pack/directory",
  "force": false,
  "skip_tests": false
}

Response (201 Created):

{
  "data": {
    "pack": {
      "id": 1,
      "ref": "mypack",
      "label": "My Pack",
      "version": "1.0.0",
      ...
    },
    "test_result": {
      "pack_ref": "mypack",
      "pack_version": "1.0.0",
      "status": "passed",
      "total_tests": 10,
      "passed": 10,
      "failed": 0,
      "skipped": 0,
      "pass_rate": 1.0,
      "duration_ms": 1234,
      "test_suites": [...]
    },
    "tests_skipped": false
  },
  "message": "Pack registered successfully"
}

Error Response (400 Bad Request) - Tests Failed:

{
  "error": "Pack registration failed: tests did not pass. Use force=true to register anyway.",
  "code": "BAD_REQUEST"
}

Install Pack

Endpoint: POST /api/v1/packs/install

Status: Returns 501 Not Implemented

Request Body:

{
  "source": "https://github.com/attune/pack-slack.git",
  "ref_spec": "main",
  "force": false,
  "skip_tests": false
}

This endpoint will be implemented in a future release.

Test Execution Behavior

Default Behavior (Tests Enabled)

When registering a pack with tests configured in pack.yaml:

Tests are automatically executed after pack creation
Test results are stored in the pack_test_execution table
Registration fails if any test fails
Pack record is rolled back if tests fail (unless --force is used)

Skip Tests (`--skip-tests` flag)

Use this flag when:

Tests are known to be slow or flaky
Testing in a non-standard environment
Manually verifying pack functionality later

attune pack register /path/to/pack --skip-tests

Behavior:

Tests are not executed
No test results are stored
Registration always succeeds (no validation)
Response includes "tests_skipped": true

Force Registration (`--force` flag)

Use this flag when:

Pack already exists and you want to reinstall
Tests are failing but you need to proceed anyway
Developing and iterating on pack tests

attune pack register /path/to/pack --force

Behavior:

Deletes existing pack if it exists
Tests still run (unless --skip-tests is also used)
Registration succeeds even if tests fail
Warning logged if tests fail

Combined Flags

# Force reinstall and skip tests entirely
attune pack register /path/to/pack --force --skip-tests

CLI Output Examples

Successful Registration with Tests

✓ Pack 'core' registered successfully
  Version: 0.1.0
  ID: 1
✓ All tests passed
  Tests: 76/76 passed

Failed Registration (Tests Failed)

✗ Error: Pack registration failed: tests did not pass. Use force=true to register anyway.

Registration with Skipped Tests

✓ Pack 'core' registered successfully
  Version: 0.1.0
  ID: 1
  Tests were skipped

Forced Registration with Failed Tests

⚠ Pack 'mypack' registered successfully
  Version: 1.0.0
  ID: 2
✗ Some tests failed
  Tests: 8/10 passed

Testing Requirements

For a pack to support automatic testing during installation:

pack.yaml must include a testing section:

testing:
  enabled: true
  test_suites:
    - name: "Unit Tests"
      runner: "python_unittest"
      working_dir: "tests"
      test_files:
        - "test_*.py"
      timeout_seconds: 60

Test files must exist in the specified locations
Tests must be executable in the target environment

See Pack Testing Framework for detailed test configuration.

Database Storage

All test executions are stored in the attune.pack_test_execution table:

pack_id: Reference to the pack
pack_version: Version tested
trigger_reason: How tests were triggered (e.g., "register", "manual")
total_tests, passed, failed, skipped: Test counts
pass_rate: Percentage of tests passed
duration_ms: Total execution time
result: Full test results as JSON

Query test history:

attune pack test-history core

Or via API:

curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/api/v1/packs/core/tests

Error Handling

Pack Directory Not Found

{
  "error": "Pack directory does not exist: /invalid/path",
  "code": "BAD_REQUEST"
}

Missing pack.yaml

{
  "error": "pack.yaml not found in directory: /path/to/pack",
  "code": "BAD_REQUEST"
}

No Testing Configuration

{
  "error": "No testing configuration found in pack.yaml for pack 'mypack'",
  "code": "BAD_REQUEST"
}

Testing Disabled

{
  "error": "Testing is disabled for pack 'mypack'",
  "code": "BAD_REQUEST"
}

Pack Already Exists

{
  "error": "Pack 'mypack' already exists. Use force=true to reinstall.",
  "code": "CONFLICT"
}

Best Practices

Development Workflow

During active development: Use --skip-tests for faster iteration
```
attune pack register ./my-pack --force --skip-tests
```
Before committing: Run tests explicitly to validate
```
attune pack test my-pack
```
For production: Let tests run automatically during registration
```
attune pack register ./my-pack
```

CI/CD Integration

In your CI/CD pipeline:

# Register pack (tests will run automatically)
attune pack register ./pack-directory

# Exit code will be non-zero if tests fail
echo $?  # 0 = success, 1 = failure

Production Deployment

For production deployments:

Never skip tests unless you have a specific reason
Use --force only when redeploying a known-good version
Monitor test results via the API or database
Set up alerts for test failures

Future Enhancements

Planned improvements to pack installation:

Remote Pack Installation: Install packs from git repositories
Dependency Resolution: Automatically install required packs
Version Management: Support multiple versions of the same pack
Async Testing: Return immediately and poll for test results
Test Result Comparison: Compare test results across versions
Webhooks: Notify external systems of test results
Pack Registry: Central repository for discovering and installing packs

Pack Testing Framework - Complete testing guide
Pack Testing API Reference - API documentation
Pack Development Guide - Creating packs
Pack Structure - pack.yaml format

Examples

See the packs/core directory for a complete example of a pack with testing enabled:

packs/core/pack.yaml - Testing configuration
packs/core/tests/ - Test files
packs/core/actions/ - Actions being tested

attune pack register packs/core

8.7 KiB Raw Blame History