attune-system/attune

Fork 0

Files

David Culbreth 3b14c65998 re-uploading work

2026-02-04 17:46:30 -06:00

13 KiB

Raw Blame History

Pack Installation from Git Repositories

Last Updated: 2025-01-27
Status: Production Ready

Overview

Attune supports installing packs directly from git repositories, enabling teams to:

Version Control: Track pack versions using git tags, branches, and commits
Collaboration: Share packs across teams via git hosting services
Automation: Integrate pack deployment into CI/CD pipelines
Development: Test packs from feature branches before release

Quick Start

Web UI Installation

Navigate to Packs page
Click Add Pack dropdown → Install from Remote
Select Git Repository as source type
Enter repository URL (HTTPS or SSH)
Optionally specify a git reference (branch, tag, or commit)
Configure installation options
Click Install Pack

CLI Installation

# Install from default branch
attune pack install https://github.com/example/pack-slack.git

# Install from specific tag
attune pack install https://github.com/example/pack-slack.git --ref v2.1.0

# Install from branch
attune pack install https://github.com/example/pack-slack.git --ref main

# Install from commit hash
attune pack install https://github.com/example/pack-slack.git --ref a1b2c3d

# SSH URL
attune pack install git@github.com:example/pack-slack.git --ref v2.1.0

# Skip tests and dependency validation (use with caution)
attune pack install https://github.com/example/pack-slack.git --skip-tests --skip-deps

API Installation

curl -X POST http://localhost:8080/api/v1/packs/install \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "source": "https://github.com/example/pack-slack.git",
    "ref_spec": "v2.1.0",
    "force": false,
    "skip_tests": false,
    "skip_deps": false
  }'

Supported Git Sources

HTTPS URLs

Public repositories:

https://github.com/username/pack-name.git
https://gitlab.com/username/pack-name.git
https://bitbucket.org/username/pack-name.git

Private repositories with credentials:

https://username:token@github.com/username/pack-name.git

Security Note: For private repositories, use SSH keys or configure git credential helpers instead of embedding credentials in URLs.

SSH URLs

Standard format:

git@github.com:username/pack-name.git
git@gitlab.com:username/pack-name.git

SCP-style:

user@server:path/to/pack.git

Requirements:

SSH keys must be configured on the Attune server
User running Attune service must have access to private key
Host must be in ~/.ssh/known_hosts

Git References

The ref_spec parameter accepts any valid git reference:

Branches

# Default branch (usually main or master)
--ref main

# Development branch
--ref develop

# Feature branch
--ref feature/new-action

Commit Hashes

# Full commit hash
--ref a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0

# Short commit hash (7+ characters)
--ref a1b2c3d

Special References

# HEAD of default branch
--ref HEAD

# No ref specified = default branch
# (equivalent to --depth 1 clone)

Pack Structure Requirements

The git repository must contain a valid pack structure:

Option 1: Root-Level Pack

repository-root/
├── pack.yaml          # Required
├── actions/           # Optional
├── sensors/           # Optional
├── triggers/          # Optional
├── rules/             # Optional
├── workflows/         # Optional
└── README.md          # Recommended

Option 2: Pack Subdirectory

repository-root/
├── pack/
│   ├── pack.yaml      # Required
│   ├── actions/
│   └── ...
├── docs/
└── README.md

The installer will automatically detect and use the pack directory.

Installation Process

When you install a pack from git, Attune performs these steps:

1. Clone Repository

git clone [--depth 1] <url> <temp-dir>

If a ref_spec is provided:

git checkout <ref_spec>

2. Locate Pack Directory

Search for pack.yaml at repository root
If not found, search in pack/ subdirectory
Fail if pack.yaml not found in either location

3. Validate Dependencies (unless skipped)

Extract runtime dependencies from pack.yaml
Extract pack dependencies
Verify all dependencies are satisfied
Fail if dependencies are missing (unless force enabled)

4. Register Pack

Parse pack.yaml metadata
Create database entry
Auto-sync workflows if present

5. Execute Tests (unless skipped)

Run pack test suite if configured
Fail if tests don't pass (unless force enabled)

6. Copy to Permanent Storage

Move pack files to {packs_base_dir}/{pack_ref}/
Calculate directory checksum
Store installation metadata

7. Record Installation

Store installation record in pack_installation table
Record source URL, git ref, timestamp, checksum
Link to installing user

Installation Options

Force Installation

Flag: --force (CLI) or force: true (API)

Enables:

Reinstall pack even if it already exists (replaces existing)
Proceed even if dependencies are missing
Proceed even if tests fail

Use Cases:

Upgrading pack to new version
Recovering from failed installation
Development/testing workflows

Warning: Force mode bypasses safety checks. Use cautiously in production.

Skip Tests

Flag: --skip-tests (CLI) or skip_tests: true (API)

Skip executing pack test suite
Faster installation
Useful when tests are slow or not available

Use Cases:

Installing trusted packs
Tests not yet implemented
Development environments

Skip Dependencies

Flag: --skip-deps (CLI) or skip_deps: true (API)

Skip validation of runtime dependencies
Skip validation of pack dependencies
May result in runtime failures if dependencies truly missing

Use Cases:

Dependencies will be installed separately
Custom runtime environment
Development/testing

Example Workflows

Development Workflow

# 1. Install pack from feature branch for testing
attune pack install https://github.com/myorg/pack-custom.git \
  --ref feature/new-sensor \
  --skip-tests \
  --force

# 2. Test the pack
attune pack test custom

# 3. When satisfied, install from main/release tag
attune pack install https://github.com/myorg/pack-custom.git \
  --ref v1.0.0 \
  --force

Production Deployment

# Install specific version with full validation
attune pack install https://github.com/myorg/pack-slack.git \
  --ref v2.1.0

# Verify installation
attune pack list | grep slack
attune pack test slack

CI/CD Pipeline

# GitHub Actions example
- name: Install pack on staging
  run: |
    attune pack install https://github.com/${{ github.repository }}.git \
      --ref ${{ github.sha }} \
      --force
  env:
    ATTUNE_API_TOKEN: ${{ secrets.ATTUNE_TOKEN }}

- name: Run tests
  run: attune pack test $(basename ${{ github.repository }})

Private Repository with SSH

# 1. Set up SSH key on Attune server
ssh-keygen -t ed25519 -C "attune@example.com"
cat ~/.ssh/id_ed25519.pub  # Add to GitHub/GitLab

# 2. Add host to known_hosts
ssh-keyscan github.com >> ~/.ssh/known_hosts

# 3. Install pack
attune pack install git@github.com:myorg/private-pack.git --ref main

Troubleshooting

Git Clone Fails

Error: Git clone failed: Permission denied

Solutions:

Verify SSH keys are configured correctly
Check repository access permissions
For HTTPS, verify credentials or token
Add host to ~/.ssh/known_hosts

Ref Not Found

Error: Git checkout failed: pathspec 'v1.0.0' did not match any file(s) known to git

Solutions:

Verify tag/branch exists in repository
Check spelling and case sensitivity
Ensure ref is pushed to remote
Try with full commit hash

Pack.yaml Not Found

Error: pack.yaml not found in directory

Solutions:

Ensure pack.yaml exists at repository root or in pack/ subdirectory
Check file name spelling (case-sensitive on Linux)
Verify correct branch/tag is checked out

Dependency Validation Failed

Error: Pack dependency validation failed: pack 'core' version '^1.0.0' not found

Solutions:

Install missing dependencies first
Use --skip-deps to bypass validation (not recommended)
Use --force to install anyway

Test Failures

Error: Pack registration failed: tests did not pass

Solutions:

Review test output for specific failures
Fix issues in pack code
Use --skip-tests to install without testing
Use --force to install despite test failures

Security Considerations

SSH Keys

Use dedicated SSH key for Attune service
Restrict key permissions (read-only access preferred)
Rotate keys periodically
Use SSH agent for key management

HTTPS Authentication

Never embed credentials directly in URLs
Use git credential helpers
Consider personal access tokens with limited scope
Rotate tokens regularly

Code Review

Review pack code before installation
Install from tagged releases, not branches
Verify pack author/source
Check for malicious code in actions/sensors

Git References

Production: Use specific tags (e.g., v1.2.3)
Staging: Use release branches (e.g., release-*)
Development: Feature branches acceptable
Avoid: main/master in production (may change unexpectedly)

Advanced Topics

Submodules

If pack repository uses git submodules:

# Clone with submodules
git clone --recurse-submodules <url>

Note

: Current implementation does not automatically clone submodules. Manual configuration required.

Large Repositories

For large repositories, use shallow clones:

# Default behavior when no ref_spec
git clone --depth 1 <url>

When specific ref is needed:

git clone <url>
git checkout <ref>

Monorepos

For repositories containing multiple packs:

monorepo/
├── pack-a/
│   └── pack.yaml
├── pack-b/
│   └── pack.yaml
└── pack-c/
    └── pack.yaml

Limitation: Current implementation expects one pack per repository. For monorepos, use filesystem registration or create separate repositories.

Database Schema

Installation metadata is stored in the pack_installation table:

CREATE TABLE pack_installation (
  id BIGSERIAL PRIMARY KEY,
  pack_id BIGINT NOT NULL REFERENCES pack(id),
  source_type VARCHAR(50) NOT NULL,        -- 'git'
  source_url TEXT,                          -- Git repository URL
  source_ref TEXT,                          -- Branch/tag/commit
  checksum TEXT,                            -- Directory checksum
  checksum_verified BOOLEAN DEFAULT FALSE,
  installed_by BIGINT REFERENCES identity(id),
  installation_method VARCHAR(50),          -- 'api', 'cli', 'web'
  storage_path TEXT NOT NULL,               -- File system path
  meta JSONB,                               -- Additional metadata
  created TIMESTAMP DEFAULT NOW(),
  updated TIMESTAMP DEFAULT NOW()
);

API Reference

Install Pack Endpoint

POST /api/v1/packs/install

Request Body:

{
  "source": "https://github.com/example/pack-slack.git",
  "ref_spec": "v2.1.0",
  "force": false,
  "skip_tests": false,
  "skip_deps": false
}

Response (201 Created):

{
  "data": {
    "pack": {
      "id": 42,
      "ref": "slack",
      "label": "Slack Pack",
      "version": "2.1.0",
      "description": "Slack integration pack",
      "is_standard": false,
      ...
    },
    "test_result": {
      "status": "passed",
      "total_tests": 10,
      "passed": 10,
      "failed": 0,
      ...
    },
    "tests_skipped": false
  }
}

Error Responses:

400: Invalid request, dependency validation failed, or tests failed
409: Pack already exists (use force: true to override)
500: Internal error (git clone failed, filesystem error, etc.)

Future Enhancements

Planned Features

Git submodule support
Monorepo support (install specific subdirectory)
Pack version upgrade workflow
Automatic version detection from tags
Git LFS support
Signature verification for signed commits

Registry Integration

When pack registry system is implemented:

Registry can reference git repositories
Automatic version discovery from git tags
Centralized pack metadata
See pack-registry-spec.md for details

Pack Structure - Pack directory and file format
Pack Registry Specification - Registry-based installation
Pack Testing Framework - Testing packs
Configuration - Server configuration
Production Deployment - Deployment guide

Examples Repository

Example packs demonstrating git-based installation:

# Simple action pack
attune pack install https://github.com/attune-examples/pack-hello-world.git

# Complex pack with dependencies
attune pack install https://github.com/attune-examples/pack-kubernetes.git --ref v1.0.0

# Private repository (SSH)
attune pack install git@github.com:mycompany/pack-internal.git --ref main

Support

For issues or questions:

GitHub Issues: https://github.com/attune-io/attune/issues
Documentation: https://docs.attune.io/packs/installation
Community: https://community.attune.io

13 KiB Raw Blame History

Pack Installation from Git Repositories

Overview

Quick Start

Web UI Installation

CLI Installation

API Installation

Supported Git Sources

HTTPS URLs

SSH URLs

Git References

Branches

Tags

Commit Hashes

Special References

Pack Structure Requirements

Option 1: Root-Level Pack

Option 2: Pack Subdirectory

Installation Process

1. Clone Repository

2. Locate Pack Directory

3. Validate Dependencies (unless skipped)

4. Register Pack

5. Execute Tests (unless skipped)

6. Copy to Permanent Storage

7. Record Installation

Installation Options

Force Installation

Skip Tests

Skip Dependencies

Example Workflows

Development Workflow

Production Deployment

CI/CD Pipeline

Private Repository with SSH

Troubleshooting

Git Clone Fails

Ref Not Found

Pack.yaml Not Found

Dependency Validation Failed

Test Failures

Security Considerations

SSH Keys

HTTPS Authentication

Code Review

Git References

Advanced Topics

Submodules

Large Repositories

Monorepos

Database Schema

API Reference

Install Pack Endpoint

Future Enhancements

Planned Features

Registry Integration

Related Documentation

Examples Repository

Support

13 KiB

Raw Blame History