attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
343488b3ebc02af2e7c3a23a4ea1f614e0b93ebd
attune/docs/packs/pack-registry-spec.md
David Culbreth 3b14c65998 re-uploading work
2026-02-04 17:46:30 -06:00

841 lines
27 KiB
Markdown
Raw Blame History

# Pack Registry and Installation Specification
**Last Updated**: 2024-01-20
**Status**: Specification (Pre-Implementation)
---
## Overview
This document specifies the pack registry and installation system for Attune, enabling community-driven pack distribution. The system allows packs to be:
- Published to independent registries (no central authority required)
- Installed from git repositories, HTTP/HTTPS URLs, or local sources
- Discovered through configurable registry indices
- Validated and tested during installation
---
## Design Goals
1. **Decentralized**: No single point of failure; anyone can host a registry
2. **CI/CD Friendly**: Integrate with existing build and artifact storage systems
3. **Flexible Sources**: Support multiple installation sources (git, HTTP, local)
4. **Priority-Based Discovery**: Search multiple registries in configured order
5. **Secure**: Validate checksums and signatures (future)
6. **Automated**: Install dependencies, run tests, register components automatically
---
## Pack Index File Format
### Index Structure
Each registry hosts an **index file** (typically `index.json`) that catalogs available packs.
**Format**: JSON
**Location**: Configurable URL (HTTPS recommended)
**Filename Convention**: `index.json` or `registry.json`
### Index Schema
```json
{
"registry_name": "Attune Community Registry",
"registry_url": "https://registry.attune.io",
"version": "1.0",
"last_updated": "2024-01-20T12:00:00Z",
"packs": [
{
"ref": "slack",
"label": "Slack Integration",
"description": "Send messages, upload files, and monitor Slack channels",
"version": "2.1.0",
"author": "Attune Team",
"email": "team@attune.io",
"homepage": "https://github.com/attune-io/pack-slack",
"repository": "https://github.com/attune-io/pack-slack",
"license": "Apache-2.0",
"keywords": ["slack", "messaging", "notifications"],
"runtime_deps": ["python3"],
"
"install_sources": [
{
"type": "git",
"url": "https://github.com/attune-io/pack-slack.git",
"ref": "v2.1.0",
"checksum": "sha256:abc123..."
},
{
"type": "archive",
"url": "https://github.com/attune-io/pack-slack/archive/refs/tags/v2.1.0.zip",
"checksum": "sha256:def456..."
}
],
"contents": {
"actions": [
{
"name": "send_message",
"description": "Send a message to a Slack channel"
},
{
"name": "upload_file",
"description": "Upload a file to Slack"
}
],
"sensors": [
{
"name": "message_sensor",
"description": "Monitor Slack messages"
}
],
"triggers": [
{
"name": "message_received",
"description": "Fires when a message is received"
}
],
"rules": [],
"workflows": []
},
"dependencies": {
"attune_version": ">=0.1.0",
"python_version": ">=3.9",
"packs": []
},
"meta": {
"downloads": 1543,
"stars": 87,
"tested_attune_versions": ["0.1.0", "0.2.0"]
}
}
]
}
```
### Field Definitions
#### Registry Metadata
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `registry_name` | string | Yes | Human-readable registry name |
| `registry_url` | string | Yes | Registry homepage URL |
| `version` | string | Yes | Index format version (semantic versioning) |
| `last_updated` | string | Yes | ISO 8601 timestamp of last update |
| `packs` | array | Yes | Array of pack entries |
#### Pack Entry
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `ref` | string | Yes | Unique pack identifier (matches pack.yaml) |
| `label` | string | Yes | Human-readable pack name |
| `description` | string | Yes | Brief pack description |
| `version` | string | Yes | Semantic version (latest available) |
| `author` | string | Yes | Pack author/maintainer name |
| `email` | string | No | Contact email |
| `homepage` | string | No | Pack homepage URL |
| `repository` | string | No | Source repository URL |
| `license` | string | Yes | SPDX license identifier |
| `keywords` | array[string] | No | Searchable keywords/tags |
| `runtime_deps` | array[string] | Yes | Required runtimes (python3, nodejs, shell) |
| `install_sources` | array[object] | Yes | Available installation sources (see below) |
| `contents` | object | Yes | Pack components summary |
| `dependencies` | object | No | Pack dependencies |
| `meta` | object | No | Additional metadata |
#### Install Source
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `type` | string | Yes | Source type: "git" or "archive" |
| `url` | string | Yes | Source URL |
| `ref` | string | No | Git ref (tag, branch, commit) for git type |
| `checksum` | string | Yes | Format: "algorithm:hash" (e.g., "sha256:abc...") |
#### Contents Object
| Field | Type | Description |
|-------|------|-------------|
| `actions` | array[object] | List of actions with name and description |
| `sensors` | array[object] | List of sensors with name and description |
| `triggers` | array[object] | List of triggers with name and description |
| `rules` | array[object] | List of bundled rules |
| `workflows` | array[object] | List of bundled workflows |
#### Dependencies Object
| Field | Type | Description |
|-------|------|-------------|
| `attune_version` | string | Semver requirement (e.g., ">=0.1.0", "^1.0.0") |
| `python_version` | string | Python version requirement |
| `nodejs_version` | string | Node.js version requirement |
| `packs` | array[string] | Pack dependencies (format: "ref@version") |
---
## Pack Sources
Packs can be installed from multiple source types:
### 1. Git Repository
Install directly from a git repository:
```bash
attune pack install https://github.com/example/pack-slack.git
attune pack install https://github.com/example/pack-slack.git --ref v2.1.0
attune pack install https://github.com/example/pack-slack.git --ref main
attune pack install git@github.com:example/pack-slack.git --ref v2.1.0
```
**Requirements**:
- Repository must contain valid pack structure at root or in `pack/` subdirectory
- `pack.yaml` must be present
- Git client must be installed on system
### 2. Archive URL
Install from a zip or tar.gz archive:
```bash
attune pack install https://example.com/packs/slack-2.1.0.zip
attune pack install https://example.com/packs/slack-2.1.0.tar.gz
```
**Requirements**:
- Archive must contain pack directory structure
- Archive root or single top-level directory must contain `pack.yaml`
- Supported formats: `.zip`, `.tar.gz`, `.tgz`
### 3. Local Directory
Install from a local filesystem path:
```bash
attune pack install /path/to/pack-slack
attune pack install ./packs/my-pack
```
**Requirements**:
- Directory must contain valid pack structure
- `pack.yaml` must be present
- Used for development and testing
### 4. Local Archive
Upload and install from a local archive file:
```bash
attune pack install /path/to/pack-slack-2.1.0.zip
attune pack install ./my-pack.tar.gz
```
**Requirements**:
- Archive must contain valid pack structure
- Archive is uploaded to Attune API before installation
- Used for air-gapped or offline installations
### 5. Registry Reference
Install by pack reference, searching configured registries:
```bash
attune pack install slack
attune pack install slack@2.1.0
attune pack install slack@latest
```
**Requirements**:
- At least one registry must be configured
- Pack reference must exist in one of the registries
- Registries searched in configured priority order
---
## Configuration
### Registry Configuration
Add registry URLs to service configuration files:
**YAML Configuration** (`config.yaml`):
```yaml
pack_registry:
enabled: true
indices:
- url: https://registry.attune.io/index.json
priority: 1
enabled: true
name: "Official Attune Registry"
- url: https://company-internal.example.com/attune-registry.json
priority: 2
enabled: true
name: "Company Internal Registry"
headers:
Authorization: "Bearer ${REGISTRY_TOKEN}"
- url: file:///opt/attune/local-registry.json
priority: 3
enabled: true
name: "Local Filesystem Registry"
# Cache settings
cache_ttl: 3600 # Cache index for 1 hour
cache_enabled: true
# Download settings
timeout: 120
verify_checksums: true
allow_http: false # Only allow HTTPS
```
**Environment Variables**:
```bash
# Enable/disable registry
export ATTUNE__PACK_REGISTRY__ENABLED=true
# Set registry URLs (comma-separated, in priority order)
export ATTUNE__PACK_REGISTRY__INDICES="https://registry.attune.io/index.json,https://internal.example.com/registry.json"
# Cache settings
export ATTUNE__PACK_REGISTRY__CACHE_TTL=3600
export ATTUNE__PACK_REGISTRY__VERIFY_CHECKSUMS=true
```
### Priority-Based Search
Registries are searched in **priority order** (lowest priority number first):
1. **Priority 1**: Official Attune Registry (public packs)
2. **Priority 2**: Company Internal Registry (private packs)
3. **Priority 3**: Local Filesystem Registry (development packs)
When installing by reference (e.g., `attune pack install slack`):
- Search priority 1 registry first
- If not found, search priority 2
- If not found, search priority 3
- If not found in any registry, return error
**Use Cases**:
- **Override public packs**: Company registry can provide custom version of "slack" pack
- **Private packs**: Internal registry can host proprietary packs
- **Development**: Local registry can provide development versions
### Registry Headers
For authenticated registries, configure custom HTTP headers:
```yaml
pack_registry:
indices:
- url: https://private-registry.example.com/index.json
headers:
Authorization: "Bearer ${PRIVATE_REGISTRY_TOKEN}"
X-Custom-Header: "value"
```
---
## CLI Commands
### Install Pack
```bash
# From registry (by reference)
attune pack install <pack-ref>[@version]
# From git repository
attune pack install <git-url> [--ref <branch|tag|commit>]
# From archive URL
attune pack install <https-url>
# From local directory
attune pack install <local-path>
# From local archive
attune pack install <local-archive-path>
# Options
--force # Force reinstall if already exists
--skip-tests # Skip running pack tests
--skip-deps # Skip installing dependencies
--registry <name> # Use specific registry (skip priority search)
--no-registry # Don't search registries (direct install only)
```
### Examples
```bash
# Install latest version from registry
attune pack install slack
# Install specific version from registry
attune pack install slack@2.1.0
# Install from git repository (latest tag)
attune pack install https://github.com/example/pack-slack.git
# Install from git repository (specific tag)
attune pack install https://github.com/example/pack-slack.git --ref v2.1.0
# Install from git repository (branch)
attune pack install https://github.com/example/pack-slack.git --ref main
# Install from archive URL
attune pack install https://example.com/packs/slack-2.1.0.zip
# Install from local directory (development)
attune pack install ./packs/my-pack
# Install from local archive
attune pack install ./slack-2.1.0.zip
# Force reinstall
attune pack install slack --force
# Skip tests (faster, but not recommended)
attune pack install slack --skip-tests
```
### Generate Index Entry
For pack maintainers, generate an index entry from a pack:
```bash
attune pack index-entry \
--pack-dir <path-to-pack> \
--version <version> \
--git-url <git-repo-url> \
--git-ref <tag-or-branch> \
--archive-url <archive-url>
# Output to stdout (JSON)
attune pack index-entry --pack-dir ./pack-slack --version 2.1.0 \
--git-url https://github.com/example/pack-slack.git \
--git-ref v2.1.0 \
--archive-url https://example.com/packs/slack-2.1.0.zip
# Append to existing index file
attune pack index-entry --pack-dir ./pack-slack --version 2.1.0 \
--git-url https://github.com/example/pack-slack.git \
--git-ref v2.1.0 \
--archive-url https://example.com/packs/slack-2.1.0.zip \
--index-file registry.json \
--output registry.json
```
**Output Example**:
```json
{
"ref": "slack",
"label": "Slack Integration",
"description": "Send messages, upload files, and monitor Slack channels",
"version": "2.1.0",
"author": "Example Team",
"email": "team@example.com",
"license": "Apache-2.0",
"runtime_deps": ["python3"],
"install_sources": [
{
"type": "git",
"url": "https://github.com/example/pack-slack.git",
"ref": "v2.1.0",
"checksum": "sha256:abc123..."
},
{
"type": "archive",
"url": "https://example.com/packs/slack-2.1.0.zip",
"checksum": "sha256:def456..."
}
],
"contents": {
"actions": [...],
"sensors": [...],
"triggers": [...]
}
}
```
### Update Index File
Merge multiple index entries or update an existing index:
```bash
# Add entry to index
attune pack index-update --index registry.json --entry entry.json
# Merge multiple indices
attune pack index-merge --output combined.json registry1.json registry2.json
# Update pack version in index
attune pack index-update --index registry.json --pack slack --version 2.1.1 \
--git-ref v2.1.1 --archive-url https://example.com/packs/slack-2.1.1.zip
```
### List Registries
```bash
attune pack registries
# Output:
# Priority | Name | URL | Status
# ---------|-------------------------|------------------------------------------|--------
# 1 | Official Attune Registry| https://registry.attune.io/index.json | Online
# 2 | Company Internal | https://internal.example.com/registry.json| Online
# 3 | Local Development | file:///opt/attune/local-registry.json | Online
```
### Search Registry
```bash
# Search all registries
attune pack search <keyword>
# Search specific registry
attune pack search <keyword> --registry "Official Attune Registry"
# Example
attune pack search slack
# Output:
# Ref | Version | Description | Registry
# -------|---------|------------------------------------------|-------------------------
# slack | 2.1.0 | Send messages and monitor Slack channels | Official Attune Registry
```
---
## Installation Process
### Installation Workflow
```
┌─────────────────────────────────────────────────────────────────────┐
│ 1. Source Resolution │
│ - Registry reference → Search indices → Resolve install source │
│ - Direct URL → Use provided source │
│ - Local path → Use local filesystem │
└────────────────┬────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ 2. Download/Copy Pack │
│ - Git: Clone repository to temp directory │
│ - Archive: Download and extract to temp directory │
│ - Local: Copy to temp directory │
└────────────────┬────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ 3. Validate Pack Structure │
│ - Verify pack.yaml exists and is valid │
│ - Verify pack ref matches (if installing from registry) │
│ - Verify version matches (if specified) │
│ - Validate pack structure (actions, sensors, triggers) │
└────────────────┬────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ 4. Check Dependencies │
│ - Verify Attune version compatibility │
│ - Check runtime dependencies (Python, Node.js, etc.) │
│ - Verify dependent packs are installed │
│ - Check Python/Node.js version requirements │
└────────────────┬────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ 5. Setup Worker Environment │
│ - Python: Create virtualenv, install requirements.txt │
│ - Node.js: Create node_modules, run npm install │
│ - Shell: Verify scripts are executable │
└────────────────┬────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ 6. Run Pack Tests (if present) │
│ - Execute test suite defined in pack │
│ - Verify all tests pass │
│ - Skip if --skip-tests flag provided │
└────────────────┬────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ 7. Register Pack Components │
│ - Insert pack metadata into database │
│ - Register actions, sensors, triggers │
│ - Register bundled rules and workflows (if any) │
│ - Copy pack files to permanent location │
└────────────────┬────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ 8. Cleanup │
│ - Remove temporary directory │
│ - Log installation success │
│ - Return pack ID and metadata │
└─────────────────────────────────────────────────────────────────────┘
```
### Pack Storage Location
Installed packs are stored in the configured packs directory:
```
/var/lib/attune/packs/
├── slack/
│ ├── pack.yaml
│ ├── actions/
│ ├── sensors/
│ ├── triggers/
│ ├── requirements.txt
│ ├── .venv/ # Python virtualenv (if applicable)
│ └── metadata.json # Installation metadata
├── aws/
└── github/
```
Installation metadata includes:
```json
{
"pack_ref": "slack",
"version": "2.1.0",
"installed_at": "2024-01-20T12:00:00Z",
"installed_from": {
"type": "git",
"url": "https://github.com/example/pack-slack.git",
"ref": "v2.1.0"
},
"checksum": "sha256:abc123...",
"registry": "Official Attune Registry"
}
```
---
## Checksum Verification
To ensure pack integrity, checksums are verified during installation:
### Supported Algorithms
- `sha256` (recommended)
- `sha512` (recommended)
- `sha1` (legacy, not recommended)
- `md5` (legacy, not recommended)
### Checksum Format
```
algorithm:hash
```
Examples:
- `sha256:abc123def456...`
- `sha512:789xyz...`
### Generating Checksums
For pack maintainers:
```bash
# Git repository (tar.gz snapshot)
sha256sum pack-slack-2.1.0.tar.gz
# Zip archive
sha256sum pack-slack-2.1.0.zip
# Using attune CLI
attune pack checksum ./pack-slack-2.1.0.zip
```
### Verification Process
1. Download/extract pack to temporary location
2. Calculate checksum of downloaded content
3. Compare with checksum in index file
4. If mismatch, abort installation and report error
5. If `verify_checksums: false` in config, skip verification (not recommended)
---
## CI/CD Integration
### GitHub Actions Example
Automate pack building and registry updates:
```yaml
name: Build and Publish Pack
on:
push:
tags:
- 'v*'
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Create pack archive
run: |
VERSION=${GITHUB_REF#refs/tags/v}
zip -r pack-slack-${VERSION}.zip . -x ".git/*" ".github/*"
- name: Calculate checksum
id: checksum
run: |
CHECKSUM=$(sha256sum pack-slack-*.zip | awk '{print $1}')
echo "checksum=sha256:${CHECKSUM}" >> $GITHUB_OUTPUT
- name: Upload to artifact storage
run: |
VERSION=${GITHUB_REF#refs/tags/v}
aws s3 cp pack-slack-${VERSION}.zip s3://my-bucket/packs/
- name: Generate registry entry
run: |
VERSION=${GITHUB_REF#refs/tags/v}
attune pack index-entry \
--pack-dir . \
--version ${VERSION} \
--git-url https://github.com/example/pack-slack.git \
--git-ref ${GITHUB_REF#refs/tags/} \
--archive-url https://my-bucket.s3.amazonaws.com/packs/pack-slack-${VERSION}.zip \
--checksum ${{ steps.checksum.outputs.checksum }} \
> entry.json
- name: Update registry index
run: |
# Download current index
wget https://registry.example.com/index.json
# Add new entry
attune pack index-update \
--index index.json \
--entry entry.json \
--output index.json
# Upload updated index
aws s3 cp index.json s3://registry.example.com/
```
---
## Error Handling
### Installation Errors
| Error | Cause | Resolution |
|-------|-------|------------|
| Pack not found in registry | Pack ref doesn't exist in any configured registry | Check pack name, verify registry is online |
| Checksum mismatch | Downloaded pack doesn't match expected checksum | Pack may be corrupted or tampered with; contact pack maintainer |
| Pack already installed | Pack with same ref already exists | Use `--force` to reinstall |
| Dependency not met | Required Attune version, runtime, or pack not available | Update Attune, install runtime, or install dependency pack |
| Invalid pack structure | pack.yaml missing or invalid | Fix pack structure |
| Tests failed | Pack tests did not pass | Fix pack code or use `--skip-tests` (not recommended) |
### Registry Errors
| Error | Cause | Resolution |
|-------|-------|------------|
| Registry unreachable | Network error, DNS failure | Check network, verify URL |
| Invalid index format | Index JSON is malformed | Contact registry maintainer |
| Authentication failed | Registry requires authentication but token is invalid | Update registry token in configuration |
---
## Security Considerations
### 1. HTTPS Only (Recommended)
Configure `allow_http: false` to reject non-HTTPS registries:
```yaml
pack_registry:
allow_http: false # Only allow HTTPS
```
### 2. Checksum Verification
Always enable checksum verification in production:
```yaml
pack_registry:
verify_checksums: true
```
### 3. Registry Authentication
For private registries, use secure token storage:
```bash
export REGISTRY_TOKEN=$(cat /run/secrets/registry_token)
export ATTUNE__PACK_REGISTRY__INDICES="https://registry.example.com/index.json"
```
### 4. Code Review
- Review pack code before installation
- Use `--skip-tests` cautiously
- Test packs in non-production environment first
### 5. Signature Verification (Future)
Future enhancement: GPG signature verification for pack archives:
```json
{
"type": "archive",
"url": "https://example.com/packs/slack-2.1.0.zip",
"checksum": "sha256:abc123...",
"signature": "https://example.com/packs/slack-2.1.0.zip.sig",
"signing_key": "0x1234567890ABCDEF"
}
```
---
## Future Enhancements
### Version 1.1
- **Semantic version matching**: `slack@^2.0.0`, `slack@~2.1.0`
- **Pack updates**: `attune pack update <ref>` to upgrade to latest version
- **Dependency resolution**: Automatic installation of pack dependencies
### Version 1.2
- **GPG signature verification**: Cryptographic verification of pack authenticity
- **Pack ratings and reviews**: Community feedback in registry
- **Usage statistics**: Download counts, popularity metrics
### Version 1.3
- **Private pack authentication**: Token-based authentication for private packs
- **Pack mirroring**: Automatic mirroring of registry indices for redundancy
- **Delta updates**: Only download changed files when updating packs
---
## Related Documentation
- [Pack Structure](./pack-structure.md)
- [Pack Management Architecture](./pack-management-architecture.md)
- [CLI Documentation](./cli.md)
- [Configuration Guide](./configuration.md)
- [Pack Testing Framework](./pack-testing-framework.md)
Reference in New Issue View Git Blame Copy Permalink