re-uploading work
This commit is contained in:
587
docs/packs/pack-installation-git.md
Normal file
587
docs/packs/pack-installation-git.md
Normal file
@@ -0,0 +1,587 @@
|
||||
# Pack Installation from Git Repositories
|
||||
|
||||
**Last Updated**: 2025-01-27
|
||||
**Status**: Production Ready
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Attune supports installing packs directly from git repositories, enabling teams to:
|
||||
|
||||
- **Version Control**: Track pack versions using git tags, branches, and commits
|
||||
- **Collaboration**: Share packs across teams via git hosting services
|
||||
- **Automation**: Integrate pack deployment into CI/CD pipelines
|
||||
- **Development**: Test packs from feature branches before release
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Web UI Installation
|
||||
|
||||
1. Navigate to **Packs** page
|
||||
2. Click **Add Pack** dropdown → **Install from Remote**
|
||||
3. Select **Git Repository** as source type
|
||||
4. Enter repository URL (HTTPS or SSH)
|
||||
5. Optionally specify a git reference (branch, tag, or commit)
|
||||
6. Configure installation options
|
||||
7. Click **Install Pack**
|
||||
|
||||
### CLI Installation
|
||||
|
||||
```bash
|
||||
# Install from default branch
|
||||
attune pack install https://github.com/example/pack-slack.git
|
||||
|
||||
# Install from specific tag
|
||||
attune pack install https://github.com/example/pack-slack.git --ref v2.1.0
|
||||
|
||||
# Install from branch
|
||||
attune pack install https://github.com/example/pack-slack.git --ref main
|
||||
|
||||
# Install from commit hash
|
||||
attune pack install https://github.com/example/pack-slack.git --ref a1b2c3d
|
||||
|
||||
# SSH URL
|
||||
attune pack install git@github.com:example/pack-slack.git --ref v2.1.0
|
||||
|
||||
# Skip tests and dependency validation (use with caution)
|
||||
attune pack install https://github.com/example/pack-slack.git --skip-tests --skip-deps
|
||||
```
|
||||
|
||||
### API Installation
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8080/api/v1/packs/install \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"source": "https://github.com/example/pack-slack.git",
|
||||
"ref_spec": "v2.1.0",
|
||||
"force": false,
|
||||
"skip_tests": false,
|
||||
"skip_deps": false
|
||||
}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Supported Git Sources
|
||||
|
||||
### HTTPS URLs
|
||||
|
||||
**Public repositories:**
|
||||
```
|
||||
https://github.com/username/pack-name.git
|
||||
https://gitlab.com/username/pack-name.git
|
||||
https://bitbucket.org/username/pack-name.git
|
||||
```
|
||||
|
||||
**Private repositories with credentials:**
|
||||
```
|
||||
https://username:token@github.com/username/pack-name.git
|
||||
```
|
||||
|
||||
> **Security Note**: For private repositories, use SSH keys or configure git credential helpers instead of embedding credentials in URLs.
|
||||
|
||||
### SSH URLs
|
||||
|
||||
**Standard format:**
|
||||
```
|
||||
git@github.com:username/pack-name.git
|
||||
git@gitlab.com:username/pack-name.git
|
||||
```
|
||||
|
||||
**SCP-style:**
|
||||
```
|
||||
user@server:path/to/pack.git
|
||||
```
|
||||
|
||||
**Requirements:**
|
||||
- SSH keys must be configured on the Attune server
|
||||
- User running Attune service must have access to private key
|
||||
- Host must be in `~/.ssh/known_hosts`
|
||||
|
||||
---
|
||||
|
||||
## Git References
|
||||
|
||||
The `ref_spec` parameter accepts any valid git reference:
|
||||
|
||||
### Branches
|
||||
```bash
|
||||
# Default branch (usually main or master)
|
||||
--ref main
|
||||
|
||||
# Development branch
|
||||
--ref develop
|
||||
|
||||
# Feature branch
|
||||
--ref feature/new-action
|
||||
```
|
||||
|
||||
### Tags
|
||||
```bash
|
||||
# Semantic version tag
|
||||
--ref v1.2.3
|
||||
|
||||
# Release tag
|
||||
--ref release-2024-01-27
|
||||
```
|
||||
|
||||
### Commit Hashes
|
||||
```bash
|
||||
# Full commit hash
|
||||
--ref a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0
|
||||
|
||||
# Short commit hash (7+ characters)
|
||||
--ref a1b2c3d
|
||||
```
|
||||
|
||||
### Special References
|
||||
```bash
|
||||
# HEAD of default branch
|
||||
--ref HEAD
|
||||
|
||||
# No ref specified = default branch
|
||||
# (equivalent to --depth 1 clone)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pack Structure Requirements
|
||||
|
||||
The git repository must contain a valid pack structure:
|
||||
|
||||
### Option 1: Root-Level Pack
|
||||
```
|
||||
repository-root/
|
||||
├── pack.yaml # Required
|
||||
├── actions/ # Optional
|
||||
├── sensors/ # Optional
|
||||
├── triggers/ # Optional
|
||||
├── rules/ # Optional
|
||||
├── workflows/ # Optional
|
||||
└── README.md # Recommended
|
||||
```
|
||||
|
||||
### Option 2: Pack Subdirectory
|
||||
```
|
||||
repository-root/
|
||||
├── pack/
|
||||
│ ├── pack.yaml # Required
|
||||
│ ├── actions/
|
||||
│ └── ...
|
||||
├── docs/
|
||||
└── README.md
|
||||
```
|
||||
|
||||
> The installer will automatically detect and use the pack directory.
|
||||
|
||||
---
|
||||
|
||||
## Installation Process
|
||||
|
||||
When you install a pack from git, Attune performs these steps:
|
||||
|
||||
### 1. Clone Repository
|
||||
```
|
||||
git clone [--depth 1] <url> <temp-dir>
|
||||
```
|
||||
|
||||
If a `ref_spec` is provided:
|
||||
```
|
||||
git checkout <ref_spec>
|
||||
```
|
||||
|
||||
### 2. Locate Pack Directory
|
||||
- Search for `pack.yaml` at repository root
|
||||
- If not found, search in `pack/` subdirectory
|
||||
- Fail if `pack.yaml` not found in either location
|
||||
|
||||
### 3. Validate Dependencies (unless skipped)
|
||||
- Extract runtime dependencies from `pack.yaml`
|
||||
- Extract pack dependencies
|
||||
- Verify all dependencies are satisfied
|
||||
- Fail if dependencies are missing (unless `force` enabled)
|
||||
|
||||
### 4. Register Pack
|
||||
- Parse `pack.yaml` metadata
|
||||
- Create database entry
|
||||
- Auto-sync workflows if present
|
||||
|
||||
### 5. Execute Tests (unless skipped)
|
||||
- Run pack test suite if configured
|
||||
- Fail if tests don't pass (unless `force` enabled)
|
||||
|
||||
### 6. Copy to Permanent Storage
|
||||
- Move pack files to `{packs_base_dir}/{pack_ref}/`
|
||||
- Calculate directory checksum
|
||||
- Store installation metadata
|
||||
|
||||
### 7. Record Installation
|
||||
- Store installation record in `pack_installation` table
|
||||
- Record source URL, git ref, timestamp, checksum
|
||||
- Link to installing user
|
||||
|
||||
---
|
||||
|
||||
## Installation Options
|
||||
|
||||
### Force Installation
|
||||
**Flag**: `--force` (CLI) or `force: true` (API)
|
||||
|
||||
Enables:
|
||||
- Reinstall pack even if it already exists (replaces existing)
|
||||
- Proceed even if dependencies are missing
|
||||
- Proceed even if tests fail
|
||||
|
||||
**Use Cases**:
|
||||
- Upgrading pack to new version
|
||||
- Recovering from failed installation
|
||||
- Development/testing workflows
|
||||
|
||||
**Warning**: Force mode bypasses safety checks. Use cautiously in production.
|
||||
|
||||
### Skip Tests
|
||||
**Flag**: `--skip-tests` (CLI) or `skip_tests: true` (API)
|
||||
|
||||
- Skip executing pack test suite
|
||||
- Faster installation
|
||||
- Useful when tests are slow or not available
|
||||
|
||||
**Use Cases**:
|
||||
- Installing trusted packs
|
||||
- Tests not yet implemented
|
||||
- Development environments
|
||||
|
||||
### Skip Dependencies
|
||||
**Flag**: `--skip-deps` (CLI) or `skip_deps: true` (API)
|
||||
|
||||
- Skip validation of runtime dependencies
|
||||
- Skip validation of pack dependencies
|
||||
- May result in runtime failures if dependencies truly missing
|
||||
|
||||
**Use Cases**:
|
||||
- Dependencies will be installed separately
|
||||
- Custom runtime environment
|
||||
- Development/testing
|
||||
|
||||
---
|
||||
|
||||
## Example Workflows
|
||||
|
||||
### Development Workflow
|
||||
|
||||
```bash
|
||||
# 1. Install pack from feature branch for testing
|
||||
attune pack install https://github.com/myorg/pack-custom.git \
|
||||
--ref feature/new-sensor \
|
||||
--skip-tests \
|
||||
--force
|
||||
|
||||
# 2. Test the pack
|
||||
attune pack test custom
|
||||
|
||||
# 3. When satisfied, install from main/release tag
|
||||
attune pack install https://github.com/myorg/pack-custom.git \
|
||||
--ref v1.0.0 \
|
||||
--force
|
||||
```
|
||||
|
||||
### Production Deployment
|
||||
|
||||
```bash
|
||||
# Install specific version with full validation
|
||||
attune pack install https://github.com/myorg/pack-slack.git \
|
||||
--ref v2.1.0
|
||||
|
||||
# Verify installation
|
||||
attune pack list | grep slack
|
||||
attune pack test slack
|
||||
```
|
||||
|
||||
### CI/CD Pipeline
|
||||
|
||||
```yaml
|
||||
# GitHub Actions example
|
||||
- name: Install pack on staging
|
||||
run: |
|
||||
attune pack install https://github.com/${{ github.repository }}.git \
|
||||
--ref ${{ github.sha }} \
|
||||
--force
|
||||
env:
|
||||
ATTUNE_API_TOKEN: ${{ secrets.ATTUNE_TOKEN }}
|
||||
|
||||
- name: Run tests
|
||||
run: attune pack test $(basename ${{ github.repository }})
|
||||
```
|
||||
|
||||
### Private Repository with SSH
|
||||
|
||||
```bash
|
||||
# 1. Set up SSH key on Attune server
|
||||
ssh-keygen -t ed25519 -C "attune@example.com"
|
||||
cat ~/.ssh/id_ed25519.pub # Add to GitHub/GitLab
|
||||
|
||||
# 2. Add host to known_hosts
|
||||
ssh-keyscan github.com >> ~/.ssh/known_hosts
|
||||
|
||||
# 3. Install pack
|
||||
attune pack install git@github.com:myorg/private-pack.git --ref main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Git Clone Fails
|
||||
|
||||
**Error**: `Git clone failed: Permission denied`
|
||||
|
||||
**Solutions**:
|
||||
- Verify SSH keys are configured correctly
|
||||
- Check repository access permissions
|
||||
- For HTTPS, verify credentials or token
|
||||
- Add host to `~/.ssh/known_hosts`
|
||||
|
||||
### Ref Not Found
|
||||
|
||||
**Error**: `Git checkout failed: pathspec 'v1.0.0' did not match any file(s) known to git`
|
||||
|
||||
**Solutions**:
|
||||
- Verify tag/branch exists in repository
|
||||
- Check spelling and case sensitivity
|
||||
- Ensure ref is pushed to remote
|
||||
- Try with full commit hash
|
||||
|
||||
### Pack.yaml Not Found
|
||||
|
||||
**Error**: `pack.yaml not found in directory`
|
||||
|
||||
**Solutions**:
|
||||
- Ensure `pack.yaml` exists at repository root or in `pack/` subdirectory
|
||||
- Check file name spelling (case-sensitive on Linux)
|
||||
- Verify correct branch/tag is checked out
|
||||
|
||||
### Dependency Validation Failed
|
||||
|
||||
**Error**: `Pack dependency validation failed: pack 'core' version '^1.0.0' not found`
|
||||
|
||||
**Solutions**:
|
||||
- Install missing dependencies first
|
||||
- Use `--skip-deps` to bypass validation (not recommended)
|
||||
- Use `--force` to install anyway
|
||||
|
||||
### Test Failures
|
||||
|
||||
**Error**: `Pack registration failed: tests did not pass`
|
||||
|
||||
**Solutions**:
|
||||
- Review test output for specific failures
|
||||
- Fix issues in pack code
|
||||
- Use `--skip-tests` to install without testing
|
||||
- Use `--force` to install despite test failures
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### SSH Keys
|
||||
- Use dedicated SSH key for Attune service
|
||||
- Restrict key permissions (read-only access preferred)
|
||||
- Rotate keys periodically
|
||||
- Use SSH agent for key management
|
||||
|
||||
### HTTPS Authentication
|
||||
- Never embed credentials directly in URLs
|
||||
- Use git credential helpers
|
||||
- Consider personal access tokens with limited scope
|
||||
- Rotate tokens regularly
|
||||
|
||||
### Code Review
|
||||
- Review pack code before installation
|
||||
- Install from tagged releases, not branches
|
||||
- Verify pack author/source
|
||||
- Check for malicious code in actions/sensors
|
||||
|
||||
### Git References
|
||||
- **Production**: Use specific tags (e.g., `v1.2.3`)
|
||||
- **Staging**: Use release branches (e.g., `release-*`)
|
||||
- **Development**: Feature branches acceptable
|
||||
- **Avoid**: `main`/`master` in production (may change unexpectedly)
|
||||
|
||||
---
|
||||
|
||||
## Advanced Topics
|
||||
|
||||
### Submodules
|
||||
|
||||
If pack repository uses git submodules:
|
||||
|
||||
```bash
|
||||
# Clone with submodules
|
||||
git clone --recurse-submodules <url>
|
||||
```
|
||||
|
||||
> **Note**: Current implementation does not automatically clone submodules. Manual configuration required.
|
||||
|
||||
### Large Repositories
|
||||
|
||||
For large repositories, use shallow clones:
|
||||
|
||||
```bash
|
||||
# Default behavior when no ref_spec
|
||||
git clone --depth 1 <url>
|
||||
```
|
||||
|
||||
When specific ref is needed:
|
||||
```bash
|
||||
git clone <url>
|
||||
git checkout <ref>
|
||||
```
|
||||
|
||||
### Monorepos
|
||||
|
||||
For repositories containing multiple packs:
|
||||
|
||||
```
|
||||
monorepo/
|
||||
├── pack-a/
|
||||
│ └── pack.yaml
|
||||
├── pack-b/
|
||||
│ └── pack.yaml
|
||||
└── pack-c/
|
||||
└── pack.yaml
|
||||
```
|
||||
|
||||
**Limitation**: Current implementation expects one pack per repository. For monorepos, use filesystem registration or create separate repositories.
|
||||
|
||||
---
|
||||
|
||||
## Database Schema
|
||||
|
||||
Installation metadata is stored in the `pack_installation` table:
|
||||
|
||||
```sql
|
||||
CREATE TABLE pack_installation (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
pack_id BIGINT NOT NULL REFERENCES pack(id),
|
||||
source_type VARCHAR(50) NOT NULL, -- 'git'
|
||||
source_url TEXT, -- Git repository URL
|
||||
source_ref TEXT, -- Branch/tag/commit
|
||||
checksum TEXT, -- Directory checksum
|
||||
checksum_verified BOOLEAN DEFAULT FALSE,
|
||||
installed_by BIGINT REFERENCES identity(id),
|
||||
installation_method VARCHAR(50), -- 'api', 'cli', 'web'
|
||||
storage_path TEXT NOT NULL, -- File system path
|
||||
meta JSONB, -- Additional metadata
|
||||
created TIMESTAMP DEFAULT NOW(),
|
||||
updated TIMESTAMP DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## API Reference
|
||||
|
||||
### Install Pack Endpoint
|
||||
|
||||
**POST** `/api/v1/packs/install`
|
||||
|
||||
**Request Body**:
|
||||
```json
|
||||
{
|
||||
"source": "https://github.com/example/pack-slack.git",
|
||||
"ref_spec": "v2.1.0",
|
||||
"force": false,
|
||||
"skip_tests": false,
|
||||
"skip_deps": false
|
||||
}
|
||||
```
|
||||
|
||||
**Response** (201 Created):
|
||||
```json
|
||||
{
|
||||
"data": {
|
||||
"pack": {
|
||||
"id": 42,
|
||||
"ref": "slack",
|
||||
"label": "Slack Pack",
|
||||
"version": "2.1.0",
|
||||
"description": "Slack integration pack",
|
||||
"is_standard": false,
|
||||
...
|
||||
},
|
||||
"test_result": {
|
||||
"status": "passed",
|
||||
"total_tests": 10,
|
||||
"passed": 10,
|
||||
"failed": 0,
|
||||
...
|
||||
},
|
||||
"tests_skipped": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Error Responses**:
|
||||
- `400`: Invalid request, dependency validation failed, or tests failed
|
||||
- `409`: Pack already exists (use `force: true` to override)
|
||||
- `500`: Internal error (git clone failed, filesystem error, etc.)
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
- Git submodule support
|
||||
- Monorepo support (install specific subdirectory)
|
||||
- Pack version upgrade workflow
|
||||
- Automatic version detection from tags
|
||||
- Git LFS support
|
||||
- Signature verification for signed commits
|
||||
|
||||
### Registry Integration
|
||||
When pack registry system is implemented:
|
||||
- Registry can reference git repositories
|
||||
- Automatic version discovery from git tags
|
||||
- Centralized pack metadata
|
||||
- See `pack-registry-spec.md` for details
|
||||
|
||||
---
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Pack Structure](pack-structure.md) - Pack directory and file format
|
||||
- [Pack Registry Specification](pack-registry-spec.md) - Registry-based installation
|
||||
- [Pack Testing Framework](pack-testing-framework.md) - Testing packs
|
||||
- [Configuration](../configuration/configuration.md) - Server configuration
|
||||
- [Production Deployment](../deployment/production-deployment.md) - Deployment guide
|
||||
|
||||
---
|
||||
|
||||
## Examples Repository
|
||||
|
||||
Example packs demonstrating git-based installation:
|
||||
|
||||
```bash
|
||||
# Simple action pack
|
||||
attune pack install https://github.com/attune-examples/pack-hello-world.git
|
||||
|
||||
# Complex pack with dependencies
|
||||
attune pack install https://github.com/attune-examples/pack-kubernetes.git --ref v1.0.0
|
||||
|
||||
# Private repository (SSH)
|
||||
attune pack install git@github.com:mycompany/pack-internal.git --ref main
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Support
|
||||
|
||||
For issues or questions:
|
||||
- GitHub Issues: https://github.com/attune-io/attune/issues
|
||||
- Documentation: https://docs.attune.io/packs/installation
|
||||
- Community: https://community.attune.io
|
||||
Reference in New Issue
Block a user