attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
da8055cb79ed5398cd78c02839193f8ffbea32eb
attune/work-summary/2026-02-05-pack-installation-consolidation.md

11 KiB
Raw Blame History

Pack Installation Consolidation - Work Summary

Date: 2026-02-05
Type: Schema Simplification (Pre-Production)
Status: ✅ Complete

Overview

Consolidated the separate pack_installation table into the pack table by adding nullable installation metadata columns. This simplifies the schema by eliminating an unnecessary 1:1 relationship table, reducing joins and making the data model more intuitive.

Problem Statement

The pack_installation table tracked installation metadata (source type, URL, checksum, storage path, etc.) in a separate table with a 1:1 relationship to the pack table. This design:

  • Required joins to retrieve complete pack information
  • Added unnecessary complexity to queries
  • Created a separate repository layer for what was essentially pack metadata
  • Had no use case for multiple installation records per pack

The relationship was strictly 1:1 (one installation record per pack), making it a perfect candidate for denormalization.

Solution

Merged installation metadata directly into the pack table as nullable columns. Packs that are not installed will have these fields as NULL.

Changes Made

1. Database Migration (migrations/20250101000002_pack_system.sql)

Added columns to pack table:

-- Installation metadata (nullable for non-installed packs)
source_type TEXT,
source_url TEXT,
source_ref TEXT,
checksum TEXT,
checksum_verified BOOLEAN DEFAULT FALSE,
installed_at TIMESTAMPTZ,
installed_by BIGINT,
installation_method TEXT,
storage_path TEXT,

Added indexes:

CREATE INDEX idx_pack_installed_at ON pack(installed_at DESC) WHERE installed_at IS NOT NULL;
CREATE INDEX idx_pack_installed_by ON pack(installed_by) WHERE installed_by IS NOT NULL;
CREATE INDEX idx_pack_source_type ON pack(source_type) WHERE source_type IS NOT NULL;

Added foreign key constraint (in migrations/20250101000003_identity_and_auth.sql):

ALTER TABLE pack
    ADD CONSTRAINT fk_pack_installed_by
    FOREIGN KEY (installed_by)
    REFERENCES identity(id)
    ON DELETE SET NULL;

2. Model Changes (crates/common/src/models.rs)

Updated Pack struct:

  • Added installation metadata fields (all Option<T> types)
  • Removed PackInstallation struct entirely
  • Removed CreatePackInstallation struct

New fields in Pack:

pub source_type: Option<String>,
pub source_url: Option<String>,
pub source_ref: Option<String>,
pub checksum: Option<String>,
pub checksum_verified: Option<bool>,
pub installed_at: Option<DateTime<Utc>>,
pub installed_by: Option<Id>,
pub installation_method: Option<String>,
pub storage_path: Option<String>,

3. Repository Changes

Removed:

  • crates/common/src/repositories/pack_installation.rs (entire file deleted)
  • PackInstallationRepository from repositories/mod.rs

Updated PackRepository (crates/common/src/repositories/pack.rs):

Added new method:

pub async fn update_installation_metadata(
    executor: E,
    id: i64,
    source_type: String,
    source_url: Option<String>,
    source_ref: Option<String>,
    checksum: Option<String>,
    checksum_verified: bool,
    installed_by: Option<i64>,
    installation_method: String,
    storage_path: String,
) -> Result<Pack>

Added helper methods:

  • is_installed() - Check if a pack has installation metadata
  • list_installed() - List all installed packs
  • list_by_source_type() - Filter packs by installation source

Updated all SELECT queries to include installation fields in:

  • find_by_id()
  • find_by_ref()
  • list()
  • create()
  • update()
  • list_paginated()
  • find_by_tag()
  • find_standard()
  • search()

Updated input structs:

  • Added installers: JsonDict to CreatePackInput
  • Added installers: Option<JsonDict> to UpdatePackInput

4. API Route Changes (crates/api/src/routes/packs.rs)

Updated install_pack endpoint:

  • Removed PackInstallationRepository::new() and create() calls
  • Replaced with direct call to PackRepository::update_installation_metadata()

Before:

let installation_metadata = CreatePackInstallation { ... };
installation_repo.create(installation_metadata).await?;

After:

PackRepository::update_installation_metadata(
    &state.db,
    pack_id,
    source_type.to_string(),
    source_url,
    source_ref,
    checksum.clone(),
    installed.checksum.is_some() && checksum.is_some(),
    user_id,
    "api".to_string(),
    final_path.to_string_lossy().to_string(),
).await?;

5. Test Updates

Updated all test files to include installers: json!({}) in CreatePackInput:

  • crates/api/tests/helpers.rs
  • crates/api/tests/sse_execution_stream_tests.rs
  • crates/api/tests/webhook_api_tests.rs
  • crates/api/tests/webhook_security_tests.rs
  • crates/api/tests/pack_registry_tests.rs
  • crates/common/tests/helpers.rs
  • crates/common/tests/pack_repository_tests.rs
  • crates/common/tests/permission_repository_tests.rs
  • crates/common/tests/repository_runtime_tests.rs
  • crates/executor/tests/fifo_ordering_integration_test.rs
  • crates/executor/tests/policy_enforcer_tests.rs

Updated pack registry tests to use Pack fields instead of PackInstallation:

// Before
let installation = installation_repo.get_by_pack_id(pack_id).await?;
assert_eq!(installation.source_type, "local_directory");

// After
let pack = PackRepository::find_by_id(&ctx.pool, pack_id).await?;
assert_eq!(pack.source_type.as_deref(), Some("local_directory"));

6. Migration Schema Fixes (Missing Columns)

Fixed missing is_adhoc columns and rule table during migration consolidation:

Issues Found:

  • action.is_adhoc column was missing from migration
  • sensor.is_adhoc column was missing from migration
  • rule table was completely missing from migrations
  • ActionRepository::update() missing is_adhoc in RETURNING clause

Fixes:

  • Added is_adhoc BOOLEAN NOT NULL DEFAULT FALSE to action table (migration 005)
  • Added is_adhoc BOOLEAN NOT NULL DEFAULT FALSE to sensor table (migration 004)
  • Created complete rule table with is_adhoc in migration 006 (after action exists)
  • Added foreign key constraints for enforcement.rule and event.rule
  • Updated ActionRepository::update() RETURNING clause to include is_adhoc
  • Added proper indexes, triggers, and comments for all tables

7. CLI Test Fixes (Unrelated but Fixed)

Fixed failing whoami tests in crates/cli/tests/test_auth.rs:

Issue: Mock endpoint path was /auth/whoami but actual API uses /auth/me

Fixes:

  • Updated mock_whoami_success() to use /auth/me path
  • Fixed mock response structure to match CurrentUserResponse (removed extra fields)
  • Changed test assertions from "username" to "login"
  • Changed parameter from email to display_name

Breaking Changes

Note: This is a pre-production change with no deployments or users, so breaking changes are acceptable.

  • Database schema change: pack_installation table removed, new columns added to pack
  • Model API change: PackInstallation and CreatePackInstallation types removed
  • Repository API change: PackInstallationRepository removed, new methods added to PackRepository

Benefits

  1. Simpler Schema: One less table to manage (17 tables instead of 18)
  2. No Joins Required: All pack information available in a single query
  3. Clearer Data Model: Installation is a property of a pack, not a separate entity
  4. Reduced Code: Eliminated ~170 lines of repository code
  5. Better Performance: Fewer joins, simpler queries, partial indexes on nullable fields

Migration Notes

For users migrating from the old schema (when v1.0 releases):

  1. Drop and recreate database (acceptable since pre-production)
  2. Run consolidated migrations from scratch
  3. Reload pack data using pack installation API

Testing

  • ✅ All workspace compilation successful
  • ✅ Pack registry tests updated and passing
  • ✅ CLI auth tests fixed and passing
  • ✅ No compilation warnings

Files Modified

Migrations (4 files)

  • migrations/20250101000002_pack_system.sql - Added installation columns to pack table
  • migrations/20250101000003_identity_and_auth.sql - Added foreign key constraint
  • migrations/20250101000004_trigger_sensor_event_rule.sql - Added is_adhoc to sensor, added indexes/triggers/comments
  • migrations/20250101000005_action.sql - Added is_adhoc to action, added indexes/triggers/comments
  • migrations/20250101000006_execution_system.sql - Added complete rule table with is_adhoc and foreign key constraints

Models (1 file)

  • crates/common/src/models.rs - Updated Pack struct, removed PackInstallation module

Repositories (3 files)

  • crates/common/src/repositories/pack.rs - Added installation methods, updated all queries
  • crates/common/src/repositories/action.rs - Fixed update() RETURNING clause to include is_adhoc
  • crates/common/src/repositories/mod.rs - Removed PackInstallationRepository export
  • Deleted: crates/common/src/repositories/pack_installation.rs

API Routes (1 file)

  • crates/api/src/routes/packs.rs - Updated install_pack to use new repository method

Tests (14 files)

  • crates/api/tests/helpers.rs
  • crates/api/tests/pack_registry_tests.rs
  • crates/api/tests/sse_execution_stream_tests.rs
  • crates/api/tests/webhook_api_tests.rs
  • crates/api/tests/webhook_security_tests.rs
  • crates/common/tests/helpers.rs
  • crates/common/tests/pack_repository_tests.rs
  • crates/common/tests/permission_repository_tests.rs
  • crates/common/tests/repository_runtime_tests.rs
  • crates/executor/tests/fifo_ordering_integration_test.rs
  • crates/executor/tests/policy_enforcer_tests.rs
  • crates/cli/tests/common/mod.rs - Fixed whoami mock
  • crates/cli/tests/test_auth.rs - Fixed whoami tests

Total: 22 files modified, 1 file deleted

Verification Steps

To verify the changes after dropping and recreating the database:

# 1. Drop and recreate databases
make db-reset
make db-test-setup

# 2. Run migrations
make db-migrate

# 3. Verify schema
psql attune -c "\d pack" | grep -E "installed_at|source_type|storage_path"

# 4. Run tests
cargo test --workspace

# 5. Test pack installation via API
# (Start services and test via web UI or CLI)

Related Documentation

  • Migration consolidation was completed on 2026-01-17
  • This change continues the pre-production schema refinement effort
  • See docs/migrations/CONSOLIDATION-COMPLETE.md for full migration history

Lessons Learned

  1. Migration consolidation requires careful verification - Missing tables/columns can slip through when consolidating migrations
  2. Test everything after schema changes - Running repository tests revealed missing is_adhoc columns
  3. Dependency ordering matters - Rule table needed to be in migration 006 after action table (migration 005)
  4. RETURNING clauses must be complete - All model fields must be included in UPDATE...RETURNING queries

Next Steps

  1. ✅ Drop existing databases and re-run migrations
  2. ✅ Fix missing is_adhoc columns in migrations
  3. ✅ Add missing rule table to migrations
  4. ✅ Verify all repository tests pass
  5. Test pack installation workflow end-to-end
  6. Update any documentation referencing pack_installation table
  7. Consider similar consolidations for other 1:1 relationships if any exist
Reference in New Issue View Git Blame Copy Permalink