re-uploading work

reference/models.md

@@ -0,0 +1,509 @@
+# Attune Data Models Reference
+
+This document describes the SQLAlchemy ORM models that define the Attune automation platform's database schema.
+
+## Overview
+
+Attune is an event-driven automation and orchestration platform with built-in multi-tenancy, RBAC, and workflow capabilities. The data models are organized into several functional areas:
+
+- **Packaging & Distribution**: Organization of automation components
+- **Runtime Environment**: Execution environments and workers
+- **Event-Driven Architecture**: Triggers, sensors, and events
+- **Actions & Automation**: Executable operations and rules
+- **Execution & Orchestration**: Action execution and workflow management
+- **User Interactions**: Asynchronous user input and approvals in workflows
+- **Policy & Permissions**: Access control and execution policies
+- **Identity & Access**: User and service account management
+- **Configuration & Secrets**: Secure key-value storage
+- **Notifications**: Real-time event streaming
+- **Artifacts & Storage**: Execution output management
+
+---
+
+## Core Infrastructure
+
+### `Base`
+**Purpose**: SQLAlchemy declarative base class for all models.
+- Configured with "attune" schema metadata
+- Parent class for all database models
+
+---
+
+## Packaging & Distribution
+
+### `Pack`
+**Purpose**: A package/bundle of automation components (sensors, actions, rules, policies, etc.).
+
+**Key Fields**:
+- `ref`: Unique reference (e.g., "slack", "github")
+- `label`: Human-readable name
+- `version`: Semantic version string (validated)
+- `conf_schema`: JSON schema for configuration
+- `config`: Configuration values
+- `runtime_deps`: List of required runtime references
+- `is_standard`: Whether this is a core/built-in pack
+
+**Relationships**:
+- Contains: runtimes, triggers, sensors, actions, rules, permission_sets, policies
+
+**Purpose**: Primary organizational unit for distributing and managing related automation components.
+
+---
+
+## Runtime Environment
+
+### `RuntimeType` (Enum)
+**Values**: `action`, `sensor`
+
+**Purpose**: Distinguishes between action execution environments and sensor monitoring environments.
+
+### `Runtime`
+**Purpose**: Defines an execution environment for actions or sensors.
+
+**Key Fields**:
+- `ref`: Unique reference (format: `pack.(action|sensor).name`)
+- `runtime_type`: Type of runtime (action or sensor)
+- `name`: Runtime name (e.g., "python3.11", "nodejs20")
+- `distributions`: JSON describing available distributions
+- `installation`: JSON describing installation requirements
+- `pack`: Parent pack ID
+
+**Relationships**:
+- Belongs to: pack
+- Used by: workers, sensors, actions
+
+**Purpose**: Defines how to install and execute code (Python, Node.js, containers, etc.).
+
+### `WorkerType` (Enum)
+**Values**: `local`, `remote`, `container`
+
+**Purpose**: Categorizes worker deployment types.
+
+### `WorkerStatus` (Enum)
+**Values**: `active`, `inactive`, `busy`, `error`
+
+**Purpose**: Tracks current operational state of workers.
+
+### `Worker`
+**Purpose**: Represents a worker process/container that executes actions or monitors sensors.
+
+**Key Fields**:
+- `name`: Worker identifier
+- `worker_type`: Deployment type
+- `runtime`: Associated runtime ID
+- `host`, `port`: Connection details for remote workers
+- `status`: Current operational state
+- `capabilities`: JSON describing worker capabilities
+- `last_heartbeat`: Last health check timestamp
+
+**Relationships**:
+- Uses: runtime
+
+**Purpose**: Manages the actual compute resources that run automation code.
+
+---
+
+## Event-Driven Architecture
+
+### `Trigger`
+**Purpose**: Defines an event type that can activate rules.
+
+**Key Fields**:
+- `ref`: Unique reference (format: `pack.name`)
+- `label`: Display name
+- `enabled`: Whether trigger is active
+- `param_schema`: JSON schema for trigger parameters
+- `out_schema`: JSON schema for trigger output/payload
+- `pack`: Parent pack ID
+
+**Relationships**:
+- Belongs to: pack
+- Referenced by: sensors, rules, events
+
+**Examples**: "webhook_received", "file_modified", "schedule_fired"
+
+### `Sensor`
+**Purpose**: Monitors for trigger conditions and generates events.
+
+**Key Fields**:
+- `ref`: Unique reference (format: `pack.name`)
+- `entrypoint`: Code entry point (e.g., "sensors/webhook.py")
+- `runtime`: Execution environment
+- `trigger`: Associated trigger
+- `enabled`: Whether sensor is active
+- `param_schema`: JSON schema for configuration parameters
+
+**Relationships**:
+- Belongs to: pack
+- Uses: runtime
+- Monitors for: trigger
+
+**Purpose**: Active monitoring component that detects when triggers fire and creates events.
+
+### `Event`
+**Purpose**: An instance of a trigger firing.
+
+**Key Fields**:
+- `trigger`: Trigger ID (with SET NULL on delete)
+- `trigger_ref`: Trigger reference (preserved)
+- `config`: Snapshot of trigger/sensor configuration at event time
+- `payload`: Event data
+- `source`: Sensor that generated the event
+- `source_ref`: Source sensor reference
+
+**Relationships**:
+- Instance of: trigger
+- Generated by: sensor
+- Triggers: enforcements
+
+**Purpose**: Records that a trigger occurred, including all context needed for rule evaluation.
+
+---
+
+## Actions & Automation
+
+### `Action`
+**Purpose**: An executable task/operation.
+
+**Key Fields**:
+- `ref`: Unique reference (format: `pack.name`)
+- `label`: Display name
+- `entrypoint`: Code entry point (e.g., "actions/send_email.py")
+- `runtime`: Execution environment
+- `param_schema`: JSON schema for input parameters
+- `out_schema`: JSON schema for output/results
+- `pack`: Parent pack ID
+
+**Relationships**:
+- Belongs to: pack
+- Uses: runtime
+- Executed by: executions
+- Used in: rules
+
+**Examples**: "send_email", "create_ticket", "deploy_service"
+
+### `Rule`
+**Purpose**: Connects triggers to actions with conditional logic.
+
+**Key Fields**:
+- `ref`: Unique reference (format: `pack.name`)
+- `trigger`: Trigger that activates this rule
+- `action`: Action to execute when conditions are met
+- `conditions`: JSON array of condition expressions
+- `enabled`: Whether rule is active
+- `pack`: Parent pack ID
+
+**Relationships**:
+- Belongs to: pack
+- When: trigger
+- Then: action
+- Creates: enforcements
+
+**Purpose**: Automation logic that says "when trigger X fires, if conditions Y are met, execute action Z".
+
+### `EnforcementStatus` (Enum)
+**Values**: `created`, `processed`, `disabled`
+
+**Purpose**: Tracks processing state of rule enforcements.
+
+### `EnforcementCondition` (Enum)
+**Values**: `any`, `all`
+
+**Purpose**: Defines logical operator for multiple conditions (OR vs AND).
+
+### `Enforcement`
+**Purpose**: An instance of a rule being triggered by an event.
+
+**Key Fields**:
+- `rule`: Rule being enforced
+- `rule_ref`: Rule reference (preserved)
+- `trigger_ref`: Trigger reference (preserved)
+- `event`: Event that triggered this enforcement
+- `config`: Snapshot of rule/trigger configuration
+- `status`: Processing state
+- `payload`: Event payload for rule evaluation
+- `condition`: Logical operator (any/all)
+- `conditions`: Condition expressions to evaluate
+
+**Relationships**:
+- Instance of: rule
+- Triggered by: event
+- Creates: executions
+
+**Purpose**: Records that a rule was triggered and tracks whether it should execute its action.
+
+---
+
+## Execution & Orchestration
+
+### `ExecutionStatus` (Enum)
+**Values**: `requested`, `scheduling`, `scheduled`, `running`, `completed`, `failed`, `canceling`, `cancelled`, `timeout`, `abandoned`
+
+**Purpose**: Tracks detailed lifecycle state of action executions.
+
+### `Execution`
+**Purpose**: Represents a single action execution (can be part of a workflow).
+
+**Key Fields**:
+- `action`: Action being executed
+- `action_ref`: Action reference (preserved)
+- `config`: Snapshot of action configuration
+- `parent`: Parent execution ID (for workflows)
+- `enforcement`: Enforcement that triggered this execution
+- `executor`: Identity that initiated execution
+- `status`: Current execution state
+- `result`: JSON output from the action
+
+**Relationships**:
+- Executes: action
+- Triggered by: enforcement (if rule-driven) or executor (if manual)
+- Parent/child: execution (for workflows)
+- Executed by: identity
+
+**Purpose**: Tracks individual action runs, supports nested workflows, and stores results.
+
+### `InquiryStatus` (Enum)
+**Values**: `pending`, `responded`, `timeout`, `cancelled`
+
+**Purpose**: Tracks lifecycle state of user inquiries in workflows.
+
+### `Inquiry`
+**Purpose**: Represents an asynchronous user interaction within a workflow execution.
+
+**Key Fields**:
+- `execution`: Execution that is waiting on this inquiry
+- `prompt`: Question or prompt text for the user
+- `response_schema`: JSON schema defining expected response format
+- `assigned_to`: Identity who should respond to this inquiry
+- `status`: Current state of the inquiry
+- `response`: JSON response data from the user
+- `timeout_at`: When this inquiry expires
+- `responded_at`: When the response was received
+
+**Relationships**:
+- Belongs to: execution
+- Assigned to: identity
+
+**Purpose**: Enables workflows to pause and wait for human input/approval. When an action needs user interaction (e.g., approval, additional information, decision-making), it creates an Inquiry. The workflow execution pauses until the inquiry is responded to, times out, or is cancelled. This allows for human-in-the-loop automation patterns.
+
+**Use Cases**:
+- Approval workflows (deploy approval, expense approval)
+- Information gathering (incident details, configuration choices)
+- Decision points (which path to take in a workflow)
+- Manual verification steps
+
+---
+
+## Policy & Permissions
+
+### `PermissionGrant` (TypedDict)
+**Purpose**: Defines structure for permission grants.
+
+**Fields**:
+- `type`: `system`, `pack`, or `user`
+- `scope`: Pack name for pack-scoped permissions
+- `components`: List of component references
+
+**Purpose**: Structured permission definition for RBAC.
+
+### `PermissionSet`
+**Purpose**: A named collection of permissions (like a role).
+
+**Key Fields**:
+- `ref`: Unique reference (format: `pack.name`)
+- `label`: Display name
+- `grants`: Array of permission grants
+- `pack`: Parent pack (optional, for pack-defined roles)
+
+**Relationships**:
+- Belongs to: pack (optional)
+- Assigned to: identities via permission_assignment
+
+**Purpose**: Groups permissions together for assignment to users/services.
+
+### `PermissionAssignment`
+**Purpose**: Links identities to permission sets (many-to-many).
+
+**Key Fields**:
+- `identity`: Identity ID
+- `permset`: Permission set ID
+
+**Purpose**: Grants permissions to users/services.
+
+### `PolicyMethod` (Enum)
+**Values**: `cancel`, `enqueue`
+
+**Purpose**: Defines how to handle policy violations.
+
+### `Policy`
+**Purpose**: Defines execution policies for actions (rate limiting, concurrency control, etc.).
+
+**Key Fields**:
+- `ref`: Unique reference (format: `pack.name`)
+- `action`: Action this policy applies to
+- `parameters`: List of parameter names used for policy grouping
+- `method`: How to handle policy violations
+- `threshold`: Numeric limit (e.g., max concurrent executions)
+- `pack`: Parent pack
+
+**Relationships**:
+- Belongs to: pack
+- Applies to: action
+
+**Examples**: "Max 5 concurrent executions", "Rate limit to 100/hour"
+
+---
+
+## Identity & Access
+
+### `Identity`
+**Purpose**: Represents a user or service account.
+
+**Key Fields**:
+- `login`: Unique login identifier
+- `display_name`: Human-readable name
+- `attributes`: JSON for custom attributes (email, groups, etc.)
+
+**Relationships**:
+- Has: permission_assignments
+- Executes: executions
+- Owns: keys
+- Receives: inquiries (assigned tasks requiring response)
+
+**Purpose**: User/service account management for authentication and authorization.
+
+---
+
+## Configuration & Secrets
+
+### `OwnerType` (Enum)
+**Values**: `system`, `identity`, `pack`, `action`, `sensor`
+
+**Purpose**: Defines ownership scope for keys.
+
+### `Key`
+**Purpose**: Stores configuration values and secrets with ownership scoping.
+
+**Key Fields**:
+- `ref`: Unique reference (format: `[owner.]name`)
+- `owner_type`: Type of owner
+- `owner`: Owner identifier (auto-populated)
+- `owner_identity`, `owner_pack`, `owner_action`, `owner_sensor`: Foreign keys to owners
+- `name`: Key name
+- `encrypted`: Whether value is encrypted
+- `encryption_key_hash`: Hash of encryption key used
+- `value`: The actual value (encrypted if `encrypted=true`)
+
+**Constraints**:
+- Unique on: (owner_type, owner, name)
+- Exactly one owner FK must be set (validated by trigger)
+
+**Purpose**: Secure, scoped key-value storage for configuration and secrets.
+
+---
+
+## Notifications
+
+### `NotificationState` (Enum)
+**Values**: `created`, `queued`, `processing`, `error`
+
+**Purpose**: Tracks notification delivery state.
+
+### `Notification`
+**Purpose**: System notifications about entity changes.
+
+**Key Fields**:
+- `channel`: Notification channel (typically table name)
+- `entity_type`: Type of entity (table name)
+- `entity`: Entity identifier (typically ID or ref)
+- `activity`: Activity type (e.g., "created", "updated", "completed")
+- `state`: Processing state
+- `content`: JSON payload with notification details
+
+**Behavior**:
+- Automatically sends PostgreSQL `pg_notify` on insert
+- Used by triggers throughout the system to notify about changes
+
+**Purpose**: Real-time event streaming for UI updates and system integration.
+
+---
+
+## Artifacts & Storage
+
+### `ArtifactType` (Enum)
+**Values**: 
+- `file_binary`: Binary file for download
+- `file_datatable`: Tabular data for display
+- `file_image`: Image for display
+- `file_text`: Text file for display
+- `url`: URL reference
+- `progress`: Progress tracking data
+- `other`: Other types
+
+**Purpose**: Categorizes artifact display/handling behavior.
+
+### `RetentionPolicyType` (Enum)
+**Values**: `versions`, `days`, `hours`, `minutes`
+
+**Purpose**: Defines retention policy type.
+
+### `Artifact`
+**Purpose**: Manages execution output artifacts with retention policies.
+
+**Key Fields**:
+- `ref`: Artifact reference/path
+- `scope`: Owner type (system, identity, pack, action, sensor)
+- `owner`: Owner identifier
+- `type`: Artifact type
+- `retention_policy`: How to retain artifacts
+- `retention_limit`: Numeric limit for retention
+
+**Purpose**: Tracks files, logs, and other outputs from executions with automatic cleanup.
+
+---
+
+## Database Functions & Triggers
+
+The models file defines several PostgreSQL functions and triggers that provide:
+
+1. **Auto-population**: Automatically fill pack references, foreign keys, and refs
+2. **Validation**: Ensure referential integrity and format constraints
+3. **Configuration Snapshots**: Capture configuration at event/enforcement/execution creation time
+4. **Notifications**: Automatically create notification records on entity changes
+5. **Ownership Validation**: Enforce key ownership constraints
+
+These ensure data consistency and provide audit trails throughout the system.
+
+---
+
+## Common Patterns
+
+### Reference Format
+Most components use a `ref` field with format `pack.name` (e.g., `slack.webhook_trigger`). Runtimes use `pack.(action|sensor).name`.
+
+### Ref vs ID
+- Foreign key relationships use IDs
+- `*_ref` fields store the string reference and are preserved even if the referenced entity is deleted
+- Triggers auto-populate IDs from refs and vice versa
+
+### Configuration Snapshots
+Events, enforcements, and executions capture configuration snapshots in their `config` field to maintain execution context even if definitions change later.
+
+### Soft Cascades
+Many foreign keys use `SET NULL` on delete and preserve `*_ref` fields, allowing historical tracking even after components are removed.
+
+---
+
+## System Architecture Flow
+
+1. **Pack Installation**: Packs are installed, containing triggers, sensors, actions, rules, and policies
+2. **Sensor Monitoring**: Sensors watch for trigger conditions
+3. **Event Generation**: When a trigger fires, an event is created
+4. **Rule Evaluation**: Events trigger enforcements of matching rules
+5. **Execution**: Enforcements create executions of actions
+6. **Worker Processing**: Workers pick up and execute actions
+7. **Human Interaction** (optional): Actions can create inquiries to pause workflows and wait for user input
+8. **Result Storage**: Execution results and artifacts are stored
+9. **Notifications**: Changes flow through the notification system for real-time updates
+
+This architecture supports both rule-driven automation and manual action execution, with full audit trails, workflow orchestration capabilities, and human-in-the-loop patterns via inquiries.