attune-system/attune
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
attune/work-summary/status/TODO.md
David Culbreth 3b14c65998 re-uploading work
2026-02-04 17:46:30 -06:00

50 KiB
Raw Permalink Blame History

Attune Implementation TODO

Last Updated: 2026-01-28 (OpenAPI Nullable Fields Fix)
Current Status: Core Platform Complete - Type-Safe API Client - E2E Tests Ready - Pack Testing Framework Designed

Note

: This is a clean, streamlined TODO with only outstanding tasks.
See TODO.OLD.md for the complete historical record of all completed work.

📊 Project Status Overview

✅ Completed Core Platform

All 5 Core Services Implemented:

  • ✅ API Service - REST API with JWT auth, 40+ endpoints, OpenAPI docs
  • ✅ Executor Service - Enforcement processing, execution scheduling, policy enforcement
  • ✅ Worker Service - Python/Shell/Local runtimes, artifact management, secret injection
  • ✅ Sensor Service - Timer triggers, event generation, rule evaluation
  • ✅ Notifier Service - Real-time notifications via WebSocket

CLI Tool:

  • ✅ attune-cli - Comprehensive command-line interface
    • Authentication (login, logout, whoami)
    • Pack management (install, list, show, uninstall, register)
    • Action execution with parameters
    • Rule management (create, enable, disable, list)
    • Execution monitoring with advanced filtering (pack, action, status, result search)
    • Raw result extraction for piping to other tools
    • Trigger and sensor inspection
    • Configuration management
    • Multiple output formats (table, JSON, YAML)
    • Shorthand output flags (-j for JSON, -y for YAML)
    • Interactive prompts and colored output
    • Unix-friendly piping and interoperability

Database & Infrastructure:

  • ✅ 22 tables across 5 consolidated migration files
  • ✅ Repository layer with full CRUD operations
  • ✅ RabbitMQ message queue infrastructure
  • ✅ PostgreSQL with LISTEN/NOTIFY for real-time updates

Advanced Features:

  • ✅ FIFO execution ordering with per-action queues
  • ✅ Workflow orchestration (YAML-based, task graphs, variables)
  • ✅ JWT authentication with access/refresh tokens
  • ✅ Secret management (stdin injection, not env vars)
  • ✅ Per-pack Python virtual environments (dependency isolation)
  • ✅ Log size limits (10MB default, streaming)
  • ✅ Human-in-the-loop inquiries

Web Frontend:

  • ✅ Auto-generated TypeScript client from OpenAPI spec
  • ✅ 90+ type definitions, 13 service classes
  • ✅ Full type safety and compile-time schema validation
  • ✅ React 18 + TypeScript + Vite + TanStack Query
  • ✅ JWT authentication with token refresh
  • ✅ Protected routes and main layout

Testing:

  • ✅ 96 common library tests passing
  • ✅ 55 executor unit tests + 8 integration tests
  • ✅ 43 worker tests with security validation
  • ✅ 46 API tests (16 health + auth tests passing)
  • ✅ 76 core pack tests (36 bash + 38 Python) - 100% action coverage

🎯 Current Focus: Production Deployment

Priority 0: Pack Registry System (Phases 1-6)

Status: ✅ COMPLETE - All Phases Implemented & Tested
Priority: P0 - Core infrastructure for pack distribution
Design Doc: docs/pack-registry.md

Goal: Enable secure, versioned pack distribution with multiple installation sources and dependency validation

Phase 1: Registry Infrastructure ✅ COMPLETE

  • Registry index format (JSON with pack metadata)
  • PackIndexEntry model with install sources
  • RegistryClient for index fetching and searching
  • Configuration for registry URLs and caching
  • Multi-registry priority-based search

Phase 2: Installation Sources ✅ COMPLETE

  • Git repository cloning (HTTPS/SSH)
  • Archive download and extraction (.zip, .tar.gz, .tgz)
  • Local directory copying
  • Local archive file extraction
  • Registry reference resolution
  • Checksum verification

Phase 3: Enhanced Installation ✅ COMPLETE

  • pack_installation database table for metadata tracking
  • PackInstallationRepository for CRUD operations
  • PackStorage utility for versioned storage management
  • Checksum calculation utilities (SHA256)
  • Installation metadata (source, user, timestamp, checksum)
  • CLI attune pack checksum command

Phase 4: Dependency Validation & Tools ✅ COMPLETE

  • DependencyValidator for runtime and pack dependencies
  • Semver version parsing and constraint matching
  • Runtime version detection (Python, Node.js, shell)
  • Progress reporting infrastructure (ProgressEvent, ProgressCallback)
  • CLI attune pack index-entry command

Phase 5: Integration & Testing ✅ COMPLETE

  • Wire progress events to CLI (enhanced output)
  • Dependency validation in install flow (API)
  • CLI flags: --skip-deps, --skip-tests
  • CLI attune pack index-update command
  • CLI attune pack index-merge command
  • CI/CD examples documentation (548 lines)
  • Test scenarios documented

Phase 6: Comprehensive Integration Testing ✅ COMPLETE

  • CLI integration tests (17/17 passing - 100%)
  • Pack checksum command tests
  • Pack index-entry generation tests
  • Pack index-update tests
  • Pack index-merge tests
  • Error handling and edge case tests
  • Output format validation (JSON, YAML, table)
  • API integration tests (14/14 passing - 100%)
  • Pack installation error handling (400 for validation errors, 404 for missing sources)
  • Dependency validation with wildcard version support (*)
  • Missing pack.yaml detection and proper error responses

Phase 7: Production Readiness (NEXT - Optional)

  • Git clone from remote repositories (integration test)
  • Archive download from HTTP URLs (integration test)
  • Performance testing (large packs, concurrent operations)
  • CI/CD integration (automated test execution)

Benefits Delivered:

  • ✅ Multi-source pack installation (git, archive, local, registry)
  • ✅ Automated dependency validation with wildcard version support
  • ✅ Complete installation audit trail
  • ✅ Checksum verification for security
  • ✅ Registry index management tools
  • ✅ CI/CD integration documentation
  • ✅ Production-ready CLI with 100% test coverage
  • ✅ Production-ready API with 100% test coverage
  • ✅ Proper error handling (400/404/500 status codes)

Priority 0: Pack Testing Framework (Phases 1-5)

Status: ✅ COMPLETE - All Core Features Implemented
Priority: P0 - Enables programmatic pack validation
Design Doc: docs/pack-testing-framework.md

Goal: Enable automatic test execution during pack installation to validate packs work in the target environment

Phase 1: Core Framework ✅ COMPLETE

  • Design document created (831 lines)
  • Core pack tests implemented (76 tests, 100% passing)
  • Core pack pack.yaml updated with testing configuration
  • Database migration for pack test results
    • pack_test_execution table
    • pack_test_summary view
    • Test history tracking
  • Worker test executor implementation
    • Test discovery from pack.yaml
    • Runtime-aware test execution (shell, unittest, pytest)
    • Simple output parser
    • Structured test result generation

Phase 2: CLI Integration ✅ COMPLETE

  • attune pack test <pack> command
  • Support for local pack directories and installed packs
  • Test result display (colored output, verbose, detailed modes)
  • JSON/YAML output formats for scripting
  • Exit code handling (fails on test failure)
  • Integration with pack install/register workflow
  • Force/skip test options (--skip-tests, --force)

Phase 3: API Integration ✅ COMPLETE

  • API endpoints for test execution
    • POST /packs/{ref}/test - Execute pack tests
    • GET /packs/{ref}/tests - Get test history (paginated)
    • GET /packs/{ref}/tests/latest - Get latest test result
    • POST /packs/register - Register pack with automatic testing
    • POST /packs/install - Install pack (stub for future implementation)
  • Test result storage in database
  • OpenAPI documentation (ToSchema derives)
  • API documentation (docs/api-pack-testing.md)
  • Pack install integration documentation (docs/pack-install-testing.md)
  • Web UI for viewing test history

Phase 4: Pack Install Integration ✅ COMPLETE

  • POST /packs/register endpoint for local pack registration
  • Automatic test execution during registration
  • Fail-fast validation (registration fails if tests fail)
  • --skip-tests and --force flags for flexible control
  • Test result display in CLI output
  • Rollback on test failure (unless forced)
  • Documentation (docs/pack-install-testing.md)
  • POST /packs/install for remote git sources (TODO: Future)

Phase 5: Web UI Integration ✅ COMPLETE

  • PackTestResult component for detailed test display
  • PackTestBadge component for status indicators
  • PackTestHistory component for test execution list
  • Pack detail page integration (latest test results)
  • Pack registration page with test control options
  • React Query hooks for test data (usePackTests)
  • Manual test execution from UI
  • Test history viewing with pagination
  • Documentation (docs/web-ui-pack-testing.md)

Phase 6: Advanced Features (FUTURE)

  • JUnit XML parser for pytest/Jest results
  • TAP (Test Anything Protocol) parser
  • Test caching and optimization
  • Remote pack installation from git repositories
  • Async test execution (return job ID, poll for results)
  • Webhooks for test completion notifications
  • Real-time test execution updates via WebSocket
  • Test comparison and trend analysis

Benefits:

  • Fail-fast pack installation (tests must pass to activate)
  • Dependency validation in actual environment
  • Audit trail of pack test results
  • Quality assurance for pack ecosystem

🚀 Path to Production

Priority 1: Critical Blockers (Before v1.0)

1.1 OpenAPI Schema & Client Generation

Status: ✅ COMPLETE (All phases done, 0 errors)
Priority: P0 - COMPLETED
Completed: 2026-01-28

Recent Fix (2026-01-28): Nullable Fields Issue

  • Problem: E2E tests failing with TypeError: 'NoneType' object is not iterable when generated Python client encountered null values for optional JSON object fields
  • Root Cause: utoipa wasn't marking Option<JsonValue> fields as nullable in OpenAPI spec
  • Solution: Added nullable = true attribute to 23 field annotations across 7 DTO files
  • Result: Generated Python client now correctly handles null values
  • Details: See work-summary/2026-01-28-openapi-nullable-fields-fix.md

Goal: Migrate all frontend code from manual axios calls to generated OpenAPI client

Phase 1: Core Infrastructure ✅ COMPLETE

  • Generate TypeScript client from OpenAPI spec (90+ types, 13 services)
  • Configure web/src/lib/api-config.ts with JWT token injection
  • Update web/src/types/api.ts to re-export generated types
  • Migrate AuthContext to use AuthService

Phase 2: Hooks Migration ✅ COMPLETE

  • ✅ Migrate useActions() to use ActionsService
  • ✅ Migrate useExecutions() to use ExecutionsService
  • ✅ Migrate usePacks() to use PacksService
  • ✅ Migrate useRules() to use RulesService

Phase 3: Schema Alignment ✅ COMPLETE - All files migrated (100%)

  • ✅ RuleForm.tsx - Fixed all field names and parameter names
  • ✅ ActionDetailPage.tsx - Fixed all field names, removed non-existent fields
  • ✅ ActionsPage.tsx - Fixed table display and field names
  • ✅ DashboardPage.tsx - Fixed ExecutionStatus enum, pagination metadata
  • ✅ App.tsx - Updated action routes to use :ref instead of :id
  • ✅ RuleDetailPage.tsx - Updated to use :ref routes, fixed all field mappings
  • ✅ RuleEditPage.tsx - Updated to use :ref routes and ApiResponse wrapper
  • ✅ SensorsPage.tsx - Fixed pagination, field names, ref-based routing
  • ✅ SensorDetailPage.tsx - Fixed field mappings, removed deprecated fields
  • ✅ TriggersPage.tsx - Fixed pagination, field names, ref-based routing
  • ✅ TriggerDetailPage.tsx - Fixed schema field names (param_schema, out_schema)
  • ✅ ExecutionsPage.tsx - Fixed pagination and ExecutionStatus enum
  • ✅ types/api.ts - Removed unused GeneratedExecutionStatus import
  • ✅ PackForm.tsx - Updated to use PackResponse type
  • ✅ PackEditPage.tsx - Fixed ApiResponse wrapper access
  • ✅ EventsPage.tsx - Fixed field names and pagination
  • ✅ EventDetailPage.tsx - Fixed field names
  • ✅ ExecutionDetailPage.tsx - Fixed ExecutionStatus enums, removed non-existent fields
  • ✅ useEvents.ts - Fixed EnforcementStatus type
  • ✅ useSensors.ts - Migrated to generated client
  • ✅ useTriggers.ts - Migrated to generated client

Phase 4: Testing & Validation ✅ COMPLETE

  • ✅ All TypeScript errors resolved (231 → 0)
  • ✅ Build succeeds without errors
  • ✅ Compile-time type checking verified
  • ✅ Schema alignment confirmed (all field names match backend)

Key Changes Applied:

  • Field names: name → ref/label, pack_id → pack, pack_name → pack_ref
  • Parameters: page_size → pageSize, pack_ref → packRef
  • Pagination: items → data, total → total_items
  • ExecutionStatus: String literals → Enum values (e.g., "running" → ExecutionStatus.RUNNING)
  • Removed non-existent fields: enabled, runner_type, metadata, elapsed_ms

Documentation:

  • ✅ web/src/api/README.md - Generated client usage guide
  • ✅ web/MIGRATION-TO-GENERATED-CLIENT.md - Migration guide with examples
  • ✅ web/API-CLIENT-QUICK-REFERENCE.md - Quick reference
  • ✅ docs/openapi-client-generation.md - Architecture docs
  • ✅ web/API-MIGRATION-STATUS.md - Migration progress and field mapping (updated)

Files Completed:

  • ✅ web/src/contexts/AuthContext.tsx - Migrated to AuthService
  • ✅ web/src/hooks/useActions.ts - Migrated to ActionsService
  • ✅ web/src/hooks/useExecutions.ts - Migrated to ExecutionsService
  • ✅ web/src/hooks/usePacks.ts - Migrated to PacksService
  • ✅ web/src/hooks/useRules.ts - Migrated to RulesService
  • ✅ web/src/pages/actions/ActionDetailPage.tsx - Fixed all schema issues
  • ✅ web/src/pages/actions/ActionsPage.tsx - Fixed all schema issues
  • ✅ web/src/pages/dashboard/DashboardPage.tsx - Fixed all schema issues
  • ✅ web/src/components/forms/RuleForm.tsx - Fixed all schema issues
  • ✅ web/src/App.tsx - Updated routing

TypeScript Errors: 0 (100% reduction from 231 initial errors) ✅

Migration Results:

  • ✅ 25+ files migrated to generated client
  • ✅ Zero manual axios calls remaining
  • ✅ Full compile-time type safety achieved
  • ✅ All field names aligned with backend schema
  • ✅ Build succeeds with no errors

Commands:

# Regenerate client after backend changes
cd web
npm run generate:api

# Test build for type errors
npm run build

1.2 End-to-End Integration Testing

Status: 🔄 IN PROGRESS (Tier 1: ✅ COMPLETE, Tier 2: ✅ COMPLETE, Tier 3: 🔄 81% COMPLETE)
Priority: P0 - BLOCKING
Estimated Time: 0.25 days remaining (Tier 3: 4 low-priority scenarios left)

Phase 1: E2E Environment Setup (✅ COMPLETE)

  • Document test plan with 8 scenarios
  • Create config.e2e.yaml with test-specific settings
  • Create test fixtures (test_pack with echo action)
  • Create simple workflow for testing
  • Set up test directory structure
  • Create E2E database and run migrations
  • Create database setup script (scripts/setup-e2e-db.sh)
  • Create service management scripts (start/stop)

Tier 1: Core Automation Flows (✅ COMPLETE - 8 scenarios, 33 tests)

  • T1.1: Interval Timer Automation (2 tests)
  • T1.2: Date Timer (One-Shot Execution) (3 tests)
  • T1.3: Cron Timer Execution (4 tests)
  • T1.4: Webhook Trigger with Payload (4 tests)
  • T1.5: Workflow with Array Iteration (5 tests)
  • T1.6: Key-Value Store Access (7 tests)
  • T1.7: Multi-Tenant Isolation (4 tests)
  • T1.8: Action Failure Handling (5 tests)

Tier 2: Orchestration & Data Flow (✅ COMPLETE - 13 scenarios, 37 tests)

  • T2.1: Nested Workflow Execution (2 tests)
  • T2.3: Datastore Write Operations (4 tests)
  • T2.5: Rule Criteria Evaluation (4 tests)
  • T2.6: Inquiry/Approval Workflows (4 tests)
  • T2.8: Retry Policy Execution (4 tests)

Tier 3: Advanced Features & Edge Cases (🔄 81% COMPLETE - 17/21 scenarios, 56 tests)

  • T3.1: Date Timer with Past Date (3 tests) - HIGH priority
  • T3.2: Timer Cancellation (3 tests) - HIGH priority
  • T3.3: Multiple Concurrent Timers (3 tests) - HIGH priority
  • T3.4: Webhook with Multiple Rules (2 tests) - LOW priority
  • T3.5: Webhook with Rule Criteria Filtering (4 tests) - HIGH priority
  • T3.7: Complex Workflow Orchestration (4 tests) - MEDIUM priority ✨ NEW
  • T3.8: Chained Webhook Triggers (4 tests) - MEDIUM priority ✨ NEW
  • T3.9: Multi-Step Approval Workflow (4 tests) - MEDIUM priority ✨ NEW
  • T3.10: RBAC Permission Checks (4 tests) - HIGH priority
  • T3.11: System vs User Packs (4 tests) - MEDIUM priority
  • T3.13: Invalid Action Parameters (4 tests) - HIGH priority
  • T3.14: Execution Completion Notifications (4 tests) - MEDIUM priority
  • T3.15: Inquiry Creation Notifications (4 tests) - MEDIUM priority
  • T3.16: Rule Trigger Notifications (4 tests) - MEDIUM priority ✨ NEW
  • T3.17: Container Runner Execution (4 tests) - MEDIUM priority
  • T3.18: HTTP Runner Execution (4 tests) - MEDIUM priority
  • T3.20: Secret Injection Security (4 tests) - HIGH priority
  • T3.21: Action Log Size Limits (4 tests) - MEDIUM priority
  • T3.6: Sensor-generated custom events - LOW priority
  • T3.12: Worker crash recovery - LOW priority
  • T3.19: Dependency conflict isolation - LOW priority

Files Created:

  • tests/E2E_TESTS_COMPLETE.md - Complete test documentation (832 lines)
  • tests/README.md - Test overview and quick start (300+ lines, updated)
  • tests/e2e/tier1/ - 8 test files, 33 tests (✅ COMPLETE)
  • tests/e2e/tier2/ - 5 test files, 37 tests (✅ COMPLETE)
  • tests/e2e/tier3/ - 18 test files, 56 tests (🔄 81% COMPLETE)
  • tests/helpers/ - Client, fixtures, polling utilities (~2,600 lines)
  • tests/run_e2e_tests.sh - Test runner with tier filtering
  • tests/pytest.ini - Pytest configuration with markers
  • config.e2e.yaml - E2E test configuration

Test Infrastructure:

  • ✅ AttuneClient: Full REST API wrapper with auth, retry logic
  • ✅ Pytest fixtures: client, test_pack, unique_ref generators
  • ✅ Polling helpers: wait_for_execution, wait_for_event, wait_for_inquiry_count
  • ✅ Test runner: Service health checks, automatic cleanup
  • ✅ Modular test organization: tier1/, tier2/, tier3/ directories
  • ✅ 26+ pytest markers for flexible test filtering

Acceptance Criteria:

  • All services can communicate via RabbitMQ
  • Timer triggers successfully execute actions (interval, date, cron)
  • Rules properly evaluate and create enforcements
  • Workflows execute tasks in correct order (sequential, parallel)
  • Inquiries pause and resume executions
  • Secrets are never exposed in logs/environment
  • API health checks pass
  • Authentication and JWT token generation works
  • Pack registration completes successfully
  • Actions can be created via API
  • RBAC permission enforcement works correctly
  • HTTP runner executes external API calls
  • Container runner executes in isolated Docker containers
  • Execution notifications tracked for real-time updates
  • Log size limits enforced (max 10MB)
  • Complex workflow orchestration (parallel, branching, conditionals)
  • Chained webhook triggers (multi-level cascades)
  • Multi-step approval workflows (sequential, conditional)
  • Rule trigger notifications tracked
  • Sensor-generated custom events
  • Worker crash recovery and resumption
  • Dependency conflict isolation

Next Steps:

  1. (Optional) Complete remaining 3 low-priority Tier 3 scenarios (T3.6, T3.12, T3.19)
  2. Implement WebSocket test infrastructure for real-time notification testing
  3. Add CI/CD pipeline integration (GitHub Actions)
  4. Add performance benchmarks and reporting

🎉 Milestone Achieved: All HIGH and MEDIUM priority E2E tests complete!

1.2 Deployment Infrastructure

Status: 🔄 Not Started
Priority: P0 - BLOCKING
Estimated Time: 5-7 days

Tasks:

  • Create Dockerfile for API service
  • Create Dockerfile for Executor service
  • Create Dockerfile for Worker service
  • Create Dockerfile for Sensor service
  • Create Dockerfile for Notifier service
  • Create docker-compose.yml for local development
  • Create docker-compose.production.yml
  • Add health check endpoints to all services
  • Add readiness probes to all services
  • Document Docker deployment process
  • Test multi-service Docker deployment
  • Create example .env file with all required variables

Acceptance Criteria:

  • All services can be built as Docker images
  • docker-compose brings up full stack
  • Services can communicate in Docker network
  • Health checks return proper status
  • Configuration via environment variables works
  • Database migrations run on startup

1.3 Observability & Monitoring

Status: 🔄 Not Started
Priority: P1 - HIGH
Estimated Time: 4-6 days

Tasks:

  • Standardize structured logging across all services
  • Add correlation IDs for request tracing
  • Implement OpenTelemetry spans for distributed tracing
  • Add Prometheus metrics endpoints to all services
  • Create Grafana dashboards for key metrics
  • Document log formats and levels
  • Add error tracking integration (Sentry?)
  • Create alerting rules for critical errors

Key Metrics to Track:

  • Execution throughput (executions/minute)
  • Queue lengths per action
  • Worker utilization
  • API response times
  • Database query performance
  • Message queue lag
  • Error rates by service

Acceptance Criteria:

  • All services emit structured JSON logs
  • Traces can follow requests across services
  • Metrics are exported and queryable
  • Dashboards show system health at a glance
  • Alerts fire for critical conditions

1.4 Production Configuration Management

Status: 🔄 Not Started
Priority: P1 - HIGH
Estimated Time: 2-3 days

Tasks:

  • Create config templates for dev/staging/production
  • Document all configuration options
  • Implement configuration validation on startup
  • Add JWT_SECRET generation guide
  • Add database connection pooling tuning guide
  • Document RabbitMQ configuration options
  • Create environment variable reference
  • Add configuration troubleshooting guide
  • Document secrets rotation procedures

Acceptance Criteria:

  • Configuration files exist for all environments
  • All config options are documented
  • Services validate config on startup
  • Clear error messages for misconfiguration
  • Security best practices documented

Priority 2: Production Hardening

2.1 Security Audit

Status: 🔄 Not Started
Priority: P1 - HIGH
Estimated Time: 3-4 days

Tasks:

  • Run cargo-audit for dependency vulnerabilities
  • Scan Docker images for vulnerabilities
  • Review authentication/authorization implementation
  • Audit secret handling across all services
  • Review SQL injection prevention (SQLx compile-time checks)
  • Test rate limiting on API endpoints
  • Review input validation on all endpoints
  • Document security architecture
  • Create security incident response plan

Acceptance Criteria:

  • No critical vulnerabilities in dependencies
  • Docker images pass security scans
  • Authentication properly enforced
  • Secrets never logged or exposed
  • SQL injection not possible
  • Input validation comprehensive

2.2 Performance Testing & Optimization

Status: 🔄 Not Started
Priority: P2 - MEDIUM
Estimated Time: 4-5 days

Tasks:

  • Set up load testing environment (k6 or Locust)
  • Create load test scenarios
    • High API request volume
    • Many concurrent executions
    • Large workflow executions
    • Queue buildup and processing
  • Run performance benchmarks
  • Identify bottlenecks
  • Optimize database queries
  • Tune connection pool sizes
  • Optimize message queue throughput
  • Document performance characteristics
  • Set performance SLOs

Target Metrics:

  • API p99 latency < 100ms
  • Support 1000+ concurrent executions
  • Process 100+ events/second
  • Handle 10,000+ queued executions

Acceptance Criteria:

  • Load tests run successfully
  • Performance bottlenecks identified and documented
  • Critical optimizations implemented
  • Performance metrics baselined

2.3 Error Handling & Resilience

Status: 🔄 Not Started
Priority: P2 - MEDIUM
Estimated Time: 3-4 days

Tasks:

  • Implement retry logic with exponential backoff
  • Add circuit breakers for external dependencies
  • Implement graceful degradation patterns
  • Add connection retry logic for database
  • Add connection retry logic for RabbitMQ
  • Test failure scenarios
    • Database connection loss
    • RabbitMQ connection loss
    • Worker crashes during execution
    • Out of memory conditions
    • Disk full scenarios
  • Document failure modes and recovery
  • Create chaos testing suite

Acceptance Criteria:

  • Services recover from transient failures
  • No data loss on service restart
  • Executions resume after crash
  • Clear error messages for operators

Priority 3: Documentation & Developer Experience

3.1 User Documentation

Status: 🔄 Partial
Priority: P2 - MEDIUM
Estimated Time: 4-5 days

Completed:

  • ✅ API endpoint documentation (10 files)
  • ✅ Workflow orchestration guide
  • ✅ Queue architecture documentation
  • ✅ Configuration reference
  • ✅ Authentication guide

Outstanding:

  • Getting Started guide
  • Deployment guide (Docker/Kubernetes)
  • Pack development tutorial
  • Workflow authoring guide
  • Troubleshooting guide
  • Operations runbook
  • Migration guide from StackStorm
  • API client examples (curl, Python, JavaScript)
  • Architecture overview document

3.2 Developer Setup Improvements

Status: 🔄 Not Started
Priority: P3 - LOW
Estimated Time: 2-3 days

Tasks:

  • Create setup script for local development
  • Add Makefile with common commands
  • Document IDE setup (VS Code, IntelliJ)
  • Create development database seeding script
  • Add example packs for testing
  • Document debugging procedures
  • Create development troubleshooting guide

Acceptance Criteria:

  • New developer can get system running in < 30 minutes
  • Common tasks automated via Makefile
  • Clear debugging documentation

Priority 4: Feature Completeness

4.1 Pack Management Features

Status: 🔄 In Progress
Priority: P2 - MEDIUM
Estimated Time: 8-10 days

Tasks:

Phase 1: Pack Registry System (3-4 days) ✅ COMPLETE

  • Design and implement pack index file format (JSON schema)
  • Implement registry configuration system (multi-registry, priority-based)
  • Create registry client for fetching and parsing index files
  • Implement registry caching (TTL-based)
  • Add registry authentication support (custom headers)
  • Create registry search functionality
  • Add attune pack registries CLI command
  • Add attune pack search CLI command

Phase 2: Pack Installation Sources (3-4 days) ✅ COMPLETE

  • Implement git repository installation source
    • Clone repository to temp directory
    • Support branch/tag/commit refs
    • Handle SSH and HTTPS URLs
  • Implement archive URL installation source
    • Download and extract .zip, .tar.gz, .tgz
    • Verify checksums (sha256, sha512)
  • Implement local directory installation source
  • Implement local archive upload installation source
  • Implement registry reference resolution
    • Search registries in priority order
    • Resolve install source from index entry
    • Handle version specifications (@version, @latest)

Phase 3: Enhanced Installation Process (2-3 days) ✅ COMPLETE

  • Add checksum generation and verification utilities
  • Implement installation metadata tracking (database table)
  • Add pack storage management (permanent versioned location)
  • Add attune pack checksum CLI command for pack authors
  • Track installation source, checksum, timestamp, and user
  • Improve error handling with I/O error types
  • Move packs to permanent storage with versioning

Phase 4: Dependency Validation & Enhanced Features (2-3 days) ✅ COMPLETE

  • Implement runtime dependency validation (Python/Node.js/shell versions)
  • Add pack dependency resolution and validation
  • Add version constraint checking (semver with ^, ~, >=, etc.)
  • Progress indicators infrastructure during installation
  • Implement attune pack index-entry CLI command
    • Parse pack.yaml and extract metadata
    • Generate install sources with checksums
    • Output JSON index entry

Phase 5: Additional Tools & Integration Testing (1-2 days)

  • CLI integration for progress reporting
  • API integration for dependency validation
  • Implement attune pack index-update CLI command
    • Add/update pack entry in existing index
    • Validate index schema
  • Implement attune pack index-merge CLI command
  • Create CI/CD integration examples (GitHub Actions)
  • Comprehensive integration tests for all Phase 1-4 features

Documentation:

  • Pack registry specification (docs/pack-registry-spec.md)
  • Example registry index file (docs/examples/registry-index.json)
  • Registry configuration example (config.registry-example.yaml)
  • Phase 1 work summary (work-summary/2024-01-21-pack-registry-phase1.md)
  • Phase 2 work summary (work-summary/2024-01-21-pack-registry-phase2.md)
  • Phase 3 work summary (work-summary/2024-01-22-pack-registry-phase3.md)
  • Phase 4 work summary (work-summary/2024-01-22-pack-registry-phase4.md)
  • Pack installation guide (user-facing)
  • Registry hosting guide
  • CI/CD integration guide

Acceptance Criteria:

  • ✅ Packs can be installed from git URLs, HTTP archives, local directories, and local archives
  • ✅ Packs can be discovered and installed by reference from configured registries
  • ✅ Multiple registries can be configured with priority-based search
  • ✅ Checksums are verified during installation
  • ✅ Pack maintainers can generate index entries via CLI
  • ✅ Registry indices can be hosted independently (decentralized)
  • ✅ Installation metadata tracked in database
  • ✅ Dependency validation system implemented
  • 🔄 Installation process validates dependencies (needs integration)
  • 🔄 Comprehensive integration tests (needs implementation)

4.2 Advanced Workflow Features

Status: 🔄 Partial (Phase 1 & 2 Complete)
Priority: P3 - LOW
Estimated Time: 3-4 weeks

Completed:

  • ✅ Workflow parser and validator
  • ✅ Task graph execution engine
  • ✅ Variable system (task, vars, params)
  • ✅ Sequential and parallel task execution
  • ✅ With-items iteration support
  • ✅ Conditional task execution (when clauses)
  • ✅ Retry and timeout support

Outstanding (Phase 3 - Advanced Features):

  • Error handling (on-error, on-complete, on-success)
  • Nested workflows (workflow calling workflow)
  • Dynamic task generation
  • Complex join logic (join: all, any, count)
  • Task result transformation
  • Advanced variable scoping

Outstanding (Phase 4 - API & Tools):

  • Workflow execution control API (pause/resume/cancel)
  • Workflow visualization endpoint
  • Workflow dry-run/validation endpoint
  • Visual workflow editor (future)

Outstanding (Phase 5 - Testing & Docs):

  • Comprehensive workflow integration tests
  • Workflow authoring tutorial
  • Workflow pattern library
  • Performance testing for large workflows

4.3 Example Packs

Status: ✅ COMPLETE (Core Pack)
Priority: P3 - LOW
Estimated Time: N/A

Core Pack: ✅ COMPLETE - PRODUCTION READY

  • core.intervaltimer trigger (interval-based timer)
  • core.crontimer trigger (cron-based timer)
  • core.datetimetimer trigger (one-shot datetime timer)
  • core.interval_timer_sensor sensor (built-in)
  • core.echo action (shell)
  • core.sleep action (shell)
  • core.noop action (shell)
  • core.http_request action (Python, full HTTP client)
  • Pack structure with proper YAML metadata
  • Comprehensive README documentation
  • Loader script (scripts/load_core_pack.py)
  • Shell wrapper (scripts/load-core-pack.sh)
  • Setup documentation (packs/core/SETUP.md)
  • README integration (Getting Started section)
  • core.webhook trigger (future Phase 4)
  • core.local_command action (future)

Utilities Pack:

  • utils.json_parse action
  • utils.template action (Jinja2)
  • utils.sleep action
  • utils.fail action (for testing)

Notification Pack:

  • notify.email action
  • notify.slack action
  • notify.webhook action

Git Pack:

  • git.clone action
  • git.push trigger (webhook)
  • git.pull_request trigger

Priority 5: Future Enhancements

5.1 RBAC (Role-Based Access Control)

Status: 🔄 Not Started
Priority: P3 - LOW
Estimated Time: 2-3 weeks

Tasks:

  • Define permission model
  • Implement role-based permissions
  • Add permission checks to API endpoints
  • Add pack-level permissions
  • Add action-level permissions
  • Create admin UI for permission management
  • Document RBAC model

5.2 Multi-Tenancy

Status: 🔄 Not Started
Priority: P4 - FUTURE
Estimated Time: 3-4 weeks

Tasks:

  • Add organization/tenant concept
  • Isolate data by tenant
  • Add tenant-aware authentication
  • Implement tenant quotas
  • Add tenant billing integration
  • Document multi-tenancy architecture

5.3 Web UI

Status: 🔄 In Progress
Priority: P4 - FUTURE
Estimated Time: 8-12 weeks

Completed:

  • Project setup (React 18 + TypeScript + Vite)
  • Authentication system (JWT login/logout)
  • Protected routes with auth guards
  • Main layout with sidebar navigation
  • API client with axios (token refresh)
  • TanStack Query configuration
  • Tailwind CSS styling setup
  • Type definitions for API models
  • Custom React hooks (usePacks, useActions, useExecutions, useRules)
  • Packs list page with data fetching
  • Actions list page with data fetching
  • Executions list page with real-time auto-refresh
  • Pack detail page (full info, actions list, management)
  • Action detail page (parameters, execution form, recent executions)
  • Execution detail page (status, timeline, parameters, result)
  • Rules list page with filtering and management
  • Rule detail page (full info, criteria, action parameters)
  • Dashboard with live metrics (packs, rules, actions, executions)
  • Real-time execution updates via SSE
  • Status distribution charts and success rate
  • Recent activity feed with live updates
  • Events list page with trigger filtering
  • Event detail page (payload display, metadata)
  • Triggers list page with pack filtering
  • Trigger detail page (schemas, quick links)
  • Sensors list page with enable/disable toggle
  • Sensor detail page (entry point, poll interval, trigger types)
  • Custom React hooks (useEvents, useTriggers, useSensors)
  • Updated sidebar navigation with all entity types
  • Rule detail page: Enforcements tab (audit trail view)
  • Rule create/edit form with pack, trigger, action selection
  • Pack registration form with config schema definition
  • RuleCreatePage and PackCreatePage routes
  • Create buttons on Rules and Packs list pages

Architecture Notes:

  • Actions and Sensors are code-based, registered via pack installation (NOT editable in UI)
  • Triggers are pack-based except for ad-hoc packs (only ad-hoc triggers are UI-configurable)
  • Workflow actions will be UI-configurable in the future (separate feature)

Completed (2026-01-19):

  • Edit functionality integration for rules and packs
  • Rule edit page with form integration
  • Pack edit page with system/ad-hoc constraints
  • Prominent enable/disable toggle on rule detail page
  • YAML source export for rules (pack deployment)
  • Edit buttons on rule and pack detail pages

In Progress:

  • Trigger create/edit form (for ad-hoc packs only)

TODO:

  • Trigger create/edit form implementation
  • Log viewer with filtering
  • User management interface
  • Visual workflow editor (React Flow) - includes workflow action configuration
  • Settings page
  • Automated tests (Vitest + Playwright)
  • API client code generation from OpenAPI spec (optional)

📝 Critical Next Steps (This Week)

Current Focus: End-to-End Integration Testing

Phase 1: Setup (Current - 80% Complete)

  1. Create E2E database: createdb attune_e2e ✅ TODO
  2. Run migrations against E2E database ✅ TODO
  3. Verify all 5 services start with config.e2e.yaml ✅ TODO
  4. Create test helper utilities (api_client, service_manager) ✅ TODO

Phase 2: Basic Tests (Next - 2-3 days)

  1. Timer automation test - Full chain verification
  2. Workflow execution test - 3-task sequential workflow
  3. FIFO ordering test - Verify execution ordering

Phase 3: Advanced Tests (Following - 2-3 days)

  1. Secret management test
  2. Inquiry (human-in-the-loop) test
  3. Error handling and retry test
  4. Real-time notification test
  5. Dependency isolation test

Then Proceed To:

  • Docker Infrastructure (3-4 days)
  • Basic Observability (parallel, 2-3 days)

📝 Recent Changes

Session 11 (2026-01-28) - OpenAPI Nullable Fields Fix

Status: ✅ COMPLETE
Priority: P0 - CRITICAL BLOCKER

Problem Solved:

  • E2E tests were crashing with TypeError: 'NoneType' object is not iterable
  • Generated Python OpenAPI client couldn't handle null values in optional JSON fields
  • Fields like param_schema, out_schema, config were causing deserialization failures

Solution:

  • Added nullable = true to 23 Option<JsonValue> field annotations across 7 DTO files
  • Added #[serde(skip_serializing_if = "Option::is_none")] to request DTOs
  • Regenerated Python client with fixed OpenAPI spec

Files Modified:

  • crates/api/src/dto/action.rs - 6 fields
  • crates/api/src/dto/trigger.rs - 7 fields
  • crates/api/src/dto/event.rs - 2 fields
  • crates/api/src/dto/inquiry.rs - 3 fields
  • crates/api/src/dto/pack.rs - 3 fields
  • crates/api/src/dto/rule.rs - 3 fields
  • crates/api/src/dto/workflow.rs - 4 fields
  • tests/generated_client/ - Entire directory regenerated

Impact:

  • ✅ E2E tests can now run without TypeError crashes
  • ✅ OpenAPI spec correctly shows "type": ["object", "null"] for nullable fields
  • ✅ Generated client matches API schema exactly
  • ✅ No manual patching required for generated code

Time: 2 hours

Session 10 (2026-01-23) - E2E Test Field Name Fixes & API Schema Updates

E2E Test Infrastructure Fixes:

  • ✅ Fixed field name mismatches between tests and API responses
    • Updated all tests to use trigger['label'] instead of trigger['name']
    • Updated all tests to use trigger['ref'] instead of trigger['type']
    • Updated all tests to use rule['label'] instead of rule['name']
    • Removed invalid action['runner_type'] assertions
  • ✅ Updated timer helper functions to create sensors consistently
    • create_interval_timer() - Already created sensor correctly
    • create_date_timer() - Now creates sensor with date config
    • create_cron_timer() - Now creates sensor with cron config
    • All return consistent structure with trigger + sensor info
  • ✅ Updated AttuneClient to handle new API schema
    • create_trigger() - Already supported new schema (ref/label)
    • create_rule() - Maps legacy fields to new schema (trigger_id→trigger_ref, name→ref/label, criteria→conditions, action_parameters→action_params)
    • create_action() - Maps legacy fields to new schema (name→ref/label, runner_type→runtime lookup)
    • list_runtimes() - Added new method to support runtime ID lookup
  • ✅ Added sensor service restart mechanism
    • restart_sensor_service() helper function
    • Tries docker-compose, systemctl, or falls back to wait
    • Called automatically after creating timer sensors
    • Ensures sensors are loaded and can generate events

Files Modified:

  • tests/e2e/tier1/test_t1_01_interval_timer.py - Fixed field names
  • tests/e2e/tier1/test_t1_02_date_timer.py - Fixed field names, updated assertions
  • tests/e2e/tier1/test_t1_03_cron_timer.py - Fixed field names, updated assertions
  • tests/e2e/tier1/test_t1_04_webhook_trigger.py - Fixed field names
  • tests/e2e/tier1/test_t1_08_action_failure.py - Fixed field names
  • tests/helpers/client.py - Updated create_rule(), create_action(), added list_runtimes()
  • tests/helpers/fixtures.py - Updated timer helpers, added restart_sensor_service()

OpenAPI Client Generator (Better Solution):

  • ✅ Implemented automatic Python client generation from OpenAPI spec
    • Created scripts/generate-python-client.sh to download spec and generate client
    • Generates type-safe Pydantic models for all 71 API endpoints
    • Eliminates field name mapping issues at the source
    • Client stays in sync with API automatically
    • Supports both sync and async operations
  • ✅ Generated client installed as attune-client package
    • Full type safety with Pydantic validation
    • Auto-completion in IDEs
    • Exact API schema, no manual mapping needed
  • ✅ Database migrations applied
    • Fixed missing webhook_enabled, webhook_key, webhook_config columns
    • All 6 webhook migrations now applied to E2E database

Documentation:

  • ✅ Created work-summary/2026-01-23-e2e-field-fixes.md - Comprehensive guide to all changes
  • ✅ Created work-summary/2026-01-23-openapi-client-generator.md - Complete OpenAPI client generator guide with migration plan

Next Steps:

  1. Create backward-compatible wrapper around generated client
  2. Update test fixtures to use generated client internally
  3. Run Tier 1 tests with new client
  4. Gradually migrate tests to use generated client directly
  5. Remove manual client code once migration complete

Session 9 (2026-01-20) - Core Pack Setup Complete

Core Pack Loader:

  • ✅ Created Python loader script (scripts/load_core_pack.py)
    • Parses pack YAML files and loads into database
    • Idempotent upsert operations (safe to re-run)
    • Transaction-based with rollback on error
    • Loads pack metadata, triggers, actions, and sensors
    • Creates required runtime entries
    • Comprehensive error handling and reporting
  • ✅ Created shell wrapper script (scripts/load-core-pack.sh)
    • Prerequisites checking (Python, packages, database)
    • Interactive package installation
    • Colored output and help documentation
    • Environment variable support
  • ✅ Comprehensive setup documentation (packs/core/SETUP.md)
    • 3 loading methods (Python, SQL, CLI)
    • Verification and testing procedures
    • Troubleshooting guide
    • Development workflow
    • CI/CD integration examples
  • ✅ Updated main README with core pack loading instructions
  • ✅ Core pack now ready for production use

Core Pack Contents:

  • 3 triggers (intervaltimer, crontimer, datetimetimer)
  • 4 actions (echo, sleep, noop, http_request)
  • 1 sensor (interval_timer_sensor)
  • Complete YAML definitions and implementations

Testing:

  • ✅ Comprehensive webhook test suite (32 tests total)
    • Repository tests: 6 tests
    • API management tests: 9 tests
    • Security feature tests: 17 tests
  • ✅ Test documentation (docs/webhook-testing.md)
  • ✅ Quick reference (crates/api/tests/README.md)

Session 8 (2026-01-20) - Webhook System Phase 2 & 3 Complete

Webhook System - API Endpoints:

  • ✅ Implemented webhook receiver endpoint: POST /api/v1/webhooks/:webhook_key
  • ✅ Implemented webhook management endpoints:
    • POST /api/v1/triggers/:ref/webhooks/enable
    • POST /api/v1/triggers/:ref/webhooks/disable
    • POST /api/v1/triggers/:ref/webhooks/regenerate
  • ✅ Created webhook DTOs (WebhookReceiverRequest, WebhookReceiverResponse)
  • ✅ Updated TriggerResponse and TriggerSummary to include webhook fields
  • ✅ Added webhook endpoints to OpenAPI documentation
  • ✅ Created comprehensive integration tests for all webhook endpoints
  • ✅ Implemented event creation from webhook payloads with metadata
  • ✅ Added proper error handling and validation
  • ✅ Webhook receiver endpoint is public (no auth required)
  • ✅ Management endpoints are protected with JWT authentication

Technical Details:

  • Uses static repository pattern with trait implementations
  • Webhook keys generated with wh_ prefix (40 random alphanumeric chars)
  • Event config includes webhook metadata (source IP, headers, user agent)
  • Proper Option handling and error mapping
  • All tests follow integration test patterns

Files Added/Modified:

  • crates/api/src/routes/webhooks.rs - Webhook routes (268 lines)
  • crates/api/src/dto/webhook.rs - Webhook DTOs (41 lines)
  • crates/api/src/dto/trigger.rs - Added webhook fields
  • crates/api/src/openapi.rs - Added webhook endpoints to spec
  • crates/api/tests/webhook_api_tests.rs - Integration tests (513 lines)
  • docs/webhook-system-architecture.md - Updated status to Phase 2 Complete

Next Steps:

  • Phase 3: Advanced webhook features (HMAC signatures, rate limiting, IP whitelist)
  • Phase 4: Web UI integration for webhook management
  • Phase 5: Webhook event history and analytics

Session 7 (2026-01-18) - CLI Tool Implementation + Output Enhancements

CLI Output Enhancements:

  • ✅ Added shorthand output flags: -j for JSON, -y for YAML
  • ✅ Implemented flag conflict handling (mutually exclusive)
  • ✅ Added execution result command to get raw execution results
  • ✅ Enabled piping to Unix tools (jq, yq, grep, awk)
  • ✅ Updated all documentation with shorthand examples
  • ✅ Added scripting examples for data extraction

Execution Search Enhancement:

  • ✅ Added pack-based filtering for executions (API and CLI)
  • ✅ Added result content search (case-insensitive substring match)
  • ✅ Enhanced CLI with --pack and --result options
  • ✅ Fixed query parameter alignment (action → action_ref, limit → per_page)
  • ✅ Added URL encoding for query parameters
  • ✅ Updated API documentation with new filters
  • ✅ Updated CLI documentation with usage examples

CLI Tool Implementation: Completed:

  • ✅ Created attune-cli crate with comprehensive command structure
  • ✅ Implemented authentication commands (login, logout, whoami)
  • ✅ Implemented pack management (list, show, install, uninstall, register)
  • ✅ Implemented action commands (list, show, execute with wait support)
  • ✅ Implemented rule management (list, show, enable, disable, create, delete)
  • ✅ Implemented execution monitoring (list, show, logs, cancel)
  • ✅ Implemented trigger and sensor inspection commands
  • ✅ Implemented CLI configuration management
  • ✅ Added HTTP client with JWT authentication
  • ✅ Added configuration file management (~/.config/attune/config.yaml)
  • ✅ Added multiple output formats (table, JSON, YAML)
  • ✅ Added colored and formatted table output
  • ✅ Added interactive prompts for confirmations
  • ✅ Created comprehensive CLI documentation
  • ✅ Updated main README with CLI usage

Technical Details:

  • Uses clap for argument parsing with subcommands
  • Uses reqwest for HTTP API calls
  • Uses comfy-table for formatted output
  • Uses dialoguer for interactive prompts
  • Uses colored for terminal colors
  • Config stored in standard XDG config directory
  • JWT tokens persisted in config file
  • Supports global flags (--api-url, --output, -j, -y, --verbose)
  • Shorthand flags with conflict handling
  • Raw result extraction for Unix tool integration

Session 6 (2026-01-17) - TODO Cleanup & E2E Test Setup

TODO Cleanup:

  • ✅ Moved bloated TODO.md to TODO.OLD.md (2,200+ lines)
  • ✅ Created streamlined TODO.md with only outstanding tasks
  • ✅ Verified API authentication already complete
  • ✅ Consolidated migration files (5 migrations, 22 tables)
  • ✅ Documented current project status clearly

E2E Test Infrastructure (Phase 1 - 80% Complete):

  • ✅ Created comprehensive test plan with 8 scenarios
  • ✅ Created config.e2e.yaml (204 lines)
  • ✅ Created test pack with echo action and simple workflow
  • ✅ Set up tests/ directory structure
  • ✅ Documented test procedures and debugging
  • ⏳ Next: Create E2E database and verify service startup

🎯 Success Criteria for v1.0 Release

  • ✅ All 5 core services implemented
  • ✅ Core automation features working (triggers, rules, actions)
  • ✅ Workflow orchestration functional
  • ✅ Security properly enforced (JWT auth, secrets)
  • ✅ Dependency isolation working
  • ⏳ End-to-end integration tests passing
  • ⏳ Docker deployment working
  • ⏳ Basic observability (logs, metrics, health checks)
  • ⏳ Production configuration documented
  • ⏳ Security audit completed
  • ⏳ Performance benchmarked
  • ⏳ Core documentation complete

📚 Reference Documentation

Completed Documentation

  • docs/workflow-orchestration.md - Complete workflow design (1,063 lines)
  • docs/queue-architecture.md - FIFO queue implementation (564 lines)
  • docs/dependency-isolation.md - Per-pack Python venvs
  • docs/log-size-limits.md - Bounded log writers
  • docs/ops-runbook-queues.md - Operations procedures (851 lines)
  • work-summary/2026-01-27-api-authentication-fix.md - Security fix details
  • work-summary/2025-01-secret-passing-complete.md - Secure secret handling
  • API documentation: 10 files covering all endpoints

Outstanding Documentation

  • Getting Started guide
  • Deployment guide
  • Operations runbook (general)
  • Troubleshooting guide
  • Pack development tutorial

⏱️ Estimated Timeline to Production

Minimum Viable Production (MVP):

  • Integration testing: 2-3 days
  • Docker setup: 3-4 days
  • Observability basics: 2-3 days
  • Configuration/docs: 2-3 days
  • Total: 2-3 weeks

Production-Ready with Confidence:

  • Add security audit: +3-4 days
  • Add performance testing: +4-5 days
  • Add comprehensive docs: +4-5 days
  • Total: 4-6 weeks

🔗 Related Documents

  • TODO.OLD.md - Historical TODO with completed tasks
  • work-summary/ - Session summaries and detailed implementation docs
  • docs/ - Technical documentation and API references
  • CHANGELOG.md - Version history and completed features
Reference in New Issue View Git Blame Copy Permalink