nix/DOCKER_UPDATE_PLAN.md

Docker Container Update Automation Plan

Current State

• Hecuba (Hetzner cloud host) runs Docker containers
• WUD (Watchtower) is already running as a docker container
• No declarative docker configuration in NixOS
• Manual container management currently

Goals

Automate docker container updates on hecuba with proper declarative management

Evaluation: Update Approaches

Option 1: WUD (Watchtower)

Pros:

• Already deployed and working
• Simple, single-purpose tool
• Good monitoring capabilities via web UI
• Can schedule update windows
• Supports multiple strategies (always, weekly, etc.)

Cons:

• Not declarative
• Requires manual docker-compose or container management
• No NixOS integration

Option 2: Watchtower (original)

Pros:

• More popular and battle-tested
• Simpler configuration
• Wide community support

Cons:

• Same as WUD - not declarative

Option 3: NixOS Virtualisation.OCI-Containers

Pros:

• Fully declarative
• Reproducible builds
• Integrated with NixOS system
• Automatic rollback capability
• Can be managed via colmena

Cons:

• More complex setup
• Learning curve for OCI containers syntax
• Update automation still needs to be handled separately

Option 4: NixOS + Auto-Update

Pros:

• Declarative containers
• Automatic system updates can trigger container updates
• Full NixOS ecosystem integration

Cons:

• Most complex approach
• Overkill for simple use case

Implementation Plan

Phase 1: Inventory Current Setup

• Document all existing docker containers on hecuba
• Document current WUD configuration
• Document update schedules and preferences
• Identify containers that should NOT auto-update
• Map container dependencies

Phase 2: Choose Strategy

• Evaluate trade-offs between WUD vs declarative approach
• Decision: Hybrid approach (declarative + WUD) OR full NixOS

Option A: Hybrid (Recommended Short-term)

• Keep WUD for automation
• Add OCI containers to NixOS for declarative config
• Gradually migrate containers one by one

Option B: Full NixOS

• Replace WUD with declarative containers
• Use systemd timers for update schedules
• More complex but fully reproducible

Phase 3: Implementation (Hybrid Approach)

Step 1: Create Docker Module

Create modules/docker/containers.nix:

{ config, lib, ... }:
{
  virtualisation.oci-containers = {
    backend = "docker";
    containers = {
      # Container definitions here
    };
  };
}

Step 2: Define Containers

• Add WUD container to declarative config
• Add other existing containers to declarative config
• Configure container restart policies
• Set up container-specific networks if needed

Step 3: Persistent Storage

• Document volumes for each container
• Add volume management to NixOS config
• Ensure backup processes cover container data

Step 4: WUD Configuration

• Add WUD config to NixOS module
• Configure watch intervals
• Set up notifications
• Configure containers to exclude from auto-update

Step 5: Deployment

• Test configuration locally first
• Deploy to hecuba via colmena
• Monitor container restarts
• Verify WUD still works

Phase 4: Maintenance & Monitoring

• Set up container health checks
• Configure alerts for failed updates
• Document rollback procedure
• Schedule regular container audits

Container Inventory Template

Container Name:
Purpose:
Image:
Exposed Ports:
Volumes:
Network:
Auto-Update: yes/no
Restart Policy:
Notes:

Example NixOS OCI Container Definition

# modules/docker/containers.nix
{ config, lib, pkgs, ... }:
{
  virtualisation.oci-containers = {
    backend = "docker";
    containers = {
      wud = {
        image = "containrrr/watchtower:latest";
        ports = [ "8080:8080" ];
        volumes = [
          "/var/run/docker.sock:/var/run/docker.sock"
        ];
        environment = {
          WATCHTOWER_CLEANUP = "true";
          WATCHTOWER_SCHEDULE = "0 2 * * *";
        };
      };
      # Add other containers here
    };
  };
}

Migration Strategy

1. Document First: Before changing anything, document current state
2. Test Locally: Use colmena's local deployment if possible
3. Migrate One by One: Move containers individually to minimize risk
4. Monitor Closely: Watch logs after each migration
5. Keep Backups: Ensure data is backed up before major changes

WUD vs Watchtower Clarification

There are two different tools:

• Watchtower: Original tool, more popular
• WUD: Different implementation with web UI

Since you already have WUD running, we should:

1. Document its current configuration
2. Either keep it and make it declarative, OR
3. Switch to Watchtower if it better fits your needs

Next Steps

1. Immediate: Document all current containers and their configs
2. Decision: Choose between hybrid or full NixOS approach
3. Implementation: Create docker containers module
4. Testing: Deploy to hecuba and verify

Questions to Answer

• Which containers are currently running?
• How critical is uptime for each container?
• Any containers that should NEVER auto-update?
• Preferred update schedule (daily, weekly)?
• How should update failures be handled (retry, notify, manual)?
• Do you have backups of container data currently?

Risk Considerations

• Auto-updates can break applications
• Need to test updates before production (maybe staging)
• Some containers have configuration changes between versions
• Data loss risk if volumes are misconfigured
• Network disruption during updates

Monitoring Setup

Consider adding monitoring for:

• Container health status
• Update success/failure rates
• Disk space usage
• Resource consumption
• Backup verification