nix/DOCKER_UPDATE_PLAN.md

# Docker Container Update Automation Plan

## Current State
- Hecuba (Hetzner cloud host) runs Docker containers
- WUD (Watchtower) is already running as a docker container
- No declarative docker configuration in NixOS
- Manual container management currently

## Goals
Automate docker container updates on hecuba with proper declarative management

## Evaluation: Update Approaches

### Option 1: WUD (Watchtower)
**Pros:**
- Already deployed and working
- Simple, single-purpose tool
- Good monitoring capabilities via web UI
- Can schedule update windows
- Supports multiple strategies (always, weekly, etc.)

**Cons:**
- Not declarative
- Requires manual docker-compose or container management
- No NixOS integration

### Option 2: Watchtower (original)
**Pros:**
- More popular and battle-tested
- Simpler configuration
- Wide community support

**Cons:**
- Same as WUD - not declarative

### Option 3: NixOS Virtualisation.OCI-Containers
**Pros:**
- Fully declarative
- Reproducible builds
- Integrated with NixOS system
- Automatic rollback capability
- Can be managed via colmena

**Cons:**
- More complex setup
- Learning curve for OCI containers syntax
- Update automation still needs to be handled separately

### Option 4: NixOS + Auto-Update
**Pros:**
- Declarative containers
- Automatic system updates can trigger container updates
- Full NixOS ecosystem integration

**Cons:**
- Most complex approach
- Overkill for simple use case

## Implementation Plan

### Phase 1: Inventory Current Setup
- [ ] Document all existing docker containers on hecuba
- [ ] Document current WUD configuration
- [ ] Document update schedules and preferences
- [ ] Identify containers that should NOT auto-update
- [ ] Map container dependencies

### Phase 2: Choose Strategy
- [ ] Evaluate trade-offs between WUD vs declarative approach
- [ ] Decision: Hybrid approach (declarative + WUD) OR full NixOS

#### Option A: Hybrid (Recommended Short-term)
- Keep WUD for automation
- Add OCI containers to NixOS for declarative config
- Gradually migrate containers one by one

#### Option B: Full NixOS
- Replace WUD with declarative containers
- Use systemd timers for update schedules
- More complex but fully reproducible

### Phase 3: Implementation (Hybrid Approach)

#### Step 1: Create Docker Module
Create `modules/docker/containers.nix`:
```nix
{ config, lib, ... }:
{
  virtualisation.oci-containers = {
    backend = "docker";
    containers = {
      # Container definitions here
    };
  };
}
```

#### Step 2: Define Containers
- [ ] Add WUD container to declarative config
- [ ] Add other existing containers to declarative config
- [ ] Configure container restart policies
- [ ] Set up container-specific networks if needed

#### Step 3: Persistent Storage
- [ ] Document volumes for each container
- [ ] Add volume management to NixOS config
- [ ] Ensure backup processes cover container data

#### Step 4: WUD Configuration
- [ ] Add WUD config to NixOS module
- [ ] Configure watch intervals
- [ ] Set up notifications
- [ ] Configure containers to exclude from auto-update

#### Step 5: Deployment
- [ ] Test configuration locally first
- [ ] Deploy to hecuba via colmena
- [ ] Monitor container restarts
- [ ] Verify WUD still works

### Phase 4: Maintenance & Monitoring
- [ ] Set up container health checks
- [ ] Configure alerts for failed updates
- [ ] Document rollback procedure
- [ ] Schedule regular container audits

## Container Inventory Template

```
Container Name:
Purpose:
Image:
Exposed Ports:
Volumes:
Network:
Auto-Update: yes/no
Restart Policy:
Notes:
```

## Example NixOS OCI Container Definition

```nix
# modules/docker/containers.nix
{ config, lib, pkgs, ... }:
{
  virtualisation.oci-containers = {
    backend = "docker";
    containers = {
      wud = {
        image = "containrrr/watchtower:latest";
        ports = [ "8080:8080" ];
        volumes = [
          "/var/run/docker.sock:/var/run/docker.sock"
        ];
        environment = {
          WATCHTOWER_CLEANUP = "true";
          WATCHTOWER_SCHEDULE = "0 2 * * *";
        };
      };
      # Add other containers here
    };
  };
}
```

## Migration Strategy

1. **Document First**: Before changing anything, document current state
2. **Test Locally**: Use colmena's local deployment if possible
3. **Migrate One by One**: Move containers individually to minimize risk
4. **Monitor Closely**: Watch logs after each migration
5. **Keep Backups**: Ensure data is backed up before major changes

## WUD vs Watchtower Clarification

There are two different tools:
- **Watchtower**: Original tool, more popular
- **WUD**: Different implementation with web UI

Since you already have WUD running, we should:
1. Document its current configuration
2. Either keep it and make it declarative, OR
3. Switch to Watchtower if it better fits your needs

## Next Steps

1. **Immediate**: Document all current containers and their configs
2. **Decision**: Choose between hybrid or full NixOS approach
3. **Implementation**: Create docker containers module
4. **Testing**: Deploy to hecuba and verify

## Questions to Answer

- Which containers are currently running?
- How critical is uptime for each container?
- Any containers that should NEVER auto-update?
- Preferred update schedule (daily, weekly)?
- How should update failures be handled (retry, notify, manual)?
- Do you have backups of container data currently?

## Risk Considerations

- Auto-updates can break applications
- Need to test updates before production (maybe staging)
- Some containers have configuration changes between versions
- Data loss risk if volumes are misconfigured
- Network disruption during updates

## Monitoring Setup

Consider adding monitoring for:
- Container health status
- Update success/failure rates
- Disk space usage
- Resource consumption
- Backup verification