Fix: Use nix flake check for hooks, simplify module, remove apps
This commit is contained in:
217
DOCKER_UPDATE_PLAN.md
Normal file
217
DOCKER_UPDATE_PLAN.md
Normal file
@@ -0,0 +1,217 @@
|
||||
# Docker Container Update Automation Plan
|
||||
|
||||
## Current State
|
||||
- Hecuba (Hetzner cloud host) runs Docker containers
|
||||
- WUD (Watchtower) is already running as a docker container
|
||||
- No declarative docker configuration in NixOS
|
||||
- Manual container management currently
|
||||
|
||||
## Goals
|
||||
Automate docker container updates on hecuba with proper declarative management
|
||||
|
||||
## Evaluation: Update Approaches
|
||||
|
||||
### Option 1: WUD (Watchtower)
|
||||
**Pros:**
|
||||
- Already deployed and working
|
||||
- Simple, single-purpose tool
|
||||
- Good monitoring capabilities via web UI
|
||||
- Can schedule update windows
|
||||
- Supports multiple strategies (always, weekly, etc.)
|
||||
|
||||
**Cons:**
|
||||
- Not declarative
|
||||
- Requires manual docker-compose or container management
|
||||
- No NixOS integration
|
||||
|
||||
### Option 2: Watchtower (original)
|
||||
**Pros:**
|
||||
- More popular and battle-tested
|
||||
- Simpler configuration
|
||||
- Wide community support
|
||||
|
||||
**Cons:**
|
||||
- Same as WUD - not declarative
|
||||
|
||||
### Option 3: NixOS Virtualisation.OCI-Containers
|
||||
**Pros:**
|
||||
- Fully declarative
|
||||
- Reproducible builds
|
||||
- Integrated with NixOS system
|
||||
- Automatic rollback capability
|
||||
- Can be managed via colmena
|
||||
|
||||
**Cons:**
|
||||
- More complex setup
|
||||
- Learning curve for OCI containers syntax
|
||||
- Update automation still needs to be handled separately
|
||||
|
||||
### Option 4: NixOS + Auto-Update
|
||||
**Pros:**
|
||||
- Declarative containers
|
||||
- Automatic system updates can trigger container updates
|
||||
- Full NixOS ecosystem integration
|
||||
|
||||
**Cons:**
|
||||
- Most complex approach
|
||||
- Overkill for simple use case
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Inventory Current Setup
|
||||
- [ ] Document all existing docker containers on hecuba
|
||||
- [ ] Document current WUD configuration
|
||||
- [ ] Document update schedules and preferences
|
||||
- [ ] Identify containers that should NOT auto-update
|
||||
- [ ] Map container dependencies
|
||||
|
||||
### Phase 2: Choose Strategy
|
||||
- [ ] Evaluate trade-offs between WUD vs declarative approach
|
||||
- [ ] Decision: Hybrid approach (declarative + WUD) OR full NixOS
|
||||
|
||||
#### Option A: Hybrid (Recommended Short-term)
|
||||
- Keep WUD for automation
|
||||
- Add OCI containers to NixOS for declarative config
|
||||
- Gradually migrate containers one by one
|
||||
|
||||
#### Option B: Full NixOS
|
||||
- Replace WUD with declarative containers
|
||||
- Use systemd timers for update schedules
|
||||
- More complex but fully reproducible
|
||||
|
||||
### Phase 3: Implementation (Hybrid Approach)
|
||||
|
||||
#### Step 1: Create Docker Module
|
||||
Create `modules/docker/containers.nix`:
|
||||
```nix
|
||||
{ config, lib, ... }:
|
||||
{
|
||||
virtualisation.oci-containers = {
|
||||
backend = "docker";
|
||||
containers = {
|
||||
# Container definitions here
|
||||
};
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
#### Step 2: Define Containers
|
||||
- [ ] Add WUD container to declarative config
|
||||
- [ ] Add other existing containers to declarative config
|
||||
- [ ] Configure container restart policies
|
||||
- [ ] Set up container-specific networks if needed
|
||||
|
||||
#### Step 3: Persistent Storage
|
||||
- [ ] Document volumes for each container
|
||||
- [ ] Add volume management to NixOS config
|
||||
- [ ] Ensure backup processes cover container data
|
||||
|
||||
#### Step 4: WUD Configuration
|
||||
- [ ] Add WUD config to NixOS module
|
||||
- [ ] Configure watch intervals
|
||||
- [ ] Set up notifications
|
||||
- [ ] Configure containers to exclude from auto-update
|
||||
|
||||
#### Step 5: Deployment
|
||||
- [ ] Test configuration locally first
|
||||
- [ ] Deploy to hecuba via colmena
|
||||
- [ ] Monitor container restarts
|
||||
- [ ] Verify WUD still works
|
||||
|
||||
### Phase 4: Maintenance & Monitoring
|
||||
- [ ] Set up container health checks
|
||||
- [ ] Configure alerts for failed updates
|
||||
- [ ] Document rollback procedure
|
||||
- [ ] Schedule regular container audits
|
||||
|
||||
## Container Inventory Template
|
||||
|
||||
```
|
||||
Container Name:
|
||||
Purpose:
|
||||
Image:
|
||||
Exposed Ports:
|
||||
Volumes:
|
||||
Network:
|
||||
Auto-Update: yes/no
|
||||
Restart Policy:
|
||||
Notes:
|
||||
```
|
||||
|
||||
## Example NixOS OCI Container Definition
|
||||
|
||||
```nix
|
||||
# modules/docker/containers.nix
|
||||
{ config, lib, pkgs, ... }:
|
||||
{
|
||||
virtualisation.oci-containers = {
|
||||
backend = "docker";
|
||||
containers = {
|
||||
wud = {
|
||||
image = "containrrr/watchtower:latest";
|
||||
ports = [ "8080:8080" ];
|
||||
volumes = [
|
||||
"/var/run/docker.sock:/var/run/docker.sock"
|
||||
];
|
||||
environment = {
|
||||
WATCHTOWER_CLEANUP = "true";
|
||||
WATCHTOWER_SCHEDULE = "0 2 * * *";
|
||||
};
|
||||
};
|
||||
# Add other containers here
|
||||
};
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
1. **Document First**: Before changing anything, document current state
|
||||
2. **Test Locally**: Use colmena's local deployment if possible
|
||||
3. **Migrate One by One**: Move containers individually to minimize risk
|
||||
4. **Monitor Closely**: Watch logs after each migration
|
||||
5. **Keep Backups**: Ensure data is backed up before major changes
|
||||
|
||||
## WUD vs Watchtower Clarification
|
||||
|
||||
There are two different tools:
|
||||
- **Watchtower**: Original tool, more popular
|
||||
- **WUD**: Different implementation with web UI
|
||||
|
||||
Since you already have WUD running, we should:
|
||||
1. Document its current configuration
|
||||
2. Either keep it and make it declarative, OR
|
||||
3. Switch to Watchtower if it better fits your needs
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Immediate**: Document all current containers and their configs
|
||||
2. **Decision**: Choose between hybrid or full NixOS approach
|
||||
3. **Implementation**: Create docker containers module
|
||||
4. **Testing**: Deploy to hecuba and verify
|
||||
|
||||
## Questions to Answer
|
||||
|
||||
- Which containers are currently running?
|
||||
- How critical is uptime for each container?
|
||||
- Any containers that should NEVER auto-update?
|
||||
- Preferred update schedule (daily, weekly)?
|
||||
- How should update failures be handled (retry, notify, manual)?
|
||||
- Do you have backups of container data currently?
|
||||
|
||||
## Risk Considerations
|
||||
|
||||
- Auto-updates can break applications
|
||||
- Need to test updates before production (maybe staging)
|
||||
- Some containers have configuration changes between versions
|
||||
- Data loss risk if volumes are misconfigured
|
||||
- Network disruption during updates
|
||||
|
||||
## Monitoring Setup
|
||||
|
||||
Consider adding monitoring for:
|
||||
- Container health status
|
||||
- Update success/failure rates
|
||||
- Disk space usage
|
||||
- Resource consumption
|
||||
- Backup verification
|
||||
Reference in New Issue
Block a user