58 Semaphore Credentials und Secrets

58.1 SSH-Keys verwalten

58.1.1 Hinterlegen von SSH-Schlüsseln für Managed Nodes

SSH-Keys bilden die Grundlage für sichere, passwortlose Authentifizierung zwischen Semaphore und den verwalteten Zielsystemen. Die korrekte Verwaltung dieser Schlüssel ist entscheidend für die Sicherheit und Funktionalität der Ansible-Automatisierung.

58.1.1.1 SSH-Key-Arten in Semaphore

Private Keys für Host-Zugriff:

Key Type: SSH Private Key
Purpose: Authentifizierung auf Managed Nodes
Format: RSA, ECDSA, Ed25519
Storage: Verschlüsselt in Semaphore-Datenbank
Usage: Template-Assignment für Playbook-Ausführung

Deploy Keys für Repository-Zugriff:

Key Type: Git Deploy Key
Purpose: Read-Only Zugriff auf private Git-Repositories
Scope: Repository-spezifisch
Format: RSA, ECDSA, Ed25519
Usage: Automatische Playbook-Synchronisation

58.1.1.2 SSH-Key-Generierung

Empfohlene Key-Generierung:

# Ed25519 (empfohlen für neue Keys)
ssh-keygen -t ed25519 -C "semaphore-deployment" -f semaphore_ed25519

# RSA (Kompatibilität mit älteren Systemen)
ssh-keygen -t rsa -b 4096 -C "semaphore-deployment" -f semaphore_rsa

# ECDSA (Alternative zu RSA)
ssh-keygen -t ecdsa -b 521 -C "semaphore-deployment" -f semaphore_ecdsa

58.1.1.3 Public Key-Distribution

Automatisierte Public Key-Verteilung:

# Manual distribution
ssh-copy-id -i semaphore_ed25519.pub deploy@target-server

# Ansible-basierte Distribution
- name: Deploy SSH public key
  authorized_key:
    user: deploy
    state: present
    key: "{{ lookup('file', 'semaphore_ed25519.pub') }}"
    comment: "Semaphore automation key"

58.1.1.4 SSH-Key-Konfiguration in Semaphore

Key Store-Einstellungen:

Key Name: "production-deploy-key"
Type: SSH Key
Key: [Private Key Content]
Passphrase: [Optional für verschlüsselte Keys]
Description: "Production server deployment access"
Scope: Project-specific
Access Control: Project admins only

58.1.2 Globale vs. projektspezifische Keys

58.1.2.1 Globale SSH-Keys

Charakteristika: - Verfügbar für alle Projekte der Organisation - Verwaltung durch globale Administratoren - Geeignet für infrastrukturelle Zugriffe - Höhere Sicherheitsanforderungen

Anwendungsfälle:

Global Key: "infrastructure-master-key"
Usage:
├── Backup-Server (alle Projekte)
├── Monitoring-Systeme (organisationsweit)
├── Shared Services (DNS, NTP, LDAP)
├── Emergency-Access (Break-Glass-Szenarien)
└── Compliance-Auditing (Log-Server)

58.1.2.2 Projektspezifische SSH-Keys

Charakteristika: - Isoliert innerhalb einzelner Projekte - Verwaltung durch Projekt-Administratoren - Anwendungsspezifische Zugriffsrechte - Vereinfachte Rechteverwaltung

Anwendungsfälle:

Project: "E-Commerce Application"
├── web-app-deploy-key
│   ├── Purpose: Web-Server Deployment
│   ├── Access: Application-Server nur
│   └── Permissions: deploy user, sudo limited
├── db-backup-key
│   ├── Purpose: Database-Backup Operations
│   ├── Access: Database-Server nur
│   └── Permissions: postgres user, backup scripts
└── monitoring-key
    ├── Purpose: Application-Monitoring
    ├── Access: Monitoring-Agents Installation
    └── Permissions: monitoring user, service restart

58.1.2.3 Key-Scope-Entscheidungsmatrix

Kriterium Global Key Projekt-Key
Zugriff Organisationsweit Projektspezifisch
Verwaltung IT-Administratoren Projekt-Teams
Sicherheitsrisiko Hoch Mittel
Flexibilität Niedrig Hoch
Audit-Komplexität Hoch Niedrig

58.1.3 Sicherheitshinweise

58.1.3.1 Key-Permissions und Schutz

Filesystem-Permissions:

# Korrekte Berechtigungen für SSH-Keys
chmod 600 ~/.ssh/private_key        # Private Key: nur Owner read/write
chmod 644 ~/.ssh/private_key.pub    # Public Key: Owner read/write, andere read
chmod 700 ~/.ssh                    # SSH-Verzeichnis: nur Owner access

# Semaphore-Server Key-Storage
chown semaphore:semaphore /etc/semaphore/keys/
chmod 700 /etc/semaphore/keys/
chmod 600 /etc/semaphore/keys/*

58.1.3.2 Passphrase-Schutz

Verschlüsselte Private Keys:

# Key mit Passphrase erstellen
ssh-keygen -t ed25519 -N "strong-passphrase" -f secure_key

# Passphrase-Handling in Semaphore
Key Configuration:
├── Private Key: [Encrypted PEM Content]
├── Passphrase: [Stored encrypted in database]
├── Usage: Decryption at runtime only
└── Access: Authorized templates only

58.1.3.3 Key-Rotation-Strategien

Regelmäßige Schlüssel-Erneuerung:

Rotation Schedule:
├── Critical Systems: 90 Tage
├── Standard Systems: 180 Tage
├── Development Systems: 365 Tage
└── Emergency Rotation: Sofort bei Kompromittierung

Rotation Process:
1. Neuen SSH-Key generieren
2. Public Key auf Zielsystemen installieren
3. Semaphore-Konfiguration aktualisieren
4. Funktionstest durchführen
5. Alten Key deaktivieren
6. Alten Key nach Karenzzeit löschen

58.1.3.4 SSH-Agent-Forwarding-Sicherheit

Agent-Forwarding-Konfiguration:

# Sichere SSH-Konfiguration für Ansible
ansible_ssh_common_args: >
  -o StrictHostKeyChecking=no
  -o UserKnownHostsFile=/dev/null
  -o ForwardAgent=no
  -o ControlMaster=auto
  -o ControlPersist=60s

58.2 API-Tokens

58.2.1 Verwendung von Tokens für externe Tools

API-Tokens ermöglichen programmatischen Zugriff auf Semaphore-Funktionen und sind essentiell für die Integration in CI/CD-Pipelines, Monitoring-Systeme und externe Automatisierungstools.

58.2.1.1 Token-basierte Authentifizierung

Authentication Flow:

1. Client sendet API-Request mit Token-Header
2. Semaphore validiert Token-Signatur
3. Berechtigungen werden anhand Token-Scope geprüft
4. Request wird verarbeitet oder abgelehnt
5. Response wird mit Rate-Limiting zurückgegeben

58.2.1.2 API-Token-Header-Format

HTTP-Request-Beispiel:

GET /api/project/1/templates HTTP/1.1
Host: semaphore.company.com
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
Content-Type: application/json

58.2.2 Erzeugung und Verwaltung von Tokens

58.2.2.1 Token-Generierung über Web-Interface

User-Token-Erstellung:

User Profile → API Tokens → Create New Token
├── Token Name: "CI/CD Pipeline Integration"
├── Description: "Jenkins deployment automation"
├── Permissions: Project-specific
├── Expiration: 90 days
└── Scope: Execute templates, view jobs

58.2.2.2 API-basierte Token-Verwaltung

Programmatische Token-Erstellung:

# Token-Erstellung via API
curl -X POST https://semaphore.company.com/api/user/tokens \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "automated-deployment",
    "description": "Automated deployment token",
    "expires_at": "2024-12-31T23:59:59Z",
    "permissions": ["project:execute", "project:read"]
  }'

58.2.2.3 Token-Metadaten und -Verwaltung

Token-Information:

{
  "id": 42,
  "name": "CI/CD Pipeline Token",
  "description": "Jenkins integration for automated deployments",
  "user_id": 1,
  "created": "2024-08-15T10:30:00Z",
  "expires_at": "2024-11-15T10:30:00Z",
  "last_used": "2024-08-18T14:22:00Z",
  "permissions": [
    "project:1:execute",
    "project:1:read",
    "project:2:read"
  ],
  "active": true
}

58.2.3 User-Tokens vs. System-Tokens

58.2.3.1 User-Tokens

Charakteristika: - Verknüpft mit spezifischem Benutzeraccount - Erben Berechtigungen des Benutzers - Persönliche Verantwortlichkeit - Audit-Trail zu Benutzerkonto

Anwendungsfälle:

User Token Use Cases:
├── Personal API-Clients
├── Development Tools
├── Individual Scripts
├── Testing und Debugging
└── Temporäre Integrationen

58.2.3.2 System-Tokens (Service-Accounts)

Charakteristika: - Unabhängig von persönlichen Accounts - Spezifische, eingeschränkte Berechtigungen - Längere Gültigkeitsdauer - Service-orientierte Authentifizierung

Service-Account-Konfiguration:

Service Account: "deployment-automation"
├── Purpose: Automated CI/CD deployments
├── Permissions: 
│   ├── Execute specific templates
│   ├── Read job status
│   └── Access deployment logs
├── Token Lifetime: 1 Jahr
├── Rotation Schedule: Jährlich
└── Monitoring: API-Usage-Tracking

58.2.3.3 Token-Permissions-Matrix

Token-Typ Scope Lifetime Rotation Use Case
User Token User-based 1-90 Tage Bei Bedarf Development, Testing
Service Token Limited 90-365 Tage Scheduled CI/CD, Monitoring
Admin Token Global 30 Tage Frequent System Administration

58.2.4 Token-Sicherheit und Best Practices

58.2.4.1 Token-Storage und -Übertragung

Sichere Token-Handhabung:

# Environment Variables (empfohlen)
export SEMAPHORE_API_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."

# CI/CD Secret Management
# Jenkins Credentials Store
# GitLab CI Variables
# GitHub Actions Secrets

58.2.4.2 Token-Rotation-Automation

Automatisierte Token-Erneuerung:

#!/bin/bash
# Token-Rotation-Script
OLD_TOKEN=$SEMAPHORE_API_TOKEN

# Neuen Token erstellen
NEW_TOKEN=$(curl -X POST https://semaphore.company.com/api/user/tokens \
  -H "Authorization: Bearer $OLD_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "rotated-token", "expires_at": "2024-12-31T23:59:59Z"}' \
  | jq -r '.token')

# Konfiguration aktualisieren
update_ci_cd_config "$NEW_TOKEN"

# Alten Token deaktivieren
curl -X DELETE https://semaphore.company.com/api/user/tokens/$OLD_TOKEN_ID \
  -H "Authorization: Bearer $NEW_TOKEN"

58.3 Umgang mit Passwörtern, Vault-Passphrases, Cloud-Credentials

58.3.1 Speicherung und Schutz von Passwörtern

58.3.1.1 Credential-Typen in Semaphore

Unterstützte Credential-Formate:

Password Credentials:
├── Username/Password Combinations
├── Database Connection Strings
├── Service Account Passwords
├── Application API Keys
└── Third-party Service Tokens

Storage Format:
├── Encryption: AES-256-GCM
├── Key Management: Per-credential keys
├── Access Control: Role-based
└── Audit Logging: Vollständig protokolliert

58.3.1.2 Passwort-Konfiguration

Credential-Store-Einstellungen:

Credential Name: "production-database"
Type: Username/Password
Username: dbadmin
Password: [Encrypted and masked]
Description: "Production PostgreSQL admin access"
Scope: Project-specific
Access Control: Database-Admin-Role only
Rotation Schedule: 30 days

58.3.1.3 Sichere Passwort-Generierung

Passwort-Richtlinien:

# Starke Passwort-Generierung
openssl rand -base64 32
# Output: 7X9Kj2mN8pQ4vR6sT1uY3wZ5aB7cD9eF

# Spezielle Zeichen für Service-Accounts vermeiden
openssl rand -base64 24 | tr -d '/+=' | cut -c1-20
# Output: 8pQ4vR6sT1uY3wZ5aB7c

58.3.2 Integration von Ansible Vault-Passphrases

58.3.2.1 Vault-Password-Management

Vault-Passwort-Konfiguration:

Credential Name: "ansible-vault-production"
Type: Vault Password
Password: [Strong passphrase for vault encryption]
Description: "Production environment vault encryption key"
Usage: Automatic decryption during playbook execution
Scope: Project-specific

58.3.2.2 Vault-File-Integration

Template-Vault-Konfiguration:

Template: "Secure Application Deployment"
├── Playbook: playbooks/secure-deploy.yml
├── Vault Password: ansible-vault-production
├── Encrypted Files:
│   ├── group_vars/production/vault.yml
│   ├── host_vars/db-server/vault.yml
│   └── files/ssl-certificates.yml
└── Auto-Decryption: Enabled

58.3.2.3 Multi-Vault-Unterstützung

Mehrere Vault-Passwörter:

# ansible.cfg Konfiguration für Multiple Vaults
[defaults]
vault_password_file = /etc/semaphore/vault-passwords/default
vault_id_match = True

# Vault-ID-spezifische Passwörter
vault_identity_list = 
  prod@/etc/semaphore/vault-passwords/production,
  staging@/etc/semaphore/vault-passwords/staging,
  secrets@/etc/semaphore/vault-passwords/secrets

58.3.3 Cloud-Credentials-Verwaltung

58.3.3.1 AWS-Credentials

AWS-Access-Konfiguration:

Credential Name: "aws-production-deployment"
Type: AWS Access Key
Access Key ID: AKIAIOSFODNN7EXAMPLE
Secret Access Key: [Encrypted and masked]
Region: eu-central-1
Additional Properties:
├── Session Token: [Optional für temporary credentials]
├── MFA Device: [Optional für MFA-protected operations]
└── External ID: [Optional für Cross-Account-Access]

AWS IAM-Policy für Semaphore:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeImages",
        "s3:GetObject",
        "s3:PutObject"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:RequestedRegion": ["eu-central-1", "eu-west-1"]
        }
      }
    }
  ]
}

58.3.3.2 Azure-Credentials

Azure Service Principal:

Credential Name: "azure-service-principal"
Type: Azure Service Principal
Client ID: 12345678-1234-1234-1234-123456789012
Client Secret: [Encrypted and masked]
Tenant ID: 87654321-4321-4321-4321-210987654321
Subscription ID: abcdefgh-ijkl-mnop-qrst-uvwxyz123456

Azure RBAC-Berechtigung:

# Service Principal mit minimalen Rechten erstellen
az ad sp create-for-rbac \
  --name "semaphore-automation" \
  --role "Contributor" \
  --scopes "/subscriptions/$SUBSCRIPTION_ID/resourceGroups/production-rg"

58.3.3.3 Google Cloud Platform-Credentials

GCP Service Account:

Credential Name: "gcp-service-account"
Type: GCP Service Account Key
Service Account Email: semaphore@project-id.iam.gserviceaccount.com
Private Key: [JSON Key File Content - Encrypted]
Project ID: my-production-project

GCP IAM-Berechtigungen:

# Service Account Permissions
roles:
  - roles/compute.instanceAdmin.v1
  - roles/storage.objectAdmin
  - roles/monitoring.metricWriter
  
# Custom Role für eingeschränkte Berechtigungen
custom_role:
  title: "Semaphore Automation Role"
  permissions:
    - compute.instances.get
    - compute.instances.list
    - storage.objects.create
    - storage.objects.delete

58.3.4 Credential-Scopes

58.3.4.1 Globale Credentials

Organisationsweite Verfügbarkeit:

Global Scope Use Cases:
├── Backup-Systeme (alle Projekte)
├── Monitoring-Infrastructure
├── Shared Cloud-Resources
├── Central Logging-Services
└── Emergency-Access-Accounts

Security Considerations:
├── Höchste Sicherheitsanforderungen
├── Zentrale Verwaltung durch IT-Security
├── Umfassendes Audit-Logging
└── Regelmäßige Berechtigungsüberprüfung

58.3.4.2 Projektbezogene Credentials

Projekt-isolierte Zugriffe:

Project: "E-Commerce Platform"
├── ecommerce-db-credentials
│   ├── Scope: Database-Access nur für E-Commerce
│   ├── Permissions: Application-Database nur
│   └── Team-Access: Backend-Entwickler
├── payment-api-credentials
│   ├── Scope: Payment-Provider API
│   ├── Permissions: Transaction-Processing
│   └── Team-Access: Payment-Team
└── cdn-management-credentials
    ├── Scope: CDN-Configuration
    ├── Permissions: Cache-Management
    └── Team-Access: Frontend-Team

58.3.4.3 Teambezogene Credentials

Team-spezifische Zugriffskontrolle:

Team: "Infrastructure Team"
├── Shared Team Credentials:
│   ├── network-equipment-access
│   ├── virtualization-platform
│   └── backup-infrastructure
├── Access Control:
│   ├── Team-Members: Vollzugriff
│   ├── Project-Access: Read-Only
│   └── Organization-Access: Keine
└── Responsibility:
    ├── Credential-Maintenance: Team-Lead
    ├── Rotation-Schedule: Team-verantwortlich
    └── Incident-Response: Team-eskalation

58.4 Best Practices für Secret Management

58.4.1 Vermeidung von Klartext-Passwörtern

58.4.1.1 Code-Repository-Sicherheit

Verbotene Praktiken:

# NIEMALS in Playbooks oder Git-Repositories
database_password: "supersecret123"
api_key: "ak_live_1234567890abcdef"
ssl_private_key: |
  -----BEGIN PRIVATE KEY-----
  MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7VJTUt9Us8cKB...

Sichere Alternativen:

# Verwendung von Semaphore-Credentials
database_password: "{{ credential.db_password }}"
api_key: "{{ credential.payment_api_key }}"

# Ansible Vault für Repository-basierte Secrets
database_password: "{{ vault_db_password }}"
api_key: "{{ vault_api_key }}"

# Environment-Variable-Lookup
ssl_cert_path: "{{ lookup('env', 'SSL_CERT_PATH') }}"

58.4.1.2 Git-Hooks für Secret-Scanning

Pre-commit Hook für Secret-Detection:

#!/bin/bash
# .git/hooks/pre-commit
# Prevent accidental secret commits

if git diff --cached --name-only | xargs grep -l "password\|secret\|key" | grep -v vault; then
    echo "ERROR: Potential secrets detected in staged files"
    echo "Please use Ansible Vault or Semaphore Credentials instead"
    exit 1
fi

58.4.2 Principle of Least Privilege

58.4.2.1 Credential-Berechtigung-Minimierung

Rollenbbasierte Credential-Zugriffe:

Role: "Database Backup Operator"
├── Allowed Credentials:
│   ├── backup-database-readonly
│   └── backup-storage-write
├── Denied Credentials:
│   ├── production-admin-access
│   ├── application-secrets
│   └── infrastructure-keys
└── Audit Requirements:
    ├── Usage-Logging: Mandatory
    ├── Access-Review: Quarterly
    └── Privilege-Escalation: Approval required

58.4.2.2 Zeitlich begrenzte Zugriffe

Temporary Credential-Access:

Emergency Access Request:
├── Requester: ops-engineer-1
├── Credential: production-admin-ssh
├── Duration: 4 hours
├── Justification: "Critical production issue #INC-2024-08-001"
├── Approval: Required from Security and Manager
├── Monitoring: Enhanced logging during access period
└── Auto-Revocation: After expiration time

58.4.3 Regelmäßige Secret-Rotation

58.4.3.1 Automatisierte Rotation-Workflows

Rotation-Pipeline:

# Automated Secret Rotation Playbook
- name: Secret Rotation Workflow
  hosts: localhost
  vars:
    rotation_schedule:
      critical_systems: 30  # days
      standard_systems: 90
      development_systems: 180
  
  tasks:
    - name: Identify secrets for rotation
      uri:
        url: "{{ semaphore_api }}/credentials/rotation-due"
        headers:
          Authorization: "Bearer {{ api_token }}"
      register: rotation_candidates
      
    - name: Generate new secrets
      include_tasks: generate_new_secret.yml
      loop: "{{ rotation_candidates.json }}"
      
    - name: Update dependent systems
      include_tasks: update_systems.yml
      
    - name: Validate new credentials
      include_tasks: validate_access.yml
      
    - name: Deactivate old credentials
      include_tasks: cleanup_old_secrets.yml

58.4.3.2 Rotation-Notification-System

Stakeholder-Benachrichtigung:

Rotation Notification:
├── Advance Warning: 7 days before rotation
├── Execution Notice: During rotation process
├── Completion Confirmation: After successful rotation
├── Failure Alert: If rotation fails
└── Recipients:
    ├── Credential-Owner
    ├── System-Administrators
    ├── Security-Team
    └── Service-Dependencies

58.4.4 Integration externer Secret Manager

58.4.4.1 HashiCorp Vault Integration

Vault-Policy für Semaphore:

# vault-policy.hcl
path "secret/data/semaphore/*" {
  capabilities = ["read"]
}

path "auth/token/lookup-self" {
  capabilities = ["read"]
}

path "sys/capabilities-self" {
  capabilities = ["read"]
}

Vault-Integration-Konfiguration:

{
  "vault_enable": true,
  "vault_address": "https://vault.company.com:8200",
  "vault_auth_method": "kubernetes",
  "vault_role": "semaphore-automation",
  "vault_namespace": "automation",
  "vault_secret_engine": "kv-v2",
  "vault_secret_path": "secret/semaphore"
}

58.4.4.2 AWS Secrets Manager

AWS Secrets Manager Integration:

# Playbook mit AWS Secrets Manager Lookup
- name: Get database credentials from AWS Secrets Manager
  set_fact:
    db_credentials: "{{ lookup('amazon.aws.aws_secret', 'prod/database/credentials', region='eu-central-1') | from_json }}"
  no_log: true

- name: Connect to database
  postgresql_query:
    login_host: "{{ db_host }}"
    login_user: "{{ db_credentials.username }}"
    login_password: "{{ db_credentials.password }}"
    db: "{{ db_name }}"
    query: "SELECT version();"

58.4.4.3 Azure Key Vault

Azure Key Vault Lookup:

# Azure Key Vault Integration
- name: Retrieve SSL certificate from Key Vault
  azure_rm_keyvaultsecret_info:
    vault_uri: "https://prod-secrets.vault.azure.net/"
    name: "ssl-certificate"
  register: ssl_cert
  no_log: true

- name: Deploy SSL certificate
  copy:
    content: "{{ ssl_cert.secrets[0].value }}"
    dest: /etc/ssl/certs/application.crt
    mode: '0644'
    backup: yes

58.4.5 Security Monitoring und Compliance

58.4.5.1 Credential-Access-Auditing

Audit-Log-Format:

{
  "timestamp": "2024-08-18T14:30:00Z",
  "event_type": "credential_access",
  "user_id": "user@company.com",
  "credential_id": "prod-db-credentials",
  "template_id": 42,
  "job_id": 1234,
  "access_type": "read",
  "source_ip": "192.168.1.100",
  "user_agent": "Semaphore/2.8.0",
  "success": true,
  "risk_score": 2.5
}

58.4.5.2 Compliance-Reporting

Credential-Compliance-Dashboard:

Compliance Metrics:
├── Credential-Coverage: 98.5% (strong passwords)
├── Rotation-Compliance: 95.2% (within policy)
├── Access-Reviews: 100% (quarterly completed)
├── Vault-Usage: 87% (external secret manager)
├── Risk-Score: LOW (average 2.1/10)
└── Policy-Violations: 0 (current month)

Remediation Actions:
├── Weak-Passwords: 3 credentials need strengthening
├── Overdue-Rotation: 2 credentials require rotation
├── Excessive-Access: 1 user needs privilege review
└── Audit-Findings: 0 open items

58.4.5.3 Incident Response für Credential-Kompromittierung

Breach-Response-Playbook:

Credential Compromise Response:
├── Phase 1: Detection and Assessment (0-15 min)
│   ├── Identify compromised credentials
│   ├── Assess blast radius
│   └── Activate incident response team
├── Phase 2: Containment (15-60 min)
│   ├── Disable compromised credentials
│   ├── Block suspicious access patterns
│   └── Preserve forensic evidence
├── Phase 3: Recovery (1-24 hours)
│   ├── Generate new credentials
│   ├── Update all dependent systems
│   └── Validate system integrity
└── Phase 4: Post-Incident (24-72 hours)
    ├── Root cause analysis
    ├── Process improvements
    └── Stakeholder communication