Logs & Debugging: The Black Box Recorder

In a traditional VM, you SSH in and tail -f /var/log/syslog. In Docker, containers are ephemeral and often locked down. So where do the logs go?

1. The Architecture of docker logs

By default, Docker captures everything a container writes to Standard Output (stdout) and Standard Error (stderr).

flowchart LR
  subgraph Container
      App[Application] -->|stdout/stderr| Pipe[Docker Pipe]
  end

  Pipe -->|json-file driver| HostFile["/var/lib/docker/.../json.log"]
  HostFile -->|read| DockerClient[docker logs my-app]

1. Capture: Docker attaches to the streams of PID 1.
2. Format: It wraps each line in a JSON object with a timestamp: {"log": "Hello\n", "stream": "stdout", "time": "..."}.
3. Store: It writes to a file on the host machine.

2. The Silent Killer: Disk Exhaustion

If your app logs 1GB per day, and your container runs for 100 days, you have a 100GB file on your host disk. Docker does not rotate logs by default. This is a common cause of production outages.

The Fix: Log Rotation

You must configure the logging driver to rotate files.

# docker-compose.yml
logging:
  driver: "json-file"
  options:
    max-size: "10m"   # Rotate when file hits 10MB
    max-file: "3"     # Keep only 3 rotated files

---

3. Interactive: Log Rotation Visualizer

See how log rotation prevents disk overflow. We simulate writing logs rapidly. Watch how older files are discarded.

active.log

.1 (Old)

.2 (Older)

Deleted

Status: Idle

---

4. Code Example: Structured Logging

Since Docker captures stdout as text, multiline stack traces can be a nightmare to parse in centralized logging systems (like ELK or Datadog).

Best Practice: Log every event as a single-line JSON object.

Use logstash-logback-encoder to output JSON automatically.

<!-- logback.xml -->
<configuration>
    <appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder">
            <!-- Add custom fields -->
            <customFields>{"app_name":"my-java-service"}</customFields>
        </encoder>
    </appender>

    <root level="INFO">
        <appender-ref ref="CONSOLE" />
    </root>
</configuration>

Output:

{"@timestamp":"2023-10-05T12:00:00.000Z","message":"User login failed","user_id":"123","level":"ERROR"}

</div>

Use the standard library log/slog (Go 1.21+).

package main

import (
    "log/slog"
    "os"
)

func main() {
    // 1. Configure JSON Handler writing to Stdout
    logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

    // 2. Log with structured attributes
    logger.Info("User login attempt",
        "user_id", 42,
        "ip", "192.168.1.1",
        "status", "success",
    )
}

Output:

{"time":"2023-10-05T12:00:00Z","level":"INFO","msg":"User login attempt","user_id":42,"ip":"192.168.1.1","status":"success"}