Logging Debugging

When your codebase is tiny, print() and random try/except blocks "work". Once you have APIs, background workers, cron jobs, queues, and multiple servers, that approach falls apart. Learn professional-grade debugging + logging + error-handling systems used by real engineering teams.

Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

What You'll Master

This lesson takes you from basic debugging → production-grade logging systems used by companies handling thousands of users, multiple environments, and distributed services.

Part 1: Logging Fundamentals & Error Handling

🔥 1. Why Logging Beats print() in Real Systems

When you have thousands of users, multiple environments (dev/staging/prod), background jobs, schedulers, and queues — print() falls apart.

Feature

print()

logging

Output destination

stdout only

Console, files, HTTP, email

Severity levels

None

DEBUG → CRITICAL

Filtering

Not possible

Per module, per environment

Context

Manual string formatting

Attach user ID, request ID, etc.

Production use

Pollutes output

Professional & configurable

Rule: print() is only for quick experiments. Real services use logging everywhere.

⚙️ 2. Basic Logging Setup (Per-Module Loggers)

You should never use the root logger directly in big projects. Instead, use one logger per module:

Then in your entry point (e.g. main.py or app.py):

This gives you timestamp, log level, logger name (payments.service, users.api), and message. You can raise log level in dev: DEBUG, staging: INFO, prod: WARNING.

🧱 3. Logging Levels: A Contract for Your Team

• • DEBUG = Routine vitals check (internal details)
• • INFO = Patient admitted/discharged (normal events)
• • WARNING = Minor symptoms, monitor closely (recoverable issues)
• • ERROR = Needs immediate attention but stable (operation failures)
• • CRITICAL = Emergency! Life-threatening! (system down)

Use logging levels consistently across your team:

Level

When to Use

Example

DEBUG

Internal details, variables, timings

"User payload: {...} "

INFO

High-level "this happened"

"User registered", "Order created"

WARNING

Unusual but recoverable

"Slow query: 832ms", "Retry attempt 2"

ERROR

Operation failed, app keeps running

"Payment failed", "Email send failed"

CRITICAL

System is not usable

"DB unreachable", "Config missing"

When everyone uses the same level rules, you can alert only on ERROR/CRITICAL, filter out noisy DEBUG in production, and quickly see timeline of events with INFO logs.

🧩 4. Structured Logging (So Logs Are Actually Searchable)

For small scripts, string messages are fine. At scale, you need structured logs so tools like Datadog/Loki/ELK/CloudWatch can filter and aggregate.

Now your logging backend can filter by user_id, count logins from each IP, and group by event.

🧲 5. Handlers: Sending Logs to Multiple Destinations

A handler decides where logs go. Common ones:

Handler

Destination

Best For

StreamHandler

Console

Development, debugging

FileHandler

Regular log file

Simple file logging

RotatingFileHandler

Rotates at size limit

Production (prevents huge files)

TimedRotatingFileHandler

Rotates daily/hourly

Daily log archives

SMTPHandler

Email

Critical error alerts

HTTPHandler

Log service

Centralized monitoring

Now dev uses console, ops can inspect app.log, and files don't grow forever. In a SaaS/microservice setup, you'd usually log to stdout and let Docker/Kubernetes send logs to a central system.

🧠 6. Error Handling Strategy (Don't Just "try/except Everything")

Problems with this approach:

• • Swallows context — you don't know what went wrong
• • Hides the original traceback — debugging becomes impossible
• • The caller doesn't know it failed — silent failures are the worst!

Good pattern: Catch specific exceptions, log with traceback, decide whether to handle or re-raise

Key ideas: Use domain-level exceptions (PaymentError), wrap low-level exceptions, pass exc_info=True to capture traceback, use raise ... from e to keep the error chain.

🧨 7. Logging Tracebacks Correctly

This logs message, full traceback, and error type.

Both approaches are valid. logger.exception() is just a shortcut for logger.error(..., exc_info=True) inside an except block.

🪤 8. Global Exception Hooks (Last-Resort Safety Net)

For CLI/worker processes, you can register a global handler:

Now any uncaught error in the main thread is logged at CRITICAL. You still let the process crash (which is often correct in prod). In frameworks: Django has middleware for logging unhandled errors, FastAPI lets you add global exception handlers, Celery workers log errors from tasks.

🧪 9. Debugging Strategy: Logs + Debugger + Assertions

• long-term tracing in production
• measuring how often things break
• understanding user behaviour
• stepping through tricky paths in dev
• inspecting state live

✅ Use assertions in dev: assert total >= 0, "Total should never be negative"

In production, you can turn off certain assertions or keep only the critical ones.

🎯 10. Principles for "At Scale" Error Handling

Always log unexpected exceptions. Prefer fail loud and fast to silent corruption.

HTTP layer (FastAPI/Django view), Message queue consumer, CLI/worker entry point

DomainError → ValidationError, PaymentError, ExternalServiceError. This makes handling & logging more deliberate.

4. Keep logs human-readable AND machine-parseable

Too much DEBUG in prod = noisy. Too few logs in prod = blind.

Part 2: Production Engineering — Centralized Logging, Correlation IDs & Distributed Tracing

Part 1 covered local logging and error handling. Now we move into real production engineering — the systems used by FastAPI/Django backends, microservices, SaaS platforms, and distributed workers.

🔥 1. Centralised Log Aggregation (The Real World Standard)

As soon as you have multiple servers, background workers, containers, or functions-as-a-service — you cannot inspect logs locally anymore. Real systems send logs to a central place.

✔ ELK Stack (Elasticsearch + Logstash + Kibana) — Most customizable, open-source, works at huge scale

✔ Loki + Grafana — Insanely fast, cheap, streams logs from Docker/Kubernetes

✔ AWS CloudWatch / GCP Logging / Azure Monitor — Great if you already host on those platforms

✔ Datadog / Sentry / NewRelic — Expensive, but world-class dashboards + alerts

⚙️ 2. Logging to STDOUT in Containers (Best Practice)

In Docker/Kubernetes, you never write log files inside the container. You output logs to STDOUT:

Kubernetes will automatically: ✔ capture your stdout, ✔ send it to your log system, ✔ attach metadata (pod, namespace, service). This makes logs fully searchable across the entire cluster.

🧠 3. The Importance of Request IDs / Correlation IDs

Imagine debugging: User logs in → Their request hits API A → API A calls API B → API B queries the database → A background worker processes a message → Something fails.

Without correlation IDs, logs look like a mess. Solution: Generate a unique ID per user request.

Pass the ID through HTTP headers, background jobs, microservice calls, and log contexts. Now you can filter your log system for request_id = "1f0cd133-f9ab-4bd9-a6cd-92d3a002a415" and see EVERYTHING that happened.

This alone can save hours per week in debugging.

🧩 4. Logging Context Automatically (ContextVars)

Python provides a way to attach values (like request IDs) to all logs inside async or threaded code.

Now every log line automatically includes request_id, user_id (if added), job_id, trace_id. This is essential in async web apps like FastAPI.

🚀 5. Distributed Tracing (How Big Systems Debug)

Used by Uber, Netflix, Stripe, Google-scale systems. Distributed tracing tracks: "This request flowed through: API → Worker → DB → Cache → Queue → Another Worker"

Tools: OpenTelemetry (industry standard), Jaeger, Zipkin, Datadog APM

Now logs + traces appear linked. This gives timings for each step, slow points, bottlenecks, failed spans, and dependency maps. This is how modern SaaS teams debug performance problems.

🧵 6. Logging in Async Python (Trickier Than It Looks)

Async apps (FastAPI, aiohttp) handle hundreds/thousands of concurrent tasks. Debugging them needs extra care.

Problem 1: Interleaved logs — Two tasks printing logs at the same time → scrambled output

Tools: python-json-logger, structlog, loguru. Structured logs remove all ambiguity.

🧪 7. Debugging Async Issues

Common async issues: tasks never awaited, task cancelled too early, infinite loops in async code, race conditions modifying shared state, slow network calls blocking event loop.

You can inspect pending tasks, long-running tasks, and tasks stuck on await.

This is how real async backends prevent accidental cancellations.

⚡ 8. Advanced Exception Routing (Tiered Error Strategy)

In large systems, errors must be: 1. Logged, 2. Categorized, 3. Reported to monitoring tools, 4. Potentially retried, 5. Potentially suppressed, 6. Potentially escalated

Logged at WARNING, Message shown to user, No alerts

Logged at ERROR, Trigger retry/backoff, Alert if repeated failures

Logged at CRITICAL, Immediate alert, Fallback mode or shutdown

This prevents alert fatigue and improves reliability.

🔔 9. Error Tracking Tools (Sentry, Rollbar, Bugsnag)

Instead of forcing developers to read logs manually, use automatic error monitoring.

Sentry provides: stack trace, breadcrumbs (log history), user ID + metadata, environment (dev/staging/prod), frequency of error, release version. You can review top crashing endpoints, sudden spikes, and regressions after deploys.

🧊 10. Debug Builds vs Production Builds

Your debugging configuration must change by environment:

Development

• • logging.DEBUG
• • colourful logs
• • stack traces everywhere
• • profiler enabled
• • hot reload

Staging

• • logging.INFO
• • Sentry enabled
• • performance profiling on
• • request IDs enabled

Production

• • logging.WARNING or ERROR
• • no sensitive data
• • structured JSON logs
• • aggressive error tracking
• • correlation IDs
• • distributed tracing

This separation keeps production clean and safe.

Part 3: Monitoring, Alerting & Observability — Production Architecture

The final deep-dive covers how large-scale systems (millions of users) keep themselves healthy, monitor failures, detect outages, and maintain reliability.

🔥 1. Observability: The "Holy Trinity"

Modern systems don't rely on just logging. True observability comes from three pillars:

Text records of what happened (requests, errors, info messages)

Numerical time-series measurements (CPU, latency, throughput, queue length)

Request flow across services (API → DB → Worker → Cache)

Together, these allow you to answer: What broke? Why? When? Where? Who was affected? How long?

This is how Google, Netflix, Uber and Stripe prevent downtime.

⚙️ 2. Metrics You MUST Track in Any Real System

A professional backend MUST export metrics like:

Performance Metrics

• • Request latency (p50/p95/p99)
• • Event loop lag
• • Worker queue size
• • CPU usage per service
• • Memory usage per process

Reliability Metrics

• • Error rate (5xx/exceptions)
• • Retry rate
• • Timeout rate
• • Dead-letter queue count

Throughput Metrics

• • Requests per second
• • Tasks executed per minute
• • DB queries per request
• • Cache hit/miss ratio

These metrics show EXACTLY where your system slows or breaks.

🧠 3. Alerting Like a Real SaaS Company

• ❌ Error rates spike (e.g., more than 2% of requests fail)
• ❌ Latency spikes (p99 over 500ms)
• ❌ CPU % hits critical (over 80% sustained)
• ❌ Memory leaks begin (memory steadily grows without dropping)
• ❌ Queue backups (worker queue length keeps increasing)

Alert channels: Email, SMS, Discord webhook, Slack, PagerDuty (industry standard)

Alerts MUST be actionable and never noisy. (Otherwise you get alert fatigue and ignore them.)

🚨 4. Creating Alert Rules (Industry Examples)

Latency Alert: IF p99_latency > 400ms FOR 5 minutes → alert

Error Rate Alert: IF 5xx_errors > 2% OF total_requests FOR 2 minutes → alert

Worker Queue Alert: IF queue_length > 1000 FOR 10 minutes → alert

Memory Leak Alert: IF memory_usage increases for 30 minutes straight → alert

This protects your system from silent failures.

🔍 5. Log Schema for Enterprise-Level Systems

Logs must be consistent, otherwise your log system becomes useless. Here's a clean, universal JSON format:

Why JSON logs? Machines can process it, log systems index it, developers can query it, easy to parse in dashboards.

Never do: Free-text logs, logs with no structure, logs with inconsistent keys.

🧩 6. Building Dashboards (Real Engineering Style)

API Dashboard

• • Incoming requests
• • Error rate
• • Latency p50/p95/p99
• • Cache hit %

Worker Dashboard

• • Queue depth
• • Processing rate
• • Task wait time
• • Failures vs retries

System Dashboard

• • CPU per service
• • Memory usage
• • Network outbound/inbound

Business Dashboard

• • Failed payments
• • Daily active users

These dashboards turn your logs + metrics into a map of system health.

🧵 7. Detecting Problems Automatically (Heuristics)

Some failures don't generate hard errors. Symptoms you must watch for:

🚨 Slow Increase → Memory Leak (Memory grows hour after hour)

🚨 Sawtooth Pattern → GC Thrashing (Frequent garbage collections)

🚨 CPU Stuck at 100% → Hot Loop (Infinite work created accidentally)

🚨 Queue Length Increasing → Bottleneck (Workers can't keep up)

🚨 Error Spikes at Same Time Daily → Scheduled Task Problem

Your monitoring system should automatically surface these patterns.

🔧 8. Production Failure Scenarios & How to Diagnose

Check: pprof/py-spy flame graph, number of tasks, bad while loops, accidental synchronous code inside async

Check: DB query count, slow endpoints, overloaded workers, missing indexes in SQL

Check: processing time per job, number of workers, retry storms, deadlocks

Check: logs suppressed accidentally, exceptions swallowed, callbacks failed silently, network retries hiding real issues

Check: tracemalloc snapshots, reference cycles, global caches growing, large objects kept alive by closures

📦 9. Production Architecture: Logging + Monitoring Stack

Logging: basic JSON logs, Monitoring: CloudWatch or simple Grafana, Error tracking: Sentry

Logging: Loki/Elasticsearch, Metrics: Prometheus, Traces: Jaeger, Error monitoring: Sentry

Distributed tracing everywhere, Multi-layer dashboards, Real-time anomaly detection, Canary deployments with rollback triggers, Full observability pipeline

🎓 10. The Engineering Mindset for Production Stability

✔ Yes: "How do I prevent this class of bug forever?"

✔ Yes: "What instrumentation do we add so slowdowns are always visible?"

🎉 Final Summary — Master Level

You now understand how real companies handle logging & debugging:

✔ Diagnosing CPU, memory, queue & latency issues

✔ Observability stack for any size of project

This is enterprise-level engineering knowledge, the type senior developers use every day.

📋 Quick Reference — Logging & Debugging

Syntax

What it does

logging.basicConfig(level=logging.INFO)

Set up basic logging config

logger = logging.getLogger(__name__)

Create a named logger

logger.exception("msg")

Log error with full traceback

pdb.set_trace()

Drop into interactive debugger

breakpoint()

Python 3.7+ built-in debugger shortcut

You can now add structured logging, use the debugger effectively, and handle errors at enterprise scale.

Up next: Testing with pytest — write fixtures, parametrised tests, and mocks like a professional.

Practice quiz

Continue this course

• Previous: Module & Package Architecture for Large Codebases
• Next: Testing with pytest: Fixtures, Parametrisation & Mocks