Devops Automation
Master Python automation for infrastructure management, deployment pipelines, monitoring, backups, and production system orchestration
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
What You'll Learn
- File and directory automation for logs, backups, and cleanup
- System command execution with subprocess
- Task scheduling and cron alternatives
- Server health monitoring and metrics collection
- API and webhook automation for CI/CD
- Docker container automation and management
- Kubernetes deployment automation
- Automated backups and data rotation
- Log processing and real-time monitoring
- Zero-downtime deployment scripts
- Infrastructure-as-Code patterns
- Building custom orchestration tools
Why Python for DevOps Automation?
Python has become the de facto standard for DevOps automation, replacing shell scripts with safer, more maintainable solutions.
Feature
Bash Scripts
Python Automation
Error handling
Cryptic exit codes
try/except with clear messages
Cross-platform
Linux/Mac only
Works everywhere
API integration
Requires curl hacks
Native requests/boto3
Maintainability
Hard to read at scale
Clean, testable code
Common Use Cases
- Deployment automation and orchestration
- Server provisioning and configuration
- Backup and disaster recovery
- Log aggregation and analysis
- Monitoring and alerting
- Infrastructure health checks
- Secret rotation and security hardening
File & Directory Automation
Every DevOps workflow involves managing files: rotating logs, cleaning temporary data, synchronizing directories, and organizing backups.
Common Tasks
- Cleanup - Remove old temporary files and logs
- Log rotation - Compress and archive logs when they exceed size limits
- Directory sync - Keep backup directories in sync
- Backup management - Create and rotate backups automatically
- File monitoring - Watch for changes and trigger actions
Real-World Example
A production CI server runs a cleanup script every hour to remove build artifacts older than 7 days, preventing disk space exhaustion. This same pattern applies to log management, cache cleanup, and temporary file handling.
System Command Execution
The subprocess module provides safe, controlled execution of system commands with proper error handling and timeout management.
Best Practices
- Always use lists - ["ls", "-la"] not "ls -la"
- Set timeouts - Prevent hanging on unresponsive commands
- Capture output - Capture stdout/stderr for logging and debugging
- Check return codes - Non-zero means failure
- Avoid shell=True - Prevents injection attacks
Common Operations
- Restarting systemd services
- Checking service status
- Running Docker and Kubernetes commands
- Executing build and deployment scripts
- Managing SSH connections
Task Scheduling
Modern DevOps requires more intelligent scheduling than traditional cron. Python provides flexible alternatives.
Tool
Best For
Complexity
Cron
Simple, one-off scripts
Low
APScheduler
In-process scheduling
Medium
Celery Beat
Distributed, high-volume
High
Scheduling Options
Traditional Cron
APScheduler (Python)
More powerful: retry on failure, parallel execution, event-based triggers, state management
Celery Beat
Distributed task queue with advanced scheduling capabilities
Typical Scheduled Tasks
- Daily database backups at 2 AM
- Log rotation every 6 hours
- Health checks every 5 minutes
- Cleanup scripts at midnight
- Certificate renewal checks weekly
Server Health Monitoring
Proactive monitoring prevents outages. Python can track system resources and alert teams before problems escalate.
Metrics to Monitor
- CPU usage - Alert on sustained high usage
- Memory consumption - Prevent OOM kills
- Disk space - Alert before running out
- Network I/O - Detect unusual traffic patterns
- Process health - Ensure critical services are running
- System uptime - Track stability
The psutil Library
psutil is the standard for cross-platform system monitoring in Python:
Provides CPU, memory, disk, network, and process information on Linux, macOS, and Windows.
API & Webhook Automation
Modern infrastructure is API-driven. Python integrates seamlessly with CI/CD systems, monitoring tools, and cloud platforms.
Common Integrations
- CI/CD triggers - GitHub Actions, GitLab CI, Jenkins
- Alerting - Slack, PagerDuty, Discord webhooks
- Monitoring - Datadog, Prometheus, Grafana APIs
- Cloud providers - AWS, GCP, Azure management APIs
- Container registries - Docker Hub, ECR, GCR
Automation Patterns
Event-driven deployment
Automated alerting
High CPU → send Slack alert → scale infrastructure
Self-healing systems
Service down → restart automatically → notify team
Docker Automation
The Docker Python SDK enables comprehensive container lifecycle management from within Python scripts.
Installation
Automation Tasks
- Cleanup - Remove stopped containers and dangling images
- Health checks - Monitor container health status
- Auto-restart - Restart unhealthy containers
- Log collection - Aggregate logs from all containers
- Image management - Build, tag, and push images
- Resource limits - Monitor and enforce CPU/memory limits
Production Use Case
A maintenance script runs nightly to clean up stopped containers and dangling images, preventing disk space issues. It also restarts any containers marked as unhealthy by Docker's health checks.
Kubernetes Automation
The Kubernetes Python client allows programmatic cluster management, enabling GitOps-style automation.
Automation Capabilities
- Deployment management - Scale, update, rollback deployments
- Pod operations - List, inspect, delete pods
- ConfigMap/Secret updates - Rotate configurations safely
- Health monitoring - Check pod and node health
- Auto-scaling - Adjust replicas based on metrics
- Resource cleanup - Remove completed jobs and old pods
Advanced Patterns
• Blue/green deployments - maintain two production environments
• Canary releases - gradually roll out changes to a subset of users
• Automatic rollback - revert on health check failure
• Multi-cluster management - orchestrate across regions
Backup Automation & Data Rotation
Regular, automated backups are essential for disaster recovery. Python orchestrates the entire backup lifecycle.
Backup Strategy
- Database dumps - MySQL, PostgreSQL, MongoDB
- File system backups - Compress and archive directories
- Cloud sync - Upload to S3, Google Cloud Storage, Azure Blob
- Rotation policy - Keep daily (7 days), weekly (4 weeks), monthly (12 months)
- Verification - Test restore capability periodically
- Encryption - Encrypt backups before storage
3-2-1 Backup Rule
3 copies of data • 2 different media types • 1 offsite copy
Python scripts can implement this automatically: local disk, network storage, cloud backup.
Log Processing & Automated Alerts
Logs contain critical information about system health, security events, and errors. Automated analysis prevents issues from going unnoticed.
Log Analysis Tasks
- Error detection - Count and categorize errors
- Pattern matching - Find security threats or anomalies
- Performance analysis - Identify slow queries and requests
- Real-time monitoring - Tail logs and alert immediately
- Aggregation - Combine logs from multiple services
- Visualization - Generate reports and dashboards
Alert Triggers
• Error threshold - Alert when error rate exceeds 1%
• Security events - Failed login attempts, suspicious patterns
• Performance degradation - Response time above threshold
• Service crashes - Application or container restarts
Zero-Downtime Deployment
Production deployments must minimize or eliminate downtime. Python orchestrates sophisticated deployment strategies.
Deployment Pipeline
Safety Mechanisms
- Rolling updates - replace pods gradually
- Health checks - verify each new pod before proceeding
- Automatic rollback on failure
- Smoke tests after deployment
- Traffic shifting strategies
Security Considerations
Automation scripts often run with elevated privileges. Security must be a top priority.
✓ Best Practices
- Store secrets in environment variables or secret managers
- Never hardcode passwords, API keys, or tokens
- Use least-privilege principles for automation accounts
- Validate all inputs before executing system commands
- Avoid shell=True in subprocess calls
- Keep logs free of sensitive information
- Implement audit trails for all automation actions
- Use encrypted connections for remote operations
✗ Security Anti-Patterns
- Hardcoding credentials in scripts
- Running automation as root unnecessarily
- Accepting user input without validation
- Logging sensitive data
- Storing backups without encryption
- Ignoring certificate validation
Building Production-Ready Automation
Professional automation systems require more than working code. They need reliability, observability, and maintainability.
Essential Components
- Logging - Comprehensive, structured logs with context
- Error handling - Graceful failure and recovery
- Monitoring - Track automation success/failure rates
- Documentation - Clear runbooks and troubleshooting guides
- Testing - Unit and integration tests for automation logic
- Version control - Git for all automation scripts
- Idempotency - Scripts can run multiple times safely
The DevOps Loop
Write automation → Test thoroughly → Deploy → Monitor → Learn from failures → Improve → Repeat
Every automation failure is an opportunity to make the system more resilient.
Key Takeaways
- Python is the industry standard for DevOps automation due to cross-platform support and rich libraries
- Automate repetitive tasks: file cleanup, backups, deployments, monitoring
- Use subprocess safely with timeouts and proper error handling
- Modern scheduling goes beyond cron - build intelligent task runners
- Monitor system health proactively with psutil and automated alerts
- Docker and Kubernetes Python SDKs enable comprehensive container orchestration
- Implement zero-downtime deployments with health checks and automatic rollback
- Security is critical - never hardcode secrets, validate inputs, use least privilege
- Production automation requires logging, monitoring, testing, and documentation
- Build self-healing systems that detect and correct issues automatically
📋 Quick Reference — DevOps Automation
Tool / Module
What it does
pathlib.Path
Modern file and directory manipulation
subprocess.run(cmd, check=True)
Run shell commands from Python
shutil.copy2 / shutil.rmtree
High-level file operations
docker SDK
Manage Docker containers from Python
psutil
Monitor CPU, memory, and processes
You can now automate deployments, manage infrastructure, and build self-healing systems using Python's DevOps toolkit.
Up next: Language Integration — call C and Rust code from Python for maximum performance.
Practice quiz
Which module is the safe, recommended way to run system commands from Python?
- os.system
- commands
- subprocess
- shlex
Answer: subprocess. subprocess.run gives controlled execution with output capture, timeouts, and return-code checking.
What is a best practice when calling subprocess for security?
- Pass the command as a list and avoid shell=True to prevent injection
- Always use shell=True
- Concatenate user input into a string
- Disable timeouts
Answer: Pass the command as a list and avoid shell=True to prevent injection. Passing a list like ['ls', '-la'] and avoiding shell=True prevents shell-injection attacks.
Why set a timeout on subprocess calls?
- To speed up the command
- To capture stderr
- It is required syntax
- To prevent the script hanging on an unresponsive command
Answer: To prevent the script hanging on an unresponsive command. A timeout stops the script from hanging indefinitely if a command never returns.
Which library is the standard for cross-platform system metrics (CPU, memory, disk)?
- os
- psutil
- sys
- platform
Answer: psutil. psutil provides CPU, memory, disk, network, and process info across Linux, macOS, and Windows.
Which class in the lesson uses the Docker SDK to clean up stopped containers and restart unhealthy ones?
- DockerAutomation
- DockerManager
- ContainerBot
- DockerClient
Answer: DockerAutomation. The DockerAutomation class wraps docker.from_env() to remove stopped containers, prune images, and restart unhealthy ones.
In the deployment workflow, what happens if the Kubernetes rollout fails its health check?
- The script ignores it
- It deletes the deployment
- It automatically rolls back with kubectl rollout undo and alerts the team
- It retries forever
Answer: It automatically rolls back with kubectl rollout undo and alerts the team. On a failed rollout the script runs kubectl rollout undo to revert and sends an alert about the failure.
What is the recommended place to store secrets like API keys in automation scripts?
- Hard-coded in the script
- Environment variables or a secret manager
- In the log files
- In the Git history
Answer: Environment variables or a secret manager. Secrets belong in environment variables or secret managers, never hard-coded in code.
What does the '3-2-1 backup rule' mean?
- 3 servers, 2 regions, 1 admin
- 3 daily backups kept for 21 days
- 3 scripts, 2 schedules, 1 alert
- 3 copies of data, 2 different media types, 1 offsite copy
Answer: 3 copies of data, 2 different media types, 1 offsite copy. The 3-2-1 rule keeps 3 copies of data on 2 media types with 1 stored offsite.
Why does the lesson favor Python over bash scripts for DevOps automation?
- Python runs only on Linux
- Better error handling, cross-platform support, and clean native API integration
- Bash cannot run commands
- Python is always faster
Answer: Better error handling, cross-platform support, and clean native API integration. Python offers try/except error handling, works across platforms, and integrates with APIs via libraries like requests and boto3.
What does it mean for an automation script to be 'idempotent'?
- It runs only once ever
- It requires root access
- It can run multiple times safely without causing harm
- It never writes to disk
Answer: It can run multiple times safely without causing harm. An idempotent script produces the same safe result whether run once or many times, a key property of reliable automation.
Continue this course
- Previous: REST API
- Next: Language Integration