Virtual Environments & Dependency Management Best Practices

🔥 1. Why You Should Never Rely on "System Python"

On most machines, you'll have System Python (used by OS tools) and Your project Python (what you control).

If you install packages globally with pip install requests, you risk:

🧪 2. What a Virtual Environment Actually Is

A virtual environment is:

• A folder containing its own Python interpreter
• With its own site-packages directory
• Isolated from global packages

Two projects can now safely require:

• Project A → Django==3.2
• Project B → Django==5.0

No conflict, because they use different environments.

⚙️ 3. Creating a Virtual Environment (venv)

Standard library way (no extra installs):

# Use the Python you want (3.10 example)
# Run in terminal: python3.10 -m venv .venv

# This creates a .venv/ folder in your project
print("Virtual environment commands:")
print("python3.10 -m venv .venv")
print("This creates a .venv/ folder in your project")

Activating it:

macOS / Linux / WSL:

# Run in terminal:
# source .venv/bin/activate

print("Activation command for macOS/Linux/WSL:")
print("source .venv/bin/activate")

Windows (PowerShell):

# Run in PowerShell:
# .venv\Scripts\Activate.ps1

print("Activation command for Windows PowerShell:")
print(".venv\\Scripts\\Activate.ps1")

Your prompt usually changes to:

From now on:

• python = the one inside .venv
• pip = the one inside .venv

Deactivating:

# Run in terminal:
# deactivate

print("Deactivation command:")
print("deactivate")

🧱 4. Best Practices for Environment Layout

Recommended pattern per project:

# Recommended project structure:
project_structure = '''
my_project/
 ├─ .venv/              # virtual environment (not committed)
 ├─ src/ or app/        # your code
 ├─ tests/
 ├─ requirements.txt    # pinned dependencies
 ├─ requirements-dev.txt
 ├─ pyproject.toml      # (optional, for modern tooling)
 └─ README.md
'''
print(project_structure)

Why .venv?

• Consistent across all your projects
• Editors like VS Code automatically detect it
• Easy to add to .gitignore

.gitignore should include:

# Add these to .gitignore:
gitignore_entries = '''
.venv/
__pycache__/
*.pyc
'''
print(".gitignore entries:")
print(gitignore_entries)

💊 5. Installing Packages the Right Way

Once your environment is activated:

# Run in terminal (with venv activated):
# python -m pip install requests fastapi uvicorn

print("Install command:")
print("python -m pip install requests fastapi uvicorn")

Why python -m pip?

• Guarantees you're using the pip tied to that Python
• Avoids weird "wrong pip" issues

Check installed packages:

# Run in terminal:
# python -m pip list

print("List installed packages:")
print("python -m pip list")

Avoid installing directly with pip outside an activated environment.

📦 6. requirements.txt & Pinning Versions

For reproducibility, you want to freeze exactly which versions you're using.

Create a requirements.txt:

# Run in terminal:
# python -m pip freeze > requirements.txt

print("Freeze command:")
print("python -m pip freeze > requirements.txt")

This file might look like:

# Example requirements.txt content:
requirements = '''
fastapi==0.115.0
uvicorn==0.32.0
pydantic==2.9.0
requests==2.32.3
'''
print("requirements.txt:")
print(requirements)

Now anyone can recreate your environment:

# Commands to recreate environment:
commands = '''
python -m venv .venv
source .venv/bin/activate  # or Windows equivalent
python -m pip install -r requirements.txt
'''
print("Recreate environment:")
print(commands)

This is critical for:

• Deploying to servers
• Running on a teammate's machine
• Long-term maintenance

🔍 7. Semantic Versioning & Safer Constraints

Not all dependencies need to be fully pinned, but you should understand version ranges:

🧪 8. Separating Runtime vs Dev Dependencies

Don't ship your test tools to production. Use two files:

requirements.txt (runtime):

# requirements.txt (runtime only):
runtime_deps = '''
fastapi==0.115.0
uvicorn==0.32.0
pydantic==2.9.0
'''
print("requirements.txt (runtime):")
print(runtime_deps)

requirements-dev.txt (development):

# requirements-dev.txt (development):
dev_deps = '''
-r requirements.txt
pytest==8.3.0
black==24.10.0
ruff==0.7.0
mypy==1.13.0
'''
print("requirements-dev.txt:")
print(dev_deps)

Then install dev tools like:

# Install dev dependencies:
print("python -m pip install -r requirements-dev.txt")

Your production environment only installs requirements.txt, keeping it:

• Lighter
• More secure
• Faster to deploy

🧰 9. Using pip-tools / Poetry / UV (Modern Workflows)

As projects get bigger, plain pip freeze becomes messy. Three common modern approaches:

1) pip-tools

Workflow:

# pip-tools workflow:
workflow = '''
pip install pip-tools

# requirements.in:
# fastapi
# uvicorn

# Compile:
pip-compile requirements.in

# Install:
pip-sync requirements.txt
'''
print("pip-tools workflow:")
print(workflow)

This ensures your environment matches exactly the file.

2) Poetry

Uses pyproject.toml + poetry.lock. You define high-level deps; Poetry resolves & locks everything.

3) UV / Rye / Hatch

Newer tools that combine:

• Env creation
• Dependency resolution
• Locking
• Scripts

🌍 10. Global Python Tools: Use pipx, Not Your Project venv

Some Python packages are tools, not libraries, e.g.:

• httpie
• black
• ruff
• pre-commit

These are better installed with pipx globally:

# Install global tools with pipx:
print("pip install pipx")
print("pipx install httpie")

This:

• Keeps them isolated from projects
• Avoids interfering with your app dependencies
• Lets you use tools from your shell, no activation needed

🧠 11. Environment Rebuild Strategy

A good habit whenever things feel "weird":

# Environment rebuild steps:
steps = '''
# 1. Delete .venv/
rm -rf .venv

# 2. Re-create environment
python -m venv .venv
source .venv/bin/activate

# 3. Reinstall from requirements.txt
python -m pip install -r requirements.txt
'''
print("Rebuild steps:")
print(steps)

If the problem disappears, it was likely:

• Stale dependencies
• Partial upgrades
• Broken wheels

This rebuild pattern is common in real teams.

🧩 12. Solving Dependency Conflicts Like a Pro

You'll eventually hit something like:

This is a dependency conflict — two packages require incompatible versions of a dependency.

How to diagnose the conflict:

Install a dependency tree visualizer:

# Diagnose dependency conflicts:
commands = '''
pip install pipdeptree
pipdeptree

# Example output:
# fastapi==0.115.0
#   - pydantic==2.9.0
# old-tool==1.3.0
#   - pydantic==1.10.0
'''
print("Dependency tree commands:")
print(commands)

How to fix it:

1. Identify which package is outdated
2. Upgrade/downgrade the conflicting package
3. Reinstall
4. If successful: pip freeze > requirements.txt

🧱 13. Lockfiles & Reproducible Installs

With basic pip:

# requirements.txt acts as a lockfile:
# - pin all versions
# - ensure identical installs for every machine

print("Upgrade and freeze:")
print("pip install --upgrade fastapi")
print("pip freeze > requirements.txt")

With pip-tools (recommended):

Keep human-friendly requirements in requirements.in:

# requirements.in (human-friendly):
requirements_in = '''
fastapi
uvicorn
'''
print("requirements.in:")
print(requirements_in)

Compile into a fully pinned lockfile:

# Compile to lockfile:
print("pip-compile requirements.in")

It produces requirements.txt with all sub-dependencies pinned.

🧬 14. Per-Environment Dependencies (Dev, Test, Production)

Large apps usually have:

• dev environment (testing, debugging tools)
• production environment (runtime packages only)
• testing environment (for CI)

A clean structure looks like:

# Multi-environment requirements structure:
structure = '''
requirements/
 ├─ base.in
 ├─ dev.in
 ├─ prod.in
 ├─ base.txt
 ├─ dev.txt
 └─ prod.txt
'''
print(structure)

Example:

requirements/base.in:

# requirements/base.in:
base = '''
fastapi
sqlalchemy
pydantic
'''
print("base.in:")
print(base)

requirements/dev.in:

# requirements/dev.in:
dev = '''
-r base.in
pytest
mypy
black
'''
print("dev.in:")
print(dev)

requirements/prod.in:

# requirements/prod.in:
prod = '''
-r base.in
gunicorn
prometheus-client
'''
print("prod.in:")
print(prod)

Compile:

# Compile all requirement files:
print("pip-compile requirements/base.in")
print("pip-compile requirements/dev.in")
print("pip-compile requirements/prod.in")

Install:

# Install for specific environment:
print("pip install -r requirements/dev.txt")
print("pip install -r requirements/prod.txt")

🔐 15. Internal Libraries & Private Packages

Businesses often create reusable internal libraries. These can be installed:

1. As editable local packages:

# Install local package in editable mode:
print("pip install -e ./core_engine")

2. From a private PyPI server:

# Install from private PyPI:
print("pip install core-engine --extra-index-url https://packages.acmetools.com/simple")

Best practices:

• Use semantic versioning (1.0.0, 1.1.0, etc.)
• Pin specific versions in your main project
• Avoid installing directly from main branches

🛡 16. Security & Supply-Chain Safety

Dependencies are attack vectors. Follow these rules:

✔ 1. Scan dependencies regularly:

# Scan for vulnerabilities:
print("pip install pip-audit")
print("pip-audit")

✔ 2. Stick with reputable packages

Prefer libraries that have:

• Lots of downloads
• Active maintenance
• Frequent updates
• Reliable documentation

✔ 3. Pin versions in production

Floating version ranges can pull in a bad update.

✔ 4. Audit new dependencies

Ask:

• Do I really need this?
• Is this package trustworthy?
• Could I write this functionality myself?

Small dependency trees = safer, easier to maintain.

🐳 17. Using Virtual Environments Inside Docker

Docker isolates processes, but venvs add clarity and consistency.

Example Dockerfile:

# Example Dockerfile with venv:
dockerfile = '''
FROM python:3.11-slim

WORKDIR /app

RUN python -m venv /app/.venv
ENV PATH="/app/.venv/bin:$PATH"

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "main.py"]
'''
print(dockerfile)

Local + Docker both use .venv → consistent setup.

⚙️ 18. Clean Install Testing in CI Pipelines

In CI tools like GitHub Actions or GitLab:

1. Create fresh venv
2. Install dependencies
3. Run tests

Example:

# GitHub Actions workflow example:
workflow = '''
- uses: actions/setup-python@v5
  with:
    python-version: "3.11"

- name: Create venv
  run: python -m venv .venv

- name: Install deps
  run: |
    source .venv/bin/activate
    pip install -r requirements-dev.txt

- name: Run tests
  run: |
    source .venv/bin/activate
    pytest
'''
print(workflow)

This ensures your project:

• Installs cleanly
• Has no hidden local dependencies
• Is production-ready

🧠 19. Workflow & Naming Conventions

Clean conventions prevent chaos:

• Always name virtual env .venv
• Use requirements.in + requirements.txt OR Poetry (don't mix)
• Document setup in README.md
• Never use global interpreter for real apps
• Recreate environment after major upgrades

Example recommended README snippet:

# README setup instructions:
readme = '''
1. python -m venv .venv
2. source .venv/bin/activate
3. pip install -r requirements-dev.txt
'''
print(readme)

🎯 20. The Master Mental Model

Everything you do around dependencies fits into this model:

• Change Python version → everything can break
• Skip virtual env → dependencies collide
• Forget to freeze versions → inconsistent behavior
• Skip lockfile → unpredictable installs

Master this, and your projects become:

• ✔ Stable
• ✔ Reproducible
• ✔ Portable
• ✔ Easy to debug
• ✔ Easy to deploy

🧪 21. Example: Clean Setup Workflow for a New Project

A solid, repeatable pattern for any new Python project:

# Complete new project setup:
setup = '''
# 1. Create the project folder
mkdir awesome-project
cd awesome-project

# 2. Create and activate a virtual environment
python -m venv .venv

# macOS / Linux
source .venv/bin/activate

# Windows (PowerShell)
.venv\\Scripts\\Activate.ps1

# 3. Install core dependencies
pip install fastapi uvicorn pydantic

# 4. Freeze versions
pip freeze > requirements.txt

# 5. Install dev-only tools
pip install pytest black mypy
pip freeze > requirements-dev.txt
'''
pri
...

Example structure:

# Project structure:
structure = '''
awesome-project/
 ├─ .venv/
 ├─ app/
 │   ├─ __init__.py
 │   └─ main.py
 ├─ tests/
 ├─ requirements.txt
 ├─ requirements-dev.txt
 └─ README.md
'''
print(structure)

Anyone who clones this can run:

# After cloning the project:
print("python -m venv .venv")
print("source .venv/bin/activate")
print("pip install -r requirements-dev.txt")

and be ready to work.

🏗 22. Example: Multi-Service Environment Structure

For a slightly bigger setup (e.g. API + worker + shared library):

# Multi-service project structure:
structure = '''
project-root/
 ├─ services/
 │   ├─ api/
 │   │   ├─ app/
 │   │   └─ requirements.txt
 │   ├─ worker/
 │   │   ├─ worker_app/
 │   │   └─ requirements.txt
 ├─ libs/
 │   └─ shared_lib/
 │       └─ setup.py or pyproject.toml
 └─ README.md
'''
print(structure)

Each service can have its own virtual environment:

# Create venv for a specific service:
print("cd services/api")
print("python -m venv .venv")
print("source .venv/bin/activate")
print("pip install -r requirements.txt")

Shared library can be installed in editable mode:

# Install shared library:
print("cd project-root")
print("pip install -e libs/shared_lib")

This keeps boundaries clear, especially when different services need different versions of frameworks.

🧩 24. Handling OS-Specific and Optional Dependencies

Some packages are only needed on certain platforms or for extra features.

Optional dependencies pattern:

# Optional dependencies pattern:
pattern = '''
# Base requirements:
# - fastapi
# - uvicorn

# Dev extras:
# - pytest
# - black

# Optional "redis" extras:
# - redis
'''
print(pattern)

Keep separate requirements-*.txt files:

# Separate requirements files:
files = '''
requirements.txt          # core
requirements-dev.txt      # dev tools
requirements-redis.txt    # extra feature
'''
print(files)

Install only what's needed:

# Install multiple requirements:
print("pip install -r requirements.txt -r requirements-redis.txt")

🧯 25. Troubleshooting Common Virtualenv & Dependency Issues

Problem 1: "It works on my machine but not on theirs"

Likely causes:

• No pinned versions (requirements.txt missing or incomplete)
• Different Python version
• Some packages installed globally by accident

Fix:

1. Ensure everyone uses the same Python major/minor (e.g. 3.11)
2. Use a venv and requirements*.txt files
3. Regenerate lockfiles after big upgrades

Problem 2: "ModuleNotFoundError even though I installed it"

Usually:

• Wrong interpreter selected in IDE
• venv not activated in the terminal
• Installed into global Python instead of the venv

Checklist:

# Troubleshooting checklist:
import sys
print("Current Python:", sys.executable)
print()
print("Terminal commands to check:")
print("which python  # or 'where python' on Windows")
print("which pip     # or 'where pip' on Windows")
print()
print("In VS Code: select interpreter pointing to .venv/bin/python")

Problem 3: "pip install works locally but fails in CI"

Possible causes:

• OS differences (Linux vs Windows)
• Missing system libraries (for packages with C extensions)
• Extra private indexes not configured in CI

Fix:

• Use the same Python version as CI locally (e.g. via pyenv)
• Install system packages in CI (e.g. libpq-dev, build-essential)
• Make sure CI has the same PIP_INDEX_URL / extra-index-url

📐 26. Structuring Requirements for Long-Term Maintainability

A common and very maintainable pattern:

# Maintainable requirements structure:
structure = '''
requirements/
 ├─ base.in
 ├─ dev.in
 ├─ prod.in
 ├─ base.txt
 ├─ dev.txt
 └─ prod.txt
'''
print(structure)

This gives:

• A clear separation of intent (*.in) vs exact resolved versions (*.txt)
• Easy upgrades by editing .in and re-compiling
• Stable installs on all machines/environments

🧠 27. Mental Models to Keep Your Environments Sane

A few simple rules keep everything under control:

1. One project → one virtual environment

   Don't reuse venvs across unrelated projects.

2. Never install app dependencies globally

   Global Python is for tooling at most, not for full stacks.

3. Pin versions for anything serious

   Experiments can be loose, but production/teaching projects should be pinned.

4. Document setup steps once, reuse everywhere

   A short "Setup" section in your README saves hours of debugging.

5. Treat requirements.txt or poetry.lock as part of your code

   They're versions of your environment — commit them and keep them updated.

✅ 28. Quick Checklist for a "Professional" Environment Setup

If you can answer yes to these, your dependency game is solid:

• Does the project have a .venv and ignore it in .gitignore?
• Are dependencies pinned somewhere (requirements.txt or lockfile)?
• Is there a clear separation between runtime deps and dev/test tools?
• Can someone new set up the project using only the README?
• Can CI recreate the environment from scratch and pass tests?
• Do you avoid installing project dependencies globally?

If any are "no", that's the next thing to improve.