Profiling
High-performance Python isn't about writing "faster code" — it's about finding bottlenecks and eliminating them with scientific precision. You cannot optimise what you do not measure.
Part of the free Python course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.
What You'll Learn in This Lesson
- • How to profile CPU usage with cProfile and line_profiler
- • How to measure memory usage with tracemalloc and memory_profiler
- • How to find the real bottleneck in your code (it's rarely where you think)
- • Practical optimisation techniques: caching, algorithm choice, data structures
- • How to write benchmarks using timeit and interpret results correctly
- • Production-level performance patterns used in real Python systems
🔥 1. Why Profiling Matters
Beginners try to "guess" what's slow. Advanced developers measure what's slow.
Approach
Method
Result
❌ Guessing
"This loop looks slow"
Waste time optimizing the wrong code
✔ Profiling
Measure actual execution time
Find and fix real bottlenecks
Optimizing the wrong 80% gives no improvement!
⚙️ 2. Timing Functions with time.perf_counter()
- two ways of looping
- two algorithms
- two function implementations
But for full programs, we need real profilers.
🧠 3. Profiling With cProfile — The Standard Tool
Run a script with profiling from command line:
⚠️ Requires local Python installation • Download Python
Column
What It Shows
ncalls
Number of times function was called
tottime
Total time in this function (excluding subcalls)
cumtime
Cumulative time (including subcalls)
📊 4. Making Results Readable With pstats
Sort By
Best For Finding
"tottime"
Time in function itself
The actual slow functions
"cumtime"
Time including all sub-calls
Functions that call slow things
"ncalls"
Number of times called
Unexpectedly hot loops
- "tottime" → slowest total functions
- "cumtime" → functions including subcalls
- "ncalls" → most-called functions
🧠 5. Line-by-Line Profiling With line_profiler
- nested loops
- ML preprocessing
- tight functions
- recursive code
🧩 6. Memory Profiling
Memory Issue
Symptom
Common Cause
Memory spike
Sudden +500MB on one line
Loading large dataset at once
Memory leak
Memory grows over time
Data accumulating in loops
High baseline
Program starts with 200MB+
Heavy imports (pandas, tensorflow)
- ✔ large lists
- ✔ numpy allocations
- ✔ memory leaks
- ✔ generators vs lists performance
⚡ 7. Real Techniques for Faster Python
Pure Python loops are slow. NumPy performs operations in C — often 50–200× faster.
Transforms slow recursive functions → instant.
🏎️ 8. Avoiding the Biggest Performance Mistakes
❌ Mistake
Why It's Slow
✅ Better Approach
Unnecessary list copies
Copies entire list in memory
Use slices or itertools
Python loops for math
Interpreted = slow
NumPy vectorized operations
String concatenation in loop
Creates new string each time
Use ''.join(list)
Opening files repeatedly
Disk I/O is expensive
Open once, read/write many
Blocking I/O in async
Blocks the entire event loop
Use run_in_executor()
- ❌ Unnecessary list copies
- ❌ Using Python loops for math
- ❌ Excessive string concatenation
- ❌ Opening files repeatedly
- ❌ Overuse of classes when simple functions work
- ❌ Blocking I/O in async code
🧪 9. Real-World Example: Speeding Up JSON Parsing
⚠️ Requires: pip install orjson • Download Python
orjson is 5–20× faster than Python's JSON parser.
🎉 Conclusion
By mastering profiling and optimisation, you gain the ability to:
Performance comes from measure → diagnose → optimise, not guessing.
📋 Quick Reference — Profiling & Performance
Tool / Syntax
What it does
cProfile.run('fn()')
Profile function call counts and time
timeit.timeit('expr', number=1000)
Benchmark small code snippets
line_profiler
Profile line-by-line execution time
memory_profiler
Track memory usage per line
__slots__
Reduce class memory footprint
You now know how to measure, diagnose, and fix performance bottlenecks — the professional workflow every senior engineer uses.
Up next: Memory Management — understand how Python's garbage collector works and prevent memory leaks.
Practice quiz
What is the core idea behind profiling before optimising?
- Always optimise the longest function first
- Rewrite everything in C
- Measure what's slow instead of guessing
- Add more print statements
Answer: Measure what's slow instead of guessing. You cannot optimise what you do not measure — profiling replaces guessing with data.
Which standard-library tool profiles function call counts and time?
- cProfile
- tracemalloc
- asyncio
- logging
Answer: cProfile. cProfile records every call, how long each takes, and how many times it runs.
In cProfile output, what does 'tottime' measure?
- Total time including all sub-calls
- Number of times the function was called
- Total program runtime
- Time in the function itself, excluding sub-calls
Answer: Time in the function itself, excluding sub-calls. tottime is time spent in the function body alone; cumtime includes time in sub-calls.
Which sort key best reveals the actually-slow functions?
- "ncalls"
- "tottime"
- "cumtime"
- "name"
Answer: "tottime". Sort by tottime to find where time is really spent; then use cumtime to trace callers.
What does functools.lru_cache do to a recursive fib function?
- Caches results so repeated calls are instant
- Slows it down by caching
- Runs it on multiple cores
- Converts it to a loop
Answer: Caches results so repeated calls are instant. lru_cache memoizes results, turning exponential recursion into near-instant lookups.
What does timeit.timeit('expr', number=1000) return?
- The result of the expression
- A profile report object
- The total time (a float) to run it that many times
- The number of calls
Answer: The total time (a float) to run it that many times. timeit returns the total elapsed time as a float for the given number of executions.
Why use a generator expression (x*x for x in nums) over a list for large data?
- It is always faster to build
- It saves memory by yielding items lazily
- It sorts the data
- It runs in parallel
Answer: It saves memory by yielding items lazily. Generators produce items one at a time instead of building the whole list in memory.
What is the recommended fix for slow string concatenation in a loop?
- Use s = s + x each time
- Use a global string
- Use print() to build it
- Use ''.join(list_of_strings)
Answer: Use ''.join(list_of_strings). ''.join builds the result in one pass; repeated + creates a new string object every iteration.
What does the 80/20 rule say about performance?
- 80% of code runs in 20% of the time
- Roughly 20% of code accounts for ~80% of runtime
- Optimise 80% of functions
- 20% of bugs cause 80% of crashes
Answer: Roughly 20% of code accounts for ~80% of runtime. A small fraction of code dominates runtime, so optimising the wrong 80% gives no improvement.
Which tool tracks memory usage line-by-line?
- cProfile
- timeit
- memory_profiler
- pstats
Answer: memory_profiler. memory_profiler shows memory growth per line; cProfile and timeit measure time, not memory.