Courses/Python/Integrating Python with Other Languages (C, Rust & More)

Lesson 38 • Advanced

Integrating Python with Other Languages (C, Rust & More)

Master Python + C/Rust integration using ctypes, cffi, Cython, and PyO3 for maximum performance while maintaining Python's productivity

Note: To run Python code on your computer, you'll need Python installed. Download Python here. Keep the website open and test our code examples with Python running on your computer.

🏎️ Real-World Analogy: Python is like a comfortable family car — easy to drive, safe, and practical for everyday use. But sometimes you need a race car (C/Rust) for the track. Language integration is like having a racing engine you can swap into your family car when you need speed, while keeping the comfortable steering wheel and seats you're used to.

What You'll Learn

Method	Difficulty	Best For	Speed Gain
ctypes	Easy	Quick prototyping, calling existing C libs	10-50x
CFFI	Medium	Cleaner C interface, better error handling	10-50x
Cython	Medium	Gradual optimization of Python code	10-100x
PyO3 (Rust)	Hard	Memory-safe, blazing fast extensions	50-1000x

Why and when to integrate Python with other languages
CPython's architecture and extension mechanism
Using ctypes to call C libraries without compilation
cffi for more ergonomic C interfacing
Writing native Python extensions in C/C++
Cython for Python-like syntax with C performance
Rust integration with PyO3 for memory safety
Embedding Python in C/Rust applications
Data marshalling and zero-copy strategies
Performance optimization and profiling
Safety considerations and testing
Building and distributing cross-platform extensions

Why Integrate Python with Other Languages?

Python is excellent for productivity, but sometimes you need more. Language integration lets you keep Python's ease of use while accessing native performance and capabilities.

Common Reasons

Performance - Move CPU-intensive loops to C/Rust (10-100x speedup)
Existing libraries - Use mature C/C++/Rust libraries without rewriting
System-level access - Hardware drivers, OS syscalls, low-level protocols
Memory safety - Rust prevents segfaults and memory leaks
Incremental optimization - Profile first, optimize only bottlenecks

The 95/5 Rule

Keep 95% of your code in Python for maintainability. Move the critical 5% (hot loops, numeric kernels) to native code. This is exactly how NumPy, pandas, and PyTorch work.

CPython's Architecture

Understanding CPython's internals helps you write better extensions.

Key Facts

CPython itself is written in C
All Python objects are C structs (PyObject, PyListObject, etc.)
The GIL (Global Interpreter Lock) serializes Python bytecode execution
C extensions can release the GIL for CPU-bound work
The Python/C API lets you create new types and call Python from C

How Extensions Work

When you import myextension, Python:

Looks for a shared library (.so, .pyd, .dll)
Loads it dynamically
Calls PyInit_myextension()
Registers functions and types with the Python runtime

ctypes: Call C Without Compilation

ctypes is Python's built-in FFI (Foreign Function Interface) for calling C libraries.

Workflow

Compile C code as shared library (.so/.dll/.dylib)
Load with ctypes.CDLL()
Declare function argument and return types
Call functions like normal Python

Pros & Cons

✓ Advantages

• No Python-specific build step
• Works with existing binaries
• Part of Python standard library
• Cross-platform

✗ Limitations

• Manual type declarations
• Easy to make mistakes (crashes)
• Limited struct/pointer support
• No automatic error handling

cffi: More Ergonomic C Interface

cffi improves on ctypes with C-style declarations and better error handling.

Key Features

C-style API definitions in Python strings
Better struct, pointer, and enum support
Two modes: ABI (existing binaries) and API (compile from source)
Automatic type conversions
Used by popular libraries like cryptography and PyPy

When to Use

Choose cffi when you need to wrap complex C APIs with many structs and pointers, or when you want more safety than raw ctypes provides.

Native Python Extensions in C/C++

The Python/C API provides maximum control and performance for extension modules.

Core Components

PyObject* - Base type for all Python objects
PyArg_ParseTuple - Parse function arguments
Py_BuildValue - Construct return values
PyMethodDef - Method table for module functions
PyModuleDef - Module definition
PyInit_* - Module initialization function

Use Cases

NumPy-style array operations
Custom Python types in C
Maximum performance numeric code
Direct CPython runtime manipulation

⚠️ Warning

C extensions are powerful but error-prone. Manual memory management, reference counting, and GIL handling make them challenging. Consider Cython or Rust instead for new projects.

Cython: Python Syntax, C Speed

Cython lets you write Python-like code with optional type annotations that compile to C.

Key Features

Familiar syntax - Looks like Python with type hints
Gradual typing - Add types where they matter for speed
C integration - Call C functions directly
NumPy support - Efficient typed array operations
Compiler directives - Disable bounds checking, wraparound

Performance Tips

• Use cdef for C-level variables and functions

• Type function parameters and return values

• Use typed memory views for arrays: double[:]

• Disable safety checks in hot loops

• Release the GIL for CPU-intensive work: with nogil:

Perfect For

Scientific computing, numeric algorithms, and data processing. If your bottleneck is a tight numeric loop, Cython is often the fastest path to optimization.

Rust Integration with PyO3

Rust offers C-like performance with memory safety guarantees. PyO3 makes Rust-Python integration seamless.

Why Rust + Python?

Memory safety - No segfaults, no data races
Performance - Comparable to C/C++
Modern tooling - Cargo, rustfmt, clippy
Rich ecosystem - Excellent libraries for crypto, networking, parsing
Fearless concurrency - Safe parallel processing

PyO3 Features

Functions

• #[pyfunction] macro
• Automatic type conversion
• Python exceptions
• Default arguments

Classes

• #[pyclass] for types
• #[pymethods] for methods
• Properties and attributes
• Magic methods (__repr__, etc.)

Building with maturin

# Install
pip install maturin

# Develop (builds and installs locally)
maturin develop

# Build wheel for distribution
maturin build --release

Embedding Python

Sometimes you want to embed Python as a scripting language inside a C/Rust application.

Use Cases

Game engines - Let users write Python mods and scripts
Plugin systems - Extensible applications with Python plugins
Configuration - Python as a config language
Trading systems - Rust/C core with Python strategies
Scientific tools - Fast core with Python analysis

Basic Workflow

Initialize Python runtime: Py_Initialize()
Import modules and run code
Call Python functions from C/Rust
Pass data between languages
Cleanup: Py_Finalize()

Data Marshalling Strategies

Efficient data transfer between languages is critical for performance.

Zero-Copy Techniques

NumPy arrays - Pass raw pointers to contiguous memory
Buffer protocol - Python's standard for binary data
Memory views - Share memory without copying
Typed memory views in Cython - double[:]

Best Practices

• Batch operations - Call once with 1M elements, not 1M times with 1 element

• Use simple types at boundaries - Primitives, strings, byte arrays

• Avoid repeated conversions - Convert once, reuse

• Check array layout - Ensure C-contiguous for efficient access

⚠️ Common Mistake

Converting large NumPy arrays to Python lists loses all performance benefits. Always pass array pointers directly to C/Rust code.

When to Choose Which Integration Path

Decision Matrix

Need numeric performance?

→ Cython or NumPy optimization

Calling existing C library?

→ ctypes (simple) or cffi (complex APIs)

Need memory safety?

→ Rust + PyO3

Complex systems code?

→ Rust + PyO3 (modern) or C++ with pybind11 (legacy)

Need scripting in native app?

→ Embed Python with CPython API or PyO3

Maximum control needed?

→ Direct C extension (but consider alternatives first)

Practical Workflow

The best approach is incremental optimization based on profiling.

Recommended Process

Write everything in pure Python
Get it working first. Premature optimization wastes time.
Profile to find bottlenecks
Use cProfile, line_profiler, or py-spy to identify hot spots.
Optimize in Python first
Use NumPy, better algorithms, caching. Native code may not be needed.
Move critical functions to native code
Only the 5-10% that's truly slow. Keep the rest in Python.
Wrap behind Python API
Callers shouldn't know it's implemented in C/Rust.
Add comprehensive tests
Test the boundary thoroughly. Native code bugs are harder to debug.

Performance & Safety

Critical Considerations

Memory management - Track lifetimes, avoid use-after-free
Reference counting - Properly incref/decref in C extensions
GIL handling - Release for CPU work, hold for Python API calls
Error propagation - Convert native errors to Python exceptions
Type safety - Validate inputs at language boundaries
Thread safety - Native code may run without GIL protection

Testing Strategy

Unit Tests

Test native code in isolation with C/Rust test frameworks

Integration Tests

Test Python API with pytest, validate behavior and errors

Fuzz Testing

Generate random inputs to find crashes and edge cases

Building & Distribution

Professional packages need cross-platform wheels that users can install without compilers.

Build Tools

For Rust (PyO3)

maturin - Handles building, packaging, and uploading to PyPI

For C/C++

setuptools with Extension or scikit-build for CMake

For Cython

setuptools with cythonize()

CI/CD for Extensions

Use GitHub Actions with cibuildwheel or maturin to build wheels for:

• Linux: manylinux (x86_64, aarch64)
• macOS: Intel and Apple Silicon
• Windows: 32-bit and 64-bit

Common Pitfalls

✗ Don't Do This

Optimize before profiling - you'll optimize the wrong thing
Convert large arrays to Python lists - destroys performance
Call native functions in tight Python loops - batch instead
Ignore error handling at boundaries - leads to crashes
Store pointers to Python memory in native code - lifecycle issues
Use shell=True in subprocess - security risk
Build giant monolithic native modules - hard to maintain

✓ Best Practices

Profile first, optimize hot paths only
Use NumPy array pointers for zero-copy data passing
Keep native modules focused and small
Test boundaries thoroughly with edge cases
Validate inputs before crossing to native code
Document memory ownership clearly
Provide fallback Python implementations

Complete Integration Example

Full example of Python and native code integration

Try it Yourself »

Python

# Python Language Integration Examples

# ============================================
# 1. CALLING C WITH CTYPES (NO COMPILATION)
# ============================================

import ctypes
import os
from pathlib import Path

# Example 1: Basic ctypes usage
def load_c_library_example():
    """Load and call functions from a shared C library"""
    # On Linux/macOS: libmath.so, on Windows: math.dll
    lib_name = "libmath.so" if os.name != "nt" else "math.dll"
    
    try:
        # Load the 
...

Key Takeaways

Python + C/Rust integration gives you productivity AND performance
Keep 95% in Python, optimize the critical 5% in native code
ctypes is easiest for simple C libraries, cffi for complex ones
Cython offers Python-like syntax with C performance for numeric code
Rust + PyO3 provides memory safety and modern tooling
Always profile before optimizing - intuition is often wrong
Zero-copy data passing via NumPy pointers is critical for performance
Test language boundaries thoroughly - bugs are harder to debug
Build cross-platform wheels for easy pip install experience
This is how NumPy, pandas, PyTorch, and most fast libraries work

📋 Quick Reference — Language Integration

Tool	What it does
ctypes.CDLL('lib.so')	Load and call a C shared library
ctypes.c_int / ctypes.c_double	C type mappings for Python
cffi	Modern C integration via inline C headers
PyO3 (Rust)	Write Python extensions in Rust
numpy array as C pointer	Zero-copy data sharing with C/Rust

🎉 Great work! You've completed this lesson.

You can now call C and Rust from Python — the same technique used in NumPy, pandas, PyTorch, and every high-performance Python library.

Up next: Architecture Patterns — apply MVC, Clean Architecture, and DDD to real Python applications.

Integrating Python with Other Languages (C, Rust & More)

What You'll Learn

Why Integrate Python with Other Languages?

Common Reasons

The 95/5 Rule

CPython's Architecture

Key Facts

How Extensions Work

ctypes: Call C Without Compilation

Workflow

Pros & Cons

✓ Advantages

✗ Limitations

cffi: More Ergonomic C Interface

Key Features

When to Use

Native Python Extensions in C/C++

Core Components

Use Cases

⚠️ Warning

Cython: Python Syntax, C Speed

Key Features

Performance Tips

Perfect For

Rust Integration with PyO3

Why Rust + Python?

PyO3 Features

Functions

Classes

Building with maturin

Embedding Python

Use Cases

Basic Workflow

Data Marshalling Strategies

Zero-Copy Techniques

Best Practices

⚠️ Common Mistake

When to Choose Which Integration Path

Decision Matrix

Practical Workflow

Recommended Process

Performance & Safety

Critical Considerations

Testing Strategy

Unit Tests

Integration Tests

Fuzz Testing

Building & Distribution

Build Tools

For Rust (PyO3)

For C/C++

For Cython

CI/CD for Extensions

Common Pitfalls

✗ Don't Do This

✓ Best Practices

Complete Integration Example

Key Takeaways

📋 Quick Reference — Language Integration

Cookie & Privacy Settings