案例研究 on Tarragon

案例：效能分析實戰

Wed, 21 Jan 2026 00:00:00 +0000

本案例基於 .claude/lib/markdown_link_checker.py 的實際程式碼，展示如何用 cProfile 和 line_profiler 進行效能分析。

先備知識

模組四基礎章節
Python 正則表達式基礎

問題背景

現有設計

markdown_link_checker.py 是一個 Markdown 連結檢查工具，核心功能是解析文件中的連結並驗證其有效性。以下是關鍵的解析方法：

 1import re
 2from typing import List, Dict
 3
 4class MarkdownLinkChecker:
 5    """Markdown link checker with precompiled regex patterns"""
 6
 7    # Precompiled regex patterns at class level (good practice!)
 8    INLINE_LINK_PATTERN = re.compile(
 9        r'(?
10    )
11
12    REFERENCE_DEF_PATTERN = re.compile(
13        r'^\s*\[([^\]]+)\]:\s*(.+)$',
14        re.MULTILINE
15    )
16
17    REFERENCE_USE_PATTERN = re.compile(
18        r'\[([^\]]+)\]\[([^\]]+)\]'
19    )
20
21    def parse_markdown_links(self, content: str) -> List[Dict]:
22        """
23        Parse all links in Markdown content
24
25        Args:
26            content: Markdown content string
27
28        Returns:
29            list[dict]: List of links with text, target, line
30        """
31        links = []
32        lines = content.split('\n')
33
34        # First, collect reference-style link definitions
35        reference_defs = {}
36        for match in self.REFERENCE_DEF_PATTERN.finditer(content):
37            ref_name = match.group(1).lower()
38            ref_target = match.group(2).strip()
39            reference_defs[ref_name] = ref_target
40
41        # Track code block state
42        in_code_block = False
43
44        # Parse inline links line by line
45        for line_num, line in enumerate(lines, start=1):
46            # Check for code block boundaries
47            if line.strip().startswith("```"):
48                in_code_block = not in_code_block
49                continue
50
51            # Skip links inside code blocks
52            if in_code_block:
53                continue
54
55            # Inline links: [text](/python-advanced/04-cpython-internals/case-studies/profiling/target)
56            for match in self.INLINE_LINK_PATTERN.finditer(line):
57                links.append({
58                    "text": match.group(1),
59                    "target": match.group(2),
60                    "line": line_num
61                })
62
63            # Reference-style links: [text][ref]
64            for match in self.REFERENCE_USE_PATTERN.finditer(line):
65                ref_name = match.group(2).lower()
66                if ref_name in reference_defs:
67                    links.append({
68                        "text": match.group(1),
69                        "target": reference_defs[ref_name],
70                        "line": line_num
71                    })
72
73        return links

效能問題

處理大型文件時可能出現：

正則表達式效率問題：複雜的 pattern 可能導致回溯
重複編譯正則表達式：若 pattern 在方法內定義，每次呼叫都會重新編譯
不必要的字串操作：split() 會建立新的字串列表
多次遍歷：分別處理引用定義和行內連結

進階解決方案

分析目標

找出效能瓶頸所在
量化各部分的時間消耗
驗證優化效果

實作步驟

步驟 1：使用 cProfile 進行函式級分析

cProfile 是 Python 標準庫的效能分析工具，可以測量每個函式的呼叫次數和執行時間。

 1import cProfile
 2import pstats
 3from io import StringIO
 4from pathlib import Path
 5
 6def profile_link_checker():
 7    """Profile the markdown link checker with cProfile"""
 8
 9    # Create test content with many links
10    test_content = generate_test_content(num_links=1000)
11
12    checker = MarkdownLinkChecker()
13
14    # Create profiler
15    profiler = cProfile.Profile()
16
17    # Run profiling
18    profiler.enable()
19    for _ in range(100):  # Run multiple times for better statistics
20        checker.parse_markdown_links(test_content)
21    profiler.disable()
22
23    # Analyze results
24    stream = StringIO()
25    stats = pstats.Stats(profiler, stream=stream)
26    stats.sort_stats('cumulative')  # Sort by cumulative time
27    stats.print_stats(20)  # Show top 20 functions
28
29    print(stream.getvalue())
30
31    return stats
32
33def generate_test_content(num_links: int) -> str:
34    """Generate test Markdown content with specified number of links"""
35    lines = ["# Test Document\n"]
36
37    for i in range(num_links):
38        if i % 3 == 0:
39            # Inline link
40            lines.append(f"Check out [Link {i}](https://example.com/{i})\n")
41        elif i % 3 == 1:
42            # Reference-style link
43            lines.append(f"See [Reference {i}][ref{i}]\n")
44        else:
45            # Plain text with potential regex traps
46            lines.append(f"This is paragraph {i} with some [text] that looks like links.\n")
47
48    # Add reference definitions at the end
49    for i in range(num_links):
50        if i % 3 == 1:
51            lines.append(f"[ref{i}]: https://example.com/ref/{i}\n")
52
53    return "".join(lines)
54
55if __name__ == "__main__":
56    profile_link_checker()

執行方式：

1# Direct execution
2python profile_link_checker.py
3
4# Using cProfile from command line
5python -m cProfile -s cumulative markdown_link_checker.py --dir ./docs/

步驟 2：使用 pstats 分析結果

pstats 模組提供更細緻的結果分析功能：

 1import cProfile
 2import pstats
 3from pstats import SortKey
 4
 5def detailed_analysis():
 6    """Perform detailed analysis with pstats"""
 7
 8    # Profile the code
 9    profiler = cProfile.Profile()
10    profiler.enable()
11
12    # Run the target function
13    checker = MarkdownLinkChecker()
14    content = generate_test_content(500)
15    for _ in range(50):
16        checker.parse_markdown_links(content)
17
18    profiler.disable()
19
20    # Create Stats object
21    stats = pstats.Stats(profiler)
22
23    # Different sorting options
24    print("=" * 70)
25    print("Top 10 by CUMULATIVE time (including sub-calls)")
26    print("=" * 70)
27    stats.sort_stats(SortKey.CUMULATIVE).print_stats(10)
28
29    print("\n" + "=" * 70)
30    print("Top 10 by TOTAL time (excluding sub-calls)")
31    print("=" * 70)
32    stats.sort_stats(SortKey.TIME).print_stats(10)
33
34    print("\n" + "=" * 70)
35    print("Top 10 by CALL count")
36    print("=" * 70)
37    stats.sort_stats(SortKey.CALLS).print_stats(10)
38
39    # Filter by function name
40    print("\n" + "=" * 70)
41    print("Functions containing 'parse' or 'match'")
42    print("=" * 70)
43    stats.sort_stats(SortKey.TIME).print_stats('parse|match', 10)
44
45    # Show callers of a specific function
46    print("\n" + "=" * 70)
47    print("Who calls 'finditer'?")
48    print("=" * 70)
49    stats.print_callers('finditer')
50
51    # Save stats to file for later analysis
52    stats.dump_stats('link_checker.prof')
53
54    return stats
55
56def load_and_compare_profiles():
57    """Load and compare saved profile data"""
58    try:
59        # Load saved profile
60        stats = pstats.Stats('link_checker.prof')
61
62        # Add another profile for comparison
63        stats.add('link_checker_optimized.prof')
64
65        # Print combined stats
66        stats.sort_stats(SortKey.CUMULATIVE).print_stats(20)
67    except FileNotFoundError:
68        print("Profile files not found. Run detailed_analysis() first.")

步驟 3：使用 line_profiler 進行行級分析

line_profiler 可以分析每一行程式碼的執行時間，適合精確定位瓶頸。

首先安裝 line_profiler：

1pip install line_profiler

使用裝飾器標記要分析的函式：

  1# profile_lines.py
  2from line_profiler import profile
  3
  4@profile
  5def parse_markdown_links_profiled(self, content: str):
  6    """
  7    Line-by-line profiled version of parse_markdown_links
  8    """
  9    links = []
 10    lines = content.split('\n')
 11
 12    # Collect reference definitions
 13    reference_defs = {}
 14    for match in self.REFERENCE_DEF_PATTERN.finditer(content):
 15        ref_name = match.group(1).lower()
 16        ref_target = match.group(2).strip()
 17        reference_defs[ref_name] = ref_target
 18
 19    in_code_block = False
 20
 21    for line_num, line in enumerate(lines, start=1):
 22        if line.strip().startswith("```"):
 23            in_code_block = not in_code_block
 24            continue
 25
 26        if in_code_block:
 27            continue
 28
 29        # These regex operations might be slow
 30        for match in self.INLINE_LINK_PATTERN.finditer(line):
 31            links.append({
 32                "text": match.group(1),
 33                "target": match.group(2),
 34                "line": line_num
 35            })
 36
 37        for match in self.REFERENCE_USE_PATTERN.finditer(line):
 38            ref_name = match.group(2).lower()
 39            if ref_name in reference_defs:
 40                links.append({
 41                    "text": match.group(1),
 42                    "target": reference_defs[ref_name],
 43                    "line": line_num
 44                })
 45
 46    return links
 47
 48# Alternative: Manual timing for specific sections
 49import time
 50
 51def parse_with_timing(self, content: str):
 52    """Manual timing for detailed analysis"""
 53    timings = {}
 54
 55    # Time: split lines
 56    start = time.perf_counter()
 57    lines = content.split('\n')
 58    timings['split_lines'] = time.perf_counter() - start
 59
 60    # Time: parse reference definitions
 61    start = time.perf_counter()
 62    reference_defs = {}
 63    for match in self.REFERENCE_DEF_PATTERN.finditer(content):
 64        ref_name = match.group(1).lower()
 65        ref_target = match.group(2).strip()
 66        reference_defs[ref_name] = ref_target
 67    timings['parse_refs'] = time.perf_counter() - start
 68
 69    # Time: main parsing loop
 70    start = time.perf_counter()
 71    links = []
 72    in_code_block = False
 73
 74    for line_num, line in enumerate(lines, start=1):
 75        if line.strip().startswith("```"):
 76            in_code_block = not in_code_block
 77            continue
 78
 79        if in_code_block:
 80            continue
 81
 82        for match in self.INLINE_LINK_PATTERN.finditer(line):
 83            links.append({
 84                "text": match.group(1),
 85                "target": match.group(2),
 86                "line": line_num
 87            })
 88
 89        for match in self.REFERENCE_USE_PATTERN.finditer(line):
 90            ref_name = match.group(2).lower()
 91            if ref_name in reference_defs:
 92                links.append({
 93                    "text": match.group(1),
 94                    "target": reference_defs[ref_name],
 95                    "line": line_num
 96                })
 97
 98    timings['main_loop'] = time.perf_counter() - start
 99
100    return links, timings

執行 line_profiler：

1# Run with kernprof
2kernprof -l -v profile_lines.py
3
4# Or use the newer approach
5python -m line_profiler profile_lines.py

步驟 4：分析正則表達式效能

正則表達式是常見的效能瓶頸，需要特別分析：

  1import re
  2import time
  3from typing import Callable
  4
  5def benchmark_regex_patterns():
  6    """Compare different regex pattern implementations"""
  7
  8    # Test content with various edge cases
  9    test_lines = [
 10        "Check out [Link](https://example.com)",
 11        "See [Text with [brackets]](/python-advanced/04-cpython-internals/case-studies/profiling/url)",
 12        "Multiple [link1](/python-advanced/04-cpython-internals/case-studies/profiling/url1) and [link2](/python-advanced/04-cpython-internals/case-studies/profiling/url2)",
 13        "No links here, just plain text",
 14        "Tricky [text](/python-advanced/04-cpython-internals/case-studies/profiling/url) with more [text](/python-advanced/04-cpython-internals/case-studies/profiling/url) links",
 15        "![Image](/python-advanced/04-cpython-internals/case-studies/profiling/image.png) should be ignored",
 16        "[Link with spaces]( url with spaces )",
 17    ] * 1000
 18
 19    test_content = "\n".join(test_lines)
 20
 21    # Pattern 1: Original pattern
 22    pattern1 = re.compile(r'(?)
 23
 24    # Pattern 2: Possessive-like (using atomic group simulation)
 25    # Note: Python re doesn't support possessive quantifiers directly
 26    pattern2 = re.compile(r'(?)
 27
 28    # Pattern 3: More specific pattern
 29    pattern3 = re.compile(r'(?)
 30
 31    # Pattern 4: Non-capturing groups where possible
 32    pattern4 = re.compile(r'(?)
 33
 34    patterns = {
 35        "original": pattern1,
 36        "non_greedy": pattern2,
 37        "more_specific": pattern3,
 38        "optimized": pattern4,
 39    }
 40
 41    results = {}
 42
 43    for name, pattern in patterns.items():
 44        # Warmup
 45        for _ in range(10):
 46            list(pattern.finditer(test_content))
 47
 48        # Benchmark
 49        start = time.perf_counter()
 50        for _ in range(100):
 51            matches = list(pattern.finditer(test_content))
 52        elapsed = time.perf_counter() - start
 53
 54        results[name] = {
 55            "time": elapsed,
 56            "matches": len(matches),
 57        }
 58
 59    # Print comparison
 60    print("Regex Pattern Performance Comparison")
 61    print("=" * 60)
 62    print(f"{'Pattern':<20} {'Time (s)':<12} {'Matches':<10}")
 63    print("-" * 60)
 64
 65    baseline = results["original"]["time"]
 66    for name, data in results.items():
 67        speedup = baseline / data["time"]
 68        print(f"{name:<20} {data['time']:<12.4f} {data['matches']:<10} ({speedup:.2f}x)")
 69
 70    return results
 71
 72def analyze_regex_backtracking():
 73    """Analyze potential regex backtracking issues"""
 74    import re
 75
 76    # Patterns that might cause backtracking
 77    problematic_patterns = [
 78        (r'\[(.+)\]\((.+)\)', "Greedy .+ can backtrack"),
 79        (r'\[([^\]]+)\]\(([^)]+)\)', "Negated class - better"),
 80        (r'\[(.*?)\]\((.*?)\)', "Non-greedy - still may backtrack"),
 81    ]
 82
 83    # Pathological input that triggers backtracking
 84    pathological = "[" + "a" * 100 + "]"  # No closing bracket pattern
 85
 86    print("Regex Backtracking Analysis")
 87    print("=" * 60)
 88
 89    for pattern_str, description in problematic_patterns:
 90        pattern = re.compile(pattern_str)
 91
 92        start = time.perf_counter()
 93        try:
 94            # Set a timeout using signal (Unix only)
 95            result = pattern.search(pathological)
 96            elapsed = time.perf_counter() - start
 97            print(f"Pattern: {pattern_str}")
 98            print(f"  Description: {description}")
 99            print(f"  Time: {elapsed:.4f}s")
100            print(f"  Match: {result}")
101            print()
102        except Exception as e:
103            print(f"Pattern: {pattern_str} - Error: {e}")
104
105def compare_compile_vs_inline():
106    """Compare precompiled vs inline regex performance"""
107
108    test_content = generate_test_content(500)
109    iterations = 100
110
111    # Test 1: Precompiled pattern (recommended)
112    compiled_pattern = re.compile(r'(?)
113
114    start = time.perf_counter()
115    for _ in range(iterations):
116        list(compiled_pattern.finditer(test_content))
117    compiled_time = time.perf_counter() - start
118
119    # Test 2: Inline pattern (compiled each time by re module cache)
120    start = time.perf_counter()
121    for _ in range(iterations):
122        list(re.finditer(r'(?, test_content))
123    inline_time = time.perf_counter() - start
124
125    # Test 3: Pattern compiled inside loop (worst case)
126    start = time.perf_counter()
127    for _ in range(iterations):
128        pattern = re.compile(r'(?)
129        list(pattern.finditer(test_content))
130    loop_compile_time = time.perf_counter() - start
131
132    print("Compile Strategy Comparison")
133    print("=" * 60)
134    print(f"{'Strategy':<25} {'Time (s)':<12} {'Relative':<10}")
135    print("-" * 60)
136    print(f"{'Precompiled (class)':<25} {compiled_time:<12.4f} {'1.00x':<10}")
137    print(f"{'Inline (re cache)':<25} {inline_time:<12.4f} {inline_time/compiled_time:.2f}x")
138    print(f"{'Compile in loop':<25} {loop_compile_time:<12.4f} {loop_compile_time/compiled_time:.2f}x")

步驟 5：優化建議與驗證

根據分析結果，實作優化版本並驗證效果：

  1import re
  2from typing import List, Dict, Tuple
  3from dataclasses import dataclass
  4
  5@dataclass(slots=True)  # Python 3.10+ optimization
  6class LinkInfo:
  7    """Link information with memory-efficient storage"""
  8    text: str
  9    target: str
 10    line: int
 11
 12class OptimizedLinkChecker:
 13    """Optimized Markdown link checker based on profiling results"""
 14
 15    # Precompiled patterns at class level
 16    INLINE_LINK_PATTERN = re.compile(r'(?)
 17    REFERENCE_DEF_PATTERN = re.compile(r'^\s*\[([^\]]+)\]:\s*(.+)$', re.MULTILINE)
 18    REFERENCE_USE_PATTERN = re.compile(r'\[([^\]]+)\]\[([^\]]+)\]')
 19    CODE_BLOCK_PATTERN = re.compile(r'^```')
 20
 21    def parse_markdown_links_optimized(self, content: str) -> List[LinkInfo]:
 22        """
 23        Optimized link parsing with reduced memory allocation
 24
 25        Optimizations:
 26        1. Use dataclass with slots for link storage
 27        2. Single pass where possible
 28        3. Avoid repeated string operations
 29        4. Use local variables for frequently accessed attributes
 30        """
 31        links = []
 32
 33        # Cache pattern references for faster access
 34        inline_pattern = self.INLINE_LINK_PATTERN
 35        ref_use_pattern = self.REFERENCE_USE_PATTERN
 36        ref_def_pattern = self.REFERENCE_DEF_PATTERN
 37
 38        # Collect reference definitions first (single pass over content)
 39        reference_defs = {
 40            match.group(1).lower(): match.group(2).strip()
 41            for match in ref_def_pattern.finditer(content)
 42        }
 43
 44        # Process line by line
 45        in_code_block = False
 46        line_start = 0
 47
 48        for line_num, line_end in enumerate(
 49            self._find_line_positions(content), start=1
 50        ):
 51            line = content[line_start:line_end]
 52
 53            # Fast code block check
 54            if line.lstrip().startswith("```"):
 55                in_code_block = not in_code_block
 56                line_start = line_end + 1
 57                continue
 58
 59            if not in_code_block:
 60                # Parse inline links
 61                for match in inline_pattern.finditer(line):
 62                    links.append(LinkInfo(
 63                        text=match.group(1),
 64                        target=match.group(2),
 65                        line=line_num
 66                    ))
 67
 68                # Parse reference links
 69                for match in ref_use_pattern.finditer(line):
 70                    ref_name = match.group(2).lower()
 71                    target = reference_defs.get(ref_name)
 72                    if target:
 73                        links.append(LinkInfo(
 74                            text=match.group(1),
 75                            target=target,
 76                            line=line_num
 77                        ))
 78
 79            line_start = line_end + 1
 80
 81        return links
 82
 83    def _find_line_positions(self, content: str):
 84        """Generator that yields line end positions"""
 85        pos = 0
 86        while True:
 87            newline = content.find('\n', pos)
 88            if newline == -1:
 89                yield len(content)
 90                break
 91            yield newline
 92            pos = newline + 1
 93
 94def verify_optimization():
 95    """Verify that optimizations maintain correctness and improve performance"""
 96    import time
 97
 98    # Generate test content
 99    test_content = generate_test_content(1000)
100
101    # Original implementation
102    original = MarkdownLinkChecker()
103
104    # Optimized implementation
105    optimized = OptimizedLinkChecker()
106
107    # Verify correctness
108    original_links = original.parse_markdown_links(test_content)
109    optimized_links = optimized.parse_markdown_links_optimized(test_content)
110
111    # Compare results
112    assert len(original_links) == len(optimized_links), \
113        f"Link count mismatch: {len(original_links)} vs {len(optimized_links)}"
114
115    for orig, opt in zip(original_links, optimized_links):
116        assert orig["text"] == opt.text, f"Text mismatch: {orig['text']} vs {opt.text}"
117        assert orig["target"] == opt.target, f"Target mismatch"
118        assert orig["line"] == opt.line, f"Line mismatch"
119
120    print("Correctness verified!")
121
122    # Benchmark
123    iterations = 100
124
125    # Original
126    start = time.perf_counter()
127    for _ in range(iterations):
128        original.parse_markdown_links(test_content)
129    original_time = time.perf_counter() - start
130
131    # Optimized
132    start = time.perf_counter()
133    for _ in range(iterations):
134        optimized.parse_markdown_links_optimized(test_content)
135    optimized_time = time.perf_counter() - start
136
137    print(f"\nPerformance Comparison ({iterations} iterations)")
138    print("=" * 50)
139    print(f"Original:  {original_time:.4f}s")
140    print(f"Optimized: {optimized_time:.4f}s")
141    print(f"Speedup:   {original_time / optimized_time:.2f}x")

完整程式碼

以下是整合所有分析功能的完整腳本：

  1#!/usr/bin/env python3
  2"""
  3Performance Profiling Script for Markdown Link Checker
  4
  5This script demonstrates various profiling techniques:
  61. cProfile for function-level analysis
  72. pstats for detailed statistics
  83. line_profiler for line-by-line analysis
  94. Regex pattern benchmarking
 105. Optimization verification
 11
 12Usage:
 13    python profiling_demo.py [--full|--quick|--regex|--verify]
 14"""
 15
 16import argparse
 17import cProfile
 18import pstats
 19import re
 20import sys
 21import time
 22from dataclasses import dataclass
 23from io import StringIO
 24from pathlib import Path
 25from pstats import SortKey
 26from typing import Dict, List, Optional
 27
 28# ===== Original Implementation =====
 29
 30class MarkdownLinkChecker:
 31    """Original Markdown link checker for comparison"""
 32
 33    INLINE_LINK_PATTERN = re.compile(r'(?)
 34    REFERENCE_DEF_PATTERN = re.compile(r'^\s*\[([^\]]+)\]:\s*(.+)$', re.MULTILINE)
 35    REFERENCE_USE_PATTERN = re.compile(r'\[([^\]]+)\]\[([^\]]+)\]')
 36
 37    def parse_markdown_links(self, content: str) -> List[Dict]:
 38        links = []
 39        lines = content.split('\n')
 40
 41        reference_defs = {}
 42        for match in self.REFERENCE_DEF_PATTERN.finditer(content):
 43            ref_name = match.group(1).lower()
 44            ref_target = match.group(2).strip()
 45            reference_defs[ref_name] = ref_target
 46
 47        in_code_block = False
 48
 49        for line_num, line in enumerate(lines, start=1):
 50            if line.strip().startswith("```"):
 51                in_code_block = not in_code_block
 52                continue
 53
 54            if in_code_block:
 55                continue
 56
 57            for match in self.INLINE_LINK_PATTERN.finditer(line):
 58                links.append({
 59                    "text": match.group(1),
 60                    "target": match.group(2),
 61                    "line": line_num
 62                })
 63
 64            for match in self.REFERENCE_USE_PATTERN.finditer(line):
 65                ref_name = match.group(2).lower()
 66                if ref_name in reference_defs:
 67                    links.append({
 68                        "text": match.group(1),
 69                        "target": reference_defs[ref_name],
 70                        "line": line_num
 71                    })
 72
 73        return links
 74
 75# ===== Test Data Generation =====
 76
 77def generate_test_content(num_links: int) -> str:
 78    """Generate test Markdown content"""
 79    lines = ["# Test Document\n\n"]
 80
 81    for i in range(num_links):
 82        if i % 3 == 0:
 83            lines.append(f"Check out [Link {i}](https://example.com/{i})\n")
 84        elif i % 3 == 1:
 85            lines.append(f"See [Reference {i}][ref{i}]\n")
 86        else:
 87            lines.append(f"Paragraph {i} with some text.\n")
 88
 89    lines.append("\n")
 90    for i in range(num_links):
 91        if i % 3 == 1:
 92            lines.append(f"[ref{i}]: https://example.com/ref/{i}\n")
 93
 94    return "".join(lines)
 95
 96# ===== Profiling Functions =====
 97
 98def run_cprofile_analysis(iterations: int = 50, num_links: int = 500):
 99    """Run cProfile analysis"""
100    print("Running cProfile Analysis")
101    print("=" * 70)
102
103    content = generate_test_content(num_links)
104    checker = MarkdownLinkChecker()
105
106    profiler = cProfile.Profile()
107    profiler.enable()
108
109    for _ in range(iterations):
110        checker.parse_markdown_links(content)
111
112    profiler.disable()
113
114    stats = pstats.Stats(profiler)
115    stats.sort_stats(SortKey.CUMULATIVE)
116
117    print(f"\nTop 15 functions by cumulative time:")
118    print("-" * 70)
119    stats.print_stats(15)
120
121    return stats
122
123def run_regex_benchmark():
124    """Benchmark different regex patterns"""
125    print("\nRegex Pattern Benchmark")
126    print("=" * 70)
127
128    content = generate_test_content(500)
129
130    patterns = {
131        "Original": re.compile(r'(?),
132        "Non-greedy inner": re.compile(r'(?),
133        "Anchored": re.compile(r'(?),
134    }
135
136    print(f"{'Pattern':<25} {'Time (ms)':<12} {'Matches':<10}")
137    print("-" * 50)
138
139    for name, pattern in patterns.items():
140        # Warmup
141        list(pattern.finditer(content))
142
143        # Benchmark
144        start = time.perf_counter()
145        for _ in range(100):
146            matches = list(pattern.finditer(content))
147        elapsed = (time.perf_counter() - start) * 1000
148
149        print(f"{name:<25} {elapsed:<12.2f} {len(matches):<10}")
150
151def run_compile_comparison():
152    """Compare precompiled vs inline regex"""
153    print("\nCompile Strategy Comparison")
154    print("=" * 70)
155
156    content = generate_test_content(500)
157    iterations = 100
158    pattern_str = r'(?
159
160    # Precompiled
161    compiled = re.compile(pattern_str)
162    start = time.perf_counter()
163    for _ in range(iterations):
164        list(compiled.finditer(content))
165    compiled_time = time.perf_counter() - start
166
167    # Inline (uses re module cache)
168    start = time.perf_counter()
169    for _ in range(iterations):
170        list(re.finditer(pattern_str, content))
171    inline_time = time.perf_counter() - start
172
173    print(f"{'Strategy':<25} {'Time (ms)':<12} {'Relative':<10}")
174    print("-" * 50)
175    print(f"{'Precompiled':<25} {compiled_time*1000:<12.2f} {'1.00x':<10}")
176    print(f"{'Inline (cached)':<25} {inline_time*1000:<12.2f} {inline_time/compiled_time:.2f}x")
177
178def main():
179    parser = argparse.ArgumentParser(description="Profiling Demo")
180    parser.add_argument('--full', action='store_true', help='Run full analysis')
181    parser.add_argument('--quick', action='store_true', help='Run quick analysis')
182    parser.add_argument('--regex', action='store_true', help='Run regex benchmark')
183    args = parser.parse_args()
184
185    if args.regex:
186        run_regex_benchmark()
187        run_compile_comparison()
188    elif args.quick:
189        run_cprofile_analysis(iterations=10, num_links=100)
190    else:
191        run_cprofile_analysis()
192        run_regex_benchmark()
193        run_compile_comparison()
194
195if __name__ == "__main__":
196    main()

分析結果範例

執行 cProfile 分析後的典型輸出：

 1Running cProfile Analysis
 2======================================================================
 3
 4Top 15 functions by cumulative time:
 5----------------------------------------------------------------------
 6         125003 function calls in 0.892 seconds
 7
 8   Ordered by: cumulative time
 9
10   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
11       50    0.012    0.000    0.892    0.018 checker.py:45(parse_markdown_links)
12    25000    0.234    0.000    0.567    0.000 {method 'finditer' of 're.Pattern'}
13       50    0.089    0.002    0.089    0.002 {method 'split' of 'str' objects}
14    50000    0.156    0.000    0.156    0.000 {method 'append' of 'list' objects}
15    25000    0.098    0.000    0.098    0.000 {method 'group' of 're.Match'}
16    12500    0.045    0.000    0.045    0.000 {method 'lower' of 'str' objects}
17    12500    0.034    0.000    0.034    0.000 {method 'strip' of 'str' objects}
18    25000    0.067    0.000    0.067    0.000 {method 'startswith' of 'str' objects}
19       50    0.023    0.000    0.023    0.000 {built-in method builtins.enumerate}

從上述結果可以觀察到：

finditer 佔用最多時間：正則表達式匹配是主要瓶頸
split 操作耗時明顯：每次解析都建立新的行列表
append 呼叫頻繁：大量的字典建立和列表操作

line_profiler 的典型輸出：

 1Timer unit: 1e-06 s
 2
 3Total time: 0.456 s
 4File: checker.py
 5Function: parse_markdown_links at line 45
 6
 7Line #      Hits         Time  Per Hit   % Time  Line Contents
 8==============================================================
 9    45                                           def parse_markdown_links(self, content):
10    46        50       1234.0     24.7      0.3      links = []
11    47        50      89012.0   1780.2     19.5      lines = content.split('\n')
12    48
13    49        50         56.0      1.1      0.0      reference_defs = {}
14    50      2500      45678.0     18.3     10.0      for match in self.REFERENCE_DEF_PATTERN.finditer(content):
15    51      2450       3456.0      1.4      0.8          ref_name = match.group(1).lower()
16    52      2450       2345.0      1.0      0.5          ref_target = match.group(2).strip()
17    53      2450       1234.0      0.5      0.3          reference_defs[ref_name] = ref_target
18    54
19    55        50         34.0      0.7      0.0      in_code_block = False
20    56
21    57     25050      12345.0      0.5      2.7      for line_num, line in enumerate(lines, start=1):
22    58     25000      34567.0      1.4      7.6          if line.strip().startswith("```"):
23    59                                                       in_code_block = not in_code_block
24    60                                                       continue
25    61
26    62     25000       5678.0      0.2      1.2          if in_code_block:
27    63                                                       continue
28    64
29    65     25000     156789.0      6.3     34.4          for match in self.INLINE_LINK_PATTERN.finditer(line):
30    66      8350      23456.0      2.8      5.1              links.append({...})
31    67
32    68     25000      78901.0      3.2     17.3          for match in self.REFERENCE_USE_PATTERN.finditer(line):
33    69      2450       1234.0      0.5      0.3              ref_name = match.group(2).lower()
34    70      2450        567.0      0.2      0.1              if ref_name in reference_defs:
35    71      2450       1234.0      0.5      0.3                  links.append({...})
36    72
37    73        50         23.0      0.5      0.0      return links

關鍵發現：

第 65 行（行內連結匹配）佔 34.4%：這是最大的瓶頸
第 47 行（split）佔 19.5%：字串分割是第二大消耗
第 68 行（引用連結匹配）佔 17.3%：也是重要的優化目標

設計權衡

面向	不分析	使用 cProfile	使用 line_profiler
開發時間	少	中等	較多
分析粒度	無	函式級	行級
效能開銷	無	低	較高
適用場景	簡單程式	一般優化	精確定位
學習成本	無	低	中等
結果準確度	-	高	非常高

什麼時候該做效能分析？

適合分析：

程式明顯變慢
處理大量資料
發布前的效能驗證
正則表達式複雜時
有迴圈處理大量項目

不建議過早優化：

功能還在開發中
使用頻率很低
效能已經足夠
可讀性更重要時

練習

基礎練習：用 cProfile 分析排序函式

 1"""
 2Exercise 1: Profile different sorting approaches
 3"""
 4import cProfile
 5import pstats
 6import random
 7from pstats import SortKey
 8
 9def bubble_sort(arr):
10    """Bubble sort implementation"""
11    n = len(arr)
12    arr = arr.copy()
13    for i in range(n):
14        for j in range(0, n - i - 1):
15            if arr[j] > arr[j + 1]:
16                arr[j], arr[j + 1] = arr[j + 1], arr[j]
17    return arr
18
19def quick_sort(arr):
20    """Quick sort implementation"""
21    if len(arr) <= 1:
22        return arr
23    pivot = arr[len(arr) // 2]
24    left = [x for x in arr if x < pivot]
25    middle = [x for x in arr if x == pivot]
26    right = [x for x in arr if x > pivot]
27    return quick_sort(left) + middle + quick_sort(right)
28
29def profile_sorting():
30    """Profile and compare sorting algorithms"""
31    data = [random.randint(0, 10000) for _ in range(1000)]
32
33    # Profile bubble sort
34    profiler = cProfile.Profile()
35    profiler.enable()
36    bubble_sort(data)
37    profiler.disable()
38
39    print("Bubble Sort Profile:")
40    stats = pstats.Stats(profiler)
41    stats.sort_stats(SortKey.TIME).print_stats(5)
42
43    # Profile quick sort
44    profiler = cProfile.Profile()
45    profiler.enable()
46    quick_sort(data)
47    profiler.disable()
48
49    print("\nQuick Sort Profile:")
50    stats = pstats.Stats(profiler)
51    stats.sort_stats(SortKey.TIME).print_stats(5)
52
53if __name__ == "__main__":
54    profile_sorting()

進階練習：用 line_profiler 找出熱點程式碼

 1"""
 2Exercise 2: Use line_profiler to find hotspots
 3
 4Instructions:
 51. Install line_profiler: pip install line_profiler
 62. Add @profile decorator to functions you want to analyze
 73. Run: kernprof -l -v exercise2.py
 8"""
 9from line_profiler import profile
10
11@profile
12def find_primes(n):
13    """Find all prime numbers up to n"""
14    primes = []
15    for num in range(2, n + 1):
16        is_prime = True
17        for i in range(2, int(num ** 0.5) + 1):
18            if num % i == 0:
19                is_prime = False
20                break
21        if is_prime:
22            primes.append(num)
23    return primes
24
25@profile
26def find_primes_sieve(n):
27    """Find primes using Sieve of Eratosthenes"""
28    sieve = [True] * (n + 1)
29    sieve[0] = sieve[1] = False
30
31    for i in range(2, int(n ** 0.5) + 1):
32        if sieve[i]:
33            for j in range(i * i, n + 1, i):
34                sieve[j] = False
35
36    return [i for i, is_prime in enumerate(sieve) if is_prime]
37
38if __name__ == "__main__":
39    print("Finding primes up to 10000...")
40    primes1 = find_primes(10000)
41    primes2 = find_primes_sieve(10000)
42    print(f"Found {len(primes1)} primes (basic)")
43    print(f"Found {len(primes2)} primes (sieve)")

挑戰題：比較不同正則表達式寫法的效能差異

 1"""
 2Exercise 3: Compare regex pattern performance
 3
 4Task: Parse email addresses from text using different patterns
 5and measure their performance.
 6"""
 7import re
 8import time
 9
10def benchmark_email_patterns():
11    """Compare different email regex patterns"""
12
13    # Test content with mixed valid and invalid emails
14    test_text = """
15    Contact us at support@example.com or sales@company.org
16    Invalid: not.an.email, @missing.com, missing@
17    Valid: user.name+tag@domain.co.uk, test123@sub.domain.com
18    Edge cases: "quoted"@domain.com, user@[192.168.1.1]
19    """ * 1000
20
21    patterns = {
22        # Simple pattern (may miss some valid emails)
23        "simple": re.compile(r'\b[\w.-]+@[\w.-]+\.\w+\b'),
24
25        # More comprehensive pattern
26        "comprehensive": re.compile(
27            r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
28        ),
29
30        # RFC 5322 inspired (complex but thorough)
31        "rfc_inspired": re.compile(
32            r'(?:[a-z0-9!#$%&\'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&\'*+/=?^_`{|}~-]+)*'
33            r'|"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]'
34            r'|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*")'
35            r'@(?:(?:[a-z0-9](/python-advanced/04-cpython-internals/case-studies/profiling/?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](/python-advanced/04-cpython-internals/case-studies/profiling/?:[a-z0-9-]*[a-z0-9])?'
36            r'|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}'
37            r'(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:'
38            r'(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]'
39            r'|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])',
40            re.IGNORECASE
41        ),
42    }
43
44    print("Email Pattern Performance Comparison")
45    print("=" * 60)
46    print(f"{'Pattern':<20} {'Time (ms)':<12} {'Matches':<10}")
47    print("-" * 60)
48
49    for name, pattern in patterns.items():
50        # Warmup
51        list(pattern.findall(test_text))
52
53        # Benchmark
54        start = time.perf_counter()
55        for _ in range(100):
56            matches = pattern.findall(test_text)
57        elapsed = (time.perf_counter() - start) * 1000
58
59        print(f"{name:<20} {elapsed:<12.2f} {len(matches):<10}")
60
61    # Your task: Add more patterns and analyze the results
62    # Consider: What trade-offs exist between accuracy and speed?
63
64if __name__ == "__main__":
65    benchmark_email_patterns()

延伸閱讀

下一章：記憶體優化

案例：記憶體優化

Wed, 21 Jan 2026 00:00:00 +0000

本案例基於 .claude/lib/config_loader.py 的實際程式碼，展示如何用 __slots__ 和 weakref 優化記憶體使用。

先備知識

問題背景

現有設計

config_loader.py 使用全域字典作為快取，這是一個常見的設計模式：

 1# Global cache variables
 2_agents_config_cache: Optional[dict] = None
 3_quality_rules_cache: Optional[dict] = None
 4
 5def load_agents_config() -> dict:
 6    """
 7    載入代理人配置
 8
 9    使用模組層級變數作為快取，避免重複讀取檔案。
10    """
11    global _agents_config_cache
12    if _agents_config_cache is None:
13        try:
14            _agents_config_cache = load_config("agents")
15        except FileNotFoundError:
16            _agents_config_cache = _get_default_agents_config()
17    return _agents_config_cache
18
19def clear_config_cache() -> None:
20    """清除配置快取（用於測試或配置熱更新）"""
21    global _agents_config_cache, _quality_rules_cache
22    _agents_config_cache = None
23    _quality_rules_cache = None

這種設計簡單直觀，但當系統需要快取更複雜的物件時，會遇到記憶體問題。

記憶體問題

當快取大量物件時：

Python 字典有額外開銷：每個字典需要維護 hash table、keys、values
物件的 __dict__ 佔用記憶體：每個實例都有自己的屬性字典
快取可能導致記憶體洩漏：強引用阻止物件被回收

讓我們用一個更複雜的快取場景來說明問題：

 1import sys
 2
 3class ConfigItem:
 4    """配置項目，模擬複雜的快取物件"""
 5    def __init__(self, key: str, value: str, metadata: dict):
 6        self.key = key
 7        self.value = value
 8        self.metadata = metadata
 9        self.access_count = 0
10        self.last_accessed = None
11
12# Create a config item and measure memory
13item = ConfigItem("database.host", "localhost", {"type": "string"})
14
15# Object size
16print(f"ConfigItem size: {sys.getsizeof(item)} bytes")
17# ConfigItem size: 48 bytes
18
19# But the real cost is in __dict__
20print(f"__dict__ size: {sys.getsizeof(item.__dict__)} bytes")
21# __dict__ size: 184 bytes

當快取數萬個這樣的物件時，記憶體開銷會非常可觀。

進階解決方案

優化目標

減少每個物件的記憶體佔用
避免快取導致的記憶體洩漏
保持 API 不變

實作步驟

步驟 1：使用 `slots` 減少物件大小

__slots__ 告訴 Python 這個類別只會有哪些屬性，讓直譯器可以用更緊湊的方式儲存資料：

 1import sys
 2
 3class ConfigItemWithoutSlots:
 4    """標準類別，使用 __dict__"""
 5    def __init__(self, key: str, value: str, metadata: dict):
 6        self.key = key
 7        self.value = value
 8        self.metadata = metadata
 9        self.access_count = 0
10        self.last_accessed = None
11
12class ConfigItemWithSlots:
13    """使用 __slots__ 優化記憶體"""
14    __slots__ = ['key', 'value', 'metadata', 'access_count', 'last_accessed']
15
16    def __init__(self, key: str, value: str, metadata: dict):
17        self.key = key
18        self.value = value
19        self.metadata = metadata
20        self.access_count = 0
21        self.last_accessed = None
22
23# Compare memory usage
24item_without = ConfigItemWithoutSlots("db.host", "localhost", {"type": "str"})
25item_with = ConfigItemWithSlots("db.host", "localhost", {"type": "str"})
26
27print(f"Without __slots__: {sys.getsizeof(item_without)} bytes")
28print(f"With __slots__:    {sys.getsizeof(item_with)} bytes")
29
30# The real difference is __dict__
31print(f"__dict__ overhead: {sys.getsizeof(item_without.__dict__)} bytes")
32# item_with has no __dict__!
33try:
34    item_with.__dict__
35except AttributeError as e:
36    print(f"No __dict__: {e}")

記憶體結構比較

 1沒有 __slots__:
 2┌──────────────────────────────────┐
 3│ PyObject header         (16 B)   │
 4│ __dict__ 指標            (8 B)   │
 5│ __weakref__ 指標         (8 B)   │
 6│                                  │
 7│ __dict__ (separate object):      │
 8│   - hash table          (64 B)   │
 9│   - keys array          (40 B)   │
10│   - values array        (40 B)   │
11│   - key strings        (~80 B)   │
12│                                  │
13│ Total: ~256 bytes per object     │
14└──────────────────────────────────┘
15
16有 __slots__:
17┌──────────────────────────────────┐
18│ PyObject header         (16 B)   │
19│ key slot                 (8 B)   │
20│ value slot               (8 B)   │
21│ metadata slot            (8 B)   │
22│ access_count slot        (8 B)   │
23│ last_accessed slot       (8 B)   │
24│                                  │
25│ Total: ~56 bytes per object      │
26└──────────────────────────────────┘

大量物件的記憶體節省

 1import sys
 2import tracemalloc
 3
 4def measure_memory(cls, count=10000):
 5    """Measure memory for creating multiple objects"""
 6    tracemalloc.start()
 7
 8    objects = [
 9        cls(f"key_{i}", f"value_{i}", {"index": i})
10        for i in range(count)
11    ]
12
13    current, peak = tracemalloc.get_traced_memory()
14    tracemalloc.stop()
15
16    return current, peak, objects
17
18# Measure both classes
19mem_without, peak_without, _ = measure_memory(ConfigItemWithoutSlots)
20mem_with, peak_with, _ = measure_memory(ConfigItemWithSlots)
21
22print(f"Without __slots__: {mem_without / 1024 / 1024:.2f} MB")
23print(f"With __slots__:    {mem_with / 1024 / 1024:.2f} MB")
24print(f"Savings:           {(mem_without - mem_with) / 1024 / 1024:.2f} MB")
25print(f"Ratio:             {mem_without / mem_with:.1f}x")
26
27# Typical output:
28# Without __slots__: 3.82 MB
29# With __slots__:    1.15 MB
30# Savings:           2.67 MB
31# Ratio:             3.3x

步驟 2：使用 weakref 避免強引用

weakref 讓我們可以引用物件，但不阻止它被垃圾回收：

 1import weakref
 2
 3class CacheableConfig:
 4    """可以被弱引用的配置物件"""
 5    __slots__ = ['key', 'value', '_data', '__weakref__']  # Note: __weakref__ slot
 6
 7    def __init__(self, key: str, value: str):
 8        self.key = key
 9        self.value = value
10        self._data = None
11
12    def __repr__(self):
13        return f"CacheableConfig({self.key!r}, {self.value!r})"
14
15# Create object and weak reference
16config = CacheableConfig("app.name", "MyApp")
17weak_ref = weakref.ref(config)
18
19print(f"Object exists: {weak_ref()}")
20# Object exists: CacheableConfig('app.name', 'MyApp')
21
22# Delete the strong reference
23del config
24
25print(f"After del: {weak_ref()}")
26# After del: None

使用 callback 追蹤物件回收

 1import weakref
 2
 3def on_finalize(ref):
 4    """Callback when object is garbage collected"""
 5    print(f"Object was garbage collected!")
 6
 7config = CacheableConfig("db.port", "5432")
 8weak_ref = weakref.ref(config, on_finalize)
 9
10print("Deleting object...")
11del config
12# Output: Object was garbage collected!

步驟 3：使用 WeakValueDictionary

WeakValueDictionary 是實作自動清理快取的利器：

 1import weakref
 2from typing import Callable, TypeVar, Generic
 3
 4T = TypeVar('T')
 5
 6class WeakCache(Generic[T]):
 7    """
 8    Auto-cleaning cache using weak references.
 9
10    Objects are automatically removed from cache when no strong
11    references exist outside the cache.
12    """
13
14    def __init__(self):
15        self._cache: weakref.WeakValueDictionary[str, T] = (
16            weakref.WeakValueDictionary()
17        )
18        self._hits = 0
19        self._misses = 0
20
21    def get(self, key: str, factory: Callable[[], T]) -> T:
22        """
23        Get item from cache, creating it if necessary.
24
25        Args:
26            key: Cache key
27            factory: Function to create value if not cached
28
29        Returns:
30            Cached or newly created value
31        """
32        value = self._cache.get(key)
33        if value is not None:
34            self._hits += 1
35            return value
36
37        self._misses += 1
38        value = factory()
39        self._cache[key] = value
40        return value
41
42    def __len__(self) -> int:
43        return len(self._cache)
44
45    def stats(self) -> dict:
46        """Return cache statistics"""
47        total = self._hits + self._misses
48        hit_rate = self._hits / total if total > 0 else 0
49        return {
50            "hits": self._hits,
51            "misses": self._misses,
52            "hit_rate": f"{hit_rate:.1%}",
53            "size": len(self._cache),
54        }
55
56# Demo: automatic cleanup
57cache = WeakCache[CacheableConfig]()
58
59# Create and cache object
60config1 = cache.get("app.name", lambda: CacheableConfig("app.name", "MyApp"))
61config2 = cache.get("app.name", lambda: CacheableConfig("app.name", "MyApp"))  # Cache hit
62
63print(f"Cache size: {len(cache)}")  # 1
64print(f"Same object: {config1 is config2}")  # True
65print(f"Stats: {cache.stats()}")  # hits=1, misses=1
66
67# Delete strong reference
68del config1
69del config2
70
71# Object is garbage collected, cache is auto-cleaned
72import gc
73gc.collect()
74
75print(f"Cache size after cleanup: {len(cache)}")  # 0

步驟 4：測量記憶體使用

使用 sys.getsizeof 和 tracemalloc 進行精確測量：

 1import sys
 2import tracemalloc
 3from pympler import asizeof  # pip install pympler
 4
 5def measure_object_size(obj, label="Object"):
 6    """Measure object size using different methods"""
 7
 8    # Basic size (doesn't include referenced objects)
 9    basic = sys.getsizeof(obj)
10
11    # Deep size (includes all referenced objects)
12    # Using pympler for accurate measurement
13    deep = asizeof.asizeof(obj)
14
15    print(f"{label}:")
16    print(f"  sys.getsizeof: {basic:,} bytes")
17    print(f"  pympler deep:  {deep:,} bytes")
18
19    return basic, deep
20
21# Compare different object types
22item_without = ConfigItemWithoutSlots("key", "value", {"a": 1, "b": 2})
23item_with = ConfigItemWithSlots("key", "value", {"a": 1, "b": 2})
24
25measure_object_size(item_without, "Without __slots__")
26measure_object_size(item_with, "With __slots__")
27
28# Using tracemalloc for allocation tracking
29def track_allocations():
30    """Track memory allocations during execution"""
31    tracemalloc.start()
32
33    # Simulate creating many cached objects
34    items = []
35    for i in range(1000):
36        items.append(ConfigItemWithSlots(
37            f"config.item.{i}",
38            f"value_{i}",
39            {"index": i, "active": True}
40        ))
41
42    # Get snapshot
43    snapshot = tracemalloc.take_snapshot()
44    top_stats = snapshot.statistics('lineno')
45
46    print("\nTop 5 memory allocations:")
47    for stat in top_stats[:5]:
48        print(f"  {stat}")
49
50    # Get traced memory
51    current, peak = tracemalloc.get_traced_memory()
52    print(f"\nCurrent memory: {current / 1024:.1f} KB")
53    print(f"Peak memory:    {peak / 1024:.1f} KB")
54
55    tracemalloc.stop()
56    return items
57
58track_allocations()

比較記憶體差異的完整腳本

 1import sys
 2import gc
 3import tracemalloc
 4from dataclasses import dataclass
 5
 6class StandardConfig:
 7    """Standard class with __dict__"""
 8    def __init__(self, key, value, metadata):
 9        self.key = key
10        self.value = value
11        self.metadata = metadata
12        self.hits = 0
13
14class SlottedConfig:
15    """Optimized with __slots__"""
16    __slots__ = ['key', 'value', 'metadata', 'hits']
17
18    def __init__(self, key, value, metadata):
19        self.key = key
20        self.value = value
21        self.metadata = metadata
22        self.hits = 0
23
24@dataclass
25class DataclassConfig:
26    """Using dataclass"""
27    key: str
28    value: str
29    metadata: dict
30    hits: int = 0
31
32@dataclass(slots=True)  # Python 3.10+
33class SlottedDataclass:
34    """Dataclass with __slots__"""
35    key: str
36    value: str
37    metadata: dict
38    hits: int = 0
39
40def benchmark_memory(cls, count=10000, label=""):
41    """Benchmark memory usage for a class"""
42    gc.collect()
43    tracemalloc.start()
44
45    objects = [
46        cls(f"key_{i}", f"value_{i}", {"index": i})
47        for i in range(count)
48    ]
49
50    current, peak = tracemalloc.get_traced_memory()
51    tracemalloc.stop()
52
53    per_object = current / count
54    print(f"{label or cls.__name__:25} | "
55          f"{current/1024:8.1f} KB | "
56          f"{per_object:6.1f} B/obj")
57
58    return objects  # Keep reference to prevent GC
59
60print(f"{'Class':25} | {'Total':>10} | {'Per Object':>10}")
61print("-" * 55)
62
63benchmark_memory(StandardConfig)
64benchmark_memory(SlottedConfig)
65benchmark_memory(DataclassConfig)
66benchmark_memory(SlottedDataclass)
67
68# Typical output:
69# Class                     |      Total |  Per Object
70# -------------------------------------------------------
71# StandardConfig            |   2578.5 KB |  263.6 B/obj
72# SlottedConfig             |    859.4 KB |   87.9 B/obj
73# DataclassConfig           |   2656.3 KB |  271.6 B/obj
74# SlottedDataclass          |    898.4 KB |   91.9 B/obj

完整程式碼

以下是整合所有優化技術的完整實作：

  1"""
  2Memory-optimized configuration cache system.
  3
  4This module demonstrates:
  5- Using __slots__ to reduce object memory footprint
  6- Using weakref for automatic cache cleanup
  7- Using tracemalloc for memory profiling
  8
  9Based on patterns from .claude/lib/config_loader.py
 10"""
 11
 12import weakref
 13import sys
 14import gc
 15import tracemalloc
 16from typing import Any, Callable, Generic, Optional, TypeVar
 17from pathlib import Path
 18from datetime import datetime
 19
 20T = TypeVar('T')
 21
 22class ConfigEntry:
 23    """
 24    Memory-optimized configuration entry.
 25
 26    Uses __slots__ to reduce memory footprint by ~3x compared
 27    to regular classes.
 28    """
 29    __slots__ = [
 30        'key', 'value', 'source', 'loaded_at',
 31        'access_count', '__weakref__'
 32    ]
 33
 34    def __init__(
 35        self,
 36        key: str,
 37        value: Any,
 38        source: Optional[str] = None
 39    ):
 40        self.key = key
 41        self.value = value
 42        self.source = source
 43        self.loaded_at = datetime.now()
 44        self.access_count = 0
 45
 46    def __repr__(self) -> str:
 47        return (
 48            f"ConfigEntry(key={self.key!r}, "
 49            f"value={self.value!r}, "
 50            f"accesses={self.access_count})"
 51        )
 52
 53    def touch(self) -> None:
 54        """Record an access to this entry"""
 55        self.access_count += 1
 56
 57class SmartConfigCache:
 58    """
 59    Smart configuration cache with automatic memory management.
 60
 61    Features:
 62    - Weak references for automatic cleanup
 63    - Memory usage tracking
 64    - Hit/miss statistics
 65    - Optional size limits
 66
 67    Example:
 68        cache = SmartConfigCache(max_size=1000)
 69
 70        # Get or create config
 71        config = cache.get_or_create(
 72            "database.host",
 73            lambda: ConfigEntry("database.host", "localhost", "env")
 74        )
 75
 76        # Check stats
 77        print(cache.stats())
 78    """
 79
 80    def __init__(self, max_size: Optional[int] = None):
 81        """
 82        Initialize the cache.
 83
 84        Args:
 85            max_size: Maximum number of entries. None for unlimited.
 86        """
 87        self._cache: weakref.WeakValueDictionary[str, ConfigEntry] = (
 88            weakref.WeakValueDictionary()
 89        )
 90        self._strong_refs: dict[str, ConfigEntry] = {}  # Keep important items
 91        self._max_size = max_size
 92        self._hits = 0
 93        self._misses = 0
 94        self._evictions = 0
 95
 96    def get(self, key: str) -> Optional[ConfigEntry]:
 97        """
 98        Get entry from cache.
 99
100        Args:
101            key: Configuration key
102
103        Returns:
104            ConfigEntry if found, None otherwise
105        """
106        # Check strong refs first
107        entry = self._strong_refs.get(key)
108        if entry is not None:
109            self._hits += 1
110            entry.touch()
111            return entry
112
113        # Then check weak refs
114        entry = self._cache.get(key)
115        if entry is not None:
116            self._hits += 1
117            entry.touch()
118            return entry
119
120        self._misses += 1
121        return None
122
123    def get_or_create(
124        self,
125        key: str,
126        factory: Callable[[], ConfigEntry],
127        keep_strong: bool = False
128    ) -> ConfigEntry:
129        """
130        Get existing entry or create new one.
131
132        Args:
133            key: Configuration key
134            factory: Function to create entry if not found
135            keep_strong: If True, keep a strong reference (won't auto-cleanup)
136
137        Returns:
138            Existing or newly created ConfigEntry
139        """
140        entry = self.get(key)
141        if entry is not None:
142            return entry
143
144        # Create new entry
145        entry = factory()
146        self._cache[key] = entry
147
148        if keep_strong:
149            self._enforce_size_limit()
150            self._strong_refs[key] = entry
151
152        return entry
153
154    def _enforce_size_limit(self) -> None:
155        """Evict old entries if cache is full"""
156        if self._max_size is None:
157            return
158
159        while len(self._strong_refs) >= self._max_size:
160            # Evict least accessed entry
161            if not self._strong_refs:
162                break
163
164            min_key = min(
165                self._strong_refs.keys(),
166                key=lambda k: self._strong_refs[k].access_count
167            )
168            del self._strong_refs[min_key]
169            self._evictions += 1
170
171    def pin(self, key: str) -> bool:
172        """
173        Pin an entry to prevent automatic cleanup.
174
175        Args:
176            key: Configuration key
177
178        Returns:
179            True if entry was pinned, False if not found
180        """
181        entry = self._cache.get(key)
182        if entry is None:
183            return False
184
185        self._enforce_size_limit()
186        self._strong_refs[key] = entry
187        return True
188
189    def unpin(self, key: str) -> bool:
190        """
191        Unpin an entry to allow automatic cleanup.
192
193        Args:
194            key: Configuration key
195
196        Returns:
197            True if entry was unpinned, False if not found
198        """
199        if key in self._strong_refs:
200            del self._strong_refs[key]
201            return True
202        return False
203
204    def clear(self) -> None:
205        """Clear all cached entries"""
206        self._cache.clear()
207        self._strong_refs.clear()
208
209    def stats(self) -> dict:
210        """
211        Get cache statistics.
212
213        Returns:
214            Dict with hits, misses, hit_rate, size, pinned, evictions
215        """
216        total = self._hits + self._misses
217        hit_rate = self._hits / total if total > 0 else 0.0
218
219        return {
220            "hits": self._hits,
221            "misses": self._misses,
222            "hit_rate": f"{hit_rate:.1%}",
223            "total_size": len(self._cache) + len(self._strong_refs),
224            "weak_refs": len(self._cache),
225            "pinned": len(self._strong_refs),
226            "evictions": self._evictions,
227        }
228
229    def memory_usage(self) -> dict:
230        """
231        Estimate memory usage of cached entries.
232
233        Returns:
234            Dict with entry_count, estimated_bytes, per_entry_bytes
235        """
236        all_entries = list(self._cache.values()) + list(self._strong_refs.values())
237
238        if not all_entries:
239            return {
240                "entry_count": 0,
241                "estimated_bytes": 0,
242                "per_entry_bytes": 0,
243            }
244
245        # Estimate based on first entry
246        sample = all_entries[0]
247        per_entry = sys.getsizeof(sample)
248
249        return {
250            "entry_count": len(all_entries),
251            "estimated_bytes": per_entry * len(all_entries),
252            "per_entry_bytes": per_entry,
253        }
254
255def demo_memory_optimization():
256    """Demonstrate memory optimization techniques"""
257
258    print("=" * 60)
259    print("Memory Optimization Demo")
260    print("=" * 60)
261
262    # Start memory tracking
263    tracemalloc.start()
264    gc.collect()
265    snapshot1 = tracemalloc.take_snapshot()
266
267    # Create cache and populate
268    cache = SmartConfigCache(max_size=100)
269
270    # Simulate loading many configurations
271    entries = []
272    for i in range(1000):
273        entry = cache.get_or_create(
274            f"config.item.{i}",
275            lambda i=i: ConfigEntry(
276                f"config.item.{i}",
277                f"value_{i}",
278                "demo"
279            ),
280            keep_strong=(i < 100)  # Pin first 100
281        )
282        entries.append(entry)
283
284    # Take snapshot after creation
285    gc.collect()
286    snapshot2 = tracemalloc.take_snapshot()
287
288    # Print stats
289    print("\nCache Statistics:")
290    for key, value in cache.stats().items():
291        print(f"  {key}: {value}")
292
293    print("\nMemory Usage:")
294    for key, value in cache.memory_usage().items():
295        print(f"  {key}: {value:,}")
296
297    # Show memory diff
298    diff = snapshot2.compare_to(snapshot1, 'lineno')
299    print("\nTop 5 Memory Allocations:")
300    for stat in diff[:5]:
301        print(f"  {stat}")
302
303    # Demo weak reference cleanup
304    print("\n" + "-" * 60)
305    print("Weak Reference Cleanup Demo")
306    print("-" * 60)
307
308    print(f"Before cleanup - Cache size: {cache.stats()['total_size']}")
309
310    # Delete external references to unpinned entries
311    del entries
312    gc.collect()
313
314    print(f"After cleanup  - Cache size: {cache.stats()['total_size']}")
315    print("(Only pinned entries remain)")
316
317    tracemalloc.stop()
318
319if __name__ == "__main__":
320    demo_memory_optimization()

使用範例

 1from memory_optimized_cache import SmartConfigCache, ConfigEntry
 2
 3# Initialize cache with size limit
 4cache = SmartConfigCache(max_size=1000)
 5
 6# Load configuration
 7def load_database_config():
 8    """Factory function to load database config"""
 9    return ConfigEntry(
10        key="database",
11        value={
12            "host": "localhost",
13            "port": 5432,
14            "name": "myapp"
15        },
16        source="config.yaml"
17    )
18
19# Get or create (with strong reference for important config)
20db_config = cache.get_or_create(
21    "database",
22    load_database_config,
23    keep_strong=True  # Keep in memory
24)
25
26print(f"Database host: {db_config.value['host']}")
27
28# Temporary config (will be auto-cleaned when not referenced)
29temp_config = cache.get_or_create(
30    "temp.setting",
31    lambda: ConfigEntry("temp.setting", "temporary", "runtime")
32)
33
34# Check statistics
35print(cache.stats())
36# {'hits': 0, 'misses': 2, 'hit_rate': '0.0%',
37#  'total_size': 2, 'weak_refs': 1, 'pinned': 1, 'evictions': 0}
38
39# Memory usage
40print(cache.memory_usage())
41# {'entry_count': 2, 'estimated_bytes': 112, 'per_entry_bytes': 56}

設計權衡

`slots` vs 標準類別

面向	標準類別	`__slots__`
記憶體佔用	較多（有 `__dict__`）	較少（節省 ~60-70%）
動態屬性	支援 `obj.new_attr = x`	不支援（除非加 `__dict__`）
繼承	簡單	子類別需要自己的 `__slots__`
弱引用	預設支援	需要加入 `__weakref__` slot
Pickle	直接支援	需要 `__getstate__`/`__setstate__`
多重繼承	正常運作	多個父類別不能都有非空 `__slots__`

強引用 vs 弱引用快取

面向	強引用快取	弱引用快取
記憶體管理	需要手動清理	自動清理
資料保證	資料一定存在	資料可能被回收
適用場景	關鍵配置	暫時性資料
實作複雜度	簡單	稍微複雜

何時使用哪種技術？

 1決策樹：
 2
 3物件數量多嗎？
 4├── 是 → 考慮 __slots__
 5│   └── 需要動態屬性嗎？
 6│       ├── 是 → __slots__ = [..., '__dict__']
 7│       └── 否 → __slots__ = [...]
 8└── 否 → 標準類別即可
 9
10快取可能無限增長嗎？
11├── 是 → 使用 WeakValueDictionary 或 LRU
12└── 否 → 普通字典即可
13
14資料可以被回收嗎？
15├── 是 → weakref
16└── 否 → 強引用

什麼時候該用這個技術？

適合使用

建立大量小物件：如資料點、事件、配置項目
記憶體使用是瓶頸：經過 profiling 確認
快取可能無限增長：如用戶 session、請求資料
長時間運行的服務：如 web server、daemon

不建議使用

物件數量很少：優化效果不明顯
需要動態新增屬性：__slots__ 會限制彈性
過早優化：先確認是否真的有問題
程式碼可讀性優先：標準類別更直觀

優化決策流程

 1# Step 1: Profile first!
 2# Don't optimize until you know where the problem is
 3
 4import tracemalloc
 5
 6tracemalloc.start()
 7# ... run your code ...
 8snapshot = tracemalloc.take_snapshot()
 9top_stats = snapshot.statistics('lineno')
10
11for stat in top_stats[:10]:
12    print(stat)
13
14# Step 2: If memory is the issue, identify the class
15# Look for classes with many instances
16
17import gc
18from collections import Counter
19
20counter = Counter(type(obj).__name__ for obj in gc.get_objects())
21print(counter.most_common(10))
22
23# Step 3: Only then apply __slots__ to hot classes

練習

基礎練習：比較有無 `slots` 的記憶體差異

撰寫一個腳本，比較以下三種類別建立 100,000 個實例的記憶體使用：

 1# Exercise: Complete this benchmark script
 2
 3import sys
 4import tracemalloc
 5from dataclasses import dataclass
 6
 7# 1. Standard class
 8class PointStandard:
 9    def __init__(self, x, y, z):
10        self.x = x
11        self.y = y
12        self.z = z
13
14# 2. Class with __slots__
15class PointSlots:
16    __slots__ = ['x', 'y', 'z']
17    def __init__(self, x, y, z):
18        self.x = x
19        self.y = y
20        self.z = z
21
22# 3. Named tuple
23from collections import namedtuple
24PointNamed = namedtuple('PointNamed', ['x', 'y', 'z'])
25
26# TODO: Write benchmark function
27def benchmark(cls, count=100000):
28    """Measure memory for creating `count` instances"""
29    pass  # Implement this
30
31# TODO: Compare results
32# Expected: PointSlots uses ~3x less memory than PointStandard

進階練習：實作 weakref 快取

建立一個 ImageCache 類別，具有以下功能：

 1# Exercise: Implement ImageCache
 2
 3class ImageCache:
 4    """
 5    Cache for image data with automatic cleanup.
 6
 7    Requirements:
 8    - Use WeakValueDictionary for auto-cleanup
 9    - Track hit/miss statistics
10    - Support maximum size limit
11    - Provide memory usage estimation
12
13    Example usage:
14        cache = ImageCache(max_size=100)
15
16        img = cache.get_or_load("photo.jpg", lambda: load_image("photo.jpg"))
17
18        print(cache.stats())
19        # {'hits': 0, 'misses': 1, 'size': 1}
20    """
21
22    def __init__(self, max_size=None):
23        # TODO: Initialize cache
24        pass
25
26    def get_or_load(self, key, loader):
27        # TODO: Implement get or load logic
28        pass
29
30    def stats(self):
31        # TODO: Return cache statistics
32        pass

挑戰題：用 tracemalloc 追蹤記憶體洩漏

給定以下有記憶體洩漏的程式碼，使用 tracemalloc 找出問題並修復：

 1# Exercise: Find and fix the memory leak
 2
 3class EventHandler:
 4    _handlers = []  # Class variable - potential leak!
 5
 6    def __init__(self, name):
 7        self.name = name
 8        self.callbacks = []
 9        EventHandler._handlers.append(self)  # Leak: strong reference
10
11    def register(self, callback):
12        self.callbacks.append(callback)
13
14    def fire(self, event):
15        for cb in self.callbacks:
16            cb(event)
17
18def process_events():
19    """This function creates handlers but never cleans them up"""
20    for i in range(1000):
21        handler = EventHandler(f"handler_{i}")
22        handler.register(lambda e: print(e))
23        handler.fire(f"event_{i}")
24        # handler goes out of scope but is still in _handlers!
25
26# TODO:
27# 1. Use tracemalloc to measure memory growth
28# 2. Identify the leak
29# 3. Fix EventHandler to use weak references
30# 4. Verify the fix with tracemalloc

延伸閱讀

上一章：效能分析實戰 返回：模組四：CPython 內部機制

案例研究 on Tarragon

案例：效能分析實戰

先備知識

問題背景

現有設計

效能問題

進階解決方案

分析目標

實作步驟

步驟 1：使用 cProfile 進行函式級分析

步驟 2：使用 pstats 分析結果

步驟 3：使用 line_profiler 進行行級分析

步驟 4：分析正則表達式效能

步驟 5：優化建議與驗證

完整程式碼

分析結果範例

設計權衡

什麼時候該做效能分析？

練習

基礎練習：用 cProfile 分析排序函式

進階練習：用 line_profiler 找出熱點程式碼

挑戰題：比較不同正則表達式寫法的效能差異

延伸閱讀

案例：記憶體優化

先備知識

問題背景

現有設計

記憶體問題

進階解決方案

優化目標

實作步驟

步驟 1：使用 __slots__ 減少物件大小

記憶體結構比較

大量物件的記憶體節省

步驟 2：使用 weakref 避免強引用

使用 callback 追蹤物件回收

步驟 3：使用 WeakValueDictionary

步驟 4：測量記憶體使用

比較記憶體差異的完整腳本

完整程式碼

使用範例

設計權衡

__slots__ vs 標準類別

強引用 vs 弱引用快取

何時使用哪種技術？

什麼時候該用這個技術？

適合使用

不建議使用

優化決策流程

練習

基礎練習：比較有無 __slots__ 的記憶體差異

進階練習：實作 weakref 快取

挑戰題：用 tracemalloc 追蹤記憶體洩漏

延伸閱讀

步驟 1：使用 `slots` 減少物件大小

`slots` vs 標準類別

基礎練習：比較有無 `slots` 的記憶體差異