深度学习推理性能评估体系

在大模型推理加速实践中，建立科学的性能评估体系是优化工作的基础。本文将从实际工程角度，介绍一套可复现的推理性能评估方法。

核心指标定义

主要关注三个核心指标：

吞吐量：每秒处理样本数（samples/sec）
延迟：单个样本平均处理时间（ms/sample）
内存占用：GPU/内存使用率

评估框架实现

import torch
import time
from torch.utils.data import DataLoader

def benchmark_inference(model, dataloader, device='cuda'):
    model = model.to(device)
    model.eval()
    
    # 预热
    for i, batch in enumerate(dataloader):
        if i >= 3: break
        _ = model(batch.to(device))
    
    # 正式测试
    times = []
    total_samples = 0
    start_time = time.time()
    
    for batch in dataloader:
        batch_size = len(batch)
        total_samples += batch_size
        
        torch.cuda.synchronize()
        start = time.time()
        with torch.no_grad():
            outputs = model(batch.to(device))
        torch.cuda.synchronize()
        end = time.time()
        
        times.append(end - start)
    
    total_time = time.time() - start_time
    
    throughput = total_samples / total_time
    avg_latency = sum(times) / len(times) * 1000  # ms
    
    return {
        'throughput': throughput,
        'avg_latency': avg_latency,
        'total_time': total_time,
        'samples': total_samples
    }