基于日志分析的模型异常请求识别与告警

基于日志分析的模型异常请求识别与告警

问题背景

在生产环境中，我们发现模型推理服务出现间歇性响应超时，但常规监控指标未触发告警。通过深入分析日志发现，部分请求在特定时间段内出现显著的延迟模式。

监控指标配置

# prometheus监控配置
- job_name: 'model_service'
  metrics_path: /metrics
  static_configs:
    - targets: ['localhost:8080']
  metric_relabel_configs:
    - source_labels: [__name__]
      regex: 'request_duration_seconds'
      target_label: metric_type
      replacement: duration
    - source_labels: [__name__]
      regex: 'request_count'
      target_label: metric_type
      replacement: count

# 日志分析指标
- log_patterns:
  - pattern: 'REQUEST.*duration=(\d+)ms.*status=(\d+)'
    labels:
      duration_ms: $1
      status_code: $2

告警规则设置

# alerting rules
- alert: HighLatencyRequests
  expr: 
    rate(request_duration_seconds_count[5m]) > 100
    AND
    histogram_quantile(0.95, request_duration_seconds_sum / request_duration_seconds_count) > 2000
  for: 3m
  labels:
    severity: warning
    service: model-inference
  annotations:
    summary: "模型请求延迟过高"
    description: "95%请求延迟超过2秒，当前延迟{{ $value }}ms"

- alert: AbnormalRequestPattern
  expr:
    rate(log_requests{level="ERROR"}[10m]) > 5
    AND
    histogram_quantile(0.99, log_duration_ms) > 3000
  for: 2m
  labels:
    severity: critical
    service: model-logging

复现步骤

1. 模拟高并发请求：ab -n 1000 -c 50 http://localhost:8080/predict
2. 查看Prometheus指标：curl http://localhost:9090/api/v1/query?query=request_duration_seconds
3. 观察告警触发：kubectl get alerts

解决方案

通过设置基于分位数的动态阈值，实现了对异常请求模式的精准识别。当95%请求延迟超过2秒且请求量大于100次/分钟时触发告警。