基于Prometheus的LLM服务监控告警

基于Prometheus的LLM服务监控告警

随着大模型服务的微服务化改造，监控告警体系成为保障服务稳定运行的关键。本文将介绍如何基于Prometheus构建LLM服务的监控告警系统。

监控指标收集

首先需要在LLM服务中集成Prometheus客户端库，以收集关键指标：

from prometheus_client import start_http_server, Counter, Histogram

# 定义计数器和直方图
request_count = Counter('llm_requests_total', 'Total requests', ['method', 'endpoint'])
request_duration = Histogram('llm_request_duration_seconds', 'Request duration')

@app.route('/predict')
def predict():
    with request_duration.time():
        # 业务逻辑
        request_count.labels(method='POST', endpoint='/predict').inc()
        return response

Prometheus配置

在prometheus.yml中添加目标：

scrape_configs:
  - job_name: 'llm-service'
    static_configs:
      - targets: ['localhost:8000']

告警规则配置

创建告警规则文件alerting_rules.yml：

groups:
- name: llm-alerts
  rules:
  - alert: HighLatency
    expr: histogram_quantile(0.95, sum(rate(llm_request_duration_seconds_bucket[5m])) by (le)) > 10
    for: 2m
    labels:
      severity: page
    annotations:
      summary: "LLM服务延迟过高"

通过以上配置，当95%请求延迟超过10秒时，将触发告警通知。