大模型训练前数据预处理流程设计

在大模型训练中，数据预处理是决定模型性能的关键环节。本文将介绍一套完整的数据预处理流程，帮助数据科学家构建高质量的训练数据集。

1. 数据清洗与去重

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer

# 加载数据
df = pd.read_csv('raw_data.csv')

# 去除重复行
df_cleaned = df.drop_duplicates(subset=['text_column'], keep='first')

# 移除空值
df_cleaned = df_cleaned.dropna(subset=['text_column'])

# 文本清洗函数
def clean_text(text):
    text = text.lower()
    text = re.sub(r'[^a-zA-Z0-9\s]', '', text)
    return text.strip()

df_cleaned['cleaned_text'] = df_cleaned['text_column'].apply(clean_text)

2. 数据格式标准化

将文本数据转换为模型可接受的格式，包括tokenization和padding操作：

from transformers import AutoTokenizer

# 加载tokenizer
model_name = 'bert-base-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_name)

def tokenize_and_pad(texts, max_length=512):
    return tokenizer(
        texts,
        truncation=True,
        padding='max_length',
        max_length=max_length,
        return_tensors='pt'
    )

# 应用tokenization
encoded_data = tokenize_and_pad(df_cleaned['cleaned_text'].tolist())

3. 特征工程处理

对于结构化数据，进行特征编码和标准化：

from sklearn.preprocessing import StandardScaler, LabelEncoder

# 数值特征标准化
scaler = StandardScaler()
numeric_features = ['feature1', 'feature2']
df_cleaned[numeric_features] = scaler.fit_transform(df_cleaned[numeric_features])

# 分类特征编码
label_encoders = {}
for col in ['category1', 'category2']:
    le = LabelEncoder()
    df_cleaned[col] = le.fit_transform(df_cleaned[col])
    label_encoders[col] = le

4. 数据集划分

将处理后的数据划分为训练集、验证集和测试集：

from sklearn.model_selection import train_test_split

# 划分训练集和测试集
X_train, X_test, y_train, y_test = train_test_split(
    encoded_data['input_ids'], 
    df_cleaned['labels'], 
    test_size=0.1, 
    random_state=42
)

# 验证集从训练集中进一步划分
X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, 
    test_size=0.11, 
    random_state=42
)