40 章完整教程 · 从 torch.tensor() 到工业级部署 · 深度学习一网打尽
在学习 PyTorch 之前,我们先搞清楚一个基本问题:什么是深度学习?
想象你教一个小孩认猫。你不会告诉他"猫有三角形的耳朵、胡须、竖瞳",你只是给他看很多猫的照片,看多了他自然就认识了。深度学习就是这个原理——给计算机看大量的数据,让它自己"学会"规律,而不是由人来写规则。
深度学习是机器学习的一个子领域,核心是用多层神经网络从数据中自动学习特征。它在以下领域取得了突破性成果:
人工智能是一个大概念,指让机器表现出类似人类的智能。机器学习是实现 AI 的一种方法——让机器从数据中学习,而不是人手写规则。深度学习是机器学习的一个子集——使用多层神经网络来学习。
打个比方:AI 是"造一辆车",机器学习是"用发动机驱动",深度学习是"用 V8 涡轮增压发动机",PyTorch 就是"造这台发动机的工具箱"。
| 框架 | 开发者 | 特点 | 适用场景 | 社区活跃度 |
|---|---|---|---|---|
| PyTorch | Meta (Facebook) | 动态图、Pythonic、调试方便 | 研究 + 工业部署 | ⭐⭐⭐⭐⭐ |
| TensorFlow | 静态图(TF2 改善)、生态完善 | 工业部署、移动端 | ⭐⭐⭐⭐ | |
| JAX | 函数式、自动微分强大 | 科研、高性能计算 | ⭐⭐⭐ | |
| PaddlePaddle | 百度 | 中文文档好、国产生态 | 国内企业项目 | ⭐⭐⭐ |
| MXNet | Apache | 高效、支持多语言 | 分布式训练 | ⭐⭐(已停止维护) |
在正式开始学习之前,先对 PyTorch 的核心概念有个大体印象。不需要现在就完全理解,后面每一章都会详细讲解:
| 时间 | 事件 | 意义 |
|---|---|---|
| 2016 | PyTorch 前身 Torch7 发布 | Lua 语言的深度学习框架 |
| 2017 | PyTorch 0.1 发布 | Facebook AI Research 推出 Python 版本 |
| 2018 | PyTorch 1.0 发布 | 正式版,支持生产环境部署 |
| 2020 | PyTorch 成为最流行的框架 | 超过 TensorFlow,学术界首选 |
| 2023 | PyTorch 2.0 发布 | torch.compile 大幅提升性能 |
| 2024 | PyTorch 2.x 持续更新 | FSDP、export 等新特性 |
深度学习用到的数学其实不多,主要是以下几块:
深度学习框架(如 PyTorch)已经帮你处理了所有数学计算。你需要理解的是概念(什么是梯度、什么是损失函数),而不是手算公式。就像开车不需要懂发动机原理一样。
在正式学习 PyTorch 之前,先了解几个基本概念:
神经元是神经网络的基本单位。它接收多个输入,加权求和后通过一个激活函数输出。
Python# 一个人工神经元的计算过程
import torch
# 输入
x = torch.tensor([1.0, 2.0, 3.0])
# 权重
w = torch.tensor([0.5, 0.3, 0.2])
# 偏置
b = torch.tensor(0.1)
# 加权求和
z = torch.dot(x, w) + b
print(f"加权求和: {z}") # 1*0.5 + 2*0.3 + 3*0.2 + 0.1 = 1.8
# 激活函数(ReLU)
output = torch.relu(z)
print(f"输出: {output}") # 1.8
数据从输入层经过隐藏层到输出层的过程,叫做前向传播。就像工厂的流水线,原材料经过一道道工序变成成品。
计算损失函数对每个参数的梯度的过程,叫做反向传播。PyTorch 的 autograd 帮你自动完成。
深度学习的训练就是不断重复:前向传播 → 计算损失 → 反向传播 → 更新参数。就像不断练习考试,每次错的题都改正,成绩就会越来越好。
Anaconda 是 Python 的科学计算发行版,自带了大量常用库和包管理工具 conda。强烈建议使用 Anaconda 管理你的深度学习环境。
conda --version 验证虚拟环境的作用是隔离不同项目的依赖,避免版本冲突。
想象你有两个项目:项目 A 需要 PyTorch 1.13,项目 B 需要 PyTorch 2.0。如果没有虚拟环境,装了 A 就没法同时用 B。虚拟环境就像给每个项目一个独立的"房间",互不影响。你可以在不同的"房间"里安装不同版本的库。
# 创建名为 pytorch 的虚拟环境,Python 版本 3.10
conda create -n pytorch python=3.10
# 激活环境(每次使用前都要激活)
conda activate pytorch
# 确认 Python 版本
python --version
# Python 3.10.x
# 查看当前环境安装了哪些包
pip list
# 退出当前环境(一般不需要)
conda deactivate
# 查看所有环境
conda env list
# 删除环境
conda env remove -n pytorch
# 导出环境(方便分享给他人)
conda env export > environment.yml
# 从文件创建环境
conda env create -f environment.yml
访问 PyTorch 官网 https://pytorch.org/get-started/locally/,根据你的系统选择对应的安装命令。
pip install torch torchvision torchaudio
# 第一步:确认你的 CUDA 版本
nvidia-smi
# 在输出的右上角能看到 CUDA Version: 12.x 之类的
# 第二步:去 PyTorch 官网选对应版本
# 例如 CUDA 12.1:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
# 例如 CUDA 11.8:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
安装 GPU 版本前,一定先用 nvidia-smi 查看你的 CUDA 版本。PyTorch 的 CUDA 版本必须 ≤ 你系统的 CUDA 版本。比如你系统是 CUDA 11.8,那就装 cu118,不能装 cu121。
没有 NVIDIA 显卡? 没关系,装 CPU 版本就行,学习阶段完全够用。训练慢一点而已。
AMD 显卡? PyTorch 也支持 AMD(ROCm),但配置比较复杂,建议先用 CPU 版本。
import torch
# 查看版本
print(f"PyTorch 版本: {torch.__version__}")
# 查看 CUDA 是否可用
print(f"CUDA 可用: {torch.cuda.is_available()}")
print(f"CUDA 版本: {torch.version.cuda}")
print(f"GPU 数量: {torch.cuda.device_count()}")
if torch.cuda.is_available():
print(f"GPU 名称: {torch.cuda.get_device_name(0)}")
print(f"显存大小: {torch.cuda.get_device_properties(0).total_mem / 1024**3:.1f} GB")
# 快速测试:在 GPU 上做矩阵乘法
if torch.cuda.is_available():
a = torch.randn(1000, 1000, device='cuda')
b = torch.randn(1000, 1000, device='cuda')
c = a @ b
print(f"GPU 矩阵乘法成功!结果形状: {c.shape}")
print(f"结果设备: {c.device}") # cuda:0
# CPU 测试
a = torch.randn(100, 100)
b = torch.randn(100, 100)
c = a @ b
print(f"CPU 矩阵乘法成功!结果形状: {c.shape}")
| 工具 | 优点 | 缺点 | 推荐场景 |
|---|---|---|---|
| Jupyter Notebook | 交互式、可视化好、适合实验 | 不适合大型项目 | 学习和实验(强烈推荐初学者) |
| VS Code | 轻量、插件丰富、免费 | 需要配置 Python 环境 | 日常开发 |
| PyCharm | 功能强大、调试方便 | 占内存大、收费(社区版免费) | 大型项目开发 |
| Google Colab | 免费 GPU、无需安装任何东西 | 需要科学上网、会话有时限 | 没有 GPU 的同学 |
pip install jupyter notebook
# 启动
jupyter notebook
# 会自动打开浏览器,创建 .ipynb 文件即可开始编程
# 安装 VS Code 后,安装以下扩展:
# 1. Python(微软官方)
# 2. Jupyter(微软官方)
# 3. Pylance(代码补全)
# 在 VS Code 中选择 Python 解释器:
# Ctrl+Shift+P → "Python: Select Interpreter" → 选你的 conda 环境
import torch
# 创建两个张量
x = torch.tensor([1.0, 2.0, 3.0])
y = torch.tensor([4.0, 5.0, 6.0])
# 基本运算
print("加法:", x + y) # tensor([5., 7., 9.])
print("乘法:", x * y) # tensor([ 4., 10., 18.])
print("点积:", torch.dot(x, y)) # tensor(32.)
# 创建一个简单的"神经网络"
w = torch.randn(3, requires_grad=True) # 权重,需要求导
b = torch.randn(1, requires_grad=True) # 偏置
# 前向传播
input_data = torch.tensor([1.0, 2.0, 3.0])
prediction = w @ input_data + b # 线性变换
print(f"预测值: {prediction.item():.4f}")
# 假设目标值是 10
target = torch.tensor(10.0)
loss = (prediction - target) ** 2 # 均方误差
print(f"损失: {loss.item():.4f}")
# 反向传播
loss.backward()
print(f"权重梯度: {w.grad}")
print(f"偏置梯度: {b.grad}")
虽然简单,但它包含了深度学习的完整流程:创建数据 → 前向传播 → 计算损失 → 反向传播。后面我们只是把这个过程变得更复杂、更强大。
| 特性 | PyTorch | TensorFlow |
|---|---|---|
| 计算图 | 动态图(define-by-run) | 静态图(TF1)/ 动态图(TF2) |
| 调试 | 和 Python 一样方便 | TF1 很难调试,TF2 改善 |
| API 设计 | Pythonic,简洁直观 | API 较多,有时不一致 |
| 部署 | TorchServe、ONNX | TF Serving、TFLite |
| 移动端 | 支持(但不如 TF) | TFLite 非常成熟 |
| 社区 | 学术界首选 | 工业界较多 |
| 学习曲线 | 低(会 Python 就能用) | 中等 |
pip install torch -i https://pypi.tuna.tsinghua.edu.cn/simplenvidia-smi 查看 CUDA 版本,再选对应的 PyTorchtorch.device('mps')# 常用快捷键
# Shift+Enter:运行当前单元格
# Esc+A:在上方插入新单元格
# Esc+B:在下方插入新单元格
# Esc+DD:删除当前单元格
# Esc+M:转为 Markdown
# Esc+Y:转为代码
# 魔法命令
%matplotlib inline # 在 notebook 中显示图片
%timeit sum(range(1000)) # 测量代码运行时间
%load_ext autoreload
%autoreload 2 # 自动重新加载修改过的模块
# 查看 GPU 信息
!nvidia-smi
# 安装包
!pip install torch torchvision
张量就是"多维数组"的别名。如果你学过 NumPy,张量就是 ndarray 的 PyTorch 版本,但多了两个超能力:① 可以在 GPU 上运算;② 支持自动求导(自动算梯度)。你可以把张量想象成一个"容器",里面装着数字,可以是 0 维(一个数字)、1 维(一行数字)、2 维(一个表格)……任意维度。
张量按维度(阶/dim)分类:
torch.tensor(3.14),shape 为 []。就像一个点[1, 2, 3],shape 为 [3]。就像一条线[[1,2],[3,4]],shape 为 [2,2]。就像一张纸[通道, 高, 宽],如 [3, 224, 224]。就像一本书(多页)[批大小, 通道, 高, 宽],如 [32, 3, 224, 224]。就像一摞书[批大小, 时间帧, 通道, 高, 宽],如 [8, 16, 3, 224, 224]import torch
# ==========================================
# 1. 从 Python 列表创建
# ==========================================
a = torch.tensor([1, 2, 3])
print(a) # tensor([1, 2, 3])
print(a.dtype) # torch.int64 (自动推断为整数)
b = torch.tensor([1.0, 2.0, 3.0])
print(b.dtype) # torch.float32 (自动推断为浮点数)
# 指定数据类型
c = torch.tensor([1, 2, 3], dtype=torch.float32)
print(c.dtype) # torch.float32
# 创建二维张量(矩阵)
d = torch.tensor([[1, 2, 3],
[4, 5, 6]])
print(d.shape) # torch.Size([2, 3]) → 2行3列
print(d.ndim) # 2 (维度数)
# 创建三维张量
e = torch.tensor([[[1, 2], [3, 4]],
[[5, 6], [7, 8]]])
print(e.shape) # torch.Size([2, 2, 2])
# ==========================================
# 2. 全零张量
# ==========================================
zeros = torch.zeros(3, 4)
print(zeros)
# tensor([[0., 0., 0., 0.],
# [0., 0., 0., 0.],
# [0., 0., 0., 0.]])
# 保持和其他张量一样的形状
x = torch.randn(2, 3)
zeros_like_x = torch.zeros_like(x) # 形状和 x 一样,全是0
# ==========================================
# 3. 全一张量
# ==========================================
ones = torch.ones(2, 3)
# tensor([[1., 1., 1.],
# [1., 1., 1.]])
# ==========================================
# 4. 单位矩阵(对角线为1)
# ==========================================
eye = torch.eye(4) # 4x4 单位矩阵
# tensor([[1., 0., 0., 0.],
# [0., 1., 0., 0.],
# [0., 0., 1., 0.],
# [0., 0., 0., 1.]])
# ==========================================
# 5. 随机张量(最常用!)
# ==========================================
# 均匀分布 [0, 1)
rand = torch.rand(2, 3)
print(rand)
# tensor([[0.4963, 0.7682, 0.0885],
# [0.1320, 0.3074, 0.6341]])
# 标准正态分布(均值0,标准差1)
randn = torch.randn(2, 3)
# 整数随机张量
randint = torch.randint(0, 10, (3,)) # 从0到10中随机选3个
print(randint) # tensor([7, 2, 9])
# ==========================================
# 6. 序列张量
# ==========================================
# 等差序列
arange = torch.arange(0, 10, 2) # 从0到10,步长2
print(arange) # tensor([0, 2, 4, 6, 8])
# 等间隔序列
linspace = torch.linspace(0, 1, 5) # 从0到1,分成5份
print(linspace) # tensor([0.0000, 0.2500, 0.5000, 0.7500, 1.0000])
# 对数间隔
logspace = torch.logspace(0, 3, 4) # 10^0 到 10^3
print(logspace) # tensor([ 1., 10., 100., 1000.])
torch.tensor([1, 2, 3]) — 小写 t,根据输入推断 dtype,推荐使用torch.Tensor(2, 3) — 大写 T,创建未初始化的 2x3 float32 张量(值是随机垃圾)torch.FloatTensor([1, 2, 3]) — 和大写 T 一样,指定类型的构造器记忆方法:小写 tensor 是工厂函数(推荐),大写 Tensor 是类构造器。日常用小写的就行。
x = torch.randn(2, 3, 4)
print(x.shape) # torch.Size([2, 3, 4]) 形状
print(x.size()) # torch.Size([2, 3, 4]) 同上(两种写法等价)
print(x.size(0)) # 2 (第0维的大小)
print(x.dim()) # 3 (维度数/阶)
print(x.numel()) # 24 (元素总数 = 2*3*4)
print(x.dtype) # torch.float32 (数据类型)
print(x.device) # cpu (所在设备)
print(x.requires_grad) # False (是否需要求导)
print(x.is_contiguous()) # True (内存是否连续)
| 类型 | torch dtype | 字节数 | 说明 |
|---|---|---|---|
| 32位浮点 | torch.float32 |
4 | 默认类型,模型训练常用 |
| 16位浮点 | torch.float16 |
2 | 混合精度训练,节省显存 |
| BF16 | torch.bfloat16 |
2 | Google 提出的 16 位格式,A100/H100 显卡常用 |
| 64位浮点 | torch.float64 |
8 | 高精度计算,深度学习很少用 |
| 64位整数 | torch.int64 |
8 | 索引、标签常用(默认整数类型) |
| 32位整数 | torch.int32 |
4 | 整数运算 |
| 16位整数 | torch.int16 |
2 | 节省空间的整数 |
| 8位整数 | torch.int8 |
1 | 模型量化常用 |
| 布尔型 | torch.bool |
1 | 掩码操作、条件筛选 |
# 类型转换的各种方式
x = torch.tensor([1, 2, 3]) # int64
print(x.dtype) # torch.int64
y = x.float() # → float32
z = x.to(torch.float16) # → float16
w = x.double() # → float64
u = x.type(torch.float32) # 另一种写法
# 检查类型
print(x.dtype == torch.int64) # True
print(x.is_floating_point()) # False
print(y.is_floating_point()) # True
张量的索引和切片规则与 Python 列表、NumPy 完全一致。如果你已经会 Python,这部分几乎不用学。
Pythonx = torch.tensor([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
# ==========================================
# 基本索引
# ==========================================
print(x[0]) # tensor([1, 2, 3]) 第0行
print(x[1]) # tensor([4, 5, 6]) 第1行
print(x[-1]) # tensor([7, 8, 9]) 最后一行
print(x[1, 2]) # tensor(6) 第1行第2列
print(x[0][1]) # tensor(2) 等价写法
# ==========================================
# 切片(和 Python 一模一样)
# ==========================================
print(x[0:2]) # 前2行
print(x[:, 0]) # tensor([1, 4, 7]) 所有行的第0列
print(x[:, 1:]) # 第1列之后的所有列
print(x[0:2, 1:3]) # 前2行,第1-2列
# 步长切片
print(x[::2]) # 每隔一行取一行 → 第0行和第2行
print(x[:, ::2]) # 每隔一列取一列
# ==========================================
# 布尔索引(条件筛选)
# ==========================================
mask = x > 5
print(mask)
# tensor([[False, False, False],
# [False, False, True],
# [ True, True, True]])
print(x[mask]) # tensor([6, 7, 8, 9]) 取出大于5的元素
# 更简洁的写法
print(x[x > 5]) # tensor([6, 7, 8, 9])
print(x[x != 5]) # 除去5以外的所有元素
# ==========================================
# 花式索引(高级索引)
# ==========================================
indices = torch.tensor([0, 2])
print(x[indices]) # 取第0行和第2行
# 用列表索引
print(x[[0, 2], [1, 2]]) # 取 (0,1) 和 (2,2) 位置的元素 → tensor([2, 9])
x = torch.arange(12)
print(x) # tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
# ==========================================
# reshape / view:改变形状
# ==========================================
y = x.reshape(3, 4)
print(y)
# tensor([[ 0, 1, 2, 3],
# [ 4, 5, 6, 7],
# [ 8, 9, 10, 11]])
# -1 表示自动推算
z = x.reshape(2, -1) # 2行,列自动算 → 2x6
print(z.shape) # torch.Size([2, 6])
w = x.reshape(-1, 6) # 列固定6,行自动算 → 2x6
print(w.shape) # torch.Size([2, 6])
# ==========================================
# flatten:展平为一维
# ==========================================
flat = y.flatten()
print(flat) # tensor([ 0, 1, ..., 11])
# 只展平指定维度
m = torch.randn(2, 3, 4)
print(m.flatten(1).shape) # torch.Size([2, 12]) 从第1维开始展平
# ==========================================
# unsqueeze:增加维度(非常常用!)
# ==========================================
a = torch.tensor([1, 2, 3]) # shape: [3]
b = a.unsqueeze(0) # shape: [1, 3] 在第0维增加
c = a.unsqueeze(1) # shape: [3, 1] 在第1维增加
print(a.shape, b.shape, c.shape)
# torch.Size([3]) torch.Size([1, 3]) torch.Size([3, 1])
# 为什么需要 unsqueeze?
# 深度学习中经常需要添加 batch 维度
# 单张图片 shape: [3, 224, 224]
# 加 batch 后: [1, 3, 224, 224]
img = torch.randn(3, 224, 224)
img_batch = img.unsqueeze(0) # [1, 3, 224, 224]
# ==========================================
# squeeze:删除大小为1的维度
# ==========================================
d = b.squeeze(0) # shape: [3]
print(d.shape) # torch.Size([3])
# 如果维度大小不是1,squeeze 不会删除
e = torch.randn(1, 3, 1)
print(e.squeeze().shape) # torch.Size([3]) 只删除大小为1的维度
print(e.squeeze(0).shape) # torch.Size([3, 1])
# ==========================================
# transpose:交换两个维度
# ==========================================
m = torch.randn(2, 3)
print(m.t().shape) # torch.Size([3, 2])
print(m.transpose(0, 1).shape) # torch.Size([3, 2]) 等价
# ==========================================
# permute:任意重排维度(深度学习常用!)
# ==========================================
# 图片格式转换
img_chw = torch.randn(3, 224, 224) # [通道, 高, 宽] PyTorch 格式
img_hwc = img_chw.permute(1, 2, 0) # [高, 宽, 通道] 显示格式
print(img_hwc.shape) # torch.Size([224, 224, 3])
# 从 [B, H, W, C] → [B, C, H, W]
x = torch.randn(2, 224, 224, 3)
x = x.permute(0, 3, 1, 2)
print(x.shape) # torch.Size([2, 3, 224, 224])
# ==========================================
# cat:拼接张量
# ==========================================
a = torch.randn(2, 3)
b = torch.randn(2, 3)
c = torch.cat([a, b], dim=0) # 沿第0维拼接 → [4, 3]
d = torch.cat([a, b], dim=1) # 沿第1维拼接 → [2, 6]
# ==========================================
# stack:堆叠张量(增加新维度)
# ==========================================
e = torch.stack([a, b], dim=0) # → [2, 2, 3]
f = torch.stack([a, b], dim=1) # → [2, 2, 3] (堆叠方式不同)
# cat vs stack 的区别:
# cat: 沿已有维度拼接,不增加维度
# stack: 创建新维度后堆叠
view():要求内存连续,不连续会报错,但效率略高reshape():不连续时会自动拷贝,更安全,推荐使用记忆:日常用 reshape,确定连续时用 view。如果你不确定,就用 reshape。
import numpy as np
# ==========================================
# Tensor → NumPy
# ==========================================
t = torch.tensor([1.0, 2.0, 3.0])
n = t.numpy()
print(type(n)) # <class 'numpy.ndarray'>
# 注意:默认是共享内存的!
t[0] = 100
print(n[0]) # 100.0 ← 也变了!
# 如果想要独立副本
n2 = t.clone().numpy()
t[0] = 999
print(n2[0]) # 100.0 ← 不受影响
# ==========================================
# NumPy → Tensor
# ==========================================
n3 = np.array([4.0, 5.0, 6.0])
t3 = torch.from_numpy(n3) # 共享内存
print(type(t3)) # <class 'torch.Tensor'>
# 独立副本
t4 = torch.tensor(n3) # 复制数据
# 在 GPU 上的张量转 NumPy 需要先移到 CPU
if torch.cuda.is_available():
t_gpu = torch.randn(3, device='cuda')
n4 = t_gpu.cpu().numpy() # 先 .cpu() 再 .numpy()
当你用 torch.from_numpy() 或 .numpy() 转换时,两者共享同一块内存。修改一个,另一个也会变!如果不想这样,用 torch.tensor() 或 .clone().numpy()。
# ==========================================
# 创建在 CPU 上的张量
# ==========================================
x = torch.randn(3, 4)
print(x.device) # cpu
# ==========================================
# 移到 GPU(三种写法)
# ==========================================
if torch.cuda.is_available():
x_gpu1 = x.to('cuda') # 方法1:to()(推荐)
x_gpu2 = x.cuda() # 方法2:cuda()
x_gpu3 = x.to(device='cuda:0') # 方法3:指定GPU编号
print(x_gpu1.device) # cuda:0
# 移回 CPU
x_cpu1 = x_gpu1.to('cpu')
x_cpu2 = x_gpu1.cpu()
# ==========================================
# 最佳实践:定义 device 变量
# ==========================================
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"使用设备: {device}")
# 直接在目标设备上创建张量
y = torch.randn(3, 4, device=device)
# 模型也用同样的方式
# model = MyModel().to(device)
# 多 GPU 时指定
# device = torch.device('cuda:1') # 用第二个GPU
三个地方需要移到同一个设备:
model.to(device)data = data.to(device)target = target.to(device)如果这三者不在同一个设备上,会报错:Expected all tensors to be on the same device
# 连续内存 vs 非连续内存
x = torch.randn(2, 3, 4)
print(x.is_contiguous()) # True
y = x.transpose(0, 1) # 转置后内存不连续
print(y.is_contiguous()) # False
# 如果需要连续内存
z = y.contiguous()
# 查看张量占用的内存
x = torch.randn(1000, 1000)
print(f"元素数: {x.numel()}") # 1000000
print(f"每个元素字节: {x.element_size()}") # 4 (float32)
print(f"总字节: {x.numel() * x.element_size()}") # 4000000
print(f"总 MB: {x.numel() * x.element_size() / 1024**2:.1f}") # 3.8 MB
| 操作 | 代码 | 说明 |
|---|---|---|
| 创建 | torch.tensor([1,2,3]) |
从列表创建 |
| 全零 | torch.zeros(3,4) |
创建全零张量 |
| 随机 | torch.randn(3,4) |
标准正态分布 |
| 改变形状 | x.reshape(2,6) |
reshape |
| 增加维度 | x.unsqueeze(0) |
在第0维增加 |
| 删除维度 | x.squeeze() |
删除大小为1的维度 |
| 转置 | x.t() |
2D张量转置 |
| 重排维度 | x.permute(2,0,1) |
任意重排 |
| 拼接 | torch.cat([a,b], dim=0) |
沿维度拼接 |
| 堆叠 | torch.stack([a,b], dim=0) |
增加新维度后堆叠 |
| 移到GPU | x.to('cuda') |
设备迁移 |
| 转NumPy | x.numpy() |
转为NumPy数组 |
理解张量的内存布局有助于调试和优化性能:
Python# 连续内存 vs 非连续内存
x = torch.randn(2, 3, 4)
print(x.is_contiguous()) # True
# 转置后内存不连续
y = x.transpose(0, 1)
print(y.is_contiguous()) # False
# 如果需要连续内存
z = y.contiguous()
# stride:每个维度移动一步需要跳过的元素数
print(x.stride()) # (12, 4, 1) 第0维移动1步跳12个元素
print(y.stride()) # (4, 12, 1) 转置后stride变了
# 存储大小 vs 实际大小
print(x.storage().size()) # 24 存储中的元素数
print(x.nelement()) # 24 张量的元素数
x.add_(1) 比 x = x + 1 更快(但要注意梯度)reshape 比 contiguous().view() 更安全PyTorch 的数学运算和 NumPy 几乎一模一样,而且全部支持 GPU 加速。所有运算都是逐元素的(element-wise),即对每个元素独立操作。
Pythona = torch.tensor([1.0, 2.0, 3.0])
b = torch.tensor([4.0, 5.0, 6.0])
# ==========================================
# 逐元素运算
# ==========================================
print(a + b) # tensor([5., 7., 9.]) 加法
print(a - b) # tensor([-3., -3., -3.]) 减法
print(a * b) # tensor([ 4., 10., 18.]) 乘法
print(a / b) # tensor([0.25, 0.40, 0.50]) 除法
print(a ** 2) # tensor([1., 4., 9.]) 幂运算
print(a % 2) # tensor([1., 0., 1.]) 取模
# ==========================================
# 数学函数
# ==========================================
print(a.sqrt()) # 平方根
print(a.exp()) # 指数函数 e^x
print(a.log()) # 自然对数 ln(x)
print(a.log10()) # 以10为底的对数
print(a.abs()) # 绝对值
print(a.sin()) # 正弦
print(a.cos()) # 余弦
print(a.tanh()) # 双曲正切
print(a.sigmoid()) # Sigmoid 函数 1/(1+e^-x)
print(a.relu()) # ReLU: max(0, x)
# 也可以用函数形式
print(torch.add(a, b))
print(torch.mul(a, b))
print(torch.sqrt(a))
# ==========================================
# 就地操作(in-place):以 _ 结尾
# ==========================================
c = torch.tensor([1.0, 2.0, 3.0])
c.add_(10) # c 变成 [11., 12., 13.] ← 直接修改 c
c.mul_(2) # c 变成 [22., 24., 26.]
c.relu_() # 就地 ReLU
print(c)
# 注意:就地操作会修改原张量,可能影响梯度计算
# 一般不推荐在需要求导的张量上使用
A = torch.tensor([[1.0, 2.0],
[3.0, 4.0]])
B = torch.tensor([[5.0, 6.0],
[7.0, 8.0]])
# ==========================================
# 矩阵乘法(三种写法等价)
# ==========================================
C1 = A @ B # 写法1:@ 运算符(推荐,最简洁)
C2 = torch.mm(A, B) # 写法2:mm 函数(只用于2D矩阵)
C3 = A.matmul(B) # 写法3:matmul 方法
print(C1)
# tensor([[19., 22.],
# [43., 50.]])
# 计算过程:C[0,0] = 1*5 + 2*7 = 19
# ==========================================
# 批量矩阵乘法(深度学习常用!)
# ==========================================
batch_A = torch.randn(32, 10, 20) # 32个 10x20 的矩阵
batch_B = torch.randn(32, 20, 30) # 32个 20x30 的矩阵
batch_C = batch_A @ batch_B # → [32, 10, 30] 自动批量运算
# ==========================================
# 向量运算
# ==========================================
v1 = torch.tensor([1.0, 2.0, 3.0])
v2 = torch.tensor([4.0, 5.0, 6.0])
# 点积(内积)
dot = torch.dot(v1, v2) # 1*4 + 2*5 + 3*6 = 32
print(dot) # tensor(32.)
# 外积
outer = torch.outer(v1, v2)
print(outer.shape) # torch.Size([3, 3])
# ==========================================
# 转置
# ==========================================
print(A.T) # 简写
print(A.transpose(0, 1)) # 完整写法
print(A.permute(1, 0)) # permute 写法
x = torch.tensor([[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0]])
# tensor([[1, 2, 3],
# [4, 5, 6]])
# ==========================================
# 求和
# ==========================================
print(x.sum()) # tensor(21.) 全部元素之和
print(x.sum(dim=0)) # tensor([5., 7., 9.]) 按列求和(压缩行)
print(x.sum(dim=1)) # tensor([ 6., 15.]) 按行求和(压缩列)
print(x.sum(dim=0, keepdim=True)) # tensor([[5., 7., 9.]]) 保持维度
# ==========================================
# 均值
# ==========================================
print(x.mean()) # tensor(3.5000)
print(x.mean(dim=0)) # tensor([2.5, 3.5, 4.5]) 每列的均值
print(x.mean(dim=1)) # tensor([2., 5.]) 每行的均值
# ==========================================
# 最大值 / 最大值
# ==========================================
print(x.max()) # tensor(6.) 最大值
print(x.min()) # tensor(1.) 最小值
# 返回 (values, indices)
values, indices = x.max(dim=1)
print(values) # tensor([3., 6.]) 每行的最大值
print(indices) # tensor([2, 2]) 每行最大值的列索引
# ==========================================
# argmax / argmin:最大值的索引
# ==========================================
print(x.argmax()) # tensor(5) 展平后的索引
print(x.argmax(dim=1)) # tensor([2, 2]) 每行最大值的列索引
print(x.argmax(dim=0)) # tensor([1, 1, 1]) 每列最大值的行索引
# ==========================================
# 标准差、方差
# ==========================================
print(x.std()) # 标准差
print(x.var()) # 方差
# ==========================================
# 累积运算
# ==========================================
print(torch.cumsum(x, dim=0)) # 累积和
print(torch.cumprod(x, dim=1)) # 累积积
# ==========================================
# topk:取最大的 k 个
# ==========================================
values, indices = x.topk(2, dim=1)
print(values) # tensor([[3., 2.], [6., 5.]]) 每行最大的2个
# ==========================================
# 排序
# ==========================================
sorted_x, sorted_idx = x.sort(dim=1)
print(sorted_x) # 每行排序后的结果
广播是 PyTorch(和 NumPy)中非常重要的概念:当两个张量形状不同时,自动扩展较小的张量以匹配较大的张量。
想象你在给一列人发钱。你有 3 个人([3]),还有 3 组金币([3, 4])。广播机制会自动把 3 个人"复制"成 [3, 4] 的矩阵,然后逐个发钱。不需要你手动复制——PyTorch 帮你做了。这就像 Excel 里你把一行数字拖下来填充一样。
# ==========================================
# 广播示例
# ==========================================
# 标量 + 张量:标量自动广播
a = torch.tensor([1.0, 2.0, 3.0])
print(a + 10) # tensor([11., 12., 13.])
# 不同形状的广播
A = torch.randn(3, 4) # [3, 4]
b = torch.randn(4) # [4] → 自动变成 [1, 4] → 再变成 [3, 4]
print((A + b).shape) # torch.Size([3, 4])
# 更复杂的广播
x = torch.randn(2, 1, 3) # [2, 1, 3]
y = torch.randn(4, 3) # [4, 3] → 变成 [1, 4, 3] → [2, 4, 3]
print((x + y).shape) # torch.Size([2, 4, 3])
# 不能广播的例子
a = torch.randn(3, 4)
b = torch.randn(3, 5)
# a + b # RuntimeError! 4 != 5 且都不是1
例:[3, 4] + [4] → [3, 4] + [1, 4] → [3, 4] ✅
例:[3, 4] + [3, 5] → 4 ≠ 5 且都不是1 → ❌ 报错
a = torch.tensor([1.0, 2.0, 3.0])
b = torch.tensor([2.0, 2.0, 2.0])
# 比较运算(返回布尔张量)
print(a == b) # tensor([False, True, False])
print(a != b) # tensor([ True, False, True])
print(a > b) # tensor([False, False, True])
print(a >= b) # tensor([False, True, True])
print(a < b) # tensor([ True, False, False])
# 逻辑运算
c = torch.tensor([True, True, False])
d = torch.tensor([True, False, False])
print(c & d) # tensor([ True, False, False]) AND
print(c | d) # tensor([ True, True, False]) OR
print(~c) # tensor([False, False, True]) NOT
# 取满足条件的元素
print(a[a > 1.5]) # tensor([2., 3.])
# where:条件选择(类似 numpy.where)
result = torch.where(a > 1.5, a, b)
print(result) # tensor([2., 2., 3.]) 大于1.5取a,否则取b
# allclose:判断两个张量是否近似相等
x = torch.tensor([1.0, 2.0, 3.0])
y = torch.tensor([1.0, 2.0, 3.0001])
print(torch.allclose(x, y, atol=1e-3)) # True
torch.einsum 是一种强大的张量运算表达方式,可以用一行代码实现复杂的矩阵运算。它用字符串来描述维度的关系。
A = torch.randn(3, 4)
B = torch.randn(4, 5)
# 矩阵乘法:C[i,j] = Σ_k A[i,k] * B[k,j]
C = torch.einsum('ik,kj->ij', A, B)
print(C.shape) # torch.Size([3, 5])
# 转置
A_T = torch.einsum('ij->ji', A)
print(A_T.shape) # torch.Size([4, 3])
# 对角线元素
M = torch.randn(4, 4)
diag = torch.einsum('ii->i', M)
print(diag.shape) # torch.Size([4])
# 迹(对角线之和)
trace = torch.einsum('ii->', M)
# 批量矩阵乘法
A_batch = torch.randn(32, 10, 20)
B_batch = torch.randn(32, 20, 30)
C_batch = torch.einsum('bij,bjk->bik', A_batch, B_batch)
print(C_batch.shape) # torch.Size([32, 10, 30])
# 向量点积
v1 = torch.randn(10)
v2 = torch.randn(10)
dot = torch.einsum('i,i->', v1, v2)
# 外积
outer = torch.einsum('i,j->ij', v1, v2)
print(outer.shape) # torch.Size([10, 10])
广播是 NumPy 和 PyTorch 中最强大的特性之一,但也是最容易出错的地方:
Python# 广播规则示例
# 规则:从最后一个维度开始比较,满足以下条件之一可以广播:
# 1. 两个维度相等
# 2. 其中一个维度为 1
# 3. 其中一个维度缺失(默认为 1)
# 示例1:标量 + 张量
a = torch.tensor([1, 2, 3]) # [3]
b = 10 # 标量
print(a + b) # [11, 12, 13] 标量广播到 [3]
# 示例2:不同形状
A = torch.randn(3, 4) # [3, 4]
b = torch.randn(4) # [4] → 自动变成 [1, 4] → [3, 4]
print((A + b).shape) # [3, 4]
# 示例3:更复杂的广播
A = torch.randn(2, 1, 3) # [2, 1, 3]
B = torch.randn(4, 3) # [4, 3] → [1, 4, 3] → [2, 4, 3]
print((A + B).shape) # [2, 4, 3]
# 示例4:不能广播的情况
A = torch.randn(3, 4)
B = torch.randn(3, 5)
# A + B # RuntimeError! 4 != 5 且都不是1
# 实际应用:给每行加不同的偏置
weights = torch.randn(3, 4) # 3个样本,4个特征
bias = torch.tensor([1.0, 2.0, 3.0]) # 3个偏置
# weights + bias # 不能直接加,需要调整形状
bias = bias.unsqueeze(1) # [3, 1]
print((weights + bias).shape) # [3, 4] 每行加不同的偏置
# 实际应用1:计算余弦相似度
def cosine_similarity(a, b):
"""计算两个向量的余弦相似度"""
return torch.dot(a, b) / (torch.norm(a) * torch.norm(b))
v1 = torch.tensor([1.0, 2.0, 3.0])
v2 = torch.tensor([4.0, 5.0, 6.0])
print(f"余弦相似度: {cosine_similarity(v1, v2):.4f}")
# 实际应用2:批量计算相似度矩阵
def batch_cosine_similarity(vectors):
"""计算所有向量两两之间的相似度"""
# 归一化
norms = torch.norm(vectors, dim=1, keepdim=True)
normalized = vectors / norms
# 相似度矩阵
return torch.mm(normalized, normalized.t())
vectors = torch.randn(10, 128) # 10个128维向量
sim_matrix = batch_cosine_similarity(vectors)
print(f"相似度矩阵形状: {sim_matrix.shape}") # [10, 10]
# 实际应用3:one-hot 编码
def one_hot(labels, num_classes):
"""将标签转换为 one-hot 编码"""
return torch.zeros(labels.size(0), num_classes).scatter_(1, labels.unsqueeze(1), 1)
labels = torch.tensor([0, 2, 1, 3])
one_hot_labels = one_hot(labels, 4)
print(one_hot_labels)
# tensor([[1, 0, 0, 0],
# [0, 0, 1, 0],
# [0, 1, 0, 0],
# [0, 0, 0, 1]])
想象你在爬山,目标是找到山谷最低点。你需要知道"往哪个方向走是下坡"——这个方向就是梯度。自动求导就是 PyTorch 帮你自动计算这个方向的技术。没有它,你就得手写每个函数的导数公式,对于深度神经网络(几百万个参数)来说,这是不可能的。
深度学习的训练过程就是:
import torch
# ==========================================
# 最简单的例子:y = x^2,求 dy/dx
# ==========================================
x = torch.tensor(3.0, requires_grad=True) # 告诉 PyTorch:追踪这个张量
print(x) # tensor(3., requires_grad=True)
y = x ** 2 # y = x^2 = 9
print(y) # tensor(9., grad_fn=<PowBackward0>)
# grad_fn 记录了这个张量是怎么算出来的
# 反向传播:计算 dy/dx
y.backward()
# x 的梯度
print(x.grad) # tensor(6.) → dy/dx = 2x = 2*3 = 6 ✅
它告诉 PyTorch:"请追踪在这个张量上做的所有操作,这样我后面可以调用 .backward() 来算梯度。" 默认是 False。模型的参数(权重和偏置)默认就是 requires_grad=True。
当你对张量做操作时,PyTorch 会在后台自动构建一个有向无环图(DAG),记录所有操作。这个图叫做计算图。
Pythonx = torch.tensor(2.0, requires_grad=True)
y = x ** 2 # y = x^2
z = 3 * y + 5 # z = 3x^2 + 5
# 计算图:x → (x^2) → y → (3*y) → (3*y+5) → z
# ↑ ↑
# PowBackward MulBackward
# 查看计算图
print(y.grad_fn) # <PowBackward0 object>
print(z.grad_fn) # <AddBackward0 object>
# 反向传播
z.backward()
print(x.grad) # tensor(12.) → dz/dx = 6x = 6*2 = 12 ✅
自动求导的核心是链式法则:如果 y = f(g(x)),则 dy/dx = f'(g(x)) × g'(x)。PyTorch 自动帮你做这个乘法。
Pythonx = torch.tensor(2.0, requires_grad=True)
# 复合函数:c = sin((x+1)^2)
a = x + 1 # a = x + 1 = 3
b = a ** 2 # b = (x+1)^2 = 9
c = b.sin() # c = sin((x+1)^2) = sin(9)
c.backward()
# 链式法则:dc/dx = cos((x+1)^2) * 2(x+1)
print(x.grad) # tensor(-5.4668)
# 验证
import math
manual_grad = math.cos(9) * 6
print(f"手算: {manual_grad:.4f}") # -5.4668 ✅
.backward() 只能对标量(单个数字)调用。如果 loss 是一个向量或矩阵,需要先 .sum() 或传入 gradient 参数。
x = torch.randn(3, requires_grad=True)
# ❌ 错误:y 是向量,不能直接 backward
# y = x * 2
# y.backward() # RuntimeError: grad can be implicitly created only for scalar outputs
# ✅ 方法1:求和变成标量(最常用)
y = x * 2
y.sum().backward()
print(x.grad) # tensor([2., 2., 2.])
# ✅ 方法2:传入 gradient 参数(权重)
x.grad = None # 清零之前的梯度
y = x * 2
y.backward(gradient=torch.tensor([1.0, 1.0, 1.0]))
print(x.grad) # tensor([2., 2., 2.])
PyTorch 中梯度是累加的,每次调用 .backward() 后,新梯度会加到旧梯度上。所以在每次反向传播前,必须调用 optimizer.zero_grad() 或 x.grad.zero_() 清零。这是新手最常犯的错误之一!
x = torch.tensor(3.0, requires_grad=True)
# 第一次
y = x ** 2
y.backward()
print(x.grad) # tensor(6.)
# 第二次(梯度累加了!)
y = x ** 2
y.backward()
print(x.grad) # tensor(12.) ← 6 + 6 = 12 ❌ 不是我们想要的
# 正确做法:每次 backward 前清零
x.grad.zero_() # 或者 optimizer.zero_grad()
y = x ** 2
y.backward()
print(x.grad) # tensor(6.) ✅
有时候你不希望 PyTorch 追踪某些操作的梯度(比如推理阶段、或者某些中间变量)。有三种方式:
Pythonx = torch.tensor(3.0, requires_grad=True)
# ==========================================
# 方法1:torch.no_grad()(最常用,推荐!)
# ==========================================
with torch.no_grad():
y = x ** 2
print(y.requires_grad) # False
# 在这个代码块里,所有操作都不会被追踪
# 推理时一定要用,可以节省内存和加速
# ==========================================
# 方法2:detach()
# ==========================================
y = x ** 2
z = y.detach() # 创建一个不追踪梯度的"副本"
print(z.requires_grad) # False
# 注意:z 和 y 共享内存,但 z 不在计算图中
# ==========================================
# 方法3:requires_grad_(False)(就地操作)
# ==========================================
x.requires_grad_(False)
print(x.requires_grad) # False
with torch.no_grad(): 可以节省显存、加速计算当 PyTorch 内置的操作不够用时,你可以自定义前向和反向传播逻辑:
Pythonclass MyReLU(torch.autograd.Function):
"""自定义 ReLU 激活函数"""
@staticmethod
def forward(ctx, x):
# ctx 是上下文对象,用于保存反向传播需要的数据
ctx.save_for_backward(x)
return x.clamp(min=0) # 小于0的设为0
@staticmethod
def backward(ctx, grad_output):
x, = ctx.saved_tensors
grad_input = grad_output.clone()
grad_input[x < 0] = 0 # x < 0 时梯度为0
return grad_input
# 使用自定义函数(注意用 .apply 调用)
x = torch.tensor([-1.0, 2.0, -3.0, 4.0], requires_grad=True)
y = MyReLU.apply(x)
print(y) # tensor([0., 2., 0., 4.])
y.sum().backward()
print(x.grad) # tensor([0., 1., 0., 1.])
在实际的深度学习训练中,autograd 的使用流程是这样的:
Python# 标准训练循环中的 autograd 使用
model = ... # 你的模型
optimizer = ... # 优化器
data, labels = ... # 数据
# 1. 前向传播
predictions = model(data) # 计算预测值
loss = loss_fn(predictions, labels) # 计算损失
# 2. 反向传播
optimizer.zero_grad() # 清零梯度(重要!)
loss.backward() # 自动计算所有参数的梯度
# 3. 更新参数
optimizer.step() # 用梯度更新参数
# 4. 查看梯度
for name, param in model.named_parameters():
if param.grad is not None:
print(f"{name}: grad norm = {param.grad.norm():.4f}")
torch.autograd.grad 可以计算二阶、三阶导数register_hook 可以在反向传播时拦截梯度torch.autograd.gradcheck 验证自定义 Function 的正确性loss.backward(retain_graph=True) 允许多次 backward# 高阶导数
x = torch.tensor(2.0, requires_grad=True)
y = x ** 3
# 一阶导数
grad1 = torch.autograd.grad(y, x, create_graph=True)[0]
print(f"一阶导数: {grad1}") # 3x^2 = 12
# 二阶导数
grad2 = torch.autograd.grad(grad1, x)[0]
print(f"二阶导数: {grad2}") # 6x = 12
# 三阶导数
grad3 = torch.autograd.grad(grad2, x)[0]
print(f"三阶导数: {grad3}") # 6
# Jacobian 矩阵
x = torch.randn(3, requires_grad=True)
y = x ** 2
jacobian = torch.autograd.functional.jacobian(lambda x: x ** 2, x)
print(f"Jacobian:\n{jacobian}")
retain_graph=True 或重新前向传播requires_grad=Truetorch.no_grad() 中optimizer.zero_grad()神经网络就像一个"函数工厂":你给它输入(比如一张图片),它经过一系列变换后给你输出(比如"这是一只猫")。每一层就像工厂里的一个加工站,把数据加工成更有用的形式。第一层可能学会识别"边缘",第二层学会识别"纹理",第三层学会识别"耳朵"……最终组合出"这是一只猫"的判断。
一个最简单的神经网络包含:
torch.nn.Module 是 PyTorch 中所有神经网络的基类。你自定义的模型必须继承它。它提供了很多便利功能:参数管理、设备迁移、保存/加载等。
import torch
import torch.nn as nn
class SimpleNet(nn.Module):
def __init__(self):
super().__init__() # 必须调用父类构造函数
# 定义网络层("买零件")
self.linear1 = nn.Linear(784, 256) # 全连接层:784 → 256
self.relu1 = nn.ReLU() # 激活函数
self.linear2 = nn.Linear(256, 128) # 全连接层:256 → 128
self.relu2 = nn.ReLU()
self.linear3 = nn.Linear(128, 10) # 全连接层:128 → 10
def forward(self, x):
# 定义前向传播逻辑("组装零件")
x = self.linear1(x) # 线性变换:y = xW^T + b
x = self.relu1(x) # 激活函数:max(0, x)
x = self.linear2(x)
x = self.relu2(x)
x = self.linear3(x) # 输出层(不加激活函数)
return x
# 创建模型
model = SimpleNet()
print(model)
输出:
SimpleNet(
(linear1): Linear(in_features=784, out_features=256, bias=True)
(relu1): ReLU()
(linear2): Linear(in_features=256, out_features=128, bias=True)
(relu2): ReLU()
(linear3): Linear(in_features=128, out_features=10, bias=True)
)
__init__:定义网络有哪些层("买零件")——只执行一次forward:定义数据怎么流过这些层("组装零件")——每次前向传播都执行你永远不需要直接调用 forward(),而是用 model(x),PyTorch 会自动调用。
如果你的网络是"一层接一层"的简单结构,可以用 nn.Sequential 简化代码:
# ==========================================
# 方法1:使用 Sequential(最简洁)
# ==========================================
model = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Linear(256, 128),
nn.ReLU(),
nn.Linear(128, 10)
)
# ==========================================
# 方法2:使用 OrderedDict(给每层命名,推荐!)
# ==========================================
from collections import OrderedDict
model = nn.Sequential(OrderedDict([
('fc1', nn.Linear(784, 256)),
('relu1', nn.ReLU()),
('fc2', nn.Linear(256, 128)),
('relu2', nn.ReLU()),
('dropout', nn.Dropout(0.5)),
('fc3', nn.Linear(128, 10)),
]))
# 命名后可以用名字访问层
print(model.fc1) # Linear(in_features=784, out_features=256, bias=True)
# ==========================================
# 使用模型
# ==========================================
x = torch.randn(1, 784) # 一个样本,784个特征
output = model(x)
print(output.shape) # torch.Size([1, 10])
print(output) # 10个类别的"分数"(logits)
# nn.Linear(in_features, out_features, bias=True)
# 计算:y = xW^T + b
layer = nn.Linear(100, 50)
x = torch.randn(32, 100) # batch_size=32, 特征数=100
y = layer(x)
print(y.shape) # torch.Size([32, 50])
# 查看参数
print(f"权重形状: {layer.weight.shape}") # torch.Size([50, 100])
print(f"偏置形状: {layer.bias.shape}") # torch.Size([50])
print(f"参数量: {layer.weight.numel() + layer.bias.numel()}") # 5050
# 不带偏置
layer_no_bias = nn.Linear(100, 50, bias=False)
# ==========================================
# ReLU:最常用,max(0, x)
# ==========================================
relu = nn.ReLU()
x = torch.tensor([-2.0, -1.0, 0.0, 1.0, 2.0])
print(relu(x)) # tensor([0., 0., 0., 1., 2.])
# ==========================================
# LeakyReLU:ReLU 的改进,负半轴也有小梯度
# ==========================================
leaky = nn.LeakyReLU(0.1)
print(leaky(x)) # tensor([-0.2000, -0.1000, 0.0000, 1.0000, 2.0000])
# ==========================================
# Sigmoid:输出 (0, 1),常用于二分类输出层
# ==========================================
sigmoid = nn.Sigmoid()
print(sigmoid(torch.tensor([-1.0, 0.0, 1.0])))
# tensor([0.2689, 0.5000, 0.7311])
# ==========================================
# Tanh:输出 (-1, 1)
# ==========================================
tanh = nn.Tanh()
print(tanh(x))
# ==========================================
# GELU:Transformer 中常用(BERT、GPT 都用它)
# ==========================================
gelu = nn.GELU()
print(gelu(x))
# ==========================================
# SiLU/Swish:EfficientNet 使用
# ==========================================
silu = nn.SiLU()
print(silu(x))
# ==========================================
# Softmax:多分类输出层,将输出变成概率
# ==========================================
softmax = nn.Softmax(dim=1)
logits = torch.tensor([[2.0, 1.0, 0.1]])
probs = softmax(logits)
print(probs) # tensor([[0.6590, 0.2424, 0.0986]]) 加起来等于1
print(probs.sum()) # tensor(1.0000)
model = nn.Sequential(
nn.Linear(10, 20),
nn.ReLU(),
nn.Linear(20, 5)
)
# ==========================================
# 查看所有参数
# ==========================================
for name, param in model.named_parameters():
print(f"{name}: shape={param.shape}, requires_grad={param.requires_grad}")
# 0.weight: shape=torch.Size([20, 10]), requires_grad=True
# 0.bias: shape=torch.Size([20]), requires_grad=True
# 2.weight: shape=torch.Size([5, 20]), requires_grad=True
# 2.bias: shape=torch.Size([5]), requires_grad=True
# ==========================================
# 统计参数量
# ==========================================
total_params = sum(p.numel() for p in model.parameters())
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"总参数量: {total_params:,}") # 325
print(f"可训练参数: {trainable_params:,}") # 325
# ==========================================
# 冻结某些层(不参与训练)
# ==========================================
for param in model[0].parameters(): # 冻结第一层
param.requires_grad = False
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f"冻结后可训练参数: {trainable_params:,}") # 125 (少了20*10+20=220)
# ==========================================
# 移到 GPU
# ==========================================
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
# 确认参数在哪个设备
for name, param in model.named_parameters():
print(f"{name}: {param.device}")
break
class Classifier(nn.Module):
def __init__(self, input_dim, hidden_dim, num_classes):
super().__init__()
self.network = nn.Sequential(
nn.Linear(input_dim, hidden_dim),
nn.BatchNorm1d(hidden_dim), # 批归一化:加速训练、稳定梯度
nn.ReLU(),
nn.Dropout(0.3), # Dropout:防止过拟合
nn.Linear(hidden_dim, hidden_dim // 2),
nn.BatchNorm1d(hidden_dim // 2),
nn.ReLU(),
nn.Dropout(0.3),
nn.Linear(hidden_dim // 2, num_classes)
)
def forward(self, x):
return self.network(x)
# 创建模型
model = Classifier(input_dim=100, hidden_dim=256, num_classes=10)
# 测试
x = torch.randn(32, 100) # 32个样本,每个100维
logits = model(x)
print(f"输出形状: {logits.shape}") # torch.Size([32, 10])
print(f"参数量: {sum(p.numel() for p in model.parameters()):,}")
理解 nn.Module 的生命周期有助于正确使用它:
Pythonclass MyModel(nn.Module):
def __init__(self):
super().__init__()
# 1. 定义层(只执行一次)
self.fc1 = nn.Linear(784, 256)
self.fc2 = nn.Linear(256, 10)
def forward(self, x):
# 2. 前向传播(每次调用 model(x) 都执行)
x = torch.relu(self.fc1(x))
return self.fc2(x)
# 使用流程
model = MyModel()
# 3. 移到设备
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
# 4. 训练模式
model.train()
# ... 训练代码 ...
# 5. 评估模式
model.eval()
# ... 评估代码 ...
# 6. 保存模型
torch.save(model.state_dict(), 'model.pth')
# 7. 加载模型
model.load_state_dict(torch.load('model.pth'))
def count_parameters(model):
"""统计模型参数量"""
total_params = sum(p.numel() for p in model.parameters())
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
return total_params, trainable_params
total, trainable = count_parameters(model)
print(f"总参数量: {total:,}")
print(f"可训练参数: {trainable:,}")
print(f"不可训练参数: {total - trainable:,}")
# 查看每层的参数量
for name, param in model.named_parameters():
print(f"{name}: {param.numel():,} parameters")
# 使用 torchinfo 查看模型结构(需要安装 torchinfo)
# pip install torchinfo
from torchinfo import summary
model = MyModel()
summary(model, input_size=(1, 784))
# 输出:
# =================================================================
# Layer (type:depth-idx) Param #
# =================================================================
# MyModel --
# ├─Linear: 1-1 200,960
# ├─Linear: 1-2 2,570
# =================================================================
# Total params: 203,530
# Trainable params: 203,530
# Non-trainable params: 0
损失函数就像"考试的扣分标准"。模型做一次预测,损失函数就告诉它"你错得有多离谱"。损失越小,模型越准。训练的目标就是让损失尽可能小。不同的任务用不同的损失函数,就像不同的考试有不同的评分标准。
用于预测连续值(如房价、温度、股票价格):
Pythonimport torch
import torch.nn as nn
# ==========================================
# 均方误差(MSE Loss):最常用的回归损失
# L = mean((y_pred - y_true)^2)
# ==========================================
mse_loss = nn.MSELoss()
pred = torch.tensor([2.5, 0.0, 2.0])
true = torch.tensor([3.0, -0.5, 2.0])
loss = mse_loss(pred, true)
print(f"MSE Loss: {loss.item():.4f}") # 0.1667
# 手动计算验证:mean((2.5-3)^2, (0-(-0.5))^2, (2-2)^2) = mean(0.25, 0.25, 0) = 0.1667
# ==========================================
# 平均绝对误差(MAE/L1 Loss):对异常值更鲁棒
# L = mean(|y_pred - y_true|)
# ==========================================
mae_loss = nn.L1Loss()
print(f"MAE Loss: {mae_loss(pred, true).item():.4f}") # 0.3333
# ==========================================
# Smooth L1 Loss(Huber Loss):结合 MSE 和 MAE 的优点
# 小误差用 MSE(平方),大误差用 MAE(绝对值)
# ==========================================
huber_loss = nn.SmoothL1Loss()
print(f"Huber Loss: {huber_loss(pred, true).item():.4f}")
# ==========================================
# Huber Loss vs MSE vs MAE 的选择
# ==========================================
# MSE:对大误差惩罚很重(平方),适合异常值少的数据
# MAE:对大误差和小误差一视同仁,适合有异常值的数据
# Huber:两者折中,推荐先试这个
# ==========================================
# BCELoss:二分类交叉熵
# 通常配合 Sigmoid 使用
# ==========================================
bce_loss = nn.BCELoss()
pred_prob = torch.tensor([0.9, 0.1, 0.8]) # Sigmoid 后的概率
true_label = torch.tensor([1.0, 0.0, 1.0])
loss = bce_loss(pred_prob, true_label)
print(f"BCE Loss: {loss.item():.4f}") # 0.1446
# ==========================================
# BCEWithLogitsLoss:内置 Sigmoid,更数值稳定(推荐!)
# ==========================================
bce_logits = nn.BCEWithLogitsLoss()
logits = torch.tensor([2.0, -2.0, 1.5]) # 原始输出,未经 Sigmoid
loss = bce_logits(logits, true_label)
print(f"BCE With Logits: {loss.item():.4f}")
# ==========================================
# 为什么推荐 BCEWithLogitsLoss?
# ==========================================
# BCELoss 需要你先手动 Sigmoid,但 Sigmoid 在极端值时
# 会产生非常接近 0 或 1 的数值,导致 log(0) = -inf
# BCEWithLogitsLoss 在内部用数学技巧避免了这个问题
# ==========================================
# CrossEntropyLoss:多分类的标准损失函数
# 内置了 Softmax,所以模型输出不需要经过 Softmax!
# ==========================================
ce_loss = nn.CrossEntropyLoss()
# logits:模型的原始输出(未经 Softmax)
logits = torch.tensor([[2.0, 1.0, 0.1], # 样本1:预测为类别0
[0.5, 2.5, 0.3]]) # 样本2:预测为类别1
labels = torch.tensor([0, 1]) # 真实类别(整数索引)
loss = ce_loss(logits, labels)
print(f"Cross Entropy Loss: {loss.item():.4f}") # 0.4170
# ==========================================
# 查看 Softmax 后的概率
# ==========================================
probs = torch.softmax(logits, dim=1)
print(f"概率分布: {probs}")
# tensor([[0.6590, 0.2424, 0.0986], 样本1:65.9% 是类别0
# [0.1419, 0.7810, 0.0771]]) 样本2:78.1% 是类别1
nn.CrossEntropyLoss 内部已经包含了 Softmax,所以模型输出不要经过 Softmax!如果你先 Softmax 再传入 CE Loss,相当于做了两次 Softmax,结果会不对。
# ❌ 错误
output = model(x)
output = torch.softmax(output, dim=1) # 不需要这行!
loss = nn.CrossEntropyLoss()(output, labels)
# ✅ 正确
output = model(x) # 直接用原始 logits
loss = nn.CrossEntropyLoss()(output, labels)
# ==========================================
# KL 散度:衡量两个分布的差异(VAE、知识蒸馏常用)
# ==========================================
kl_loss = nn.KLDivLoss(reduction='batchmean')
log_probs = torch.randn(3, 5) # log 概率
target_probs = torch.softmax(torch.randn(3, 5), dim=1)
loss = kl_loss(log_probs, target_probs)
# ==========================================
# Cosine Embedding Loss:衡量相似度(对比学习常用)
# ==========================================
cos_loss = nn.CosineEmbeddingLoss()
x1 = torch.randn(3, 10)
x2 = torch.randn(3, 10)
y = torch.tensor([1, -1, 1]) # 1=相似, -1=不相似
loss = cos_loss(x1, x2, y)
# ==========================================
# Triplet Loss:三元组损失(人脸识别常用)
# anchor 与 positive 的距离应小于 anchor 与 negative 的距离
# ==========================================
triplet_loss = nn.TripletMarginLoss(margin=1.0)
anchor = torch.randn(3, 10)
positive = torch.randn(3, 10)
negative = torch.randn(3, 10)
loss = triplet_loss(anchor, positive, negative)
# ==========================================
# 自定义损失函数
# ==========================================
class FocalLoss(nn.Module):
"""Focal Loss:处理类别不平衡问题(如目标检测中正负样本比例悬殊)"""
def __init__(self, alpha=0.25, gamma=2.0):
super().__init__()
self.alpha = alpha
self.gamma = gamma
def forward(self, logits, targets):
bce = nn.functional.binary_cross_entropy_with_logits(
logits, targets, reduction='none'
)
pt = torch.exp(-bce) # 预测正确的概率
focal_weight = self.alpha * (1 - pt) ** self.gamma
return (focal_weight * bce).mean()
focal = FocalLoss()
logits = torch.randn(8, 1)
targets = torch.tensor([[1.], [0.], [1.], [1.], [0.], [0.], [1.], [0.]])
print(f"Focal Loss: {focal(logits, targets).item():.4f}")
| 任务类型 | 推荐损失函数 | 说明 |
|---|---|---|
| 二分类 | BCEWithLogitsLoss |
内置 Sigmoid,数值稳定 |
| 多分类 | CrossEntropyLoss |
内置 Softmax,最常用 |
| 回归 | MSELoss 或 L1Loss |
MSE 对异常值敏感,L1 更稳健 |
| 回归(稳健) | SmoothL1Loss |
结合 MSE 和 L1 的优点 |
| 多标签分类 | BCEWithLogitsLoss |
每个标签独立判断 |
| 类别不平衡 | CrossEntropyLoss(weight=...) |
给少数类更大权重 |
| 对比学习 | TripletMarginLoss |
拉近相似样本,推远不相似样本 |
class FocalLoss(nn.Module):
"""Focal Loss:处理类别不平衡问题"""
def __init__(self, alpha=0.25, gamma=2.0):
super().__init__()
self.alpha = alpha
self.gamma = gamma
def forward(self, logits, targets):
bce = F.binary_cross_entropy_with_logits(logits, targets, reduction='none')
pt = torch.exp(-bce)
focal_weight = self.alpha * (1 - pt) ** self.gamma
return (focal_weight * bce).mean()
class DiceLoss(nn.Module):
"""Dice Loss:图像分割常用"""
def __init__(self, smooth=1.0):
super().__init__()
self.smooth = smooth
def forward(self, predictions, targets):
predictions = torch.sigmoid(predictions)
intersection = (predictions * targets).sum()
dice = (2. * intersection + self.smooth) / (predictions.sum() + targets.sum() + self.smooth)
return 1 - dice
优化器就像"下山的向导"。损失函数告诉你"你在山上的哪个位置",梯度告诉你"哪个方向是下坡",而优化器决定"每一步走多远"(学习率)以及"怎么走更稳"(动量等技巧)。选错向导可能走得很慢,甚至走错方向。
import torch.optim as optim
model = nn.Linear(10, 2)
# ==========================================
# 基本 SGD:最简单的优化器
# 参数更新:param = param - lr * grad
# ==========================================
optimizer = optim.SGD(model.parameters(), lr=0.01)
# ==========================================
# 带动量的 SGD:更快收敛
# 动量就像"惯性",让优化器在平坦区域跑得更快
# ==========================================
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
# ==========================================
# 带 Nesterov 动量的 SGD:更精准
# ==========================================
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9, nesterov=True)
# ==========================================
# 带权重衰减(L2正则化)的 SGD
# ==========================================
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9, weight_decay=1e-4)
# ==========================================
# Adam:自适应学习率,几乎不需要调参
# 结合了动量(Momentum)和自适应学习率(RMSProp)的优点
# ==========================================
optimizer = optim.Adam(model.parameters(), lr=1e-3)
# ==========================================
# AdamW:Adam + 解耦的权重衰减
# Transformer 训练首选!比 Adam + weight_decay 更好
# ==========================================
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
# ==========================================
# Adam 的参数
# ==========================================
optimizer = optim.Adam(
model.parameters(),
lr=1e-3, # 学习率(最重要的参数)
betas=(0.9, 0.999), # 动量的衰减率(一般不用改)
eps=1e-8, # 防止除零(一般不用改)
weight_decay=0 # L2 正则化系数
)
记忆:不知道用什么就用 Adam。
学习率不是一成不变的。通常开始时大一点(快速学习),后期小一点(精细调整)。就像开车:上高速时油门踩到底,快到目的地时要减速。
Pythonoptimizer = optim.Adam(model.parameters(), lr=0.01)
# ==========================================
# StepLR:每 N 个 epoch 学习率变为原来的 gamma 倍
# ==========================================
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)
# epoch 0-29: lr=0.01, epoch 30-59: lr=0.001, epoch 60+: lr=0.0001
# ==========================================
# MultiStepLR:在指定的 epoch 降低
# ==========================================
scheduler = optim.lr_scheduler.MultiStepLR(optimizer, milestones=[30, 60, 90], gamma=0.1)
# ==========================================
# ExponentialLR:每个 epoch 都乘以 gamma
# ==========================================
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=0.95)
# ==========================================
# CosineAnnealingLR:余弦退火(最常用!)
# 学习率按余弦曲线下降,从 max_lr → min_lr
# ==========================================
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100, eta_min=1e-6)
# ==========================================
# ReduceLROnPlateau:根据验证指标调整
# 当指标不再提升时,自动降低学习率
# ==========================================
scheduler = optim.lr_scheduler.ReduceLROnPlateau(
optimizer, mode='min', patience=10, factor=0.1
)
# 使用时需要传入验证损失
# scheduler.step(val_loss)
# ==========================================
# OneCycleLR:一周期学习率策略(训练快速收敛的利器)
# 先升后降,配合 SGD 效果很好
# ==========================================
scheduler = optim.lr_scheduler.OneCycleLR(
optimizer, max_lr=0.01, total_steps=1000
)
import math
class WarmupCosineScheduler:
"""Warmup + 余弦退火:Transformer 训练的标准学习率策略"""
def __init__(self, optimizer, warmup_steps, total_steps):
self.optimizer = optimizer
self.warmup_steps = warmup_steps
self.total_steps = total_steps
self.current_step = 0
def step(self):
self.current_step += 1
# Warmup 阶段:线性增加
if self.current_step < self.warmup_steps:
lr_scale = self.current_step / self.warmup_steps
# 余弦退火阶段:按余弦曲线下降
else:
progress = (self.current_step - self.warmup_steps) / (self.total_steps - self.warmup_steps)
lr_scale = 0.5 * (1 + math.cos(math.pi * progress))
for param_group in self.optimizer.param_groups:
param_group['lr'] = param_group['initial_lr'] * lr_scale
# 使用
optimizer = optim.AdamW(model.parameters(), lr=1e-4)
for pg in optimizer.param_groups:
pg['initial_lr'] = 1e-4
scheduler = WarmupCosineScheduler(optimizer, warmup_steps=1000, total_steps=10000)
# 在训练循环中
for step in range(10000):
train_step(...)
scheduler.step()
# 标准训练循环
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
for epoch in range(100):
# ---- 训练 ----
model.train()
for batch_data, batch_labels in train_loader:
batch_data = batch_data.to(device)
batch_labels = batch_labels.to(device)
optimizer.zero_grad() # 1. 清零梯度
outputs = model(batch_data) # 2. 前向传播
loss = criterion(outputs, batch_labels) # 3. 计算损失
loss.backward() # 4. 反向传播
optimizer.step() # 5. 更新参数
scheduler.step() # 6. 更新学习率
# 打印当前学习率
current_lr = optimizer.param_groups[0]['lr']
print(f"Epoch {epoch}, LR: {current_lr:.6f}")
# 迁移学习时,预训练层用小学习率,新层用大学习率
# 这样可以保护已经学到的特征,同时让新层快速学习
optimizer = optim.Adam([
{'params': model.backbone.parameters(), 'lr': 1e-5}, # 预训练层:小学习率
{'params': model.classifier.parameters(), 'lr': 1e-3}, # 新层:大学习率
])
# 也可以用参数组的名字
optimizer = optim.AdamW([
{'params': model.layer1.parameters(), 'lr': 1e-5},
{'params': model.layer2.parameters(), 'lr': 1e-5},
{'params': model.layer3.parameters(), 'lr': 1e-4},
{'params': model.layer4.parameters(), 'lr': 1e-4},
{'params': model.fc.parameters(), 'lr': 1e-3},
])
def compare_optimizers(model_class, train_loader, test_loader, epochs=20):
"""对比不同优化器的性能"""
optimizers = {
'SGD': lambda p: optim.SGD(p, lr=0.01, momentum=0.9),
'Adam': lambda p: optim.Adam(p, lr=1e-3),
'AdamW': lambda p: optim.AdamW(p, lr=1e-3, weight_decay=0.01),
}
results = {}
for name, opt_fn in optimizers.items():
model = model_class().to(device)
optimizer = opt_fn(model.parameters())
criterion = nn.CrossEntropyLoss()
train_losses, test_accs = [], []
for epoch in range(epochs):
model.train()
for data, target in train_loader:
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
loss = criterion(model(data), target)
loss.backward()
optimizer.step()
train_losses.append(loss.item())
model.eval()
correct, total = 0, 0
with torch.no_grad():
for data, target in test_loader:
data, target = data.to(device), target.to(device)
correct += model(data).argmax(1).eq(target).sum().item()
total += target.size(0)
test_accs.append(100. * correct / total)
results[name] = {'losses': train_losses, 'accs': test_accs}
print(f"{name}: 最终准确率 {test_accs[-1]:.2f}%")
return results
import matplotlib.pyplot as plt
def plot_lr_schedulers():
"""可视化不同学习率调度器"""
model = nn.Linear(10, 2)
schedulers = {
'StepLR': optim.lr_scheduler.StepLR(optim.SGD(model.parameters(), lr=0.1), step_size=30, gamma=0.1),
'CosineAnnealing': optim.lr_scheduler.CosineAnnealingLR(optim.SGD(model.parameters(), lr=0.1), T_max=100),
'ExponentialLR': optim.lr_scheduler.ExponentialLR(optim.SGD(model.parameters(), lr=0.1), gamma=0.95),
}
plt.figure(figsize=(10, 6))
for name, scheduler in schedulers.items():
lrs = []
for _ in range(100):
lrs.append(scheduler.get_last_lr()[0])
scheduler.step()
plt.plot(lrs, label=name)
plt.xlabel('Epoch')
plt.ylabel('Learning Rate')
plt.legend()
plt.title('Learning Rate Schedulers')
plt.savefig('lr_schedulers.png', dpi=150)
想象你开了一家餐厅。Dataset 就是你的"菜单"——定义了有哪些菜品(数据)以及怎么获取(读取方式)。DataLoader 就是你的"服务员"——按照菜单批量上菜(批量加载),还能多线程同时端(多进程预取),甚至可以把菜提前热好(pin_memory 加速 GPU 传输)。
from torchvision import datasets, transforms
# ==========================================
# 加载 MNIST 手写数字数据集
# ==========================================
transform = transforms.Compose([
transforms.ToTensor(), # 转为张量,归一化到 [0, 1]
transforms.Normalize((0.1307,), (0.3081,)) # 标准化(MNIST 的均值和标准差)
])
train_dataset = datasets.MNIST(
root='./data', # 数据存放路径
train=True, # 训练集
download=True, # 如果没有就自动下载
transform=transform # 数据预处理
)
test_dataset = datasets.MNIST(
root='./data',
train=False,
transform=transform
)
print(f"训练集大小: {len(train_dataset)}") # 60000
print(f"测试集大小: {len(test_dataset)}") # 10000
# 查看一个样本
img, label = train_dataset[0]
print(f"图片形状: {img.shape}") # torch.Size([1, 28, 28])
print(f"标签: {label}") # 5
# ==========================================
# 其他内置数据集
# ==========================================
# CIFAR-10:10类彩色图片
cifar10 = datasets.CIFAR10('./data', train=True, download=True, transform=transform)
# CIFAR-100:100类彩色图片
cifar100 = datasets.CIFAR100('./data', train=True, download=True, transform=transform)
# ImageFolder:按文件夹分类的通用数据集
# 假设目录结构:data/train/cats/xxx.jpg, data/train/dogs/xxx.jpg
# dataset = datasets.ImageFolder('./data/train', transform=transform)
from torch.utils.data import Dataset
from PIL import Image
import os
class CustomImageDataset(Dataset):
"""自定义图片数据集"""
def __init__(self, root_dir, transform=None):
"""
Args:
root_dir: 图片目录
transform: 数据预处理/增强
"""
self.root_dir = root_dir
self.transform = transform
self.image_files = [f for f in os.listdir(root_dir)
if f.endswith(('.jpg', '.png', '.jpeg'))]
def __len__(self):
"""返回数据集大小(必须实现)"""
return len(self.image_files)
def __getitem__(self, idx):
"""根据索引获取一个样本(必须实现)
返回:(数据, 标签)
"""
# 1. 读取图片
img_path = os.path.join(self.root_dir, self.image_files[idx])
image = Image.open(img_path).convert('RGB')
# 2. 获取标签(假设文件名格式为 "0_xxx.jpg")
label = int(self.image_files[idx].split('_')[0])
# 3. 数据预处理
if self.transform:
image = self.transform(image)
return image, label
# 使用
transform = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225])
])
dataset = CustomImageDataset(root_dir='./images', transform=transform)
print(f"数据集大小: {len(dataset)}")
img, label = dataset[0]
print(f"图片形状: {img.shape}, 标签: {label}")
__init__:初始化,加载数据路径、读取标签文件等__len__:返回数据集大小__getitem__:根据索引返回一个样本 (数据, 标签)PyTorch 会自动调用这些方法,你只需要按格式写就行。
from torch.utils.data import DataLoader
train_loader = DataLoader(
train_dataset,
batch_size=64, # 每批 64 个样本(最重要的参数)
shuffle=True, # 打乱顺序(训练集必须打乱!)
num_workers=4, # 4 个进程并行加载(Windows 可能需要设为 0)
pin_memory=True, # 加速 GPU 传输(推荐)
drop_last=True, # 丢弃最后不足一个 batch 的数据
persistent_workers=True, # 保持 worker 进程存活(加速)
)
test_loader = DataLoader(
test_dataset,
batch_size=1000, # 测试时可以用更大的 batch
shuffle=False, # 测试集不需要打乱
num_workers=2,
)
# ==========================================
# 使用 DataLoader
# ==========================================
for batch_idx, (data, target) in enumerate(train_loader):
print(f"Batch {batch_idx}:")
print(f" data shape = {data.shape}") # [64, 1, 28, 28]
print(f" target shape = {target.shape}") # [64]
if batch_idx >= 2:
break
# 查看总共有多少个 batch
print(f"总 batch 数: {len(train_loader)}") # 60000 / 64 = 937
数据增强通过对训练图片做随机变换,人为增加数据量,防止过拟合。就像你把一张照片旋转、翻转、裁剪,就得到了好几张"不同"的照片。
Python# ==========================================
# 训练时:强数据增强
# ==========================================
train_transform = transforms.Compose([
transforms.RandomResizedCrop(224), # 随机裁剪并缩放到 224x224
transforms.RandomHorizontalFlip(p=0.5), # 随机水平翻转
transforms.RandomVerticalFlip(p=0.1), # 随机垂直翻转(概率小)
transforms.RandomRotation(15), # 随机旋转 ±15°
transforms.ColorJitter( # 颜色抖动
brightness=0.2, contrast=0.2,
saturation=0.2, hue=0.1
),
transforms.RandomAffine( # 随机仿射变换
degrees=0, translate=(0.1, 0.1)
),
transforms.RandomGrayscale(p=0.1), # 随机灰度化
transforms.ToTensor(),
transforms.Normalize(
mean=[0.485, 0.456, 0.406], # ImageNet 的均值
std=[0.229, 0.224, 0.225] # ImageNet 的标准差
),
transforms.RandomErasing(p=0.2), # 随机擦除一块区域
])
# ==========================================
# 测试时:不做增强,只做必要的预处理
# ==========================================
test_transform = transforms.Compose([
transforms.Resize(256), # 先缩放到 256
transforms.CenterCrop(224), # 再中心裁剪到 224
transforms.ToTensor(),
transforms.Normalize(
mean=[0.485, 0.456, 0.406],
std=[0.229, 0.224, 0.225]
)
])
测试/推理时只需要做 Resize + CenterCrop + Normalize,不要做随机变换,否则每次预测结果都不一样。
from torch.utils.data import WeightedRandomSampler
# 处理类别不平衡:少数类采样更多
class_counts = [100, 900] # 类别0有100个,类别1有900个
weights = [1.0 / count for count in class_counts]
sample_weights = [weights[label] for _, label in dataset]
sampler = WeightedRandomSampler(sample_weights, num_samples=len(dataset))
balanced_loader = DataLoader(dataset, batch_size=32, sampler=sampler)
# 高性能 DataLoader 配置
train_loader = DataLoader(
train_dataset,
batch_size=64,
shuffle=True,
num_workers=4, # 4个并行加载进程
pin_memory=True, # 加速 GPU 传输
persistent_workers=True, # 保持 worker 存活
prefetch_factor=2, # 每个 worker 预取 2 个 batch
drop_last=True, # 丢弃最后不完整的 batch
)
from torch.utils.data import WeightedRandomSampler
# 计算每个类别的权重
class_counts = [100, 900] # 类别0有100个,类别1有900个
class_weights = [1.0 / count for count in class_counts]
sample_weights = [class_weights[label] for _, label in dataset]
# 创建采样器
sampler = WeightedRandomSampler(sample_weights, num_samples=len(dataset))
# 使用采样器(注意:使用 sampler 时不能设置 shuffle=True)
balanced_loader = DataLoader(dataset, batch_size=32, sampler=sampler)
现在我们把前面学到的所有知识串起来,从零实现一个完整的深度学习项目:用神经网络识别手写数字(0-9)。这是深度学习的"Hello World"。
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
import time
# ==================== 1. 准备数据 ====================
print("=" * 50)
print("1. 准备数据")
print("=" * 50)
transform = transforms.Compose([
transforms.ToTensor(), # [0, 255] → [0, 1]
transforms.Normalize((0.1307,), (0.3081,)) # 标准化
])
train_dataset = datasets.MNIST('./data', train=True, download=True, transform=transform)
test_dataset = datasets.MNIST('./data', train=False, transform=transform)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True,
num_workers=2, pin_memory=True)
test_loader = DataLoader(test_dataset, batch_size=1000, shuffle=False)
print(f"训练集: {len(train_dataset)} 张图片")
print(f"测试集: {len(test_dataset)} 张图片")
print(f"每批大小: 64, 总批数: {len(train_loader)}")
# ==================== 2. 定义模型 ====================
print("\n" + "=" * 50)
print("2. 定义模型")
print("=" * 50)
class MNISTNet(nn.Module):
def __init__(self):
super().__init__()
self.flatten = nn.Flatten()
self.network = nn.Sequential(
nn.Linear(28 * 28, 512), # 输入层 → 隐藏层1
nn.ReLU(),
nn.Dropout(0.2),
nn.Linear(512, 256), # 隐藏层1 → 隐藏层2
nn.ReLU(),
nn.Dropout(0.2),
nn.Linear(256, 10) # 隐藏层2 → 输出层(10个类别)
)
def forward(self, x):
x = self.flatten(x) # [B, 1, 28, 28] → [B, 784]
return self.network(x)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = MNISTNet().to(device)
print(f"使用设备: {device}")
print(f"模型参数量: {sum(p.numel() for p in model.parameters()):,}")
print(model)
# ==================== 3. 损失函数和优化器 ====================
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=10)
# ==================== 4. 训练函数 ====================
def train_one_epoch(model, loader, criterion, optimizer, device):
model.train()
total_loss = 0
correct = 0
total = 0
for batch_idx, (data, target) in enumerate(loader):
data, target = data.to(device), target.to(device)
# 前向传播
output = model(data)
loss = criterion(output, target)
# 反向传播
optimizer.zero_grad()
loss.backward()
optimizer.step()
# 统计
total_loss += loss.item() * data.size(0)
pred = output.argmax(dim=1)
correct += pred.eq(target).sum().item()
total += target.size(0)
# 每 200 个 batch 打印一次
if (batch_idx + 1) % 200 == 0:
print(f" Batch {batch_idx+1}/{len(loader)} | "
f"Loss: {loss.item():.4f} | "
f"Acc: {100.*correct/total:.2f}%")
return total_loss / total, 100. * correct / total
# ==================== 5. 评估函数 ====================
def evaluate(model, loader, criterion, device):
model.eval()
test_loss = 0
correct = 0
total = 0
with torch.no_grad():
for data, target in loader:
data, target = data.to(device), target.to(device)
output = model(data)
test_loss += criterion(output, target).item() * data.size(0)
pred = output.argmax(dim=1)
correct += pred.eq(target).sum().item()
total += target.size(0)
return test_loss / total, 100. * correct / total
# ==================== 6. 开始训练 ====================
print("\n" + "=" * 50)
print("3. 开始训练")
print("=" * 50)
train_losses, test_losses = [], []
train_accs, test_accs = [], []
best_acc = 0
for epoch in range(10):
start_time = time.time()
train_loss, train_acc = train_one_epoch(
model, train_loader, criterion, optimizer, device
)
test_loss, test_acc = evaluate(model, test_loader, criterion, device)
scheduler.step()
elapsed = time.time() - start_time
train_losses.append(train_loss)
test_losses.append(test_loss)
train_accs.append(train_acc)
test_accs.append(test_acc)
# 保存最佳模型
if test_acc > best_acc:
best_acc = test_acc
torch.save(model.state_dict(), 'best_mnist_model.pth')
print(f"Epoch {epoch+1}/10 | "
f"Train Loss: {train_loss:.4f} | Train Acc: {train_acc:.2f}% | "
f"Test Loss: {test_loss:.4f} | Test Acc: {test_acc:.2f}% | "
f"Time: {elapsed:.1f}s")
print(f"\n最佳测试准确率: {best_acc:.2f}%")
# ==================== 7. 画图 ====================
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
ax1.plot(train_losses, label='Train', marker='o')
ax1.plot(test_losses, label='Test', marker='s')
ax1.set_xlabel('Epoch')
ax1.set_ylabel('Loss')
ax1.legend()
ax1.set_title('Loss 曲线')
ax1.grid(True, alpha=0.3)
ax2.plot(train_accs, label='Train', marker='o')
ax2.plot(test_accs, label='Test', marker='s')
ax2.set_xlabel('Epoch')
ax2.set_ylabel('Accuracy (%)')
ax2.legend()
ax2.set_title('准确率曲线')
ax2.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('training_curves.png', dpi=150, bbox_inches='tight')
plt.show()
==================================================
1. 准备数据
==================================================
训练集: 60000 张图片
测试集: 10000 张图片
每批大小: 64, 总批数: 938
==================================================
2. 定义模型
==================================================
使用设备: cuda
模型参数量: 535,818
==================================================
3. 开始训练
==================================================
Epoch 1/10 | Train Loss: 0.2134 | Train Acc: 93.52% | Test Loss: 0.0892 | Test Acc: 97.18%
Epoch 2/10 | Train Loss: 0.0876 | Train Acc: 97.31% | Test Loss: 0.0723 | Test Acc: 97.72%
Epoch 3/10 | Train Loss: 0.0632 | Train Acc: 98.04% | Test Loss: 0.0658 | Test Acc: 97.95%
...
Epoch 10/10 | Train Loss: 0.0198 | Train Acc: 99.42% | Test Loss: 0.0681 | Test Acc: 98.35%
最佳测试准确率: 98.35%
仅仅 100 多行代码,就实现了一个 98%+ 准确率的手写数字识别器。这就是 PyTorch 的魅力——简洁、直观、强大。
model.train():训练模式,Dropout 随机关闭神经元,BatchNorm 用当前 batch 的统计量model.eval():评估模式,Dropout 关闭(全部神经元都用),BatchNorm 用全局统计量忘记切换模式是新手最常见的错误之一!
# 正确的使用方式
model.train() # 开始训练前
for data, target in train_loader:
...
model.eval() # 开始评估前
with torch.no_grad():
for data, target in test_loader:
...
from PIL import Image
import torchvision.transforms as transforms
# 加载训练好的模型
model.load_state_dict(torch.load('best_mnist_model.pth'))
model.eval()
# 加载一张图片
img = Image.open('test_digit.png').convert('L') # 灰度图
transform = transforms.Compose([
transforms.Resize((28, 28)),
transforms.ToTensor(),
transforms.Normalize((0.1307,), (0.3081,))
])
img_tensor = transform(img).unsqueeze(0).to(device) # [1, 1, 28, 28]
# 预测
with torch.no_grad():
output = model(img_tensor)
probs = torch.softmax(output, dim=1)
pred = output.argmax(dim=1).item()
confidence = probs[0][pred].item()
print(f"预测结果: {pred}")
print(f"置信度: {confidence:.4f}")
print(f"各类别概率: {probs[0].tolist()}")
from sklearn.metrics import classification_report, confusion_matrix
import seaborn as sns
def evaluate_model(model, test_loader, device):
"""完整评估模型"""
model.eval()
all_preds, all_labels = [], []
with torch.no_grad():
for data, target in test_loader:
data = data.to(device)
output = model(data)
preds = output.argmax(dim=1).cpu()
all_preds.extend(preds.tolist())
all_labels.extend(target.tolist())
# 分类报告
print(classification_report(all_labels, all_preds))
# 混淆矩阵
cm = confusion_matrix(all_labels, all_preds)
plt.figure(figsize=(10, 8))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues')
plt.xlabel('预测')
plt.ylabel('真实')
plt.title('混淆矩阵')
plt.savefig('confusion_matrix.png', dpi=150)
plt.show()
return all_preds, all_labels
def predict_single_image(model, image_path, transform, device):
"""预测单张图片"""
from PIL import Image
# 加载图片
image = Image.open(image_path).convert('L') # 灰度图
image_tensor = transform(image).unsqueeze(0).to(device)
# 预测
model.eval()
with torch.no_grad():
output = model(image_tensor)
probs = torch.softmax(output, dim=1)
pred = probs.argmax(dim=1).item()
confidence = probs[0][pred].item()
# 显示结果
plt.figure(figsize=(6, 6))
plt.imshow(image, cmap='gray')
plt.title(f'预测: {pred}, 置信度: {confidence:.4f}')
plt.axis('off')
plt.show()
return pred, confidence
def plot_training_curves(train_losses, test_losses, train_accs, test_accs):
"""绘制训练曲线"""
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))
# Loss 曲线
ax1.plot(train_losses, label='Train Loss', marker='o', markersize=3)
ax1.plot(test_losses, label='Test Loss', marker='s', markersize=3)
ax1.set_xlabel('Epoch')
ax1.set_ylabel('Loss')
ax1.legend()
ax1.set_title('Loss 曲线')
ax1.grid(True, alpha=0.3)
# 准确率曲线
ax2.plot(train_accs, label='Train Acc', marker='o', markersize=3)
ax2.plot(test_accs, label='Test Acc', marker='s', markersize=3)
ax2.set_xlabel('Epoch')
ax2.set_ylabel('Accuracy (%)')
ax2.legend()
ax2.set_title('准确率曲线')
ax2.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('training_curves.png', dpi=150, bbox_inches='tight')
plt.show()
想象你看一张照片。你不是一次性看所有像素,而是先看局部特征(边缘、纹理),再组合成更大的特征(眼睛、鼻子),最后识别出"这是一张脸"。CNN 就模仿了这个过程——用卷积核在图片上滑动,逐步提取从低级到高级的特征。
全连接网络处理图片有两大问题:
CNN 的三大优势解决了这些问题:
卷积就像用一个小手电筒(卷积核)在图片上照。手电筒每次照亮一小块区域(比如 3×3),然后计算这个区域的"特征得分"。照完一个位置,往右移一步(stride),继续照。照完一行,往下移一行。最终,你就得到了一张"特征图",上面的每个点都表示原图对应区域的特征强度。
import torch
import torch.nn as nn
# 手动演示卷积操作
# 输入:1张 1通道 5x5 的图片
input_img = torch.tensor([
[1, 0, 1, 0, 1],
[0, 1, 0, 1, 0],
[1, 0, 1, 0, 1],
[0, 1, 0, 1, 0],
[1, 0, 1, 0, 1],
], dtype=torch.float32).unsqueeze(0).unsqueeze(0) # [1, 1, 5, 5]
# 卷积核:3x3 的"边缘检测器"
kernel = torch.tensor([
[1, 0, -1],
[1, 0, -1],
[1, 0, -1],
], dtype=torch.float32).unsqueeze(0).unsqueeze(0) # [1, 1, 3, 3]
# 做卷积
conv = nn.Conv2d(1, 1, 3, bias=False)
conv.weight.data = kernel
output = conv(input_img)
print(f"输入形状: {input_img.shape}") # [1, 1, 5, 5]
print(f"输出形状: {output.shape}") # [1, 1, 3, 3]
print(f"输出:\n{output.squeeze()}")
# nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
conv = nn.Conv2d(
in_channels=3, # 输入通道数(RGB=3,灰度=1)
out_channels=16, # 输出通道数 = 卷积核个数
kernel_size=3, # 卷积核大小 3x3
stride=1, # 步长(每次移动几格)
padding=1 # 填充(在边缘补零)
)
# 输入:batch_size=4, 3通道(RGB), 高32, 宽32
x = torch.randn(4, 3, 32, 32)
y = conv(x)
print(f"输入: {x.shape}") # [4, 3, 32, 32]
print(f"输出: {y.shape}") # [4, 16, 32, 32]
# 参数量 = in_channels * out_channels * kernel_size^2 + out_channels(bias)
# = 3 * 16 * 3 * 3 + 16 = 448
print(f"参数量: {sum(p.numel() for p in conv.parameters()):,}")
| 参数 | 含义 | 常用值 | 说明 |
|---|---|---|---|
in_channels |
输入通道数 | 3, 1, 64, 128 | RGB=3,灰度=1,特征图=上一层的out_channels |
out_channels |
输出通道数 | 16, 32, 64, 128, 256 | = 卷积核个数,越多特征越丰富但计算量越大 |
kernel_size |
卷积核大小 | 3(最常用), 5, 7 | 3×3 是黄金尺寸,VGG 证明了两个 3×3 等于一个 5×5 |
stride |
步长 | 1(默认), 2 | stride=2 时输出尺寸减半(下采样) |
padding |
边缘填充 | 0, 1, 2 | padding=1 + kernel=3 + stride=1 → 尺寸不变 |
groups |
分组卷积 | 1, 2, in_channels | groups=in_channels 时为深度可分离卷积(MobileNet) |
dilation |
膨胀卷积 | 1, 2 | 增大感受野而不增加参数 |
输出尺寸 = (输入尺寸 - 卷积核大小 + 2 × padding) / stride + 1
例:输入 32,kernel=3,padding=1,stride=1 → (32 - 3 + 2) / 1 + 1 = 32(尺寸不变)
例:输入 32,kernel=3,padding=0,stride=2 → (32 - 3 + 0) / 2 + 1 = 15(尺寸减半)
例:输入 224,kernel=7,padding=3,stride=2 → (224 - 7 + 6) / 2 + 1 = 112(ResNet 第一层)
池化层用于缩小特征图尺寸,减少计算量,同时增加平移不变性。
Python# 最大池化:取区域内的最大值(最常用)
max_pool = nn.MaxPool2d(kernel_size=2, stride=2)
x = torch.randn(1, 16, 32, 32)
y = max_pool(x)
print(y.shape) # [1, 16, 16, 16] 尺寸减半
# 平均池化:取区域内的平均值
avg_pool = nn.AvgPool2d(kernel_size=2, stride=2)
# 自适应平均池化(不管输入多大,输出固定大小)
adaptive_pool = nn.AdaptiveAvgPool2d(output_size=(1, 1))
x = torch.randn(1, 512, 7, 7)
y = adaptive_pool(x)
print(y.shape) # [1, 512, 1, 1]
y = y.flatten(1)
print(y.shape) # [1, 512] → 可以接全连接层了
class LeNet(nn.Module):
def __init__(self):
super().__init__()
self.features = nn.Sequential(
nn.Conv2d(1, 6, 5), # 1x32x32 → 6x28x28
nn.Tanh(),
nn.AvgPool2d(2, 2), # 6x28x28 → 6x14x14
nn.Conv2d(6, 16, 5), # 6x14x14 → 16x10x10
nn.Tanh(),
nn.AvgPool2d(2, 2), # 16x10x10 → 16x5x5
)
self.classifier = nn.Sequential(
nn.Linear(16 * 5 * 5, 120),
nn.Tanh(),
nn.Linear(120, 84),
nn.Tanh(),
nn.Linear(84, 10)
)
def forward(self, x):
x = self.features(x)
x = x.view(x.size(0), -1)
return self.classifier(x)
def vgg_block(in_ch, out_ch, num_convs):
layers = []
for i in range(num_convs):
layers += [nn.Conv2d(in_ch if i==0 else out_ch, out_ch, 3, padding=1), nn.ReLU()]
layers += [nn.MaxPool2d(2, 2)]
return nn.Sequential(*layers)
class VGG16(nn.Module):
def __init__(self, num_classes=1000):
super().__init__()
self.features = nn.Sequential(
vgg_block(3, 64, 2), # 224→112
vgg_block(64, 128, 2), # 112→56
vgg_block(128, 256, 3), # 56→28
vgg_block(256, 512, 3), # 28→14
vgg_block(512, 512, 3), # 14→7
)
self.classifier = nn.Sequential(
nn.Linear(512*7*7, 4096), nn.ReLU(), nn.Dropout(0.5),
nn.Linear(4096, 4096), nn.ReLU(), nn.Dropout(0.5),
nn.Linear(4096, num_classes),
)
def forward(self, x):
x = self.features(x)
return self.classifier(x.view(x.size(0), -1))
网络越深,训练越难。ResNet 通过跳跃连接让梯度直接流过,使得训练 100+ 层网络成为可能。
想象你要学一个很难的新技能。如果老师说"你先学和你已经会的东西的区别(残差)",那就容易多了。ResNet 就是让网络学习"输入和输出的差" F(x) = H(x) - x,而不是直接学 H(x)。如果某一层不需要变换,它只需学 F(x)=0,这比学恒等映射容易得多。
class ResidualBlock(nn.Module):
def __init__(self, channels):
super().__init__()
self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
self.bn1 = nn.BatchNorm2d(channels)
self.relu = nn.ReLU()
self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
self.bn2 = nn.BatchNorm2d(channels)
def forward(self, x):
residual = x
out = self.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
out = out + residual # 残差连接!
return self.relu(out)
block = ResidualBlock(64)
x = torch.randn(1, 64, 32, 32)
y = block(x)
print(y.shape) # [1, 64, 32, 32]
想象你看一张照片。你不是一次性看所有像素,而是先看局部特征(边缘、纹理),再组合成更大的特征(眼睛、鼻子),最后识别出"这是一张脸"。CNN 就模仿了这个过程——用卷积核在图片上滑动,逐步提取从低级到高级的特征。
全连接网络处理图片的问题:
CNN 的三大优势:
卷积操作就是用一个小的"卷积核"(如 3×3)在图片上滑动,每个位置做一次点积。
Pythonimport torch
import torch.nn as nn
# nn.Conv2d(in_channels, out_channels, kernel_size, stride, padding)
conv = nn.Conv2d(
in_channels=3, # 输入通道数(RGB=3)
out_channels=16, # 输出通道数 = 卷积核个数
kernel_size=3, # 卷积核大小 3x3
stride=1, # 步长
padding=1 # 填充(保持尺寸不变)
)
# 输入:batch_size=1, 3通道(RGB), 高32, 宽32
x = torch.randn(1, 3, 32, 32)
y = conv(x)
print(y.shape) # torch.Size([1, 16, 32, 32])
# 16个卷积核 → 16个特征图
# 参数量 = 3 * 3 * 3 * 16 + 16(偏置) = 448
print(f"参数量: {sum(p.numel() for p in conv.parameters())}")
| 参数 | 含义 | 常用值 | 说明 |
|---|---|---|---|
in_channels |
输入通道数 | 3, 1, 64, 128... | RGB=3,灰度=1,特征图=上一层的out_channels |
out_channels |
输出通道数 | 16, 32, 64, 128... | = 卷积核个数,越多特征越丰富但计算量越大 |
kernel_size |
卷积核大小 | 3(最常用), 5, 7 | 3x3 是最常用的,感受野小但层数可以加深 |
stride |
步长 | 1(默认), 2 | stride=2 时输出尺寸减半(下采样) |
padding |
边缘填充 | 0, 1, 2 | padding=1 + kernel=3 + stride=1 → 尺寸不变 |
输出尺寸 = (输入尺寸 - 卷积核大小 + 2 × padding) / stride + 1
例:输入 32,kernel=3,padding=1,stride=1 → (32 - 3 + 2) / 1 + 1 = 32(尺寸不变)
例:输入 32,kernel=3,padding=0,stride=2 → (32 - 3 + 0) / 2 + 1 = 15(尺寸减半)
例:输入 224,kernel=7,padding=3,stride=2 → (224 - 7 + 6) / 2 + 1 = 112(ResNet 第一层)
池化层用于缩小特征图尺寸,减少计算量,同时增加平移不变性。
Python# ==========================================
# 最大池化:取区域内的最大值(最常用)
# ==========================================
max_pool = nn.MaxPool2d(kernel_size=2, stride=2)
x = torch.randn(1, 16, 32, 32)
y = max_pool(x)
print(y.shape) # torch.Size([1, 16, 16, 16]) 尺寸减半
# ==========================================
# 平均池化:取区域内的平均值
# ==========================================
avg_pool = nn.AvgPool2d(kernel_size=2, stride=2)
# ==========================================
# 自适应平均池化(不管输入多大,输出固定大小)
# 这个非常有用!在分类网络的最后一层经常用
# ==========================================
adaptive_pool = nn.AdaptiveAvgPool2d(output_size=(1, 1))
x = torch.randn(1, 512, 7, 7) # 不管特征图多大
y = adaptive_pool(x)
print(y.shape) # torch.Size([1, 512, 1, 1]) 固定输出 1x1
y = y.flatten(1)
print(y.shape) # torch.Size([1, 512]) → 可以接全连接层了
# ==========================================
# 全局平均池化(GAP):替代全连接层,减少参数
# ==========================================
gap = nn.AdaptiveAvgPool2d(1)
class LeNet(nn.Module):
"""LeNet-5:最简单的 CNN,用于手写数字识别"""
def __init__(self):
super().__init__()
self.features = nn.Sequential(
nn.Conv2d(1, 6, 5), # 1x32x32 → 6x28x28
nn.Tanh(),
nn.AvgPool2d(2, 2), # 6x28x28 → 6x14x14
nn.Conv2d(6, 16, 5), # 6x14x14 → 16x10x10
nn.Tanh(),
nn.AvgPool2d(2, 2), # 16x10x10 → 16x5x5
)
self.classifier = nn.Sequential(
nn.Linear(16 * 5 * 5, 120),
nn.Tanh(),
nn.Linear(120, 84),
nn.Tanh(),
nn.Linear(84, 10)
)
def forward(self, x):
x = self.features(x)
x = x.view(x.size(0), -1) # 展平
x = self.classifier(x)
return x
model = LeNet()
print(f"参数量: {sum(p.numel() for p in model.parameters()):,}")
def make_vgg_block(in_ch, out_ch, num_convs):
"""VGG 的基本构建块:多个 3x3 卷积 + 最大池化"""
layers = []
for i in range(num_convs):
layers.append(nn.Conv2d(in_ch if i == 0 else out_ch, out_ch, 3, padding=1))
layers.append(nn.ReLU())
layers.append(nn.MaxPool2d(2, 2))
return nn.Sequential(*layers)
class VGG16(nn.Module):
def __init__(self, num_classes=1000):
super().__init__()
self.features = nn.Sequential(
make_vgg_block(3, 64, 2), # 224→112
make_vgg_block(64, 128, 2), # 112→56
make_vgg_block(128, 256, 3), # 56→28
make_vgg_block(256, 512, 3), # 28→14
make_vgg_block(512, 512, 3), # 14→7
)
self.classifier = nn.Sequential(
nn.Linear(512 * 7 * 7, 4096),
nn.ReLU(),
nn.Dropout(0.5),
nn.Linear(4096, 4096),
nn.ReLU(),
nn.Dropout(0.5),
nn.Linear(4096, num_classes),
)
def forward(self, x):
x = self.features(x)
x = x.view(x.size(0), -1)
x = self.classifier(x)
return x
网络越深,训练越难(梯度消失/爆炸)。ResNet 通过跳跃连接解决了这个问题,使得训练 100+ 层的网络成为可能。
想象你要学一个很难的新技能。如果老师让你直接学,你可能学不会。但如果老师说"你先学和你已经会的东西的区别(残差)",那就容易多了。ResNet 就是让网络学习"输入和输出的差",而不是直接学输出。如果某一层不需要做任何变换,它只需要学习 F(x)=0(残差为0),这比学习 F(x)=x(恒等映射)容易得多。
class ResidualBlock(nn.Module):
"""残差块:输出 = F(x) + x"""
def __init__(self, channels):
super().__init__()
self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
self.bn1 = nn.BatchNorm2d(channels)
self.relu = nn.ReLU()
self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
self.bn2 = nn.BatchNorm2d(channels)
def forward(self, x):
residual = x # 保存输入(恒等映射)
out = self.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
out = out + residual # 残差连接!
out = self.relu(out)
return out
# 使用
block = ResidualBlock(64)
x = torch.randn(1, 64, 32, 32)
y = block(x)
print(y.shape) # torch.Size([1, 64, 32, 32]) 形状不变
import matplotlib.pyplot as plt
def visualize_feature_maps(model, image, layer_idx=0):
"""可视化某一层的特征图,看看 CNN 学到了什么"""
features = []
def hook_fn(module, input, output):
features.append(output.detach())
# 注册前向 hook
hook = list(model.features.children())[layer_idx].register_forward_hook(hook_fn)
model.eval()
with torch.no_grad():
model(image.unsqueeze(0))
hook.remove()
# 画出前 16 个特征图
feature_map = features[0][0]
fig, axes = plt.subplots(4, 4, figsize=(10, 10))
for i, ax in enumerate(axes.flat):
if i < feature_map.shape[0]:
ax.imshow(feature_map[i].cpu().numpy(), cmap='viridis')
ax.axis('off')
plt.suptitle(f'Layer {layer_idx} Feature Maps')
plt.savefig('feature_maps.png', dpi=150)
plt.show()
Inception 的核心思想是同时用不同大小的卷积核,然后拼接结果。这样网络可以同时捕捉不同尺度的特征。
Pythonclass InceptionBlock(nn.Module):
"""Inception 模块:并行使用 1x1, 3x3, 5x5 卷积 + 池化"""
def __init__(self, in_channels, ch1x1, ch3x3_reduce, ch3x3, ch5x5_reduce, ch5x5, pool_proj):
super().__init__()
# 分支1:1x1 卷积
self.branch1 = nn.Sequential(
nn.Conv2d(in_channels, ch1x1, 1),
nn.BatchNorm2d(ch1x1), nn.ReLU()
)
# 分支2:1x1 降维 → 3x3 卷积
self.branch2 = nn.Sequential(
nn.Conv2d(in_channels, ch3x3_reduce, 1),
nn.BatchNorm2d(ch3x3_reduce), nn.ReLU(),
nn.Conv2d(ch3x3_reduce, ch3x3, 3, padding=1),
nn.BatchNorm2d(ch3x3), nn.ReLU()
)
# 分支3:1x1 降维 → 5x5 卷积
self.branch3 = nn.Sequential(
nn.Conv2d(in_channels, ch5x5_reduce, 1),
nn.BatchNorm2d(ch5x5_reduce), nn.ReLU(),
nn.Conv2d(ch5x5_reduce, ch5x5, 5, padding=2),
nn.BatchNorm2d(ch5x5), nn.ReLU()
)
# 分支4:3x3 池化 → 1x1 卷积
self.branch4 = nn.Sequential(
nn.MaxPool2d(3, stride=1, padding=1),
nn.Conv2d(in_channels, pool_proj, 1),
nn.BatchNorm2d(pool_proj), nn.ReLU()
)
def forward(self, x):
return torch.cat([self.branch1(x), self.branch2(x),
self.branch3(x), self.branch4(x)], dim=1)
# 使用
inception = InceptionBlock(192, 64, 96, 128, 16, 32, 32)
x = torch.randn(1, 192, 28, 28)
y = inception(x)
print(y.shape) # [1, 256, 28, 28] (64+128+32+32=256)
MobileNet 使用深度可分离卷积(Depthwise Separable Convolution),把标准卷积分成两步,大大减少参数量。
Pythonclass DepthwiseSeparableConv(nn.Module):
"""深度可分离卷积:先逐通道卷积,再 1x1 卷积"""
def __init__(self, in_ch, out_ch, stride=1):
super().__init__()
# 逐通道卷积(每个通道独立卷积)
self.depthwise = nn.Conv2d(in_ch, in_ch, 3, stride, 1, groups=in_ch, bias=False)
self.bn1 = nn.BatchNorm2d(in_ch)
# 逐点卷积(1x1 卷积混合通道)
self.pointwise = nn.Conv2d(in_ch, out_ch, 1, bias=False)
self.bn2 = nn.BatchNorm2d(out_ch)
self.relu = nn.ReLU()
def forward(self, x):
x = self.relu(self.bn1(self.depthwise(x)))
x = self.relu(self.bn2(self.pointwise(x)))
return x
# 参数量对比
standard_conv = nn.Conv2d(64, 128, 3, padding=1)
dw_conv = DepthwiseSeparableConv(64, 128)
print(f"标准卷积参数: {sum(p.numel() for p in standard_conv.parameters()):,}") # 73,856
print(f"深度可分离参数: {sum(p.numel() for p in dw_conv.parameters()):,}") # 8,960 (减少 8x!)
感受野(Receptive Field)是指输出特征图上的一个像素对应输入图像的区域大小。感受野越大,网络能"看到"的范围越广。
Pythondef calc_receptive_field(layers):
"""计算感受野大小
layers: [(kernel_size, stride, padding), ...]
"""
rf = 1
stride_product = 1
for k, s, p in layers:
rf = rf + (k - 1) * stride_product
stride_product *= s
return rf
# 例:VGG-16 的感受野
vgg_layers = [
(3, 1, 1), (3, 1, 1), (2, 2, 0), # Block 1
(3, 1, 1), (3, 1, 1), (2, 2, 0), # Block 2
(3, 1, 1), (3, 1, 1), (3, 1, 1), (2, 2, 0), # Block 3
]
print(f"VGG-16 Block 3 感受野: {calc_receptive_field(vgg_layers)}")
1x1 卷积看起来很奇怪(1x1 的卷积核能做什么?),但它有重要作用:
# 1x1 卷积用于通道数变换
conv_1x1 = nn.Conv2d(256, 128, 1) # 256通道 → 128通道
x = torch.randn(1, 256, 32, 32)
y = conv_1x1(x)
print(y.shape) # [1, 128, 32, 32] 尺寸不变,通道数变了
# Inception 中用 1x1 卷积降维
# 先用 1x1 把 256 通道降到 64,再用 3x3 卷积
# 这样 3x3 卷积的计算量大大减少
inception_reduce = nn.Sequential(
nn.Conv2d(256, 64, 1), # 降维
nn.ReLU(),
nn.Conv2d(64, 64, 3, padding=1), # 3x3 卷积
)
MobileNet 的核心创新:把标准卷积分成两步,大大减少计算量。
Pythonclass DepthwiseSeparableConv(nn.Module):
"""深度可分离卷积"""
def __init__(self, in_ch, out_ch, stride=1):
super().__init__()
# 第一步:逐通道卷积(每个通道独立卷积)
self.depthwise = nn.Conv2d(
in_ch, in_ch, 3, stride, 1,
groups=in_ch, # 关键:groups=in_ch
bias=False
)
self.bn1 = nn.BatchNorm2d(in_ch)
# 第二步:逐点卷积(1x1 卷积混合通道)
self.pointwise = nn.Conv2d(in_ch, out_ch, 1, bias=False)
self.bn2 = nn.BatchNorm2d(out_ch)
self.relu = nn.ReLU()
def forward(self, x):
x = self.relu(self.bn1(self.depthwise(x)))
x = self.relu(self.bn2(self.pointwise(x)))
return x
# 计算量对比
standard = nn.Conv2d(64, 128, 3, padding=1)
dw_sep = DepthwiseSeparableConv(64, 128)
standard_params = sum(p.numel() for p in standard.parameters())
dw_sep_params = sum(p.numel() for p in dw_sep.parameters())
print(f"标准卷积参数: {standard_params:,}") # 73,856
print(f"深度可分离参数: {dw_sep_params:,}") # 8,960
print(f"减少: {standard_params / dw_sep_params:.1f}x")
import matplotlib.pyplot as plt
def visualize_filters(model, layer_idx=0):
"""可视化卷积核"""
filters = list(model.children())[layer_idx].weight.data.cpu()
n_filters = min(filters.shape[0], 16)
fig, axes = plt.subplots(4, 4, figsize=(8, 8))
for i, ax in enumerate(axes.flat):
if i < n_filters:
# 取第一个通道
ax.imshow(filters[i, 0], cmap='viridis')
ax.axis('off')
plt.suptitle(f'Layer {layer_idx} Filters')
plt.savefig('filters.png', dpi=150)
plt.show()
def visualize_feature_maps(model, image, layer_idx=0):
"""可视化特征图"""
features = []
def hook(module, input, output):
features.append(output.detach())
hook = list(model.features.children())[layer_idx].register_forward_hook(hook)
model.eval()
with torch.no_grad():
model(image.unsqueeze(0))
hook.remove()
fmap = features[0][0]
fig, axes = plt.subplots(4, 4, figsize=(10, 10))
for i, ax in enumerate(axes.flat):
if i < fmap.shape[0]:
ax.imshow(fmap[i].cpu().numpy(), cmap='viridis')
ax.axis('off')
plt.savefig('feature_maps.png', dpi=150)
plt.show()
CIFAR-10 包含 10 个类别的 60000 张 32×32 彩色图片:飞机(airplane)、汽车(automobile)、鸟(bird)、猫(cat)、鹿(deer)、狗(dog)、青蛙(frog)、马(horse)、船(ship)、卡车(truck)。
Pythonimport torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision import datasets, transforms
# 数据预处理
train_transform = transforms.Compose([
transforms.RandomCrop(32, padding=4), # 随机裁剪
transforms.RandomHorizontalFlip(), # 随机翻转
transforms.ToTensor(),
transforms.Normalize((0.4914, 0.4822, 0.4465), # CIFAR-10 的均值
(0.2023, 0.1994, 0.2010)) # CIFAR-10 的标准差
])
test_transform = transforms.Compose([
transforms.ToTensor(),
transforms.Normalize((0.4914, 0.4822, 0.4465),
(0.2023, 0.1994, 0.2010))
])
train_set = datasets.CIFAR10('./data', train=True, download=True, transform=train_transform)
test_set = datasets.CIFAR10('./data', train=False, transform=test_transform)
train_loader = DataLoader(train_set, batch_size=128, shuffle=True, num_workers=2)
test_loader = DataLoader(test_set, batch_size=256, shuffle=False)
# 类别名
classes = ['airplane', 'automobile', 'bird', 'cat', 'deer',
'dog', 'frog', 'horse', 'ship', 'truck']
class CIFAR10Net(nn.Module):
"""3 个卷积块 + 全连接层的分类网络"""
def __init__(self):
super().__init__()
self.features = nn.Sequential(
# Block 1: 3 → 32
nn.Conv2d(3, 32, 3, padding=1),
nn.BatchNorm2d(32),
nn.ReLU(),
nn.Conv2d(32, 32, 3, padding=1),
nn.BatchNorm2d(32),
nn.ReLU(),
nn.MaxPool2d(2, 2), # 32x32 → 16x16
nn.Dropout2d(0.25),
# Block 2: 32 → 64
nn.Conv2d(32, 64, 3, padding=1),
nn.BatchNorm2d(64),
nn.ReLU(),
nn.Conv2d(64, 64, 3, padding=1),
nn.BatchNorm2d(64),
nn.ReLU(),
nn.MaxPool2d(2, 2), # 16x16 → 8x8
nn.Dropout2d(0.25),
# Block 3: 64 → 128
nn.Conv2d(64, 128, 3, padding=1),
nn.BatchNorm2d(128),
nn.ReLU(),
nn.Conv2d(128, 128, 3, padding=1),
nn.BatchNorm2d(128),
nn.ReLU(),
nn.MaxPool2d(2, 2), # 8x8 → 4x4
nn.Dropout2d(0.25),
)
self.classifier = nn.Sequential(
nn.Flatten(),
nn.Linear(128 * 4 * 4, 512),
nn.ReLU(),
nn.Dropout(0.5),
nn.Linear(512, 10)
)
def forward(self, x):
x = self.features(x)
x = self.classifier(x)
return x
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = CIFAR10Net().to(device)
print(f"参数量: {sum(p.numel() for p in model.parameters()):,}")
criterion = nn.CrossEntropyLoss()
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=100)
best_acc = 0
for epoch in range(100):
# 训练
model.train()
train_loss, correct, total = 0, 0, 0
for data, target in train_loader:
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
output = model(data)
loss = criterion(output, target)
loss.backward()
optimizer.step()
train_loss += loss.item() * data.size(0)
correct += output.argmax(1).eq(target).sum().item()
total += target.size(0)
scheduler.step()
# 评估
model.eval()
test_correct, test_total = 0, 0
with torch.no_grad():
for data, target in test_loader:
data, target = data.to(device), target.to(device)
output = model(data)
test_correct += output.argmax(1).eq(target).sum().item()
test_total += target.size(0)
test_acc = 100. * test_correct / test_total
if test_acc > best_acc:
best_acc = test_acc
torch.save(model.state_dict(), 'best_cifar10_model.pth')
if (epoch + 1) % 10 == 0:
print(f'Epoch {epoch+1} | Train Acc: {100.*correct/total:.2f}% | Test Acc: {test_acc:.2f}%')
print(f'Best Test Accuracy: {best_acc:.2f}%')
使用上面的架构训练 100 个 epoch,测试准确率应该能达到 91-93%。如果使用 ResNet 架构,可以达到 95%+。
from torchvision import models
# 使用预训练的 ResNet-18,修改最后一层
model = models.resnet18(weights=None) # 不用预训练权重,从零训练
model.fc = nn.Linear(model.fc.in_features, 10)
model = model.to(device)
# CIFAR-10 的图片是 32x32,比 ImageNet 的 224x224 小很多
# 所以需要修改第一个卷积层
model.conv1 = nn.Conv2d(3, 64, 3, stride=1, padding=1, bias=False)
model.maxpool = nn.Identity() # 去掉最大池化层
# 训练(和上面一样的循环)
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=200)
# 预期准确率:95%+
from sklearn.metrics import confusion_matrix, classification_report
import seaborn as sns
import matplotlib.pyplot as plt
# 收集所有预测
all_preds, all_labels = [], []
model.eval()
with torch.no_grad():
for data, target in test_loader:
data = data.to(device)
output = model(data)
preds = output.argmax(dim=1).cpu()
all_preds.extend(preds.tolist())
all_labels.extend(target.tolist())
# 混淆矩阵
cm = confusion_matrix(all_labels, all_preds)
plt.figure(figsize=(10, 8))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
xticklabels=classes, yticklabels=classes)
plt.xlabel('预测')
plt.ylabel('真实')
plt.title('混淆矩阵')
plt.savefig('confusion_matrix.png', dpi=150)
# 分类报告
print(classification_report(all_labels, all_preds, target_names=classes))
from PIL import Image
# 加载训练好的模型
model.load_state_dict(torch.load('best_cifar10_model.pth'))
model.eval()
# 加载图片
img = Image.open('test_airplane.jpg').convert('RGB')
transform = transforms.Compose([
transforms.Resize((32, 32)),
transforms.ToTensor(),
transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
])
img_tensor = transform(img).unsqueeze(0).to(device)
# 预测
with torch.no_grad():
output = model(img_tensor)
probs = torch.softmax(output, dim=1)
pred = probs.argmax(dim=1).item()
confidence = probs[0][pred].item()
print(f"预测: {classes[pred]}")
print(f"置信度: {confidence:.4f}")
# 显示所有类别的概率
for i, (cls, prob) in enumerate(zip(classes, probs[0])):
bar = '█' * int(prob * 20)
print(f" {cls:12s} {prob:.4f} {bar}")
nn.CrossEntropyLoss(label_smoothing=0.1)def cutmix_data(x, y, alpha=1.0):
"""CutMix:把一张图片的一部分剪切到另一张图片上"""
lam = torch.distributions.Beta(alpha, alpha).sample()
batch_size = x.size(0)
index = torch.randperm(batch_size).to(x.device)
# 随机选择裁剪区域
bbx1, bby1, bbx2, bby2 = rand_bbox(x.size(), lam)
# 混合
x[:, :, bbx1:bbx2, bby1:bby2] = x[index, :, bbx1:bbx2, bby1:bby2]
lam = 1 - ((bbx2 - bbx1) * (bby2 - bby1) / (x.size(-1) * x.size(-2)))
y_a, y_b = y, y[index]
return x, y_a, y_b, lam
def rand_bbox(size, lam):
"""随机生成裁剪框"""
W, H = size[2], size[3]
cut_rat = (1. - lam) ** 0.5
cut_w, cut_h = int(W * cut_rat), int(H * cut_rat)
cx, cy = torch.randint(W, (1,)), torch.randint(H, (1,))
bbx1 = torch.clamp(cx - cut_w // 2, 0, W)
bby1 = torch.clamp(cy - cut_h // 2, 0, H)
bbx2 = torch.clamp(cx + cut_w // 2, 0, W)
bby2 = torch.clamp(cy + cut_h // 2, 0, H)
return bbx1, bby1, bbx2, bby2
def test_time_augmentation(model, image, num_augments=10):
"""测试时增强:对同一张图片做多次变换,取平均预测"""
transforms_list = [
transforms.Compose([
transforms.Resize(32),
transforms.ToTensor(),
transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
]),
transforms.Compose([
transforms.Resize(32),
transforms.RandomHorizontalFlip(p=1.0),
transforms.ToTensor(),
transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
]),
]
model.eval()
all_probs = []
with torch.no_grad():
for t in transforms_list:
aug_img = t(image).unsqueeze(0).to(device)
output = model(aug_img)
probs = torch.softmax(output, dim=1)
all_probs.append(probs)
# 取平均
avg_probs = torch.stack(all_probs).mean(dim=0)
return avg_probs
from fastapi import FastAPI, UploadFile, File
from PIL import Image
import torch
import torchvision.transforms as transforms
app = FastAPI()
# 加载模型
model = CIFAR10Net()
model.load_state_dict(torch.load('best_cifar10_model.pth'))
model.eval()
classes = ['airplane', 'automobile', 'bird', 'cat', 'deer',
'dog', 'frog', 'horse', 'ship', 'truck']
transform = transforms.Compose([
transforms.Resize((32, 32)),
transforms.ToTensor(),
transforms.Normalize((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
])
@app.post("/predict")
async def predict(file: UploadFile = File(...)):
image = Image.open(file.file).convert('RGB')
img_tensor = transform(image).unsqueeze(0)
with torch.no_grad():
output = model(img_tensor)
probs = torch.softmax(output, dim=1)
pred = probs.argmax(dim=1).item()
return {
"class": classes[pred],
"confidence": probs[0][pred].item(),
"all_probs": {c: probs[0][i].item() for i, c in enumerate(classes)}
}
# 启动:uvicorn app:app --host 0.0.0.0 --port 8000
想象你在读一本书。你不会把每个字独立理解,而是会记住前面的内容来理解当前的字。比如读到"他进球了",你需要知道"他"指的是前面提到的"梅西"。RNN 就是这样——它有"记忆",能记住之前看到的信息。
CNN 擅长处理空间数据(图片),而 RNN 擅长处理序列数据:
import torch
import torch.nn as nn
# nn.RNN(input_size, hidden_size, num_layers, batch_first)
rnn = nn.RNN(
input_size=10, # 每个时间步的输入特征数
hidden_size=20, # 隐藏状态的维度
num_layers=2, # RNN 的层数
batch_first=True # 输入形状为 [batch, seq_len, features]
)
# 输入:batch_size=3, seq_len=5, input_size=10
x = torch.randn(3, 5, 10)
output, h_n = rnn(x)
print(f"output shape: {output.shape}") # [3, 5, 20] 每个时间步的输出
print(f"h_n shape: {h_n.shape}") # [2, 3, 20] 最后一个时间步的隐藏状态
基本 RNN 有一个大问题:梯度消失——当序列很长时,后面的信号传不到前面。LSTM 通过门控机制解决了这个问题。
LSTM 就像一个有传送带的工厂。信息在传送带上流动,可以通过"门"来决定:
# LSTM 和 RNN 接口几乎一样
lstm = nn.LSTM(
input_size=10,
hidden_size=20,
num_layers=2,
batch_first=True,
dropout=0.2, # 层之间的 Dropout
bidirectional=False # 单向
)
x = torch.randn(3, 5, 10)
output, (h_n, c_n) = lstm(x)
print(f"output shape: {output.shape}") # [3, 5, 20] 每个时间步的输出
print(f"h_n shape: {h_n.shape}") # [2, 3, 20] 隐藏状态
print(f"c_n shape: {c_n.shape}") # [2, 3, 20] 细胞状态(LSTM 特有)
GRU(门控循环单元)是 LSTM 的简化版本,只有 2 个门(重置门和更新门),参数更少,训练更快,效果往往和 LSTM 差不多。
Pythongru = nn.GRU(input_size=10, hidden_size=20, num_layers=2, batch_first=True)
x = torch.randn(3, 5, 10)
output, h_n = gru(x) # GRU 没有细胞状态 c_n
print(f"output shape: {output.shape}") # [3, 5, 20]
print(f"h_n shape: {h_n.shape}") # [2, 3, 20]
# bidirectional=True:同时从左到右和从右到左读取
lstm = nn.LSTM(input_size=10, hidden_size=20, num_layers=2,
batch_first=True, bidirectional=True)
x = torch.randn(3, 5, 10)
output, (h_n, c_n) = lstm(x)
print(f"output shape: {output.shape}") # [3, 5, 40] 20*2=40(两个方向拼接)
print(f"h_n shape: {h_n.shape}") # [4, 3, 20] 2层*2方向=4
nn.utils.rnn.pack_padded_sequence 处理不同长度的序列clip_grad_norm_理解 RNN 的内部计算过程有助于调试和优化:
Python# 基本 RNN 的计算:
# h_t = tanh(W_ih * x_t + b_ih + W_hh * h_{t-1} + b_hh)
# 其中:
# x_t: 当前时间步的输入
# h_{t-1}: 上一个时间步的隐藏状态
# W_ih: 输入到隐藏的权重
# W_hh: 隐藏到隐藏的权重
# LSTM 的计算(更复杂):
# f_t = sigmoid(W_if * x_t + b_if + W_hf * h_{t-1} + b_hf) # 遗忘门
# i_t = sigmoid(W_ii * x_t + b_ii + W_hi * h_{t-1} + b_hi) # 输入门
# g_t = tanh(W_ig * x_t + b_ig + W_hg * h_{t-1} + b_hg) # 候选值
# o_t = sigmoid(W_io * x_t + b_io + W_ho * h_{t-1} + b_ho) # 输出门
# c_t = f_t * c_{t-1} + i_t * g_t # 细胞状态更新
# h_t = o_t * tanh(c_t) # 隐藏状态输出
| 变体 | 特点 | 适用场景 |
|---|---|---|
| 基本 RNN | 最简单,梯度消失严重 | 几乎不用了 |
| LSTM | 门控机制,解决长期依赖 | 长序列、需要记忆的任务 |
| GRU | LSTM 的简化版,参数更少 | 数据量较少时,效果和 LSTM 差不多 |
| 双向 LSTM | 同时从前往后和从后往前读 | 文本分类、命名实体识别 |
| 多层 LSTM | 堆叠多层,提取更高级特征 | 机器翻译、文本生成 |
趋势:NLP 领域 Transformer 已经基本取代了 RNN。但在数据量小、序列短的场景,LSTM 仍然有优势。
Seq2Seq 是机器翻译、文本摘要等任务的基础架构。它由编码器(把输入序列编码成一个向量)和解码器(把向量解码成输出序列)组成。
Pythonclass Seq2SeqEncoder(nn.Module):
"""Seq2Seq 编码器"""
def __init__(self, vocab_size, embed_dim, hidden_dim, num_layers=2):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTM(embed_dim, hidden_dim, num_layers,
batch_first=True, dropout=0.2)
def forward(self, x):
embedded = self.embedding(x)
outputs, (h_n, c_n) = self.lstm(embedded)
return h_n, c_n # 返回最后的隐藏状态作为上下文
class Seq2SeqDecoder(nn.Module):
"""Seq2Seq 解码器"""
def __init__(self, vocab_size, embed_dim, hidden_dim, num_layers=2):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTM(embed_dim, hidden_dim, num_layers,
batch_first=True, dropout=0.2)
self.fc = nn.Linear(hidden_dim, vocab_size)
def forward(self, x, h_n, c_n):
embedded = self.embedding(x)
output, (h_n, c_n) = self.lstm(embedded, (h_n, c_n))
logits = self.fc(output)
return logits, h_n, c_n
class Seq2Seq(nn.Module):
"""完整的 Seq2Seq 模型"""
def __init__(self, encoder, decoder, device):
super().__init__()
self.encoder = encoder
self.decoder = decoder
self.device = device
def forward(self, src, trg, teacher_forcing_ratio=0.5):
batch_size = src.size(0)
trg_len = trg.size(1)
trg_vocab_size = self.decoder.fc.out_features
# 存储所有时间步的输出
outputs = torch.zeros(batch_size, trg_len, trg_vocab_size).to(self.device)
# 编码
h_n, c_n = self.encoder(src)
# 解码器的第一个输入是 <sos> token
decoder_input = trg[:, 0:1]
for t in range(1, trg_len):
output, h_n, c_n = self.decoder(decoder_input, h_n, c_n)
outputs[:, t:t+1, :] = output
# Teacher Forcing:有时用真实值作为下一步输入
if torch.rand(1).item() < teacher_forcing_ratio:
decoder_input = trg[:, t:t+1]
else:
decoder_input = output.argmax(dim=-1)
return outputs
# 使用
encoder = Seq2SeqEncoder(vocab_size=5000, embed_dim=256, hidden_dim=512)
decoder = Seq2SeqDecoder(vocab_size=5000, embed_dim=256, hidden_dim=512)
model = Seq2Seq(encoder, decoder, device).to(device)
传统 Seq2Seq 的问题是:编码器把整个输入压缩成一个固定长度的向量,信息会丢失。注意力机制让解码器可以"回头看"编码器的所有输出。
Pythonclass BahdanauAttention(nn.Module):
"""Bahdanau 注意力(加性注意力)"""
def __init__(self, hidden_dim):
super().__init__()
self.W1 = nn.Linear(hidden_dim, hidden_dim)
self.W2 = nn.Linear(hidden_dim, hidden_dim)
self.V = nn.Linear(hidden_dim, 1)
def forward(self, decoder_hidden, encoder_outputs):
# decoder_hidden: [batch, hidden]
# encoder_outputs: [batch, seq_len, hidden]
decoder_hidden = decoder_hidden.unsqueeze(1) # [batch, 1, hidden]
# 计算注意力分数
energy = torch.tanh(self.W1(encoder_outputs) + self.W2(decoder_hidden))
attention = self.V(energy).squeeze(-1) # [batch, seq_len]
# Softmax 归一化
weights = torch.softmax(attention, dim=1) # [batch, seq_len]
# 加权求和
context = torch.bmm(weights.unsqueeze(1), encoder_outputs) # [batch, 1, hidden]
return context.squeeze(1), weights
| 任务 | 模型 | 说明 |
|---|---|---|
| 文本分类 | LSTM + Attention | 把文本编码成向量,然后分类 |
| 机器翻译 | Seq2Seq + Attention | 编码器-解码器架构 |
| 文本生成 | 单向 LSTM | 给定前文,预测下一个字 |
| 命名实体识别 | 双向 LSTM + CRF | 序列标注任务 |
| 语音识别 | 双向 LSTM + CTC | 音频帧序列 → 文本 |
| 时间序列预测 | LSTM / GRU | 预测未来值 |
| 音乐生成 | LSTM | 生成音符序列 |
用 LSTM 对电影评论做正面/负面分类。
Pythonimport torch
import torch.nn as nn
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator
from torch.utils.data import DataLoader
from torch.nn.utils.rnn import pad_sequence
# 1. 数据处理
tokenizer = get_tokenizer('basic_english')
def yield_tokens(data_iter):
for label, text in data_iter:
yield tokenizer(text)
# 构建词汇表
train_iter = IMDB(split='train')
vocab = build_vocab_from_iterator(yield_tokens(train_iter), specials=['<unk>', '<pad>'])
vocab.set_default_index(vocab['<unk>'])
# 2. 数据集类
class IMDBDataset:
def __init__(self, split):
self.data = list(IMDB(split=split))
def __len__(self):
return len(self.data)
def __getitem__(self, idx):
label, text = self.data[idx]
tokens = tokenizer(text)
indices = vocab(tokens)
return torch.tensor(indices, dtype=torch.long), torch.tensor(label - 1, dtype=torch.long)
def collate_fn(batch):
texts, labels = zip(*batch)
texts_padded = pad_sequence(texts, batch_first=True, padding_value=vocab['<pad>'])
return texts_padded, torch.stack(labels)
train_loader = DataLoader(IMDBDataset('train'), batch_size=64,
shuffle=True, collate_fn=collate_fn)
# 3. 模型
class SentimentLSTM(nn.Module):
def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=vocab['<pad>'])
self.lstm = nn.LSTM(embed_dim, hidden_dim, num_layers=2,
batch_first=True, bidirectional=True, dropout=0.3)
self.fc = nn.Linear(hidden_dim * 2, num_classes) # *2 因为双向
self.dropout = nn.Dropout(0.5)
def forward(self, x):
embedded = self.dropout(self.embedding(x))
output, (h_n, c_n) = self.lstm(embedded)
# 取最后一层的前向和后向隐藏状态拼接
hidden = torch.cat([h_n[-2], h_n[-1]], dim=1)
return self.fc(self.dropout(hidden))
model = SentimentLSTM(len(vocab), embed_dim=128, hidden_dim=256, num_classes=2)
print(f"参数量: {sum(p.numel() for p in model.parameters()):,}")
class CharRNN(nn.Module):
"""字符级 RNN:给定前文,预测下一个字符"""
def __init__(self, vocab_size, embed_dim, hidden_dim):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTM(embed_dim, hidden_dim, num_layers=2, batch_first=True)
self.fc = nn.Linear(hidden_dim, vocab_size)
def forward(self, x, hidden=None):
x = self.embedding(x)
output, hidden = self.lstm(x, hidden)
logits = self.fc(output)
return logits, hidden
def generate(self, start_char, char_to_idx, idx_to_char, length=200, temperature=0.8):
"""生成文本"""
self.eval()
chars = [char_to_idx[start_char]]
hidden = None
with torch.no_grad():
for _ in range(length):
x = torch.tensor([[chars[-1]]])
logits, hidden = self(x, hidden)
# 温度采样
probs = torch.softmax(logits[0, -1] / temperature, dim=0)
next_char = torch.multinomial(probs, 1).item()
chars.append(next_char)
return ''.join([idx_to_char[c] for c in chars])
class TimeSeriesLSTM(nn.Module):
"""用 LSTM 预测时间序列"""
def __init__(self, input_size=1, hidden_size=64, num_layers=2):
super().__init__()
self.lstm = nn.LSTM(input_size, hidden_size, num_layers,
batch_first=True, dropout=0.2)
self.fc = nn.Linear(hidden_size, 1)
def forward(self, x):
output, (h_n, c_n) = self.lstm(x)
last_output = output[:, -1, :]
prediction = self.fc(last_output)
return prediction
# 创建滑动窗口数据集
def create_sequences(data, seq_length):
sequences, targets = [], []
for i in range(len(data) - seq_length):
sequences.append(data[i:i+seq_length])
targets.append(data[i+seq_length])
return torch.stack(sequences), torch.stack(targets)
seq_length = 30
data = torch.randn(1000)
X, y = create_sequences(data, seq_length)
print(f"X shape: {X.shape}") # [970, 30]
print(f"y shape: {y.shape}") # [970]
class TextGenerator(nn.Module):
"""基于 LSTM 的文本生成模型"""
def __init__(self, vocab_size, embed_dim=128, hidden_dim=256, num_layers=2):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.lstm = nn.LSTM(embed_dim, hidden_dim, num_layers,
batch_first=True, dropout=0.2)
self.fc = nn.Linear(hidden_dim, vocab_size)
def forward(self, x, hidden=None):
embedded = self.embedding(x)
output, hidden = self.lstm(embedded, hidden)
logits = self.fc(output)
return logits, hidden
def generate(self, start_tokens, length=200, temperature=0.8):
"""生成文本"""
self.eval()
tokens = list(start_tokens)
hidden = None
with torch.no_grad():
for _ in range(length):
x = torch.tensor([tokens[-100:]]) # 只用最后100个token
logits, hidden = self(x, hidden)
probs = torch.softmax(logits[0, -1] / temperature, dim=0)
next_token = torch.multinomial(probs, 1).item()
tokens.append(next_token)
return tokens
# 训练
model = TextGenerator(vocab_size=5000).to(device)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(50):
model.train()
total_loss = 0
for batch in train_loader:
inputs = batch[:, :-1].to(device) # 前 n-1 个 token
targets = batch[:, 1:].to(device) # 后 n-1 个 token(向右移一位)
optimizer.zero_grad()
logits, _ = model(inputs)
loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
total_loss += loss.item()
print(f"Epoch {epoch+1}, Loss: {total_loss/len(train_loader):.4f}")
clip_grad_norm_(model.parameters(), 1.0)pack_padded_sequence 和 pad_packed_sequencefrom collections import Counter
class TextPreprocessor:
"""文本预处理工具"""
def __init__(self, max_vocab_size=10000, max_seq_len=256):
self.max_vocab_size = max_vocab_size
self.max_seq_len = max_seq_len
self.word2idx = {}
self.idx2word = {}
def build_vocab(self, texts):
"""构建词汇表"""
counter = Counter()
for text in texts:
tokens = text.lower().split()
counter.update(tokens)
# 取最常见的词
most_common = counter.most_common(self.max_vocab_size - 2)
self.word2idx = {'<pad>': 0, '<unk>': 1}
for i, (word, _) in enumerate(most_common):
self.word2idx[word] = i + 2
self.idx2word = {v: k for k, v in self.word2idx.items()}
def encode(self, text):
"""文本 → 索引序列"""
tokens = text.lower().split()
indices = [self.word2idx.get(t, 1) for t in tokens] # 1 = <unk>
# 截断或填充
if len(indices) > self.max_seq_len:
indices = indices[:self.max_seq_len]
else:
indices += [0] * (self.max_seq_len - len(indices))
return indices
def decode(self, indices):
"""索引序列 → 文本"""
tokens = [self.idx2word.get(i, '<unk>') for i in indices if i != 0]
return ' '.join(tokens)
# 使用
preprocessor = TextPreprocessor(max_vocab_size=10000, max_seq_len=128)
preprocessor.build_vocab(train_texts)
encoded = preprocessor.encode("I love deep learning")
print(encoded) # [23, 456, 789, 123, 0, 0, ...]
# 完整的 RNN 训练模板
def train_rnn_model(model, train_loader, val_loader, epochs=20):
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()
scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, patience=3)
best_val_loss = float('inf')
for epoch in range(epochs):
# 训练
model.train()
train_loss = 0
for texts, labels in train_loader:
texts, labels = texts.to(device), labels.to(device)
optimizer.zero_grad()
output = model(texts)
loss = criterion(output, labels)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
train_loss += loss.item()
# 验证
model.eval()
val_loss = 0
correct, total = 0, 0
with torch.no_grad():
for texts, labels in val_loader:
texts, labels = texts.to(device), labels.to(device)
output = model(texts)
val_loss += criterion(output, labels).item()
pred = output.argmax(1)
correct += pred.eq(labels).sum().item()
total += labels.size(0)
val_loss /= len(val_loader)
val_acc = 100. * correct / total
scheduler.step(val_loss)
print(f"Epoch {epoch+1} | Train Loss: {train_loss/len(train_loader):.4f} | "
f"Val Loss: {val_loss:.4f} | Val Acc: {val_acc:.2f}%")
if val_loss < best_val_loss:
best_val_loss = val_loss
torch.save(model.state_dict(), 'best_rnn_model.pth')
当你看一张照片时,你的眼睛不会均匀地关注每个区域——你会重点关注最显眼的部分(比如人脸)。注意力机制就是让模型也学会"该关注哪里"。在翻译时,生成"爱"这个字,模型会重点关注英文的"love",而不是均匀关注所有词。
自注意力的核心思想:每个位置都和所有其他位置计算相关性。
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
class SelfAttention(nn.Module):
def __init__(self, embed_dim, num_heads):
super().__init__()
self.embed_dim = embed_dim
self.num_heads = num_heads
self.head_dim = embed_dim // num_heads
assert embed_dim % num_heads == 0
self.qkv = nn.Linear(embed_dim, 3 * embed_dim)
self.out_proj = nn.Linear(embed_dim, embed_dim)
def forward(self, x):
B, T, C = x.shape
qkv = self.qkv(x).reshape(B, T, 3, self.num_heads, self.head_dim)
qkv = qkv.permute(2, 0, 3, 1, 4)
q, k, v = qkv[0], qkv[1], qkv[2]
scale = math.sqrt(self.head_dim)
attn_scores = (q @ k.transpose(-2, -1)) / scale
attn_weights = F.softmax(attn_scores, dim=-1)
out = attn_weights @ v
out = out.transpose(1, 2).reshape(B, T, C)
return self.out_proj(out)
sa = SelfAttention(embed_dim=512, num_heads=8)
x = torch.randn(2, 10, 512)
out = sa(x)
print(out.shape) # [2, 10, 512]
# PyTorch 内置的多头注意力
mha = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
x = torch.randn(2, 10, 512)
attn_output, attn_weights = mha(x, x, x) # self-attention
print(attn_output.shape) # [2, 10, 512]
class PositionalEncoding(nn.Module):
def __init__(self, embed_dim, max_len=5000):
super().__init__()
pe = torch.zeros(max_len, embed_dim)
position = torch.arange(0, max_len).unsqueeze(1).float()
div_term = torch.exp(torch.arange(0, embed_dim, 2).float() * (-math.log(10000.0) / embed_dim))
pe[:, 0::2] = torch.sin(position * div_term)
pe[:, 1::2] = torch.cos(position * div_term)
self.register_buffer('pe', pe.unsqueeze(0))
def forward(self, x):
return x + self.pe[:, :x.size(1), :]
class TransformerEncoderLayer(nn.Module):
def __init__(self, embed_dim, num_heads, ff_dim, dropout=0.1):
super().__init__()
self.self_attn = nn.MultiheadAttention(embed_dim, num_heads, batch_first=True)
self.ff = nn.Sequential(
nn.Linear(embed_dim, ff_dim), nn.GELU(), nn.Dropout(dropout),
nn.Linear(ff_dim, embed_dim),
)
self.norm1 = nn.LayerNorm(embed_dim)
self.norm2 = nn.LayerNorm(embed_dim)
self.dropout = nn.Dropout(dropout)
def forward(self, x):
attn_out, _ = self.self_attn(x, x, x)
x = self.norm1(x + self.dropout(attn_out))
ff_out = self.ff(x)
x = self.norm2(x + self.dropout(ff_out))
return x
class SimpleTransformer(nn.Module):
def __init__(self, vocab_size, embed_dim=256, num_heads=8, num_layers=6, num_classes=2):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.pos_encoding = PositionalEncoding(embed_dim)
self.layers = nn.ModuleList([
TransformerEncoderLayer(embed_dim, num_heads, embed_dim * 4)
for _ in range(num_layers)
])
self.classifier = nn.Linear(embed_dim, num_classes)
def forward(self, x):
x = self.pos_encoding(self.embedding(x))
for layer in self.layers:
x = layer(x)
return self.classifier(x.mean(dim=1))
| 对比项 | RNN/LSTM | Transformer |
|---|---|---|
| 并行性 | 串行处理,不能并行 | 完全并行,速度快 |
| 长距离依赖 | 信息需要一步步传递,会衰减 | 直接关注任意位置,无衰减 |
| 计算复杂度 | O(n) 每步,O(n) 总共 | O(n²) 注意力矩阵 |
| 数据需求 | 小数据也能工作 | 需要大量数据 |
| 可解释性 | 隐藏状态难以解释 | 注意力权重可可视化 |
import matplotlib.pyplot as plt
def visualize_attention(model, sentence, tokenizer):
"""可视化注意力权重,看看模型关注了哪些词"""
model.eval()
tokens = tokenizer(sentence)
input_ids = torch.tensor([tokens]).to(device)
# 获取注意力权重
with torch.no_grad():
output = model(input_ids)
# 假设模型返回 (logits, attention_weights)
attn_weights = output[1] # [batch, heads, seq_len, seq_len]
# 取第一个头的注意力
attn = attn_weights[0, 0].cpu().numpy()
plt.figure(figsize=(10, 8))
plt.imshow(attn, cmap='hot')
plt.xticks(range(len(tokens)), tokens, rotation=45)
plt.yticks(range(len(tokens)), tokens)
plt.xlabel('Key')
plt.ylabel('Query')
plt.title('Self-Attention Weights')
plt.colorbar()
plt.tight_layout()
plt.savefig('attention_weights.png', dpi=150)
深入理解 Transformer 的核心公式:
Python# 自注意力的完整计算过程
def self_attention_detail(Q, K, V, mask=None):
"""
Q, K, V: [batch, seq_len, d_model]
"""
d_k = Q.size(-1)
# 1. 计算注意力分数
scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
# scores: [batch, seq_len, seq_len]
# 2. 应用掩码(解码器需要)
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9)
# 3. Softmax 归一化
attn_weights = torch.softmax(scores, dim=-1)
# attn_weights: [batch, seq_len, seq_len]
# 4. 加权求和
output = torch.matmul(attn_weights, V)
# output: [batch, seq_len, d_model]
return output, attn_weights
# 多头注意力
def multi_head_attention(Q, K, V, num_heads):
batch_size, seq_len, d_model = Q.size()
d_k = d_model // num_heads
# 分头
Q = Q.view(batch_size, seq_len, num_heads, d_k).transpose(1, 2)
K = K.view(batch_size, seq_len, num_heads, d_k).transpose(1, 2)
V = V.view(batch_size, seq_len, num_heads, d_k).transpose(1, 2)
# [batch, num_heads, seq_len, d_k]
# 计算注意力
scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
attn_weights = torch.softmax(scores, dim=-1)
output = torch.matmul(attn_weights, V)
# 合并头
output = output.transpose(1, 2).contiguous().view(batch_size, seq_len, d_model)
return output
import math
def positional_encoding_demo():
"""演示位置编码的工作原理"""
d_model = 64
max_len = 100
pe = torch.zeros(max_len, d_model)
position = torch.arange(0, max_len).unsqueeze(1).float()
div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
pe[:, 0::2] = torch.sin(position * div_term) # 偶数维用 sin
pe[:, 1::2] = torch.cos(position * div_term) # 奇数维用 cos
# 可视化
import matplotlib.pyplot as plt
plt.figure(figsize=(10, 6))
plt.imshow(pe.numpy().T, cmap='RdBu', aspect='auto')
plt.xlabel('Position')
plt.ylabel('Dimension')
plt.title('Positional Encoding')
plt.colorbar()
plt.savefig('positional_encoding.png', dpi=150)
# 为什么用 sin/cos?
# 因为 PE(pos+k) 可以用 PE(pos) 的线性变换表示
# 这让模型能学会"相对位置"的概念
label_smoothing=0.1clip_grad_norm=1.0lr = d_model^{-0.5} * min(step^{-0.5}, step * warmup^{-1.5})HuggingFace Transformers 是最流行的 NLP 库,提供了数千个预训练模型,一行代码就能使用。
Python# pip install transformers
from transformers import pipeline
# 一行代码做情感分析
classifier = pipeline('sentiment-analysis')
result = classifier("I love PyTorch!")
print(result) # [{'label': 'POSITIVE', 'score': 0.9998}]
# 文本生成
generator = pipeline('text-generation', model='gpt2')
text = generator("Once upon a time", max_length=50)
print(text[0]['generated_text'])
| 任务 | Pipeline 名称 | 示例模型 |
|---|---|---|
| 情感分析 | sentiment-analysis | distilbert-base-uncased |
| 文本生成 | text-generation | gpt2 |
| 问答 | question-answering | bert-large-uncased |
| 文本摘要 | summarization | facebook/bart-large-cnn |
| 翻译 | translation | Helsinki-NLP/opus-mt-zh-en |
| 命名实体识别 | ner | dbmdz/bert-large-cased |
| 零样本分类 | zero-shot-classification | facebook/bart-large-mnli |
| 填空 | fill-mask | bert-base-uncased |
| 图像分类 | image-classification | google/vit-base-patch16 |
from transformers import BertTokenizer, BertForSequenceClassification, Trainer, TrainingArguments
tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
model = BertForSequenceClassification.from_pretrained('bert-base-chinese', num_labels=2)
training_args = TrainingArguments(
output_dir='./results', num_train_epochs=3,
per_device_train_batch_size=16, learning_rate=2e-5,
)
trainer = Trainer(model=model, args=training_args, train_dataset=dataset)
trainer.train()
from transformers import BertTokenizer, BertForSequenceClassification
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, Dataset
# 1. 加载模型和分词器
tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
model = BertForSequenceClassification.from_pretrained('bert-base-chinese', num_labels=2)
model = model.to(device)
# 2. 自定义数据集
class ChineseSentimentDataset(Dataset):
def __init__(self, texts, labels, tokenizer, max_len=128):
self.texts = texts
self.labels = labels
self.tokenizer = tokenizer
self.max_len = max_len
def __len__(self):
return len(self.texts)
def __getitem__(self, idx):
encoding = self.tokenizer(
self.texts[idx],
max_length=self.max_len,
padding='max_length',
truncation=True,
return_tensors='pt'
)
return {
'input_ids': encoding['input_ids'].squeeze(),
'attention_mask': encoding['attention_mask'].squeeze(),
'labels': torch.tensor(self.labels[idx], dtype=torch.long)
}
# 3. 准备数据
texts = ["这部电影太好看了!", "浪费时间,不推荐", "演技很棒", "剧情太烂了"]
labels = [1, 0, 1, 0]
dataset = ChineseSentimentDataset(texts, labels, tokenizer)
loader = DataLoader(dataset, batch_size=2, shuffle=True)
# 4. 训练
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)
for epoch in range(3):
model.train()
for batch in loader:
input_ids = batch['input_ids'].to(device)
attention_mask = batch['attention_mask'].to(device)
labels = batch['labels'].to(device)
outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
loss = outputs.loss
optimizer.zero_grad()
loss.backward()
optimizer.step()
print(f"Epoch {epoch+1}, Loss: {loss.item():.4f}")
from transformers import pipeline
# 问答系统
qa = pipeline('question-answering', model='bert-large-uncased-whole-word-masking-finetuned-squad')
result = qa(question="What is PyTorch?", context="PyTorch is a deep learning framework developed by Meta.")
print(result) # {'answer': 'a deep learning framework', 'score': 0.95, ...}
# 文本摘要
summarizer = pipeline('summarization', model='facebook/bart-large-cnn')
summary = summarizer("Long article text...", max_length=130)
print(summary[0]['summary_text'])
# 命名实体识别(NER)
ner = pipeline('ner', model='dbmdz/bert-large-cased-finetuned-conll03-english')
entities = ner("My name is Wolfgang and I live in Berlin")
print(entities) # [{'entity': 'I-PER', 'word': 'Wolfgang', ...}, {'entity': 'I-LOC', 'word': 'Berlin', ...}]
# 零样本分类(不需要训练!)
classifier = pipeline('zero-shot-classification')
result = classifier("This is a great movie!", candidate_labels=["positive", "negative", "neutral"])
print(result) # {'labels': ['positive', 'negative', 'neutral'], 'scores': [0.9, 0.05, 0.05]}
class SimpleGPT(nn.Module):
"""一个简单的 GPT 模型(仅用于学习)"""
def __init__(self, vocab_size, embed_dim=256, num_heads=8,
num_layers=6, max_len=512):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim)
self.pos_encoding = PositionalEncoding(embed_dim, max_len)
# 因果掩码(确保只能看到前面的 token)
self.register_buffer('causal_mask',
torch.triu(torch.ones(max_len, max_len), diagonal=1).bool())
self.layers = nn.ModuleList([
TransformerEncoderLayer(embed_dim, num_heads, embed_dim * 4)
for _ in range(num_layers)
])
self.fc = nn.Linear(embed_dim, vocab_size)
def forward(self, x):
seq_len = x.size(1)
mask = self.causal_mask[:seq_len, :seq_len]
x = self.pos_encoding(self.embedding(x))
for layer in self.layers:
# 传入因果掩码
x = layer(x) # 注意:需要修改 layer 以支持 mask
return self.fc(x)
def generate(self, start_tokens, length=100, temperature=0.8):
"""生成文本"""
self.eval()
tokens = list(start_tokens)
with torch.no_grad():
for _ in range(length):
x = torch.tensor([tokens[-512:]]) # 只用最后 512 个 token
logits = self(x)
probs = torch.softmax(logits[0, -1] / temperature, dim=0)
next_token = torch.multinomial(probs, 1).item()
tokens.append(next_token)
return tokens
你学会了骑自行车,再学摩托车就很快——因为很多技能是可以迁移的(平衡感、方向控制)。迁移学习也一样:用在大数据集(如 ImageNet,有 1000 类、128 万张图片)上预训练好的模型,稍加修改就能用在你自己的小数据集上。你不需要从零训练——站在巨人的肩膀上!
为什么迁移学习有效?
import torch
import torch.nn as nn
from torchvision import models
# ==========================================
# 加载预训练的 ResNet-18
# ==========================================
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)
print(model) # 查看完整模型结构
# ==========================================
# 替换最后一层(适配你的分类任务)
# ==========================================
num_classes = 10 # 你的类别数
model.fc = nn.Linear(model.fc.in_features, num_classes)
# 只有最后一层是随机初始化的,其他层都用预训练权重
print(f"总参数量: {sum(p.numel() for p in model.parameters()):,}")
# ==========================================
# 其他可用的预训练模型
# ==========================================
# ResNet 系列
model = models.resnet34(weights='DEFAULT')
model = models.resnet50(weights='DEFAULT')
model = models.resnet101(weights='DEFAULT')
model = models.resnet152(weights='DEFAULT')
# VGG 系列
model = models.vgg16(weights='DEFAULT')
# EfficientNet 系列(效率更高)
model = models.efficientnet_b0(weights='DEFAULT')
model = models.efficientnet_b4(weights='DEFAULT')
# Vision Transformer(ViT)
model = models.vit_b_16(weights='DEFAULT')
# ==========================================
# 策略1:冻结所有层,只训练最后一层(最快,适合小数据集)
# ==========================================
for param in model.parameters():
param.requires_grad = False
model.fc = nn.Linear(model.fc.in_features, num_classes)
# ==========================================
# 策略2:冻结前面的层,训练后面的层
# ==========================================
for name, param in model.named_parameters():
if 'layer4' not in name and 'fc' not in name:
param.requires_grad = False
# ==========================================
# 策略3:分层学习率(推荐!效果最好)
# ==========================================
optimizer = torch.optim.Adam([
{'params': model.layer1.parameters(), 'lr': 1e-5}, # 最底层:最小学习率
{'params': model.layer2.parameters(), 'lr': 1e-5},
{'params': model.layer3.parameters(), 'lr': 1e-4},
{'params': model.layer4.parameters(), 'lr': 1e-4}, # 高层:较大学习率
{'params': model.fc.parameters(), 'lr': 1e-3}, # 新层:最大学习率
])
# 查看哪些参数需要训练
for name, param in model.named_parameters():
print(f"{name}: requires_grad={param.requires_grad}")
from torchvision import datasets, transforms, models
from torch.utils.data import DataLoader
import torch.nn as nn
import torch.optim as optim
# 数据预处理(使用 ImageNet 的均值和标准差)
transform = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
# 加载你的数据集
train_dataset = datasets.ImageFolder('./data/train', transform=transform)
test_dataset = datasets.ImageFolder('./data/test', transform=transform)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64)
# 加载预训练模型并修改
model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
model.fc = nn.Sequential(
nn.Dropout(0.5),
nn.Linear(model.fc.in_features, len(train_dataset.classes))
)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
# 冻结前面的层
for param in model.layer1.parameters():
param.requires_grad = False
for param in model.layer2.parameters():
param.requires_grad = False
# 训练
criterion = nn.CrossEntropyLoss()
optimizer = optim.AdamW(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=20)
for epoch in range(20):
model.train()
for data, target in train_loader:
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
output = model(data)
loss = criterion(output, target)
loss.backward()
optimizer.step()
scheduler.step()
print(f"Epoch {epoch+1} completed")
# ==========================================
# 方法1:特征提取(冻结所有层,只训练分类器)
# ==========================================
model = models.resnet50(weights='DEFAULT')
for param in model.parameters():
param.requires_grad = False # 冻结所有层
# 只替换最后一层
model.fc = nn.Sequential(
nn.Dropout(0.5),
nn.Linear(model.fc.in_features, num_classes)
)
# 只有 fc 层的参数会被训练
optimizer = optim.Adam(model.fc.parameters(), lr=1e-3)
# ==========================================
# 方法2:微调(解冻部分层,用小学习率)
# ==========================================
model = models.resnet50(weights='DEFAULT')
model.fc = nn.Linear(model.fc.in_features, num_classes)
# 先冻结所有层
for param in model.parameters():
param.requires_grad = False
# 解冻最后两层
for param in model.layer4.parameters():
param.requires_grad = True
for param in model.fc.parameters():
param.requires_grad = True
# 分层学习率
optimizer = optim.AdamW([
{'params': model.layer4.parameters(), 'lr': 1e-5},
{'params': model.fc.parameters(), 'lr': 1e-3},
])
# pip install timm
import timm
# 查看所有可用模型
print(timm.list_models('resnet*'))
# 加载预训练模型
model = timm.create_model('efficientnet_b0', pretrained=True, num_classes=10)
# 查看模型信息
print(timm.create_model('vit_base_patch16_224', pretrained=True))
迁移学习不仅在 CV 中有效,在 NLP 中同样重要:
Pythonfrom transformers import BertModel, BertTokenizer
# 加载预训练的 BERT
tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
bert = BertModel.from_pretrained('bert-base-chinese')
# 冻结 BERT 的参数
for param in bert.parameters():
param.requires_grad = False
# 在 BERT 上添加分类头
class BERTClassifier(nn.Module):
def __init__(self, num_classes):
super().__init__()
self.bert = bert
self.dropout = nn.Dropout(0.1)
self.fc = nn.Linear(768, num_classes) # 768 是 BERT 的隐藏维度
def forward(self, input_ids, attention_mask):
outputs = self.bert(input_ids, attention_mask=attention_mask)
pooled = outputs.pooler_output # [CLS] token 的输出
return self.fc(self.dropout(pooled))
import torch
# 检测 GPU
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(f"使用设备: {device}")
if torch.cuda.is_available():
print(f"GPU: {torch.cuda.get_device_name(0)}")
print(f"显存: {torch.cuda.get_device_properties(0).total_mem / 1024**3:.1f} GB")
# 模型和数据移到 GPU
model = model.to(device)
data = data.to(device)
target = target.to(device)
# 清理 GPU 缓存
torch.cuda.empty_cache()
混合精度训练使用 float16 代替 float32,显存减少一半,速度提升 2-3 倍,精度几乎不变。
Pythonfrom torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()
for data, target in train_loader:
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
# 前向传播用 float16
with autocast():
output = model(data)
loss = criterion(output, target)
# 反向传播用 GradScaler 处理梯度
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
# 一行代码,但效率不高(有 GIL 瓶颈)
if torch.cuda.device_count() > 1:
print(f"使用 {torch.cuda.device_count()} 个 GPU")
model = nn.DataParallel(model)
model = model.to(device)
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import DistributedSampler
def setup(rank, world_size):
dist.init_process_group("nccl", rank=rank, world_size=world_size)
torch.cuda.set_device(rank)
def cleanup():
dist.destroy_process_group()
def train(rank, world_size):
setup(rank, world_size)
model = MyModel().to(rank)
model = DDP(model, device_ids=[rank])
sampler = DistributedSampler(train_dataset, num_replicas=world_size, rank=rank)
loader = DataLoader(train_dataset, batch_size=32, sampler=sampler)
optimizer = torch.optim.Adam(model.parameters())
for epoch in range(10):
sampler.set_epoch(epoch)
for data, target in loader:
data, target = data.to(rank), target.to(rank)
optimizer.zero_grad()
loss = criterion(model(data), target)
loss.backward()
optimizer.step()
cleanup()
# 启动
world_size = torch.cuda.device_count()
mp.spawn(train, args=(world_size,), nprocs=world_size)
torch.cuda.empty_cache() 清理缓存# 梯度累积:batch_size=4, 累积4步 = 等效 batch_size=16
accumulation_steps = 4
for i, (data, target) in enumerate(train_loader):
data, target = data.to(device), target.to(device)
loss = criterion(model(data), target) / accumulation_steps
loss.backward()
if (i + 1) % accumulation_steps == 0:
optimizer.step()
optimizer.zero_grad()
from torch.utils.checkpoint import checkpoint
class CheckpointedModel(nn.Module):
"""用梯度检查点节省显存(用时间换空间)"""
def __init__(self):
super().__init__()
self.layer1 = nn.Linear(1000, 1000)
self.layer2 = nn.Linear(1000, 1000)
self.layer3 = nn.Linear(1000, 10)
def forward(self, x):
# checkpoint 不保存中间结果,反向传播时重新计算
x = checkpoint(self.layer1, x)
x = checkpoint(self.layer2, x)
x = self.layer3(x)
return x
# 显存减少约 30-50%,但训练速度慢约 20-30%
# 方式1:torchrun(推荐,PyTorch 2.0+)
torchrun --nproc_per_node=4 train.py
# 方式2:python -m torch.distributed.launch(旧方式)
python -m torch.distributed.launch --nproc_per_node=4 train.py
# 方式3:在代码中用 mp.spawn
# 见上面的 DDP 示例
# 通用的设备处理模板
def get_device():
if torch.cuda.is_available():
return torch.device('cuda')
elif hasattr(torch.backends, 'mps') and torch.backends.mps.is_available():
return torch.device('mps') # Apple Silicon
else:
return torch.device('cpu')
device = get_device()
print(f"Using device: {device}")
# 所有张量和模型都要移到同一设备
model = model.to(device)
for data, target in train_loader:
data, target = data.to(device), target.to(device)
import os
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data.distributed import DistributedSampler
def setup(rank, world_size):
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'
dist.init_process_group("nccl", rank=rank, world_size=world_size)
torch.cuda.set_device(rank)
def cleanup():
dist.destroy_process_group()
def train(rank, world_size):
setup(rank, world_size)
# 模型
model = MyModel().to(rank)
model = DDP(model, device_ids=[rank])
# 数据(分布式采样器)
sampler = DistributedSampler(
train_dataset,
num_replicas=world_size,
rank=rank,
shuffle=True
)
loader = DataLoader(
train_dataset,
batch_size=32,
sampler=sampler,
num_workers=4,
pin_memory=True
)
# 优化器
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
# 训练
for epoch in range(10):
sampler.set_epoch(epoch) # 每个 epoch 打乱顺序
model.train()
for data, target in loader:
data, target = data.to(rank), target.to(rank)
optimizer.zero_grad()
loss = criterion(model(data), target)
loss.backward()
optimizer.step()
# 只在 rank 0 上保存模型
if rank == 0:
torch.save(model.module.state_dict(), 'model.pth')
cleanup()
# 启动
if __name__ == "__main__":
world_size = torch.cuda.device_count()
torch.multiprocessing.spawn(train, args=(world_size,), nprocs=world_size)
del tensor; torch.cuda.empty_cache()relu_() 而不是 relu()import torch
# ==========================================
# 方法1:只保存参数(推荐!)
# ==========================================
torch.save(model.state_dict(), 'model_weights.pth')
# 加载
model = MyModel() # 先创建模型结构
model.load_state_dict(torch.load('model_weights.pth'))
model.eval()
# ==========================================
# 方法2:保存整个模型(不推荐,不灵活)
# ==========================================
torch.save(model, 'model_full.pth')
model = torch.load('model_full.pth')
# 保存 checkpoint
checkpoint = {
'epoch': epoch,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'scheduler_state_dict': scheduler.state_dict(),
'loss': loss,
'best_acc': best_acc,
}
torch.save(checkpoint, 'checkpoint.pth')
# 加载 checkpoint(断点续训)
checkpoint = torch.load('checkpoint.pth')
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
scheduler.load_state_dict(checkpoint['scheduler_state_dict'])
start_epoch = checkpoint['epoch'] + 1
best_acc = checkpoint['best_acc']
# 导出为 ONNX(通用格式,可在多种硬件上运行)
dummy_input = torch.randn(1, 3, 224, 224).to(device)
torch.onnx.export(
model, dummy_input, "model.onnx",
input_names=['input'], output_names=['output'],
dynamic_axes={'input': {0: 'batch_size'}},
opset_version=11,
)
# 用 ONNX Runtime 推理
import onnxruntime as ort
session = ort.InferenceSession("model.onnx")
result = session.run(None, {"input": dummy_input.cpu().numpy()})
# Trace(适合没有控制流的模型)
traced_model = torch.jit.trace(model, dummy_input)
traced_model.save("traced_model.pt")
# Script(适合有控制流的模型)
scripted_model = torch.jit.script(model)
scripted_model.save("scripted_model.pt")
# 加载(不需要 Python 环境)
loaded = torch.jit.load("traced_model.pt")
output = loaded(dummy_input)
# 安装 TorchServe
pip install torchserve torch-model-archiver
# 打包模型
torch-model-archiver --model-name resnet18 \
--version 1.0 \
--serialized-file model_weights.pth \
--handler image_classifier \
--export-path model_store
# 启动服务
torchserve --start --model-store model_store --models resnet18=resnet18.mar
# 测试
curl http://localhost:8080/predictions/resnet18 -T cat.jpg
# app.py - 用 FastAPI 部署模型(简单、高效、自动文档)
from fastapi import FastAPI, UploadFile, File
from PIL import Image
import torch
import torchvision.transforms as transforms
import io
app = FastAPI()
# 加载模型(启动时加载一次)
model = models.resnet18(num_classes=10)
model.load_state_dict(torch.load('best_model.pth', map_location='cpu'))
model.eval()
transform = transforms.Compose([
transforms.Resize((224, 224)),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
CLASSES = ['飞机', '汽车', '鸟', '猫', '鹿', '狗', '青蛙', '马', '船', '卡车']
@app.post("/predict")
async def predict(file: UploadFile = File(...)):
# 读取图片
image = Image.open(io.BytesIO(await file.read())).convert('RGB')
tensor = transform(image).unsqueeze(0)
# 推理
with torch.no_grad():
output = model(tensor)
probs = torch.softmax(output, dim=1)
pred_idx = probs.argmax(dim=1).item()
confidence = probs[0][pred_idx].item()
return {
"class": CLASSES[pred_idx],
"confidence": round(confidence, 4)
}
# 启动:uvicorn app:app --host 0.0.0.0 --port 8000
# 访问 http://localhost:8000/docs 查看自动 API 文档
state_dict(),不要保存整个模型import os
from datetime import datetime
class ModelManager:
"""模型版本管理器"""
def __init__(self, save_dir='checkpoints'):
self.save_dir = save_dir
os.makedirs(save_dir, exist_ok=True)
def save(self, model, optimizer, epoch, metrics, version=None):
"""保存模型"""
if version is None:
version = datetime.now().strftime('%Y%m%d_%H%M%S')
checkpoint = {
'version': version,
'epoch': epoch,
'model_state_dict': model.state_dict(),
'optimizer_state_dict': optimizer.state_dict(),
'metrics': metrics,
}
path = os.path.join(self.save_dir, f'model_{version}.pth')
torch.save(checkpoint, path)
print(f"模型已保存: {path}")
return path
def load(self, model, path, optimizer=None):
"""加载模型"""
checkpoint = torch.load(path)
model.load_state_dict(checkpoint['model_state_dict'])
if optimizer:
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
return checkpoint['epoch'], checkpoint['metrics']
def list_models(self):
"""列出所有保存的模型"""
files = [f for f in os.listdir(self.save_dir) if f.endswith('.pth')]
return sorted(files)
# 使用
manager = ModelManager()
manager.save(model, optimizer, epoch=10, metrics={'acc': 95.5})
manager.load(model, 'checkpoints/model_20240101_120000.pth')
import onnxruntime as ort
import numpy as np
# 创建推理会话
session = ort.InferenceSession(
"model.onnx",
providers=['CUDAExecutionProvider', 'CPUExecutionProvider'] # 优先用 GPU
)
# 获取输入输出名
input_name = session.get_inputs()[0].name
output_name = session.get_outputs()[0].name
# 推理
def predict_onnx(image):
# 预处理
img_tensor = transform(image).numpy()
# 推理
result = session.run([output_name], {input_name: img_tensor})
# 后处理
probs = np.exp(result[0]) / np.sum(np.exp(result[0]))
return probs
# 性能对比
import time
# PyTorch 推理
start = time.time()
for _ in range(100):
with torch.no_grad():
output = model(img_tensor)
pytorch_time = time.time() - start
# ONNX 推理
start = time.time()
for _ in range(100):
result = session.run([output_name], {input_name: img_numpy})
onnx_time = time.time() - start
print(f"PyTorch: {pytorch_time:.3f}s")
print(f"ONNX: {onnx_time:.3f}s")
print(f"加速: {pytorch_time / onnx_time:.2f}x")
训练时随机"关掉"一部分神经元,防止模型死记硬背(过拟合)。
Pythonclass MyModel(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(784, 256)
self.dropout = nn.Dropout(p=0.5) # 50% 的神经元被关掉
self.fc2 = nn.Linear(256, 10)
def forward(self, x):
x = F.relu(self.fc1(x))
x = self.dropout(x) # 训练时随机关闭,eval 时自动关闭
return self.fc2(x)
# BatchNorm:对每个 batch 做归一化(CNN 常用)
bn = nn.BatchNorm2d(64)
# LayerNorm:对每个样本做归一化(Transformer 常用)
ln = nn.LayerNorm(512)
import torch.nn.init as init
def init_weights(m):
if isinstance(m, nn.Linear):
init.kaiming_normal_(m.weight, mode='fan_in', nonlinearity='relu')
if m.bias is not None:
init.zeros_(m.bias)
elif isinstance(m, nn.Conv2d):
init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
elif isinstance(m, nn.BatchNorm2d):
init.ones_(m.weight)
init.zeros_(m.bias)
model.apply(init_weights)
# 防止梯度爆炸(RNN/Transformer 训练常用)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
class EarlyStopping:
def __init__(self, patience=10, min_delta=0.001):
self.patience = patience
self.min_delta = min_delta
self.counter = 0
self.best_loss = None
self.should_stop = False
def __call__(self, val_loss):
if self.best_loss is None:
self.best_loss = val_loss
elif val_loss > self.best_loss - self.min_delta:
self.counter += 1
if self.counter >= self.patience:
self.should_stop = True
else:
self.best_loss = val_loss
self.counter = 0
# 使用
early_stopping = EarlyStopping(patience=10)
for epoch in range(100):
train(...)
val_loss = validate(...)
early_stopping(val_loss)
if early_stopping.should_stop:
print(f"Early stopping at epoch {epoch}")
break
def mixup_data(x, y, alpha=0.2):
"""Mixup:混合两张图片训练,提升泛化能力"""
if alpha > 0:
lam = torch.distributions.Beta(alpha, alpha).sample()
else:
lam = 1
batch_size = x.size(0)
index = torch.randperm(batch_size).to(x.device)
mixed_x = lam * x + (1 - lam) * x[index]
y_a, y_b = y, y[index]
return mixed_x, y_a, y_b, lam
def mixup_criterion(criterion, pred, y_a, y_b, lam):
return lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)
# 使用
for data, target in train_loader:
data, target = data.to(device), target.to(device)
mixed_data, y_a, y_b, lam = mixup_data(data, target)
output = model(mixed_data)
loss = mixup_criterion(criterion, output, y_a, y_b, lam)
# Label Smoothing:软化标签,防止模型过度自信
# 原始标签:[0, 0, 1, 0] → 平滑后:[0.025, 0.025, 0.925, 0.025]
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)
| 方法 | 原理 | 适用场景 | 代码 |
|---|---|---|---|
| Dropout | 随机关闭神经元 | 全连接层 | nn.Dropout(0.5) |
| BatchNorm | 归一化每个 batch | CNN | nn.BatchNorm2d(64) |
| LayerNorm | 归一化每个样本 | Transformer | nn.LayerNorm(512) |
| 权重衰减 | L2 正则化 | 所有模型 | weight_decay=0.01 |
| 数据增强 | 增加数据多样性 | 所有模型 | transforms.RandomCrop() |
| 早停 | 验证 loss 不降时停止 | 所有模型 | EarlyStopping() |
| Mixup | 混合两张图片 | 图像分类 | mixup_data() |
| Label Smoothing | 软化标签 | 分类任务 | label_smoothing=0.1 |
# 完整的训练循环模板
def train_model(model, train_loader, val_loader, epochs=100):
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
criterion = nn.CrossEntropyLoss(label_smoothing=0.1)
optimizer = optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.01)
scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, T_max=epochs)
early_stopping = EarlyStopping(patience=10)
best_val_acc = 0
for epoch in range(epochs):
# 训练
model.train()
train_loss, train_correct, train_total = 0, 0, 0
for data, target in train_loader:
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
output = model(data)
loss = criterion(output, target)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
train_loss += loss.item() * data.size(0)
train_correct += output.argmax(1).eq(target).sum().item()
train_total += target.size(0)
scheduler.step()
# 验证
model.eval()
val_loss, val_correct, val_total = 0, 0, 0
with torch.no_grad():
for data, target in val_loader:
data, target = data.to(device), target.to(device)
output = model(data)
val_loss += criterion(output, target).item() * data.size(0)
val_correct += output.argmax(1).eq(target).sum().item()
val_total += target.size(0)
val_loss /= val_total
val_acc = 100. * val_correct / val_total
# 保存最佳模型
if val_acc > best_val_acc:
best_val_acc = val_acc
torch.save(model.state_dict(), 'best_model.pth')
# 早停
early_stopping(val_loss)
if early_stopping.should_stop:
print(f"Early stopping at epoch {epoch+1}")
break
if (epoch + 1) % 10 == 0:
print(f"Epoch {epoch+1} | Train Acc: {100.*train_correct/train_total:.2f}% | "
f"Val Acc: {val_acc:.2f}% | Best: {best_val_acc:.2f}%")
TensorBoard 是最强大的训练可视化工具,可以实时查看 loss 曲线、模型图、图片、直方图等。
Pythonfrom torch.utils.tensorboard import SummaryWriter
writer = SummaryWriter('runs/experiment_1')
for epoch in range(100):
train_loss = train(...)
writer.add_scalar('Loss/train', train_loss, epoch)
writer.add_scalar('Accuracy/train', train_acc, epoch)
writer.add_graph(model, dummy_input) # 记录模型图
writer.close()
# 启动:tensorboard --logdir=runs
from torch.utils.tensorboard import SummaryWriter
import torchvision.utils as vutils
writer = SummaryWriter('runs/experiment_full')
# 1. 记录标量(loss、accuracy、学习率等)
for epoch in range(100):
writer.add_scalar('Loss/train', train_loss, epoch)
writer.add_scalar('Loss/val', val_loss, epoch)
writer.add_scalar('Accuracy/train', train_acc, epoch)
writer.add_scalar('Accuracy/val', val_acc, epoch)
writer.add_scalar('LR', optimizer.param_groups[0]['lr'], epoch)
# 2. 记录模型图
writer.add_graph(model, dummy_input)
# 3. 记录图片
# 记录数据增强后的图片
dataiter = iter(train_loader)
images, labels = next(dataiter)
img_grid = vutils.make_grid(images[:16], nrow=4, normalize=True)
writer.add_images('Train/Images', img_grid, epoch)
# 4. 记录权重和梯度的直方图
for name, param in model.named_parameters():
writer.add_histogram(f'Weights/{name}', param, epoch)
if param.grad is not None:
writer.add_histogram(f'Gradients/{name}', param.grad, epoch)
# 5. 记录 PR 曲线
writer.add_pr_curve('PR Curve', labels, predictions, epoch)
# 6. 记录文本
writer.add_text('Config', f'lr={lr}, batch_size={batch_size}', epoch)
writer.close()
# 启动 TensorBoard
# 命令行:tensorboard --logdir=runs --port=6006
# 浏览器打开:http://localhost:6006
Grad-CAM 能可视化模型"看"了图片的哪些区域来做判断。
Pythonclass GradCAM:
def __init__(self, model, target_layer):
self.model = model
self.gradients = None
self.activations = None
target_layer.register_forward_hook(self._forward_hook)
target_layer.register_backward_hook(self._backward_hook)
def _forward_hook(self, module, input, output):
self.activations = output.detach()
def _backward_hook(self, module, grad_input, grad_output):
self.gradients = grad_output[0].detach()
def generate(self, input_tensor, target_class):
self.model.eval()
output = self.model(input_tensor)
self.model.zero_grad()
output[0, target_class].backward()
weights = self.gradients.mean(dim=(2, 3), keepdim=True)
cam = (weights * self.activations).sum(dim=1, keepdim=True)
cam = F.relu(cam)
return cam / cam.max()
# Forward Hook:获取中间层输出
activation = {}
def get_activation(name):
def hook(model, input, output):
activation[name] = output.detach()
return hook
model.layer1.register_forward_hook(get_activation('layer1'))
output = model(input)
print(activation['layer1'].shape)
# 在训练循环中记录权重和梯度的分布
for name, param in model.named_parameters():
if param.grad is not None:
writer.add_histogram(f'weights/{name}', param, epoch)
writer.add_histogram(f'gradients/{name}', param.grad, epoch)
# 检测异常值
with torch.autograd.detect_anomaly():
loss = model(data)
loss.backward() # 如果有 NaN,会报错并告诉你在哪里
from torch.utils.tensorboard import SummaryWriter
import time
class TrainingMonitor:
"""训练过程监控器"""
def __init__(self, log_dir='runs'):
self.writer = SummaryWriter(log_dir)
self.start_time = time.time()
def log_metrics(self, epoch, metrics):
"""记录指标"""
for name, value in metrics.items():
self.writer.add_scalar(name, value, epoch)
def log_gradients(self, epoch, model):
"""记录梯度分布"""
for name, param in model.named_parameters():
if param.grad is not None:
self.writer.add_histogram(f'gradients/{name}', param.grad, epoch)
def log_weights(self, epoch, model):
"""记录权重分布"""
for name, param in model.named_parameters():
self.writer.add_histogram(f'weights/{name}', param, epoch)
def log_images(self, epoch, images, tag='images'):
"""记录图片"""
self.writer.add_images(tag, images, epoch)
def log_model_graph(self, model, input_tensor):
"""记录模型图"""
self.writer.add_graph(model, input_tensor)
def close(self):
self.writer.close()
# 使用
monitor = TrainingMonitor()
for epoch in range(100):
train_loss, train_acc = train(...)
val_loss, val_acc = validate(...)
monitor.log_metrics(epoch, {
'Loss/train': train_loss,
'Loss/val': val_loss,
'Accuracy/train': train_acc,
'Accuracy/val': val_acc,
'LR': optimizer.param_groups[0]['lr'],
})
if epoch % 10 == 0:
monitor.log_gradients(epoch, model)
monitor.log_weights(epoch, model)
monitor.close()
Optuna 是最流行的超参数调优框架,使用贝叶斯搜索自动找到最优参数。
Pythonimport optuna
def objective(trial):
lr = trial.suggest_float('lr', 1e-5, 1e-2, log=True)
hidden_dim = trial.suggest_categorical('hidden_dim', [64, 128, 256, 512])
dropout = trial.suggest_float('dropout', 0.1, 0.5)
model = build_model(hidden_dim, dropout).to(device)
optimizer = optim.Adam(model.parameters(), lr=lr)
for epoch in range(20):
train_one_epoch(model, train_loader, optimizer)
val_loss = evaluate(model, val_loader)
trial.report(val_loss, epoch)
if trial.should_prune():
raise optuna.exceptions.TrialPruned()
return val_loss
study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=100)
print(f"最佳参数: {study.best_params}")
import optuna
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
# 定义模型(参数可调)
class FlexibleModel(nn.Module):
def __init__(self, input_dim, hidden_dim, num_layers, dropout, num_classes):
super().__init__()
layers = []
dim = input_dim
for _ in range(num_layers):
layers.extend([nn.Linear(dim, hidden_dim), nn.ReLU(), nn.Dropout(dropout)])
dim = hidden_dim
layers.append(nn.Linear(dim, num_classes))
self.net = nn.Sequential(*layers)
def forward(self, x):
return self.net(x)
# Optuna 目标函数
def objective(trial):
# 定义搜索空间
hidden_dim = trial.suggest_categorical('hidden_dim', [64, 128, 256, 512])
num_layers = trial.suggest_int('num_layers', 1, 4)
dropout = trial.suggest_float('dropout', 0.1, 0.5)
lr = trial.suggest_float('lr', 1e-5, 1e-2, log=True)
batch_size = trial.suggest_categorical('batch_size', [32, 64, 128])
optimizer_name = trial.suggest_categorical('optimizer', ['Adam', 'AdamW', 'SGD'])
# 构建模型和数据
model = FlexibleModel(784, hidden_dim, num_layers, dropout, 10).to(device)
loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
if optimizer_name == 'Adam':
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
elif optimizer_name == 'AdamW':
optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
else:
optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9)
# 训练
best_val_acc = 0
for epoch in range(30):
model.train()
for data, target in loader:
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
loss = nn.CrossEntropyLoss()(model(data), target)
loss.backward()
optimizer.step()
# 验证
model.eval()
correct = 0
with torch.no_grad():
for data, target in val_loader:
data, target = data.to(device), target.to(device)
pred = model(data).argmax(1)
correct += pred.eq(target).sum().item()
val_acc = correct / len(val_dataset)
# 报告和剪枝
trial.report(val_acc, epoch)
if trial.should_prune():
raise optuna.exceptions.TrialPruned()
best_val_acc = max(best_val_acc, val_acc)
return best_val_acc
# 运行搜索
study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=50, timeout=3600) # 最多 50 次或 1 小时
# 结果
print(f"最佳参数: {study.best_params}")
print(f"最佳准确率: {study.best_value:.4f}")
# 可视化
optuna.visualization.plot_optimization_history(study)
optuna.visualization.plot_param_importances(study)
def find_lr(model, train_loader, criterion, init_lr=1e-7, final_lr=10, num_steps=100):
optimizer = optim.SGD(model.parameters(), lr=init_lr)
lr_schedule = optim.lr_scheduler.ExponentialLR(optimizer, gamma=(final_lr/init_lr)**(1/num_steps))
lrs, losses = [], []
model.train()
for i, (data, target) in enumerate(train_loader):
if i >= num_steps: break
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
loss = criterion(model(data), target)
loss.backward()
optimizer.step()
lrs.append(optimizer.param_groups[0]['lr'])
losses.append(loss.item())
lr_schedule.step()
# 画图找最优点
import matplotlib.pyplot as plt
plt.plot(lrs, losses)
plt.xscale('log')
plt.xlabel('Learning Rate')
plt.ylabel('Loss')
plt.savefig('lr_finder.png', dpi=150)
| 方法 | 原理 | 优点 | 缺点 |
|---|---|---|---|
| 网格搜索 | 遍历所有组合 | 简单、全面 | 慢、维度灾难 |
| 随机搜索 | 随机采样 | 比网格搜索快 | 可能错过最优 |
| 贝叶斯搜索 | 根据历史结果智能选择下一组 | 最高效 | 需要 Optuna 等工具 |
建议:先调学习率,再调模型大小,最后调正则化。
import optuna
def objective(trial):
# 学习率(对数尺度)
lr = trial.suggest_float('lr', 1e-5, 1e-2, log=True)
# 批次大小(2 的幂次)
batch_size = trial.suggest_categorical('batch_size', [16, 32, 64, 128])
# 隐藏层维度
hidden_dim = trial.suggest_categorical('hidden_dim', [64, 128, 256, 512])
# Dropout 率
dropout = trial.suggest_float('dropout', 0.1, 0.5)
# 权重衰减
weight_decay = trial.suggest_float('weight_decay', 1e-5, 1e-2, log=True)
# 优化器
optimizer_name = trial.suggest_categorical('optimizer', ['Adam', 'AdamW', 'SGD'])
# 学习率调度器
scheduler_name = trial.suggest_categorical('scheduler', ['CosineAnnealing', 'StepLR', 'ReduceLROnPlateau'])
# 构建模型
model = MyModel(hidden_dim=hidden_dim, dropout=dropout).to(device)
# 构建优化器
if optimizer_name == 'Adam':
optimizer = optim.Adam(model.parameters(), lr=lr, weight_decay=weight_decay)
elif optimizer_name == 'AdamW':
optimizer = optim.AdamW(model.parameters(), lr=lr, weight_decay=weight_decay)
else:
optimizer = optim.SGD(model.parameters(), lr=lr, momentum=0.9, weight_decay=weight_decay)
# 训练并返回验证损失
val_loss = train_and_evaluate(model, optimizer, batch_size, epochs=20)
return val_loss
# 运行搜索
study = optuna.create_study(direction='minimize')
study.optimize(objective, n_trials=100)
print(f"最佳参数: {study.best_params}")
print(f"最佳损失: {study.best_value:.4f}")
目标检测 = 分类 + 定位:不仅要识别图片中有什么,还要标出在哪里(用边界框 bounding box)。
分类告诉你"这张图里有猫",目标检测告诉你"猫在图片的左上角,坐标是 (100, 50) 到 (300, 250)"。就像你在照片上画一个框,框住每个目标物体。
| 模型 | 类型 | 速度 | 精度 | 适用场景 |
|---|---|---|---|---|
| YOLOv8 | 单阶段 | 极快 | 高 | 实时检测、边缘设备 |
| Faster R-CNN | 两阶段 | 中等 | 最高 | 精度要求高的场景 |
| DETR | Transformer | 中等 | 高 | 端到端检测,无需 NMS |
| SSD | 单阶段 | 快 | 中等 | 多尺度检测 |
from torchvision.models.detection import fasterrcnn_resnet50_fpn
from torchvision import transforms
from PIL import Image
# 加载预训练模型
model = fasterrcnn_resnet50_fpn(weights='DEFAULT')
model.eval()
# 预处理
image = Image.open('street.jpg')
img_tensor = transforms.ToTensor()(image)
# 推理
with torch.no_grad():
predictions = model([img_tensor])
boxes = predictions[0]['boxes'] # 边界框 [x1, y1, x2, y2]
labels = predictions[0]['labels'] # 类别
scores = predictions[0]['scores'] # 置信度
COCO_CLASSES = ['person', 'bicycle', 'car', 'motorcycle', ...]
for box, label, score in zip(boxes, labels, scores):
if score > 0.5:
print(f"{COCO_CLASSES[label-1]}: {score:.2f}")
from torchvision.models.segmentation import deeplabv3_resnet101
model = deeplabv3_resnet101(weights='DEFAULT')
model.eval()
with torch.no_grad():
output = model([img_tensor])['out']
prediction = output.argmax(0) # 每个像素的类别
from torchvision.models.detection import fasterrcnn_resnet50_fpn
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
# 加载预训练模型
model = fasterrcnn_resnet50_fpn(weights='DEFAULT')
# 修改类别数(例如:背景 + 你的类别)
num_classes = 5 # 4个类别 + 1个背景
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
# 数据格式要求
# 每个样本是一个字典:
# {
# 'boxes': tensor([[x1, y1, x2, y2], ...]), # 边界框
# 'labels': tensor([1, 2, ...]), # 类别
# 'image_id': tensor([0]),
# 'area': tensor([...]),
# 'iscrowd': tensor([0, 0, ...])
# }
# 训练
model.train()
optimizer = torch.optim.SGD(model.parameters(), lr=0.005, momentum=0.9, weight_decay=0.0005)
for epoch in range(10):
for images, targets in train_loader:
images = list(img.to(device) for img in images)
targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
loss_dict = model(images, targets)
losses = sum(loss for loss in loss_dict.values())
optimizer.zero_grad()
losses.backward()
optimizer.step()
from torchvision.models.segmentation import deeplabv3_resnet101
# 加载预训练模型
model = deeplabv3_resnet101(weights='DEFAULT')
# 修改类别数
model.classifier[4] = nn.Conv2d(256, num_classes, 1)
model.aux_classifier[4] = nn.Conv2d(256, num_classes, 1)
# 评估指标:像素精度和 mIoU
def pixel_accuracy(output, target):
correct = (output == target).sum()
total = target.numel()
return correct / total
def mean_iou(output, target, num_classes):
ious = []
for cls in range(num_classes):
pred_cls = (output == cls)
target_cls = (target == cls)
intersection = (pred_cls & target_cls).sum()
union = (pred_cls | target_cls).sum()
if union > 0:
ious.append(intersection / union)
return torch.mean(torch.stack(ious))
# 目标检测的数据格式
# 每个样本是一个字典,包含:
# - image: [3, H, W] 的图片张量
# - boxes: [N, 4] 的边界框,格式为 [x1, y1, x2, y2]
# - labels: [N] 的类别标签
# - image_id: 图片的唯一标识
# - area: 每个边界框的面积
# - iscrowd: 是否是 crowd 标注
class ObjectDetectionDataset(torch.utils.data.Dataset):
def __init__(self, images_dir, annotations_file, transform=None):
self.images_dir = images_dir
self.annotations = self.load_annotations(annotations_file)
self.transform = transform
def __len__(self):
return len(self.annotations)
def __getitem__(self, idx):
ann = self.annotations[idx]
image = Image.open(os.path.join(self.images_dir, ann['file_name']))
boxes = torch.tensor(ann['boxes'], dtype=torch.float32)
labels = torch.tensor(ann['labels'], dtype=torch.int64)
target = {
'boxes': boxes,
'labels': labels,
'image_id': torch.tensor([idx]),
'area': (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0]),
'iscrowd': torch.zeros(len(boxes), dtype=torch.int64)
}
if self.transform:
image = self.transform(image)
return image, target
from torchvision.models.detection import fasterrcnn_resnet50_fpn
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
# 加载预训练模型
model = fasterrcnn_resnet50_fpn(weights='DEFAULT')
# 修改类别数(背景 + 你的类别)
num_classes = 5 # 4个类别 + 1个背景
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
# 训练
model.train()
optimizer = torch.optim.SGD(model.parameters(), lr=0.005, momentum=0.9, weight_decay=0.0005)
for epoch in range(10):
for images, targets in train_loader:
images = list(img.to(device) for img in images)
targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
loss_dict = model(images, targets)
losses = sum(loss for loss in loss_dict.values())
optimizer.zero_grad()
losses.backward()
optimizer.step()
GAN 就像造假币的(生成器)和警察(判别器)的博弈。造假币的想骗过警察,警察想识破假币。两者不断进步,最终造假币的技术越来越好,警察也越来越厉害。最终,生成器能生成以假乱真的数据。
# 原始 GAN 的损失函数(JS 散度)
# 判别器损失:让真实图片输出 1,假图片输出 0
d_loss = -torch.mean(torch.log(D(real)) + torch.log(1 - D(fake)))
# 生成器损失:让判别器对假图片输出 1
g_loss = -torch.mean(torch.log(D(fake)))
# 实际训练中常用 BCELoss
criterion = nn.BCELoss()
d_loss = criterion(D(real), real_labels) + criterion(D(fake.detach()), fake_labels)
g_loss = criterion(D(fake), real_labels) # 生成器希望判别器认为假的是真的
class Generator(nn.Module):
def __init__(self, latent_dim=100):
super().__init__()
self.main = nn.Sequential(
nn.ConvTranspose2d(latent_dim, 256, 4, 1, 0, bias=False),
nn.BatchNorm2d(256), nn.ReLU(),
nn.ConvTranspose2d(256, 128, 4, 2, 1, bias=False),
nn.BatchNorm2d(128), nn.ReLU(),
nn.ConvTranspose2d(128, 64, 4, 2, 1, bias=False),
nn.BatchNorm2d(64), nn.ReLU(),
nn.ConvTranspose2d(64, 1, 4, 2, 3, bias=False),
nn.Tanh()
)
def forward(self, x): return self.main(x)
class Discriminator(nn.Module):
def __init__(self):
super().__init__()
self.main = nn.Sequential(
nn.Conv2d(1, 64, 4, 2, 1, bias=False),
nn.LeakyReLU(0.2),
nn.Conv2d(64, 128, 4, 2, 1, bias=False),
nn.BatchNorm2d(128), nn.LeakyReLU(0.2),
nn.Conv2d(128, 1, 7, 1, 0, bias=False),
nn.Sigmoid()
)
def forward(self, x): return self.main(x).view(-1)
# 训练循环
G = Generator().to(device)
D = Discriminator().to(device)
criterion = nn.BCELoss()
opt_G = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
for epoch in range(100):
for real_imgs, _ in train_loader:
batch_size = real_imgs.size(0)
real_imgs = real_imgs.to(device)
real_labels = torch.ones(batch_size, device=device)
fake_labels = torch.zeros(batch_size, device=device)
# 训练判别器
z = torch.randn(batch_size, 100, 1, 1, device=device)
fake_imgs = G(z)
d_loss = criterion(D(real_imgs), real_labels) + criterion(D(fake_imgs.detach()), fake_labels)
opt_D.zero_grad(); d_loss.backward(); opt_D.step()
# 训练生成器
g_loss = criterion(D(fake_imgs), real_labels)
opt_G.zero_grad(); g_loss.backward(); opt_G.step()
import torchvision.utils as vutils
# 生成图片
G.eval()
with torch.no_grad():
z = torch.randn(64, 100, 1, 1, device=device)
fake_imgs = G(z).cpu()
# 保存为网格图
grid = vutils.make_grid(fake_imgs, nrow=8, normalize=True, value_range=(-1, 1))
plt.figure(figsize=(10, 10))
plt.imshow(grid.permute(1, 2, 0).numpy())
plt.axis('off')
plt.savefig('generated_images.png', dpi=150)
plt.show()
| 变体 | 特点 | 适用场景 |
|---|---|---|
| DCGAN | 用卷积层,最经典的 GAN | 图片生成 |
| WGAN | 用 Wasserstein 距离,训练更稳定 | 需要稳定训练时 |
| StyleGAN | 可以控制生成图片的风格 | 人脸生成、风格控制 |
| CycleGAN | 不需要配对数据的风格迁移 | 图片风格转换 |
| Pix2Pix | 需要配对数据的图像翻译 | 线稿→照片、黑白→彩色 |
import torchvision.utils as vutils
def train_gan(train_loader, epochs=100):
G = Generator(latent_dim=100).to(device)
D = Discriminator().to(device)
criterion = nn.BCELoss()
opt_G = torch.optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999))
opt_D = torch.optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999))
fixed_noise = torch.randn(64, 100, 1, 1, device=device) # 用于可视化
for epoch in range(epochs):
for i, (real_imgs, _) in enumerate(train_loader):
batch_size = real_imgs.size(0)
real_imgs = real_imgs.to(device)
real_labels = torch.ones(batch_size, device=device)
fake_labels = torch.zeros(batch_size, device=device)
# ---- 训练判别器 ----
z = torch.randn(batch_size, 100, 1, 1, device=device)
fake_imgs = G(z)
d_real = D(real_imgs)
d_fake = D(fake_imgs.detach())
d_loss = criterion(d_real, real_labels) + criterion(d_fake, fake_labels)
opt_D.zero_grad()
d_loss.backward()
opt_D.step()
# ---- 训练生成器 ----
d_fake = D(fake_imgs)
g_loss = criterion(d_fake, real_labels)
opt_G.zero_grad()
g_loss.backward()
opt_G.step()
# 每个 epoch 保存生成的图片
if (epoch + 1) % 10 == 0:
G.eval()
with torch.no_grad():
fake = G(fixed_noise).cpu()
grid = vutils.make_grid(fake, nrow=8, normalize=True)
plt.figure(figsize=(10, 10))
plt.imshow(grid.permute(1, 2, 0).numpy())
plt.axis('off')
plt.savefig(f'generated_epoch_{epoch+1}.png', dpi=150)
plt.close()
G.train()
print(f"Epoch {epoch+1} | D_loss: {d_loss.item():.4f} | G_loss: {g_loss.item():.4f}")
量化把 float32 参数变成 int8,模型大小减少 4 倍,推理速度提升 2-4 倍。
想象你有一本用 64 位浮点数写的书(float32),每个数字占用 4 个字节。量化就像把这本书改写成用整数(int8)表示,每个数字只占 1 个字节。书变薄了 4 倍,读起来也更快了,但数字的精度会稍微降低。
| 方式 | 原理 | 精度损失 | 使用难度 |
|---|---|---|---|
| 动态量化 | 推理时动态计算缩放因子 | 最小 | 最简单 |
| 静态量化 | 用校准数据预计算缩放因子 | 小 | 中等 |
| 量化感知训练 | 训练时模拟量化 | 最小 | 最复杂 |
# 动态量化(最简单,推荐入门使用)
quantized_model = torch.quantization.quantize_dynamic(
model, {nn.Linear, nn.LSTM}, dtype=torch.qint8
)
# 对比量化前后的模型大小
import os
torch.save(model.state_dict(), 'original.pth')
torch.save(quantized_model.state_dict(), 'quantized.pth')
orig_size = os.path.getsize('original.pth') / 1024 / 1024
quant_size = os.path.getsize('quantized.pth') / 1024 / 1024
print(f"原始模型: {orig_size:.1f} MB")
print(f"量化模型: {quant_size:.1f} MB")
print(f"压缩比: {orig_size / quant_size:.1f}x")
剪枝是移除模型中不重要的权重,让模型变得更稀疏、更小。
Pythonimport torch.nn.utils.prune as prune
# 对 Linear 层做 L1 剪枝(移除 30% 最小的权重)
for name, module in model.named_modules():
if isinstance(module, nn.Linear):
prune.l1_unstructured(module, name='weight', amount=0.3)
# 查看剪枝效果
total_params = 0
zero_params = 0
for name, param in model.named_parameters():
total_params += param.numel()
zero_params += (param == 0).sum().item()
print(f"总参数: {total_params:,}")
print(f"零参数: {zero_params:,}")
print(f"稀疏度: {zero_params / total_params * 100:.1f}%")
# 全局剪枝
parameters_to_prune = []
for name, module in model.named_modules():
if isinstance(module, (nn.Linear, nn.Conv2d)):
parameters_to_prune.append((module, 'weight'))
prune.global_unstructured(
parameters_to_prune,
pruning_method=prune.L1Unstructured,
amount=0.3 # 全局移除 30% 最小的权重
)
class DistillationLoss(nn.Module):
def __init__(self, temperature=4.0, alpha=0.7):
super().__init__()
self.temperature = temperature
self.alpha = alpha
self.ce_loss = nn.CrossEntropyLoss()
self.kl_loss = nn.KLDivLoss(reduction='batchmean')
def forward(self, student_logits, teacher_logits, labels):
hard_loss = self.ce_loss(student_logits, labels)
soft_student = F.log_softmax(student_logits / self.temperature, dim=1)
soft_teacher = F.softmax(teacher_logits / self.temperature, dim=1)
soft_loss = self.kl_loss(soft_student, soft_teacher) * (self.temperature ** 2)
return self.alpha * soft_loss + (1 - self.alpha) * hard_loss
| 方法 | 原理 | 压缩比 | 精度损失 |
|---|---|---|---|
| 量化 | float32 → int8 | 4x | 小 |
| 剪枝 | 移除不重要的权重 | 2-10x | 中 |
| 知识蒸馏 | 大模型教小模型 | 5-20x | 中 |
| 低秩分解 | 矩阵分解 | 2-4x | 小 |
| 框架 | 目标硬件 | 特点 | 适用场景 |
|---|---|---|---|
| ONNX Runtime | 跨平台 | 通用、易用 | 服务器、桌面端 |
| TensorRT | NVIDIA GPU | 最快、支持量化 | GPU 服务器、Jetson |
| Core ML | Apple 设备 | 原生支持 iPhone/Mac | iOS/macOS 应用 |
| TFLite | Android/嵌入式 | 轻量、支持移动端 | Android 应用 |
| OpenVINO | Intel 硬件 | CPU/GPU/VPU 优化 | Intel 平台 |
| ncnn | 移动端 | 极轻量、无依赖 | 移动端、嵌入式 |
import torch.quantization
# 量化感知训练:在训练时模拟量化,让模型适应量化后的精度损失
model = MyModel()
# 准备量化
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
model_prepared = torch.quantization.prepare_qat(model.train())
# 正常训练
for epoch in range(10):
for data, target in train_loader:
output = model_prepared(data)
loss = criterion(output, target)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# 转换为量化模型
model_quantized = torch.quantization.convert(model_prepared.eval())
# 量化后的模型大小和推理速度都会改善
print(f"原始模型大小: {os.path.getsize('model.pth') / 1024 / 1024:.1f} MB")
torch.save(model_quantized.state_dict(), 'model_quantized.pth')
print(f"量化模型大小: {os.path.getsize('model_quantized.pth') / 1024 / 1024:.1f} MB")
# 导出为 TorchScript(可以在移动端运行)
model.eval()
scripted_model = torch.jit.script(model)
scripted_model.save("model_mobile.pt")
# 导出为 ONNX(跨平台)
dummy_input = torch.randn(1, 3, 224, 224)
torch.onnx.export(model, dummy_input, "model.onnx",
input_names=['input'], output_names=['output'],
dynamic_axes={'input': {0: 'batch_size'}})
# 压缩效果对比示例
def compare_models(original, compressed, input_tensor):
"""对比原始模型和压缩模型"""
import time
# 模型大小
torch.save(original.state_dict(), 'orig.pth')
torch.save(compressed.state_dict(), 'comp.pth')
orig_size = os.path.getsize('orig.pth') / 1024 / 1024
comp_size = os.path.getsize('comp.pth') / 1024 / 1024
# 推理速度
original.eval()
start = time.time()
for _ in range(100):
with torch.no_grad(): original(input_tensor)
orig_time = time.time() - start
compressed.eval()
start = time.time()
for _ in range(100):
with torch.no_grad(): compressed(input_tensor)
comp_time = time.time() - start
print(f"原始模型: {orig_size:.1f}MB, {orig_time:.3f}s")
print(f"压缩模型: {comp_size:.1f}MB, {comp_time:.3f}s")
print(f"大小压缩: {orig_size/comp_size:.1f}x, 速度提升: {orig_time/comp_time:.1f}x")
PyTorch 2.0 最大的特性。一行代码,模型推理速度提升 30%-200%。
想象你在写一篇英语文章。每次写一个句子都要查字典(Python 的动态执行)。torch.compile 就像先把整篇文章翻译好,然后直接读翻译好的版本,不用再一个个查字典了。第一次翻译需要时间,但之后每次读都很快。
model = MyModel().cuda()
# 一行代码加速!
compiled_model = torch.compile(model)
# 编译模式
compiled_model = torch.compile(model, mode='reduce-overhead') # 减少开销
compiled_model = torch.compile(model, mode='max-autotune') # 最大性能
import torch
import time
# 测试 torch.compile 的加速效果
model = torch.nn.Sequential(
torch.nn.Linear(1024, 1024),
torch.nn.ReLU(),
torch.nn.Linear(1024, 1024),
torch.nn.ReLU(),
torch.nn.Linear(1024, 10)
).cuda()
compiled_model = torch.compile(model)
# 预热(第一次运行会编译,比较慢)
x = torch.randn(256, 1024, device='cuda')
_ = compiled_model(x)
# 测试原始模型
torch.cuda.synchronize()
start = time.time()
for _ in range(100):
with torch.no_grad():
_ = model(x)
torch.cuda.synchronize()
original_time = time.time() - start
# 测试编译后模型
torch.cuda.synchronize()
start = time.time()
for _ in range(100):
with torch.no_grad():
_ = compiled_model(x)
torch.cuda.synchronize()
compiled_time = time.time() - start
print(f"原始模型: {original_time:.3f}s")
print(f"编译模型: {compiled_time:.3f}s")
print(f"加速: {original_time / compiled_time:.2f}x")
| 模式 | 说明 | 适用场景 |
|---|---|---|
| default | 平衡编译时间和性能 | 大多数场景 |
| reduce-overhead | 减少 Python 开销 | 小模型、小 batch |
| max-autotune | 最大性能,编译慢 | 大模型、大 batch、追求极致性能 |
PyTorch 2.1+ 的新模型导出 API,比 TorchScript 更可靠。
Python# PyTorch 2.1+ 的新导出 API
exported = torch.export.export(model, (dummy_input,))
torch.export.save(exported, 'model.pt2')
# 加载导出的模型
loaded = torch.export.load('model.pt2')
output = loaded(dummy_input)
# 对比旧的 TorchScript
# torch.jit.trace: 不支持动态控制流
# torch.jit.script: 对 Python 语法有限制
# torch.export: 支持更多 Python 特性,更可靠
FSDP(Fully Sharded Data Parallel)是 PyTorch 的新一代分布式训练方案,比 DDP 更省显存。
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
from torch.distributed.fsdp import MixedPrecision
# 基本用法
model = MyModel().to(rank)
model = FSDP(model) # 比 DDP 更省显存
# 配合混合精度
mp_policy = MixedPrecision(
param_dtype=torch.float16,
reduce_dtype=torch.float16,
buffer_dtype=torch.float16,
)
model = FSDP(model, mixed_precision=mp_policy)
# 配合自动包装(推荐)
from torch.distributed.fsdp.wrap import size_based_auto_wrap_policy
auto_wrap_policy = size_based_auto_wrap_policy(min_num_params=1e6)
model = FSDP(model, auto_wrap_policy=auto_wrap_policy)
# PyTorch 2.0+ 的新 API
# 1. torch.compile(加速推理和训练)
model = torch.compile(model)
# 2. torch.export(新的模型导出 API)
exported = torch.export.export(model, (dummy_input,))
# 3. FSDP(更高效的分布式训练)
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP
model = FSDP(model)
# 4. scaled_dot_product_attention(更高效的注意力)
from torch.nn.functional import scaled_dot_product_attention
output = scaled_dot_product_attention(q, k, v)
# 5. torch.amp(新的混合精度 API)
with torch.amp.autocast('cuda'):
output = model(data)
torch.amp.autocast('cuda')PyTorch 2.0 内置了高效的注意力计算,自动使用 FlashAttention(如果硬件支持)。
Pythonimport torch.nn.functional as F
# 旧写法
scores = torch.matmul(Q, K.transpose(-2, -1)) / (d_k ** 0.5)
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9)
attn_weights = torch.softmax(scores, dim=-1)
output = torch.matmul(attn_weights, V)
# 新写法(自动使用 FlashAttention,更快更省显存)
output = F.scaled_dot_product_attention(Q, K, V, attn_mask=mask)
# 一行代码,自动优化,显存减少 5-20 倍,速度提升 2-4 倍
Python
# PyTorch 2.x 的性能优化代码模板
import torch
from torch.amp import autocast, GradScaler
# 1. 使用 torch.compile
model = torch.compile(model)
# 2. 使用混合精度
scaler = GradScaler()
with autocast('cuda'):
output = model(data)
loss = criterion(output, target)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
# 3. 使用更高效的注意力
from torch.nn.functional import scaled_dot_product_attention
q, k, v = ... # [batch, heads, seq_len, d_k]
output = scaled_dot_product_attention(q, k, v)
把一张图片的内容和另一张图片的风格结合起来。比如把你的照片变成梵高的《星空》风格。
想象你是一个画家,你看到一张风景照(内容),又看到梵高的《星空》(风格)。你要画一张画,风景是照片里的风景,但笔触是梵高的笔触。这就是风格迁移要做的事。
L = α × L_content + β × L_style,α 和 β 控制内容和风格的权重Gram 矩阵捕捉的是特征之间的相关性,代表了"风格"信息。比如天空和草地的颜色关系、笔触的方向分布等。
class StyleTransfer:
def __init__(self):
vgg = models.vgg19(pretrained=True).features
self.layers = nn.Sequential(*list(vgg.children())[:21])
for param in self.layers.parameters():
param.requires_grad = False
def gram_matrix(self, x):
b, c, h, w = x.size()
features = x.view(b, c, h * w)
return torch.bmm(features, features.transpose(1, 2)) / (c * h * w)
def transfer(self, content_img, style_img, steps=300):
generated = content_img.clone().requires_grad_(True)
optimizer = torch.optim.Adam([generated], lr=0.01)
for step in range(steps):
content_features = self.layers(content_img)
style_features = self.layers(style_img)
generated_features = self.layers(generated)
content_loss = F.mse_loss(generated_features, content_features)
style_loss = F.mse_loss(self.gram_matrix(generated_features),
self.gram_matrix(style_features))
total_loss = content_loss + 1e6 * style_loss
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
return generated
# 使用
st = StyleTransfer()
result = st.transfer(content_image, style_image)
# 保存结果
result_image = result.squeeze().clamp(0, 1)
plt.figure(figsize=(10, 10))
plt.imshow(result_image.permute(1, 2, 0).cpu().numpy())
plt.axis('off')
plt.savefig('style_transfer_result.png', dpi=150, bbox_inches='tight')
plt.show()
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.models as models
import torchvision.transforms as transforms
from PIL import Image
import matplotlib.pyplot as plt
# 加载图片的辅助函数
def load_image(path, size=512):
transform = transforms.Compose([
transforms.Resize((size, size)),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
image = Image.open(path).convert('RGB')
return transform(image).unsqueeze(0)
# 提取特征的层
STYLE_LAYERS = ['conv1_1', 'conv2_1', 'conv3_1', 'conv4_1', 'conv5_1']
CONTENT_LAYERS = ['conv4_2']
class VGGFeatures(nn.Module):
def __init__(self):
super().__init__()
vgg = models.vgg19(pretrained=True).features
self.layers = nn.ModuleList()
self.names = []
# 层名映射
layer_names = ['conv1_1', 'conv1_2', 'conv2_1', 'conv2_2',
'conv3_1', 'conv3_2', 'conv3_3', 'conv3_4',
'conv4_1', 'conv4_2', 'conv4_3', 'conv4_4',
'conv5_1', 'conv5_2', 'conv5_3', 'conv5_4']
i = 0
for layer in vgg.children():
if isinstance(layer, nn.Conv2d):
name = layer_names[i] if i < len(layer_names) else f'conv_{i}'
self.names.append(name)
self.layers.append(layer)
i += 1
elif isinstance(layer, nn.ReLU):
self.layers.append(nn.ReLU(inplace=False))
elif isinstance(layer, nn.MaxPool2d):
self.layers.append(layer)
for param in self.parameters():
param.requires_grad = False
def forward(self, x):
features = {}
for name, layer in zip(self.names, self.layers):
x = layer(x)
if name in STYLE_LAYERS or name in CONTENT_LAYERS:
features[name] = x
return features
def gram_matrix(x):
"""计算 Gram 矩阵(风格表示)"""
b, c, h, w = x.size()
features = x.view(b, c, h * w)
gram = torch.bmm(features, features.transpose(1, 2))
return gram / (c * h * w)
# 主程序
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
vgg = VGGFeatures().to(device).eval()
content_img = load_image('content.jpg').to(device)
style_img = load_image('style.jpg').to(device)
generated = content_img.clone().requires_grad_(True)
optimizer = torch.optim.Adam([generated], lr=0.01)
for step in range(500):
content_features = vgg(content_img)
style_features = vgg(style_img)
gen_features = vgg(generated)
# 内容损失
content_loss = F.mse_loss(gen_features['conv4_2'], content_features['conv4_2'])
# 风格损失
style_loss = 0
for name in STYLE_LAYERS:
gen_gram = gram_matrix(gen_features[name])
style_gram = gram_matrix(style_features[name])
style_loss += F.mse_loss(gen_gram, style_gram)
style_loss /= len(STYLE_LAYERS)
# 总损失
total_loss = content_loss + 1e6 * style_loss
optimizer.zero_grad()
total_loss.backward()
optimizer.step()
if (step + 1) % 100 == 0:
print(f"Step {step+1} | Content: {content_loss.item():.4f} | Style: {style_loss.item():.4f}")
# 保存结果
result = generated.squeeze().clamp(0, 1)
plt.imshow(result.permute(1, 2, 0).detach().cpu().numpy())
plt.axis('off')
plt.savefig('style_result.png', dpi=150, bbox_inches='tight')
class FastStyleTransfer(nn.Module):
"""快速风格迁移:训练一个前馈网络,一次前向传播完成风格迁移"""
def __init__(self):
super().__init__()
# 编码器
self.encoder = nn.Sequential(
nn.Conv2d(3, 32, 9, stride=1, padding=4),
nn.InstanceNorm2d(32),
nn.ReLU(),
nn.Conv2d(32, 64, 3, stride=2, padding=1),
nn.InstanceNorm2d(64),
nn.ReLU(),
nn.Conv2d(64, 128, 3, stride=2, padding=1),
nn.InstanceNorm2d(128),
nn.ReLU(),
)
# 残差块
self.residual = nn.Sequential(
ResidualBlock(128),
ResidualBlock(128),
ResidualBlock(128),
ResidualBlock(128),
ResidualBlock(128),
)
# 解码器
self.decoder = nn.Sequential(
nn.ConvTranspose2d(128, 64, 3, stride=2, padding=1, output_padding=1),
nn.InstanceNorm2d(64),
nn.ReLU(),
nn.ConvTranspose2d(64, 32, 3, stride=2, padding=1, output_padding=1),
nn.InstanceNorm2d(32),
nn.ReLU(),
nn.Conv2d(32, 3, 9, stride=1, padding=4),
nn.Tanh()
)
def forward(self, x):
x = self.encoder(x)
x = self.residual(x)
x = self.decoder(x)
return x
class ResidualBlock(nn.Module):
def __init__(self, channels):
super().__init__()
self.block = nn.Sequential(
nn.Conv2d(channels, channels, 3, padding=1),
nn.InstanceNorm2d(channels),
nn.ReLU(),
nn.Conv2d(channels, channels, 3, padding=1),
nn.InstanceNorm2d(channels),
)
def forward(self, x):
return x + self.block(x)
import re
from collections import Counter
class TextPreprocessor:
"""文本预处理器"""
def __init__(self, max_vocab_size=20000, max_len=256):
self.max_vocab_size = max_vocab_size
self.max_len = max_len
self.word2idx = {'<PAD>': 0, '<UNK>': 1}
self.idx2word = {0: '<PAD>', 1: '<UNK>'}
def clean_text(self, text):
"""清洗文本"""
text = re.sub(r'[^\w一-鿿]', ' ', text) # 保留中文和英文
text = re.sub(r'\s+', ' ', text).strip()
return text
def build_vocab(self, texts):
"""构建词汇表"""
counter = Counter()
for text in texts:
words = list(self.clean_text(text))
counter.update(words)
# 取最常见的词
for word, _ in counter.most_common(self.max_vocab_size - 2):
idx = len(self.word2idx)
self.word2idx[word] = idx
self.idx2word[idx] = word
def encode(self, text):
"""文本转索引"""
words = list(self.clean_text(text))
indices = [self.word2idx.get(w, 1) for w in words] # 1 = UNK
# 填充或截断
if len(indices) < self.max_len:
indices += [0] * (self.max_len - len(indices))
else:
indices = indices[:self.max_len]
return indices
# 使用
preprocessor = TextPreprocessor(max_vocab_size=20000, max_len=128)
preprocessor.build_vocab(train_texts)
encoded = preprocessor.encode("这部电影太好看了!")
from fastapi import FastAPI
from pydantic import BaseModel
app = FastAPI()
class SentimentRequest(BaseModel):
text: str
@app.post("/predict")
async def predict(request: SentimentRequest):
tokens = tokenize(request.text)
tensor = torch.tensor([tokens]).to(device)
with torch.no_grad():
output = model(tensor)
probs = torch.softmax(output, dim=1)
pred = probs.argmax(dim=1).item()
return {"sentiment": "正面" if pred == 1 else "负面",
"confidence": probs[0][pred].item()}
from torch.utils.data import Dataset, DataLoader
class SentimentDataset(Dataset):
"""情感分析数据集"""
def __init__(self, texts, labels, preprocessor):
self.texts = texts
self.labels = labels
self.preprocessor = preprocessor
def __len__(self):
return len(self.texts)
def __getitem__(self, idx):
encoded = self.preprocessor.encode(self.texts[idx])
return torch.tensor(encoded, dtype=torch.long), torch.tensor(self.labels[idx], dtype=torch.long)
# 构建数据集
train_dataset = SentimentDataset(train_texts, train_labels, preprocessor)
val_dataset = SentimentDataset(val_texts, val_labels, preprocessor)
train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=64)
print(f"训练集: {len(train_dataset)} 条")
print(f"验证集: {len(val_dataset)} 条")
print(f"词汇表大小: {len(preprocessor.word2idx)}")
class SentimentLSTM(nn.Module):
def __init__(self, vocab_size, embed_dim=128, hidden_dim=256, num_classes=2):
super().__init__()
self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
self.lstm = nn.LSTM(embed_dim, hidden_dim, num_layers=2,
batch_first=True, bidirectional=True, dropout=0.3)
self.attention = nn.Linear(hidden_dim * 2, 1)
self.fc = nn.Linear(hidden_dim * 2, num_classes)
self.dropout = nn.Dropout(0.5)
def forward(self, x):
embedded = self.dropout(self.embedding(x))
output, _ = self.lstm(embedded)
attn_weights = torch.softmax(self.attention(output), dim=1)
context = torch.sum(attn_weights * output, dim=1)
return self.fc(self.dropout(context))
# 训练
model = SentimentLSTM(vocab_size=5000).to(device)
optimizer = optim.Adam(model.parameters(), lr=1e-3)
for epoch in range(10):
model.train()
for texts, labels in train_loader:
texts, labels = texts.to(device), labels.to(device)
optimizer.zero_grad()
output = model(texts)
loss = F.cross_entropy(output, labels)
loss.backward()
optimizer.step()
from sklearn.metrics import classification_report, confusion_matrix
def evaluate_sentiment(model, test_loader):
model.eval()
all_preds, all_labels = [], []
with torch.no_grad():
for texts, labels in test_loader:
texts = texts.to(device)
output = model(texts)
preds = output.argmax(dim=1).cpu()
all_preds.extend(preds.tolist())
all_labels.extend(labels.tolist())
print(classification_report(all_labels, all_preds,
target_names=['负面', '正面']))
cm = confusion_matrix(all_labels, all_preds)
print(f"混淆矩阵:\n{cm}")
import matplotlib.pyplot as plt
import numpy as np
def plot_confusion_matrix(cm, classes=['负面', '正面']):
"""可视化混淆矩阵"""
fig, ax = plt.subplots(figsize=(6, 6))
im = ax.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
ax.set(xticks=[0, 1], yticks=[0, 1],
xticklabels=classes, yticklabels=classes,
xlabel='预测标签', ylabel='真实标签',
title='混淆矩阵')
# 在格子中标注数字
for i in range(2):
for j in range(2):
ax.text(j, i, str(cm[i, j]), ha='center', va='center',
color='white' if cm[i, j] > cm.max() / 2 else 'black')
plt.colorbar(im)
plt.tight_layout()
plt.savefig('confusion_matrix.png', dpi=150)
# 使用
cm = confusion_matrix(all_labels, all_preds)
plot_confusion_matrix(cm)
class MultiVarLSTM(nn.Module):
def __init__(self, input_size, hidden_size=128, num_layers=2):
super().__init__()
self.lstm = nn.LSTM(input_size, hidden_size, num_layers,
batch_first=True, dropout=0.2)
self.fc = nn.Sequential(
nn.Linear(hidden_size, 64), nn.ReLU(), nn.Linear(64, 1))
def forward(self, x):
output, _ = self.lstm(x)
return self.fc(output[:, -1, :])
# 滑动窗口数据
def create_sequences(data, seq_length):
sequences, targets = [], []
for i in range(len(data) - seq_length):
sequences.append(data[i:i+seq_length])
targets.append(data[i+seq_length])
return torch.stack(sequences), torch.stack(targets)
# 滚动预测
def rolling_predict(model, initial_seq, future_steps):
model.eval()
predictions, current_seq = [], initial_seq.clone()
with torch.no_grad():
for _ in range(future_steps):
pred = model(current_seq.unsqueeze(0))
predictions.append(pred.item())
current_seq = torch.cat([current_seq[1:], pred], dim=0)
return predictions
def evaluate_forecast(predictions, targets):
"""评估时间序列预测的常用指标"""
predictions = torch.tensor(predictions)
targets = torch.tensor(targets)
# MSE(均方误差)
mse = F.mse_loss(predictions, targets)
# RMSE(均方根误差)
rmse = torch.sqrt(mse)
# MAE(平均绝对误差)
mae = F.l1_loss(predictions, targets)
# MAPE(平均绝对百分比误差)
mape = torch.mean(torch.abs((targets - predictions) / targets)) * 100
return {'MSE': mse.item(), 'RMSE': rmse.item(),
'MAE': mae.item(), 'MAPE': mape.item()}
class TimeSeriesTransformer(nn.Module):
"""用 Transformer 做时间序列预测"""
def __init__(self, input_size, d_model=64, nhead=4, num_layers=2, pred_len=1):
super().__init__()
self.input_proj = nn.Linear(input_size, d_model)
self.pos_encoding = nn.Parameter(torch.randn(1, 500, d_model))
encoder_layer = nn.TransformerEncoderLayer(d_model=d_model, nhead=nhead, batch_first=True)
self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)
self.fc = nn.Linear(d_model, pred_len)
def forward(self, x):
# x: [batch, seq_len, input_size]
x = self.input_proj(x) + self.pos_encoding[:, :x.size(1), :]
x = self.transformer(x)
return self.fc(x[:, -1, :]) # 只用最后一个时间步的输出
# 使用
model = TimeSeriesTransformer(input_size=3, pred_len=7).to(device)
# 输入过去 30 天的数据,预测未来 7 天
x = torch.randn(32, 30, 3).to(device) # [batch=32, seq_len=30, features=3]
pred = model(x) # [32, 7]
import pandas as pd
import numpy as np
def add_time_features(df, date_col='date'):
"""添加时间特征"""
df = df.copy()
df[date_col] = pd.to_datetime(df[date_col])
# 基本时间特征
df['hour'] = df[date_col].dt.hour
df['day_of_week'] = df[date_col].dt.dayofweek # 0=周一
df['day_of_month'] = df[date_col].dt.day
df['month'] = df[date_col].dt.month
df['quarter'] = df[date_col].dt.quarter
df['day_of_year'] = df[date_col].dt.dayofyear
# 周期性编码(用 sin/cos 表示周期性)
df['hour_sin'] = np.sin(2 * np.pi * df['hour'] / 24)
df['hour_cos'] = np.cos(2 * np.pi * df['hour'] / 24)
df['dow_sin'] = np.sin(2 * np.pi * df['day_of_week'] / 7)
df['dow_cos'] = np.cos(2 * np.pi * df['day_of_week'] / 7)
df['month_sin'] = np.sin(2 * np.pi * df['month'] / 12)
df['month_cos'] = np.cos(2 * np.pi * df['month'] / 12)
# 是否周末
df['is_weekend'] = (df['day_of_week'] >= 5).astype(int)
return df
# 为什么用 sin/cos 编码?
# 因为 23:00 和 00:00 应该很接近,但直接用 23 和 0 差距很大
# sin/cos 能正确表示这种周期性关系
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
class TimeSeriesDataset(Dataset):
def __init__(self, data, seq_length, target_col=0):
self.data = torch.FloatTensor(data)
self.seq_length = seq_length
self.target_col = target_col
def __len__(self):
return len(self.data) - self.seq_length
def __getitem__(self, idx):
x = self.data[idx:idx+self.seq_length]
y = self.data[idx+self.seq_length, self.target_col]
return x, y
class LSTMPredictor(nn.Module):
def __init__(self, input_size, hidden_size=128, num_layers=2):
super().__init__()
self.lstm = nn.LSTM(input_size, hidden_size, num_layers,
batch_first=True, dropout=0.2)
self.fc = nn.Sequential(
nn.Linear(hidden_size, 64),
nn.ReLU(),
nn.Linear(64, 1)
)
def forward(self, x):
output, _ = self.lstm(x)
return self.fc(output[:, -1, :])
# 训练
model = LSTMPredictor(input_size=3).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()
for epoch in range(50):
model.train()
for x, y in train_loader:
x, y = x.to(device), y.to(device)
optimizer.zero_grad()
pred = model(x).squeeze()
loss = criterion(pred, y)
loss.backward()
optimizer.step()
全量微调一个 7B 参数的模型需要 ~60GB 显存(float16),而 LoRA 只需要 ~8GB。高效微调让普通开发者也能训大模型。
大模型微调时,权重的变化量是低秩的(low-rank)。LoRA 把权重更新分解为两个小矩阵的乘积:ΔW = A × B,其中 A 是 (d, r),B 是 (r, d),r 远小于 d。
例如:原始权重是 (4096, 4096) = 16M 参数,LoRA 用 r=16 时只需要 (4096×16)×2 = 128K 参数,减少了 125 倍。
想象一本 1000 页的教科书(基础模型)。全量微调就像重写整本书。LoRA 就像在书的空白处写笔记——你只添加了少量内容(低秩矩阵),但效果和重写整本书差不多。QLoRA 就像先把书压缩成小字版(4-bit 量化),再在空白处写笔记。
# LoRA 的数学原理
# 原始前向传播:h = W × x
# LoRA 前向传播:h = W × x + (A × B) × x
# 其中 W 冻结不动,只训练 A 和 B
# 直观理解:
# W 是一个 4096×4096 的大矩阵(16M 参数)
# A 是 4096×16,B 是 16×4096,总共 128K 参数
# 用两个小矩阵的乘积来近似权重的变化量
# pip install peft transformers accelerate bitsandbytes
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model, TaskType
import torch
# 1. 加载基础模型
model_name = "meta-llama/Llama-3.1-8B" # 或 "Qwen/Qwen2-7B"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype=torch.float16,
device_map="auto",
)
# 2. 配置 LoRA
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # LoRA 秩(越大效果越好,但参数越多)
lora_alpha=32, # 缩放因子
lora_dropout=0.05, # Dropout
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"], # 应用 LoRA 的层
)
# 3. 应用 LoRA
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# 输出:trainable params: 41,943,040 || all params: 8,072,204,288 || trainable%: 0.5196
# 4. 正常训练
optimizer = torch.optim.AdamW(model.parameters(), lr=2e-4)
for epoch in range(3):
for batch in train_loader:
outputs = model(**batch)
loss = outputs.loss
loss.backward()
optimizer.step()
optimizer.zero_grad()
# 5. 保存 LoRA 权重(只有几十 MB,不是整个模型)
model.save_pretrained("lora-llama3")
# 6. 加载和推理
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16)
model = PeftModel.from_pretrained(base_model, "lora-llama3")
model.merge_and_unload() # 合并 LoRA 到基础模型(可选)
# pip install bitsandbytes
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
# 4-bit 量化配置
bnb_config = BitsAndBytesConfig(
load_in_4bit=True, # 4-bit 加载
bnb_4bit_quant_type="nf4", # NF4 量化(最佳质量)
bnb_4bit_compute_dtype=torch.float16, # 计算用 float16
bnb_4bit_use_double_quant=True, # 双重量化(进一步压缩)
)
# 4-bit 加载模型(7B 模型只需 ~6GB 显存)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.1-8B",
quantization_config=bnb_config,
device_map="auto",
)
# 应用 LoRA(和上面一样)
lora_config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, lora_config)
# 显存占用对比:
# 全量微调 8B: ~60 GB
# LoRA (fp16): ~18 GB
# QLoRA (4-bit): ~6 GB ← 单张 RTX 4060 就能跑!
# pip install trl datasets
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset
# 加载数据集
dataset = load_dataset("tatsu-lab/alpaca", split="train")
# 格式化为对话格式
def format_prompt(example):
if example["input"]:
return f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"
else:
return f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
# 训练
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3,
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
learning_rate=2e-4,
fp16=True,
logging_steps=10,
save_strategy="epoch",
warmup_ratio=0.03,
)
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset,
formatting_func=format_prompt,
max_seq_length=2048,
)
trainer.train()
| 方法 | 可训练参数 | 显存需求 | 效果 | 适用场景 |
|---|---|---|---|---|
| 全量微调 | 100% | 极大 | 最好 | 资源充足 |
| LoRA | 0.1-1% | 小 | 接近全量 | 最常用 |
| QLoRA | 0.1-1% | 极小 | 接近全量 | 单卡微调 |
| Prefix Tuning | 0.1% | 小 | 中等 | NLP 生成任务 |
| Adapter | 1-5% | 中 | 好 | 多任务学习 |
# 合并 LoRA 权重到基础模型(推理时不需要 PEFT 库)
from peft import PeftModel
from transformers import AutoModelForCausalLM
# 加载基础模型
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")
# 加载 LoRA 权重
model = PeftModel.from_pretrained(base_model, "lora-llama3")
# 合并(LoRA 权重永久写入基础模型)
merged_model = model.merge_and_unload()
# 保存合并后的模型
merged_model.save_pretrained("merged-llama3-lora")
# 导出为 GGUF(用于 llama.cpp)
# pip install llama-cpp-python
# python convert_hf_to_gguf.py merged-llama3-lora --outtype q4_k_m
输入: "今天天气"
↓ 模型预测下一个 token
输出: "今天天气" + "真" → "今天天气真"
↓ 模型预测下一个 token
输出: "今天天气真" + "好" → "今天天气真好"
↓ 模型预测下一个 token
输出: "今天天气真好" + "。" → "今天天气真好。"
每生成一个 token,都需要一次完整的前向传播
这就是为什么大模型生成速度慢——它是串行的,不能并行
自回归生成时,每生成一个新 token,都需要计算所有 token 的注意力。但前面 token 的 Key 和 Value 不会变,所以可以缓存起来复用——这就是 KV Cache。
没有 KV Cache:生成第 N 个 token 需要计算 N 次注意力。
有了 KV Cache:生成第 N 个 token 只需要计算 1 次注意力(只算新 token 的 Q)。
代价:KV Cache 会占用大量显存。7B 模型、序列长度 4096 时,KV Cache 约占 2-4 GB。
# KV Cache 的简化实现
class CachedAttention(nn.Module):
def forward(self, x, past_kv=None):
Q, K, V = self.qkv(x).chunk(3, dim=-1)
if past_kv is not None:
past_K, past_V = past_kv
K = torch.cat([past_K, K], dim=1) # 拼接历史 K
V = torch.cat([past_V, V], dim=1) # 拼接历史 V
# 缓存当前的 K, V 供下一步使用
present_kv = (K, V)
# 注意力计算
attn = torch.matmul(Q, K.transpose(-2, -1)) / (Q.size(-1) ** 0.5)
attn = torch.softmax(attn, dim=-1)
output = torch.matmul(attn, V)
return output, present_kv
vLLM 是目前最流行的大模型推理引擎,核心技术是 PagedAttention(分页注意力),解决了 KV Cache 的显存碎片问题。
Bash# 安装 vLLM
pip install vllm
# 启动 OpenAI 兼容的 API 服务
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--tensor-parallel-size 1 \
--max-model-len 4096 \
--gpu-memory-utilization 0.9
# 使用 curl 测试
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "用Python写一个快排"}],
"max_tokens": 512
}'
Python
# 在代码中使用 vLLM
from vllm import LLM, SamplingParams
# 加载模型
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct", tensor_parallel_size=1)
# 生成
sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
prompts = ["解释什么是梯度下降", "写一个二分查找", "什么是 Transformer"]
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
print(f"Prompt: {output.prompt[:50]}...")
print(f"Generated: {output.outputs[0].text[:200]}")
print("---")
# vLLM 的核心优势:
# 1. PagedAttention: KV Cache 分页管理,显存利用率提升 2-4 倍
# 2. Continuous Batching: 动态批处理,吞吐量提升 2-24 倍
# 3. Tensor Parallelism: 多 GPU 并行推理
# 安装 Ollama(最简单的大模型本地部署方案)
# 访问 https://ollama.com 下载安装
# 拉取模型
ollama pull llama3.1:8b
ollama pull qwen2:7b
# 运行对话
ollama run llama3.1:8b
# API 调用
curl http://localhost:11434/api/generate \
-d '{"model": "llama3.1:8b", "prompt": "什么是深度学习?"}'
# Python 调用
import requests
response = requests.post("http://localhost:11434/api/generate", json={
"model": "llama3.1:8b",
"prompt": "用Python写一个快排",
"stream": False
})
print(response.json()["response"])
| 技术 | 原理 | 加速比 | 适用场景 |
|---|---|---|---|
| KV Cache | 缓存历史 K/V | 10-100x(生成) | 所有自回归模型 |
| PagedAttention | KV Cache 分页管理 | 2-4x(吞吐) | vLLM |
| Continuous Batching | 动态批处理 | 2-24x(吞吐) | 在线服务 |
| 量化(INT8/INT4) | 降低精度 | 2-4x(显存/速度) | 资源受限 |
| Speculative Decoding | 小模型草稿 + 大模型验证 | 2-3x(延迟) | 低延迟场景 |
| Tensor Parallelism | 模型切分到多 GPU | 近线性 | 多卡推理 |
GGUF 是 llama.cpp 使用的量化格式,支持 CPU 推理,是本地部署大模型的主流方案。
Python# 使用 llama-cpp-python 加载 GGUF 模型
# pip install llama-cpp-python
from llama_cpp import Llama
# 加载量化模型
llm = Llama(
model_path="llama-3.1-8b-Q4_K_M.gguf", # 4-bit 量化,约 4.5GB
n_ctx=4096, # 上下文长度
n_gpu_layers=35, # 放到 GPU 的层数(0 = 全 CPU)
)
# 生成
output = llm.create_chat_completion(
messages=[{"role": "user", "content": "用Python写一个快排"}],
max_tokens=512,
temperature=0.7,
)
print(output["choices"][0]["message"]["content"])
# 常见量化格式:
# Q4_K_M: 4-bit 量化,平衡质量和大小(推荐)
# Q5_K_M: 5-bit 量化,质量更好
# Q8_0: 8-bit 量化,质量最好但更大
# FP16: 半精度,不量化
大模型推理显存 = 模型参数 + KV Cache
模型参数(fp16):参数量 × 2 bytes
7B 模型: 7 × 10^9 × 2 = 14 GB
13B 模型: 13 × 10^9 × 2 = 26 GB
KV Cache(fp16):2 × num_layers × hidden_dim × seq_len × batch_size × 2 bytes
7B (32层, 4096维): 2 × 32 × 4096 × 4096 × 1 × 2 = 2 GB
总显存(7B, fp16, seq_len=4096, batch=1): ~16 GB
总显存(7B, 4-bit, seq_len=4096, batch=1): ~5 GB
这就是为什么 4-bit 量化 + KV Cache 对大模型推理如此重要
vLLM 和 TGI 都使用 Continuous Batching,这是它们高吞吐量的关键。
大模型预训练后,它只是一个"文字接龙机器"——给什么续什么。对齐(Alignment)是让模型学会遵循人类的指令和价值观,变得有用、安全、诚实。
预训练就像教一个人读了很多书,他知识渊博但不一定听话。SFT 就像教他按照格式回答问题。对齐就像教他什么是好的回答、什么是不好的回答——不仅要回答正确,还要安全、有帮助、诚实。
DPO 是目前最流行的对齐方法,比 RLHF 更简单稳定。核心思想:直接用偏好数据优化策略模型,不需要训练奖励模型。
DPO 的损失函数:L = -log σ(β · (log π(y_w|x) - log π(y_l|x)))
其中 y_w 是好的回答(chosen),y_l 是差的回答(rejected),β 控制偏离程度。
直觉:让模型给好的回答更高的概率,给差的回答更低的概率。β 越大,模型越倾向于区分好坏。
# pip install trl transformers datasets
from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
# 1. 加载 SFT 后的模型
model = AutoModelForCausalLM.from_pretrained("sft-model-path", torch_dtype=torch.float16)
tokenizer = AutoTokenizer.from_pretrained("sft-model-path")
# 2. 准备偏好数据集
# 数据格式:{"prompt": "...", "chosen": "好的回答", "rejected": "差的回答"}
dataset = load_dataset("Anthropic/hh-rlhf", split="train")
def format_dpo(example):
return {
"prompt": example["chosen"].split("Assistant:")[0],
"chosen": "Assistant:" + example["chosen"].split("Assistant:")[-1],
"rejected": "Assistant:" + example["rejected"].split("Assistant:")[-1],
}
dataset = dataset.map(format_dpo)
# 3. DPO 训练配置
training_args = DPOConfig(
output_dir="./dpo-model",
num_train_epochs=1,
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
learning_rate=5e-7, # DPO 学习率要比 SFT 小很多
beta=0.1, # KL 散度约束强度
loss_type="sigmoid", # DPO 损失函数类型
logging_steps=10,
save_strategy="steps",
save_steps=100,
fp16=True,
)
# 4. 训练
trainer = DPOTrainer(
model=model,
args=training_args,
train_dataset=dataset,
tokenizer=tokenizer,
max_length=1024,
max_prompt_length=512,
)
trainer.train()
# 5. 保存
trainer.save_model("./dpo-final")
RLHF 是 ChatGPT 成功的关键技术。它通过人类反馈训练一个奖励模型,然后用 PPO 优化策略模型。
# RLHF 的完整流程(使用 TRL 库)
from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead
from transformers import pipeline
# 1. 加载策略模型(带 Value Head)
model = AutoModelForCausalLMWithValueHead.from_pretrained("sft-model-path")
# 2. 加载奖励模型(之前训练好的)
reward_model = pipeline("text-classification", model="reward-model-path")
# 3. PPO 配置
ppo_config = PPOConfig(
learning_rate=1.4e-5,
batch_size=16,
mini_batch_size=4,
ppo_epochs=4,
kl_penalty="kl", # KL 散度惩罚
init_kl_coef=0.2, # 初始 KL 系数
)
# 4. PPO 训练循环
ppo_trainer = PPOTrainer(config=ppo_config, model=model, tokenizer=tokenizer)
for batch in dataloader:
# 生成回答
response_ids = model.generate(batch["input_ids"], max_new_tokens=256)
# 计算奖励
reward = reward_model(response_text)["score"]
# PPO 更新
stats = ppo_trainer.step(batch["input_ids"], response_ids, reward)
ppo_trainer.log_stats(stats, batch, [reward])
# 训练奖励模型(Reward Model)
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from torch.utils.data import DataLoader
# 加载预训练模型作为奖励模型
reward_model = AutoModelForSequenceClassification.from_pretrained(
"meta-llama/Llama-3.1-8B", num_labels=1 # 输出一个分数
)
# 偏好数据格式:(prompt, chosen, rejected)
# 训练目标:reward_model(chosen) > reward_model(rejected)
# 训练循环
for prompt, chosen, rejected in preference_data:
# 计算 chosen 和 rejected 的分数
chosen_score = reward_model(tokenize(prompt + chosen)).logits
rejected_score = reward_model(tokenize(prompt + rejected)).logits
# 损失函数:让 chosen 分数高于 rejected
loss = -torch.log(torch.sigmoid(chosen_score - rejected_score)).mean()
loss.backward()
optimizer.step()
optimizer.zero_grad()
# 偏好数据格式示例
preference_data = [
{
"prompt": "解释什么是梯度下降",
"chosen": "梯度下降是一种优化算法,通过沿着损失函数梯度的反方向迭代更新参数,逐步找到函数的最小值。就像下山时沿着最陡的方向走。",
"rejected": "梯度下降就是算梯度然后减掉。"
},
{
"prompt": "写一个排序函数",
"chosen": "def quicksort(arr):\n if len(arr) <= 1: return arr\n pivot = arr[len(arr)//2]\n left = [x for x in arr if x < pivot]\n mid = [x for x in arr if x == pivot]\n right = [x for x in arr if x > pivot]\n return quicksort(left) + mid + quicksort(right)",
"rejected": "arr.sort()\nreturn arr"
},
]
# 数据质量很重要!
# chosen 应该是:有用、准确、安全、详细
# rejected 应该是:无用、错误、有害、敷衍
| 特性 | RLHF (PPO) | DPO |
|---|---|---|
| 需要奖励模型 | 是(需要单独训练) | 否 |
| 训练稳定性 | 不稳定,调参难 | 稳定,类似 SFT |
| 计算成本 | 高(4 个模型同时在线) | 低(2 个模型) |
| 效果 | 好 | 接近或等同 RLHF |
| 工程复杂度 | 高 | 低 |
| 推荐 | 大公司、需要精确控制 | 大多数场景首选 |
建议:入门用 DPO,进阶了解 ORPO 和 SimPO。
想象你往一杯清水里滴一滴墨水(加噪过程),墨水慢慢扩散直到整杯水变浑。Diffusion 模型学的就是如何把浑水还原成清水(去噪过程)。每一步去噪一点点,最终从纯噪声生成一张清晰的图片。
| 特性 | GAN | Diffusion |
|---|---|---|
| 训练稳定性 | 不稳定(模式坍塌) | 非常稳定 |
| 生成质量 | 好 | 更好(SOTA) |
| 生成速度 | 快(一次前向) | 慢(需要多步去噪) |
| 多样性 | 容易模式坍塌 | 多样性好 |
| 可控性 | 难控制 | 容易条件控制(CFG) |
| 代表作品 | StyleGAN | Stable Diffusion, DALL-E, Sora |
# DDPM 的 PyTorch 实现(简化版)
import torch
import torch.nn as nn
class DDPM:
def __init__(self, num_timesteps=1000, beta_start=1e-4, beta_end=0.02):
self.num_timesteps = num_timesteps
# 噪声调度
self.betas = torch.linspace(beta_start, beta_end, num_timesteps)
self.alphas = 1.0 - self.betas
self.alpha_cumprod = torch.cumprod(self.alphas, dim=0)
def add_noise(self, x0, t, noise=None):
"""前向过程:给干净图片加噪"""
if noise is None:
noise = torch.randn_like(x0)
alpha_t = self.alpha_cumprod[t].view(-1, 1, 1, 1)
xt = torch.sqrt(alpha_t) * x0 + torch.sqrt(1 - alpha_t) * noise
return xt, noise
def sample(self, model, shape, device):
"""反向过程:从噪声生成图片"""
x = torch.randn(shape, device=device)
for t in reversed(range(self.num_timesteps)):
t_batch = torch.full((shape[0],), t, device=device, dtype=torch.long)
predicted_noise = model(x, t_batch)
alpha = self.alphas[t]
alpha_cumprod = self.alpha_cumprod[t]
beta = self.betas[t]
# 去噪公式
x = (1 / torch.sqrt(alpha)) * (
x - (beta / torch.sqrt(1 - alpha_cumprod)) * predicted_noise
)
if t > 0:
x += torch.sqrt(beta) * torch.randn_like(x)
return x
# 简化的 UNet 噪声预测模型
class SimpleUNet(nn.Module):
def __init__(self, in_channels=3, time_embed_dim=128):
super().__init__()
self.time_embed = nn.Embedding(1000, time_embed_dim)
self.conv1 = nn.Conv2d(in_channels, 64, 3, padding=1)
self.conv2 = nn.Conv2d(64, 64, 3, padding=1)
self.conv3 = nn.Conv2d(64, in_channels, 3, padding=1)
self.relu = nn.ReLU()
def forward(self, x, t):
t_emb = self.time_embed(t).view(-1, 128, 1, 1)
h = self.relu(self.conv1(x))
h = h + t_emb[:, :64] # 注入时间信息
h = self.relu(self.conv2(h))
return self.conv3(h)
# pip install diffusers transformers accelerate
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
import torch
# 加载 Stable Diffusion
pipe = StableDiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1",
torch_dtype=torch.float16,
)
pipe = pipe.to("cuda")
# 使用更快的调度器
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
# 生成图片
image = pipe(
prompt="a beautiful sunset over mountains, oil painting style",
negative_prompt="blurry, low quality",
num_inference_steps=25, # 步数越少越快,质量略降
guidance_scale=7.5, # 引导强度
width=512, height=512,
).images[0]
image.save("sunset.png")
# LoRA 微调 Stable Diffusion
from diffusers import StableDiffusionPipeline
import torch
pipe = StableDiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1", torch_dtype=torch.float16
)
pipe.load_lora_weights("path/to/lora-weights") # 加载 LoRA
pipe = pipe.to("cuda")
image = pipe(prompt="a photo of sks dog in a park").images[0] # sks 是微调时的触发词
ControlNet 可以在生成图片时加入精确的结构控制(边缘、姿态、深度等)。
Python# pip install controlnet-aux
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from diffusers.utils import load_image
import cv2
import numpy as np
# 加载 ControlNet(边缘检测条件)
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/sd-controlnet-canny", torch_dtype=torch.float16
)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1",
controlnet=controlnet,
torch_dtype=torch.float16,
).to("cuda")
# 准备条件图(Canny 边缘检测)
input_image = load_image("input.png")
input_image = np.array(input_image)
edges = cv2.Canny(input_image, 100, 200)
edges = edges[:, :, None]
edges = np.concatenate([edges, edges, edges], axis=2)
edges_image = Image.fromarray(edges)
# 生成(保持边缘结构,改变风格)
image = pipe(
prompt="a beautiful house, watercolor painting",
image=edges_image,
num_inference_steps=25,
).images[0]
image.save("controlnet_output.png")
| 类型 | 控制方式 | 适用场景 |
|---|---|---|
| Canny | 边缘检测 | 保持轮廓,改变风格 |
| OpenPose | 人体姿态 | 控制人物动作 |
| Depth | 深度图 | 保持空间关系 |
| Segmentation | 语义分割 | 控制物体位置 |
| Normal Map | 法线图 | 保持表面纹理 |
建议:大多数场景用 DPM-Solver++ 或 Euler a,平衡速度和质量。
// my_ops.cpp - 自定义 C++ 算子
#include <torch/extension.h>
// 自定义 ReLU 的融合版本
torch::tensor fused_relu(torch::tensor x) {
return torch::clamp_min(x, 0);
}
// 矩阵乘法 + ReLU 融合(减少一次显存读写)
torch::tensor matmul_relu(torch::tensor a, torch::tensor b) {
return torch::clamp_min(torch::mm(a, b), 0);
}
// 绑定到 Python
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
m.def("fused_relu", &fused_relu, "Fused ReLU");
m.def("matmul_relu", &matmul_relu, "MatMul + ReLU fusion");
}
Python
# 在 Python 中使用
from torch.utils.cpp_extension import load
# JIT 编译(首次运行会编译,之后会缓存)
my_ops = load(name="my_ops", sources=["my_ops.cpp"])
# 使用
x = torch.randn(1000, 1000)
result = my_ops.fused_relu(x)
a = torch.randn(1000, 1000)
b = torch.randn(1000, 1000)
result = my_ops.matmul_relu(a, b)
// vector_add_kernel.cu - CUDA Kernel
#include <torch/extension.h>
__global__ void vector_add_kernel(float* a, float* b, float* c, int n) {
int idx = blockIdx.x * blockDim.x + threadIdx.x;
if (idx < n) {
c[idx] = a[idx] + b[idx];
}
}
torch::Tensor vector_add_cuda(torch::Tensor a, torch::Tensor b) {
auto c = torch::zeros_like(a);
int n = a.numel();
int threads = 256;
int blocks = (n + threads - 1) / threads;
vector_add_kernel<<<blocks, threads>>>(a.data_ptr<float>(),
b.data_ptr<float>(),
c.data_ptr<float>(), n);
return c;
}
Triton 是 OpenAI 开发的 GPU 编程语言,比 CUDA 简单 10 倍,性能接近手写 CUDA。
# pip install triton
import triton
import triton.language as tl
@triton.jit
def add_kernel(
x_ptr, y_ptr, output_ptr,
n_elements,
BLOCK_SIZE: tl.constexpr,
):
# 获取当前程序的 ID
pid = tl.program_id(axis=0)
# 计算当前 block 处理的偏移
block_start = pid * BLOCK_SIZE
offsets = block_start + tl.arange(0, BLOCK_SIZE)
# 边界检查
mask = offsets < n_elements
# 读取数据
x = tl.load(x_ptr + offsets, mask=mask)
y = tl.load(y_ptr + offsets, mask=mask)
# 计算
output = x + y
# 写回
tl.store(output_ptr + offsets, output, mask=mask)
def add(x: torch.Tensor, y: torch.Tensor):
output = torch.empty_like(x)
n_elements = output.numel()
BLOCK_SIZE = 1024
grid = lambda meta: (triton.cdiv(n_elements, meta['BLOCK_SIZE']),)
add_kernel[grid](x, y, output, n_elements, BLOCK_SIZE=BLOCK_SIZE)
return output
# 使用
x = torch.randn(1000000, device='cuda')
y = torch.randn(1000000, device='cuda')
result = add(x, y)
@triton.jit
def softmax_kernel(
input_ptr, output_ptr,
input_row_stride, output_row_stride,
n_cols,
BLOCK_SIZE: tl.constexpr,
):
# 每个程序处理一行
row_idx = tl.program_id(0)
row_start = input_ptr + row_idx * input_row_stride
# 找最大值(数值稳定性)
max_val = -float('inf')
for off in range(0, n_cols, BLOCK_SIZE):
cols = off + tl.arange(0, BLOCK_SIZE)
mask = cols < n_cols
vals = tl.load(row_start + cols, mask=mask, other=-float('inf'))
max_val = tl.maximum(max_val, tl.max(vals))
# 计算 exp 和 sum
sum_exp = 0.0
for off in range(0, n_cols, BLOCK_SIZE):
cols = off + tl.arange(0, BLOCK_SIZE)
mask = cols < n_cols
vals = tl.load(row_start + cols, mask=mask, other=0.0)
exp_vals = tl.exp(vals - max_val)
sum_exp += tl.sum(exp_vals)
# 写入结果
row_out = output_ptr + row_idx * output_row_stride
for off in range(0, n_cols, BLOCK_SIZE):
cols = off + tl.arange(0, BLOCK_SIZE)
mask = cols < n_cols
vals = tl.load(row_start + cols, mask=mask, other=0.0)
exp_vals = tl.exp(vals - max_val)
tl.store(row_out + cols, exp_vals / sum_exp, mask=mask)
| 特性 | CUDA C | Triton |
|---|---|---|
| 语法复杂度 | 高(需理解线程模型) | 低(类似 NumPy) |
| 性能 | 极致(完全控制) | 接近 CUDA(90-99%) |
| 开发效率 | 低 | 高(10x 更快) |
| 调试 | 困难 | 相对容易 |
| 适用场景 | 极致优化、Kernel 库开发 | 自定义算子、原型开发 |
建议:先用 torch.compile 试试,如果还不够快再考虑自定义算子。90% 的情况下 torch.compile 就够了。
PyTorch Profiler 是内置的性能分析工具,可以分析 CPU 和 GPU 的时间分布、显存使用等。
Pythonimport torch
from torch.profiler import profile, record_function, ProfilerActivity
# 基本 Profiling
with profile(
activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
record_shapes=True,
profile_memory=True,
with_stack=True,
) as prof:
with record_function("model_inference"):
output = model(input_data)
# 打印摘要
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=20))
# 导出到 TensorBoard
prof.export_chrome_trace("trace.json")
# 用 chrome://tracing 打开 trace.json
# 导出到 TensorBoard
from torch.profiler import tensorboard_trace_handler
prof.export_tensorboard_trace("runs/profiler")
# 显存快照(找出显存泄漏)
torch.cuda.memory._record_memory_history()
# 运行你的代码
for i in range(100):
output = model(input_data)
loss = criterion(output, target)
loss.backward()
optimizer.step()
optimizer.zero_grad()
# 保存显存快照
torch.cuda.memory._dump_snapshot("memory_snapshot.pickle")
# 用 PyTorch 的 memory_visualization.py 可视化
# python -m torch.cuda._memory_viz memory_snapshot.pickle
# 打印显存使用情况
print(f"当前显存: {torch.cuda.memory_allocated() / 1024**3:.2f} GB")
print(f"峰值显存: {torch.cuda.max_memory_allocated() / 1024**3:.2f} GB")
print(f"缓存显存: {torch.cuda.memory_reserved() / 1024**3:.2f} GB")
总计:7B 模型全量微调需要 ~60 GB。这就是为什么需要 LoRA/QLoRA。
pip install nsight-systems 或从 NVIDIA 安装nsys profile -o report python train.pynsys-ui report.nsys-rep(图形界面)# 采集性能数据
nsys profile --trace=cuda,nvtx --output=report python train.py
# 常见瓶颈:
# 1. CPU-GPU 数据传输过多 → 用 pinned memory, prefetch
# 2. GPU Kernel 利用率低 → 用 torch.compile, 自定义算子
# 3. 显存碎片化 → 用 gradient checkpointing, 减小 batch
# 4. CPU 瓶颈(数据加载慢)→ 增加 num_workers, prefetch
# 案例:找出训练循环中的瓶颈
import torch
from torch.profiler import profile, ProfilerActivity
# 分析一个完整的训练步骤
with profile(
activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
record_shapes=True,
profile_memory=True,
) as prof:
# 数据加载
with record_function("data_loading"):
batch = next(iter(train_loader))
# 数据传输到 GPU
with record_function("to_device"):
batch = {k: v.to(device) for k, v in batch.items()}
# 前向传播
with record_function("forward"):
outputs = model(**batch)
# 计算损失
with record_function("loss"):
loss = outputs.loss
# 反向传播
with record_function("backward"):
loss.backward()
# 参数更新
with record_function("optimizer"):
optimizer.step()
optimizer.zero_grad()
# 打印结果
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
# 典型输出:
# Name | Self CUDA | CUDA time |
# --------------------|-----------|-----------|
# forward | 120ms | 150ms | ← 最耗时
# backward | 80ms | 100ms |
# data_loading | 5ms | 5ms | ← 如果这里很慢,加 num_workers
# to_device | 10ms | 10ms | ← 如果很慢,用 pin_memory
# optimizer | 15ms | 15ms |
# 完整的 Profiling 模板(复制即用)
import torch
from torch.profiler import profile, ProfilerActivity, record_function
def profile_training(model, train_loader, optimizer, num_steps=10):
"""Profile 训练循环,找出性能瓶颈"""
model.train()
criterion = torch.nn.CrossEntropyLoss()
with profile(
activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
record_shapes=True,
profile_memory=True,
) as prof:
for step, (data, target) in enumerate(train_loader):
if step >= num_steps:
break
with record_function("to_device"):
data, target = data.cuda(), target.cuda()
with record_function("forward"):
output = model(data)
with record_function("loss"):
loss = criterion(output, target)
with record_function("backward"):
loss.backward()
with record_function("optimizer"):
optimizer.step()
optimizer.zero_grad()
# 打印结果
print("=== CPU 时间 Top 10 ===")
print(prof.key_averages().table(sort_by="cpu_time_total", row_limit=10))
print("\n=== CUDA 时间 Top 10 ===")
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
# 导出到 TensorBoard
prof.export_tensorboard_trace("runs/profiler")
| 瓶颈 | 症状 | 诊断方法 | 解决方案 |
|---|---|---|---|
| 数据加载慢 | GPU 利用率低,有空闲 | Profiler 查看 CPU 时间 | num_workers, pin_memory, prefetch |
| CPU-GPU 传输 | CUDA memcpy 占比高 | nsys 查看传输时间 | pinned memory, 减少 .cpu() 调用 |
| Kernel 利用率低 | SM 利用率 < 50% | nsys 查看 SM 活动 | 增大 batch_size, torch.compile |
| 显存不足 | OOM 错误 | memory_profiler | 混合精度, 梯度累积, checkpointing |
| 显存碎片 | 显存够但分配失败 | memory snapshot | PYTORCH_CUDA_ALLOC_CONF |
num_workers > 0,pin_memory=Truetorch.amp.autocast('cuda')model = torch.compile(model).item()、.cpu() 少用F.scaled_dot_product_attention图片是网格数据(像素排列整齐),文本是序列数据(词语按顺序排列)。但社交网络、分子结构、推荐系统中的数据是图——节点之间有复杂的连接关系,不是规则的网格或序列。
图 G = (V, E)
V = {v₁, v₂, ..., vₙ} # 节点集合
E = {(vᵢ, vⱼ)} # 边集合
邻接矩阵 A:
A[i][j] = 1 如果节点 i 和 j 有边连接,否则为 0
度矩阵 D:
D[i][i] = 节点 i 的度(连接的边数)
图的拉普拉斯矩阵:
L = D - A # 非归一化拉普拉斯
L_sym = D^(-1/2) L D^(-1/2) # 对称归一化拉普拉斯
GCN 的核心公式:
H^(l+1) = σ(D̃^(-1/2) Ã D̃^(-1/2) H^(l) W^(l))
其中 Ã = A + I(加自环),D̃ 是 Ã 的度矩阵
# PyG 的图数据格式
from torch_geometric.data import Data
# 方式1:边列表(最常用)
edge_index = torch.tensor([[0, 1, 1, 2], # 源节点
[1, 0, 2, 1]], dtype=torch.long) # 目标节点
x = torch.randn(3, 16) # 3 个节点,16 维特征
data = Data(x=x, edge_index=edge_index)
# 方式2:COO 格式(稀疏矩阵)
# edge_index 就是 COO 格式的稀疏矩阵表示
# 方式3:邻接矩阵(不推荐,浪费空间)
adj = torch.tensor([[0, 1, 0],
[1, 0, 1],
[0, 1, 0]], dtype=torch.float)
# PyG 自带数据集
from torch_geometric.datasets import Planetoid, TUDataset
dataset = Planetoid(root='./data', name='Cora') # 节点分类
dataset = TUDataset(root='./data', name='MUTAG') # 图分类
# pip install torch-geometric
import torch
from torch_geometric.data import Data
from torch_geometric.nn import GCNConv, GATConv
import torch.nn.functional as F
# 创建图数据
# edge_index: [2, num_edges],表示边的连接关系
edge_index = torch.tensor([[0, 1, 1, 2, 2, 3],
[1, 0, 2, 1, 3, 2]], dtype=torch.long)
x = torch.randn(4, 16) # 4 个节点,每个 16 维特征
y = torch.tensor([0, 1, 0, 1]) # 节点标签
data = Data(x=x, edge_index=edge_index, y=y)
print(data) # Data(x=[4, 16], edge_index=[2, 6], y=[4])
class GCN(torch.nn.Module):
"""图卷积网络"""
def __init__(self, in_channels, hidden_channels, out_channels):
super().__init__()
self.conv1 = GCNConv(in_channels, hidden_channels)
self.conv2 = GCNConv(hidden_channels, out_channels)
def forward(self, data):
x, edge_index = data.x, data.edge_index
x = self.conv1(x, edge_index)
x = F.relu(x)
x = F.dropout(x, p=0.5, training=self.training)
x = self.conv2(x, edge_index)
return F.log_softmax(x, dim=1)
# 训练
from torch_geometric.datasets import Planetoid
dataset = Planetoid(root='./data', name='Cora')
data = dataset[0]
model = GCN(dataset.num_features, 16, dataset.num_classes)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)
for epoch in range(200):
model.train()
optimizer.zero_grad()
out = model(data)
loss = F.nll_loss(out[data.train_mask], data.y[data.train_mask])
loss.backward()
optimizer.step()
if (epoch + 1) % 50 == 0:
model.eval()
pred = model(data).argmax(dim=1)
acc = pred[data.test_mask].eq(data.y[data.test_mask]).float().mean()
print(f"Epoch {epoch+1} | Loss: {loss.item():.4f} | Test Acc: {acc:.4f}")
class GAT(torch.nn.Module):
"""图注意力网络:给不同邻居分配不同的注意力权重"""
def __init__(self, in_channels, hidden_channels, out_channels, heads=8):
super().__init__()
self.conv1 = GATConv(in_channels, hidden_channels, heads=heads)
self.conv2 = GATConv(hidden_channels * heads, out_channels, heads=1)
def forward(self, data):
x, edge_index = data.x, data.edge_index
x = F.elu(self.conv1(x, edge_index))
x = F.dropout(x, p=0.6, training=self.training)
x = self.conv2(x, edge_index)
return F.log_softmax(x, dim=1)
# GAT vs GCN:
# GCN:所有邻居等权平均
# GAT:学习每个邻居的注意力权重,更灵活
GNN 的核心是消息传递:每个节点聚合邻居的信息来更新自己的表示。
Python# 消息传递的直观理解
# 节点 A 的新表示 = 聚合(A 的所有邻居的旧表示) + A 的旧表示
# GCN 的聚合方式:加权平均(权重由度数决定)
# GAT 的聚合方式:注意力加权(权重由网络学习)
# PyG 的消息传递接口
from torch_geometric.nn import MessagePassing
class MyConv(MessagePassing):
def __init__(self, in_channels, out_channels):
super().__init__(aggr='mean') # 聚合方式:mean, sum, max
self.lin = nn.Linear(in_channels, out_channels)
def forward(self, x, edge_index):
return self.propagate(edge_index, x=x)
def message(self, x_j):
# x_j: 邻居节点的特征
return self.lin(x_j)
# 图分类:对整个图进行分类(而不是对节点分类)
from torch_geometric.nn import global_mean_pool, global_max_pool
class GraphClassifier(torch.nn.Module):
def __init__(self, in_channels, hidden_channels, num_classes):
super().__init__()
self.conv1 = GCNConv(in_channels, hidden_channels)
self.conv2 = GCNConv(hidden_channels, hidden_channels)
self.conv3 = GCNConv(hidden_channels, hidden_channels)
self.fc = nn.Linear(hidden_channels, num_classes)
def forward(self, data):
x, edge_index, batch = data.x, data.edge_index, data.batch
# 图卷积层
x = F.relu(self.conv1(x, edge_index))
x = F.relu(self.conv2(x, edge_index))
x = F.relu(self.conv3(x, edge_index))
# 图池化:把所有节点的特征聚合成一个图级别的特征
x = global_mean_pool(x, batch) # [batch_size, hidden_channels]
return self.fc(x)
# 图分类 vs 节点分类:
# 节点分类:每个节点一个标签(如社区检测)
# 图分类:整个图一个标签(如分子性质预测)
| 场景 | 节点 | 边 | 任务 |
|---|---|---|---|
| 社交网络 | 用户 | 关注/好友 | 社区发现、好友推荐 |
| 推荐系统 | 用户+商品 | 点击/购买 | 商品推荐 |
| 分子性质预测 | 原子 | 化学键 | 药物活性预测 |
| 交通预测 | 路口 | 道路 | 流量预测 |
| 知识图谱 | 实体 | 关系 | 链接预测、推理 |
| 金融风控 | 账户 | 交易 | 欺诈检测、反洗钱 |
想象训练一只狗:狗做对了动作就给零食(正奖励),做错了就不给(零奖励或负奖励)。狗通过反复尝试,学会了哪些动作能获得零食。强化学习就是这个过程——智能体通过与环境交互,根据奖励信号学习最优策略。
| 特性 | 监督学习 | 强化学习 |
|---|---|---|
| 数据 | 有标签的数据集 | 与环境交互产生 |
| 反馈 | 即时(每个样本有标签) | 延迟(可能很久才知道结果) |
| 目标 | 最小化损失函数 | 最大化累积奖励 |
| 探索 | 不需要 | 需要探索-利用平衡 |
| 应用 | 分类、回归 | 游戏、机器人、RLHF |
# pip install gymnasium
import gymnasium as gym
import torch
import torch.nn as nn
import random
from collections import deque
class DQN(nn.Module):
def __init__(self, state_dim, action_dim):
super().__init__()
self.net = nn.Sequential(
nn.Linear(state_dim, 128), nn.ReLU(),
nn.Linear(128, 128), nn.ReLU(),
nn.Linear(128, action_dim)
)
def forward(self, x):
return self.net(x)
class ReplayBuffer:
def __init__(self, capacity=10000):
self.buffer = deque(maxlen=capacity)
def push(self, state, action, reward, next_state, done):
self.buffer.append((state, action, reward, next_state, done))
def sample(self, batch_size):
batch = random.sample(self.buffer, batch_size)
states, actions, rewards, next_states, dones = zip(*batch)
return (torch.FloatTensor(states), torch.LongTensor(actions),
torch.FloatTensor(rewards), torch.FloatTensor(next_states),
torch.FloatTensor(dones))
# 训练 DQN
env = gym.make('CartPole-v1')
model = DQN(env.observation_space.shape[0], env.action_space.n)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
buffer = ReplayBuffer()
gamma = 0.99
epsilon = 1.0
for episode in range(500):
state, _ = env.reset()
total_reward = 0
while True:
# ε-greedy 策略
if random.random() < epsilon:
action = env.action_space.sample()
else:
with torch.no_grad():
q_values = model(torch.FloatTensor(state))
action = q_values.argmax().item()
next_state, reward, terminated, truncated, _ = env.step(action)
done = terminated or truncated
buffer.push(state, action, reward, next_state, float(done))
state = next_state
total_reward += reward
# 训练
if len(buffer.buffer) >= 64:
states, actions, rewards, next_states, dones = buffer.sample(64)
q_values = model(states).gather(1, actions.unsqueeze(1))
next_q = model(next_states).max(1)[0]
target = rewards + gamma * next_q * (1 - dones)
loss = nn.MSELoss()(q_values.squeeze(), target)
optimizer.zero_grad()
loss.backward()
optimizer.step()
if done:
break
epsilon = max(0.01, epsilon * 0.995)
if (episode + 1) % 50 == 0:
print(f"Episode {episode+1} | Reward: {total_reward} | Epsilon: {epsilon:.3f}")
PPO(Proximal Policy Optimization)是最流行的策略梯度算法,也是 ChatGPT(RLHF)的核心算法。
# PPO 的核心代码片段
class PPO:
def update(self, states, actions, old_log_probs, returns, advantages):
for _ in range(self.ppo_epochs):
# 计算新的 log 概率
new_log_probs = self.policy.log_prob(states, actions)
# 重要性采样比率
ratio = torch.exp(new_log_probs - old_log_probs)
# PPO 剪切目标
surr1 = ratio * advantages
surr2 = torch.clamp(ratio, 1 - self.clip_epsilon, 1 + self.clip_epsilon) * advantages
policy_loss = -torch.min(surr1, surr2).mean()
# 值函数损失
value_loss = nn.MSELoss()(self.value(states), returns)
# 总损失
loss = policy_loss + 0.5 * value_loss - 0.01 * entropy
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
| 算法 | 类型 | 动作空间 | 特点 | 适用场景 |
|---|---|---|---|---|
| DQN | 值函数 | 离散 | 简单、稳定 | 游戏(Atari) |
| PPO | 策略梯度 | 连续/离散 | 稳定、通用 | 机器人、RLHF |
| SAC | Actor-Critic | 连续 | 样本效率高 | 机器人控制 |
| TD3 | Actor-Critic | 连续 | DDPG 改进 | 连续控制任务 |
| A2C/A3C | Actor-Critic | 连续/离散 | 可并行 | 大规模训练 |
# pip install gymnasium
import gymnasium as gym
# 查看所有可用环境
envs = gym.envs.registry.keys()
print(f"可用环境: {len(list(envs))} 个")
# 常用环境
# CartPole: 平衡杆(离散动作,简单)
# LunarLander: 月球着陆器(离散动作,中等)
# HalfCheetah: 半猎豹(连续动作,复杂)
# Atari: 经典游戏(图像输入)
env = gym.make('CartPole-v1', render_mode='human')
obs, info = env.reset()
print(f"状态空间: {env.observation_space}") # Box([-4.8, ...], [4.8, ...], (4,))
print(f"动作空间: {env.action_space}") # Discrete(2)
for _ in range(100):
action = env.action_space.sample() # 随机动作
obs, reward, terminated, truncated, info = env.step(action)
if terminated or truncated:
obs, info = env.reset()
env.close()
如果你只是用 PyTorch 做项目,不需要看这章。但如果你想深入理解模型为什么不收敛、Loss 为什么变 NaN、梯度为什么会消失,这些数学推导会帮你从本质排查问题。
会开车不需要懂发动机原理,但赛车手必须懂。如果你只是用预训练模型做推理,不需要懂数学。但如果你要训练自己的模型、调参、排查问题,数学就是你的"发动机原理"。
以全连接层 z = Wx + b 为例,推导反向传播的梯度计算:
前向传播:
z = Wx + b # z: (d_out, 1), W: (d_out, d_in), x: (d_in, 1)
a = σ(z) # σ 是激活函数
反向传播(链式法则):
∂L/∂W = ∂L/∂z · ∂z/∂W = δ · xᵀ # δ = ∂L/∂z
∂L/∂b = ∂L/∂z = δ
∂L/∂x = Wᵀ · ∂L/∂z = Wᵀ · δ
其中 δ 的计算:
如果 z 是最后一层:δ = ∂L/∂a · σ'(z)
如果 z 是隐藏层: δ = (W_nextᵀ · δ_next) · σ'(z)
Softmax + CrossEntropy 的梯度(非常简洁):
L = -Σ yᵢ log(pᵢ), 其中 p = softmax(z)
∂L/∂z = p - y # 就是预测概率减去真实标签!
这就是为什么 CrossEntropy + Softmax 是分类任务的标配——梯度形式简单,训练稳定。
卷积层前向传播:
y = x * w + b # * 表示卷积操作
卷积层反向传播:
∂L/∂w = x * ∂L/∂y # 卷积的梯度也是卷积!
∂L/∂x = ∂L/∂y ⌣ w # 转置卷积(full convolution)
∂L/∂b = Σ ∂L/∂y # 对所有空间位置求和
关键洞察:
卷积的梯度计算也是卷积操作
这就是为什么 GPU 上卷积这么高效——可以用相同的硬件加速
BatchNorm 前向传播:
μ = mean(x) # 均值
σ² = var(x) # 方差
x̂ = (x - μ) / √(σ² + ε) # 标准化
y = γx̂ + β # 缩放和偏移
反向传播(链式法则,需要考虑 μ 和 σ² 也依赖 x):
∂L/∂γ = Σ (∂L/∂y · x̂)
∂L/∂β = Σ (∂L/∂y)
∂L/∂x = (γ / √(σ² + ε)) · [
∂L/∂y - mean(∂L/∂y) - x̂ · mean(∂L/∂y · x̂)
]
关键洞察:BatchNorm 的梯度涉及整个 batch 的统计量,所以 batch_size 太小时效果不好。
注意:BatchNorm 在 batch_size < 16 时效果很差,此时用 LayerNorm 或 GroupNorm 更好。
自注意力的计算复杂度:
Q, K, V ∈ ℝ^(N × d) # N 是序列长度,d 是维度
1. 计算注意力分数:S = QKᵀ # (N × N),复杂度 O(N²d)
2. Softmax:A = softmax(S) # (N × N),复杂度 O(N²)
3. 加权求和:O = AV # (N × d),复杂度 O(N²d)
总复杂度:O(N²d)
空间复杂度:O(N²)(存储注意力矩阵)
这就是为什么长序列(N > 4096)在 Transformer 上很慢。
FlashAttention 的解决方案(分块计算 Tiling):
不一次性计算完整的 N×N 注意力矩阵
而是把 Q, K, V 分成小块,逐块计算
用 online softmax 技巧避免存储完整的 N×N 矩阵
显存从 O(N²) 降到 O(N),速度提升 2-4 倍
标准注意力:O = softmax(QKᵀ)V # O(N²d)
线性注意力:O = φ(Q)(φ(K)ᵀV) # O(Nd²)
如果 d << N,线性注意力更快
核心技巧:用核函数 φ 近似 softmax,改变矩阵乘法顺序
先算 φ(K)ᵀV(d×d),再乘 φ(Q)(N×d)
局限性:线性注意力的效果通常不如标准注意力
FlashAttention 是更好的选择——保持标准注意力的精度,优化显存
SGD(随机梯度下降):
θ_{t+1} = θ_t - η · g_t
其中 g_t = ∇L(θ_t) 是梯度
SGD + Momentum(动量):
v_t = β · v_{t-1} + g_t # 动量(指数移动平均)
θ_{t+1} = θ_t - η · v_t
β 通常取 0.9
Adam(自适应学习率):
m_t = β₁ · m_{t-1} + (1 - β₁) · g_t # 一阶矩估计(梯度的均值)
v_t = β₂ · v_{t-1} + (1 - β₂) · g_t² # 二阶矩估计(梯度的方差)
m̂_t = m_t / (1 - β₁ᵗ) # 偏差修正
v̂_t = v_t / (1 - β₂ᵗ) # 偏差修正
θ_{t+1} = θ_t - η · m̂_t / (√v̂_t + ε)
β₁ = 0.9, β₂ = 0.999, ε = 1e-8
为什么 Adam 有时泛化不如 SGD?
- Adam 对每个参数自适应调整学习率,可能导致某些参数更新过少
- SGD 的"噪声"反而有助于跳出局部最优,找到更平坦的最小值
- 实践建议:CV 任务常用 SGD + Momentum,NLP/Transformer 常用 AdamW
反向传播通过链式法则计算梯度:∂L/∂W₁ = ∂L/∂Wₙ · ∂Wₙ/∂Wₙ₋₁ · ... · ∂W₂/∂W₁
解决方案:ReLU(梯度为 0 或 1)、残差连接(跳过连接)、BatchNorm(稳定梯度)、梯度裁剪(限制最大梯度)
∂softmax(z)ᵢ/∂zⱼ = pᵢ(δᵢⱼ - pⱼ)∂L/∂z = p - y∂ReLU(x)/∂x = 1 if x > 0, else 0∂(x + F(x))/∂x = 1 + ∂F(x)/∂xtorch.nn.utils.clip_grad_norm_)。常见于 RNN# 用代码验证数学推导
import torch
# 验证 Softmax + CrossEntropy 的梯度
z = torch.tensor([2.0, 1.0, 0.5], requires_grad=True)
y = torch.tensor(0) # 真实标签是第 0 类
# 手动计算
p = torch.softmax(z, dim=0)
print(f"Softmax 输出: {p}") # [0.5064, 0.3072, 0.1864]
# PyTorch 自动计算
loss = torch.nn.CrossEntropyLoss()(z.unsqueeze(0), y.unsqueeze(0))
loss.backward()
print(f"PyTorch 梯度: {z.grad}") # [-0.4936, 0.3072, 0.1864]
print(f"手动计算: {p - torch.nn.functional.one_hot(y, 3).float()}") # 应该相同
# 验证:梯度 = p - y(one-hot 编码的 y)
# p[0] - 1 = 0.5064 - 1 = -0.4936 ✓
# p[1] - 0 = 0.3072 ✓
# p[2] - 0 = 0.1864 ✓
# pip install wandb
import wandb
# 初始化实验
wandb.init(
project="my-llm-finetune",
config={
"model": "llama-3.1-8b",
"learning_rate": 2e-4,
"batch_size": 4,
"epochs": 3,
"lora_r": 16,
"optimizer": "adamw",
},
tags=["lora", "llama", "experiment-v1"],
)
# 训练循环
for epoch in range(3):
for step, batch in enumerate(train_loader):
loss = train_step(batch)
# 记录指标
wandb.log({
"train/loss": loss,
"train/learning_rate": scheduler.get_lr()[0],
"train/epoch": epoch,
"train/step": step,
})
# 记录图片
if step % 100 == 0:
wandb.log({"generated_samples": wandb.Image(generated_img)})
# 记录模型梯度
if step % 50 == 0:
for name, param in model.named_parameters():
if param.grad is not None:
wandb.log({f"gradients/{name}": wandb.Histogram(param.grad.cpu())})
# 保存模型到 W&B
wandb.save("model.pth")
wandb.finish()
# 安装 DVC
pip install dvc
# 初始化
dvc init
# 添加数据文件到版本控制
dvc add data/train.csv data/val.csv
# 提交(和 Git 一样)
git add data/train.csv.dvc data/val.csv.dvc .gitignore
git commit -m "add training data v1"
# 推送到远程存储(S3、GCS、Azure、SSH 等)
dvc remote add -d myremote s3://my-bucket/dvc-store
dvc push
# 切换到旧版本的数据
git checkout v1.0
dvc checkout
# 对比不同版本的数据
git diff v1.0..v2.0 data/train.csv.dvc
# DVC Pipeline(定义数据处理流程)
# dvc.yaml:
# stages:
# prepare:
# cmd: python prepare.py
# deps:
# - data/raw
# outs:
# - data/processed
# train:
# cmd: python train.py
# deps:
# - data/processed
# outs:
# - model.pth
# metrics:
# - metrics.json
# Dockerfile - PyTorch 推理服务
FROM pytorch/pytorch:2.2.0-cuda12.1-cudnn8-runtime
WORKDIR /app
# 安装依赖
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# 复制代码和模型
COPY app.py .
COPY model/ ./model/
# 暴露端口
EXPOSE 8000
# 启动服务
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]
Bash
# 构建镜像
docker build -t my-pytorch-service:v1 .
# 运行容器
docker run --gpus all -p 8000:8000 my-pytorch-service:v1
# Docker Compose(多服务)
# docker-compose.yml:
# version: '3'
# services:
# api:
# build: .
# ports:
# - "8000:8000"
# deploy:
# resources:
# reservations:
# devices:
# - driver: nvidia
# count: 1
# capabilities: [gpu]
发现漂移后,需要重新训练模型并部署新版本。
| 工具 | 功能 | 替代品 | 推荐场景 |
|---|---|---|---|
| W&B | 实验追踪、可视化 | MLflow, TensorBoard | 团队协作、论文实验 |
| MLflow | 实验追踪、模型注册 | W&B, Neptune | 企业级、开源优先 |
| DVC | 数据版本控制 | Git LFS, Pachyderm | 大数据集管理 |
| Docker | 容器化 | Podman, Singularity | 环境一致性 |
| Kubernetes | 容器编排 | Docker Swarm | 高可用、自动扩缩 |
| Triton Server | 模型推理服务 | TorchServe, vLLM | 多模型、动态 batching |
| Prometheus | 指标采集 | InfluxDB, Datadog | 监控告警 |
import torchvision
import torchvision.transforms as transforms
from torchvision import models
# 1. 加载预训练模型
model = models.resnet50(weights=models.ResNet50_Weights.DEFAULT)
model.eval()
# 2. 数据预处理(使用官方推荐的预处理参数)
transform = models.ResNet50_Weights.DEFAULT.transforms()
# 3. 加载数据集
train_dataset = torchvision.datasets.CIFAR10(
root='./data', train=True, download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True)
# 4. 使用预定义的数据增强
train_transform = transforms.Compose([
transforms.RandomResizedCrop(224),
transforms.RandomHorizontalFlip(),
transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2),
transforms.ToTensor(),
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])
# 5. 常用模型列表
print(models.list_models()) # 查看所有可用模型
# HuggingFace 不仅仅是 Transformers!
# pip install transformers datasets accelerate
# 1. Transformers:预训练模型
from transformers import pipeline
classifier = pipeline('sentiment-analysis')
print(classifier("PyTorch is amazing!"))
# 2. Datasets:数据集加载
from datasets import load_dataset
dataset = load_dataset("imdb") # 一行代码加载 IMDB 数据集
print(dataset['train'][0])
# 3. Accelerate:分布式训练
from accelerate import Accelerator
accelerator = Accelerator()
model, optimizer, train_loader = accelerator.prepare(model, optimizer, train_loader)
# 自动处理单卡/多卡/TPU,不需要改代码
# 4. 一行代码加载中文 BERT
from transformers import BertTokenizer, BertModel
tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
model = BertModel.from_pretrained('bert-base-chinese')
# 5. 模型分享和下载
# 上传模型到 HuggingFace Hub
model.push_to_hub("my-awesome-model")
tokenizer.push_to_hub("my-awesome-model")
# 下载别人分享的模型
model = BertModel.from_pretrained("username/model-name")
# pip install pytorch-lightning
import pytorch_lightning as pl
from torchmetrics import Accuracy
class LitModel(pl.LightningModule):
def __init__(self, num_classes=10):
super().__init__()
self.model = models.resnet18(num_classes=num_classes)
self.accuracy = Accuracy(task="multiclass", num_classes=num_classes)
def forward(self, x):
return self.model(x)
def training_step(self, batch, batch_idx):
x, y = batch
logits = self(x)
loss = F.cross_entropy(logits, y)
self.log('train_loss', loss)
self.log('train_acc', self.accuracy(logits, y))
return loss
def validation_step(self, batch, batch_idx):
x, y = batch
logits = self(x)
loss = F.cross_entropy(logits, y)
self.log('val_loss', loss)
self.log('val_acc', self.accuracy(logits, y))
def configure_optimizers(self):
return torch.optim.Adam(self.parameters(), lr=1e-3)
# 训练只需几行!
model = LitModel()
trainer = pl.Trainer(max_epochs=10, accelerator='auto', devices='auto')
trainer.fit(model, train_loader, val_loader)
# pip install timm
import timm
# 查看所有可用模型
print(timm.list_models('resnet*')) # 所有 ResNet
print(timm.list_models('vit*')) # 所有 ViT
print(timm.list_models('swin*')) # 所有 Swin Transformer
# 一行代码加载预训练模型
model = timm.create_model('vit_base_patch16_224', pretrained=True, num_classes=1000)
# 获取模型信息
model = timm.create_model('efficientnet_b0', pretrained=True)
print(timm.models.get_model_default_cfg(model)) # 查看默认配置
# 自定义分类头
model = timm.create_model('resnet50', pretrained=True, num_classes=10) # 10 类分类
# 获取模型的输入大小和预处理参数
data_config = timm.models.resolve_model_data_config(model)
print(data_config) # {'input_size': (3, 224, 224), 'mean': (...), 'std': (...)}
# pip install wandb
import wandb
# 初始化
wandb.init(project="my-project", config={
"learning_rate": 1e-3,
"epochs": 100,
"batch_size": 32
})
# 训练循环中记录指标
for epoch in range(100):
train_loss = train(...)
val_loss = validate(...)
wandb.log({
"train_loss": train_loss,
"val_loss": val_loss,
"epoch": epoch
})
# 记录图片
wandb.log({"generated": wandb.Image(img)})
# 记录模型
wandb.save("model.pth")
wandb.finish()
# 优势:自动记录实验、可视化对比、团队协作、超参数搜索
| 资源 | 类型 | 适合人群 | 链接 |
|---|---|---|---|
| 官方教程 | 文档 | 所有人 | pytorch.org/tutorials |
| 动手学深度学习 | 教材 | 初学者 | d2l.ai |
| CS231n | 课程 | 计算机视觉 | 斯坦福 |
| CS224n | 课程 | 自然语言处理 | 斯坦福 |
| fast.ai | 课程 | 实战派 | fast.ai |
| Papers with Code | 论文 | 进阶 | paperswithcode.com |
| PyTorch 论坛 | 社区 | 遇到问题 | discuss.pytorch.org |
nn.Module 开始深度学习是一个快速发展的领域,新的模型和技术不断涌现。保持好奇心,持续学习,动手实践——你已经走在了正确的道路上。祝你在 AI 的世界里越走越远!🚀
| 函数 | 说明 | 示例 |
|---|---|---|
torch.tensor() | 从数据创建 | torch.tensor([1,2,3]) |
torch.zeros() | 全零 | torch.zeros(3,4) |
torch.ones() | 全一 | torch.ones(2,3) |
torch.rand() | 均匀分布 [0,1) | torch.rand(2,3) |
torch.randn() | 标准正态分布 | torch.randn(2,3) |
torch.arange() | 等差序列 | torch.arange(0,10,2) |
| 函数 | 说明 |
|---|---|
x.reshape(a,b) | 改变形状 |
x.flatten() | 展平 |
x.unsqueeze(dim) | 增加维度 |
x.squeeze(dim) | 删除大小为1的维度 |
x.permute(dims) | 重排维度 |
torch.cat([a,b], dim) | 拼接 |
torch.stack([a,b], dim) | 堆叠 |
| 层 | 说明 |
|---|---|
nn.Linear(in, out) | 全连接层 |
nn.Conv2d(in, out, k) | 2D 卷积 |
nn.MaxPool2d(k) | 最大池化 |
nn.BatchNorm2d(ch) | 批归一化 |
nn.LayerNorm(dim) | 层归一化 |
nn.Dropout(p) | Dropout |
nn.Embedding(num, dim) | 嵌入层 |
nn.LSTM(in, hid) | LSTM |
nn.MultiheadAttention(dim, h) | 多头注意力 |
| 函数 | 说明 |
|---|---|
nn.MSELoss() | 均方误差(回归) |
nn.L1Loss() | 平均绝对误差(回归) |
nn.CrossEntropyLoss() | 交叉熵(多分类) |
nn.BCEWithLogitsLoss() | 二分类交叉熵(二分类) |
nn.KLDivLoss() | KL 散度 |
| 优化器 | 说明 |
|---|---|
optim.SGD(params, lr) | 随机梯度下降 |
optim.Adam(params, lr) | 自适应学习率(最常用) |
optim.AdamW(params, lr) | Adam + 解耦权重衰减 |
optim.RMSprop(params, lr) | RMSprop |
| 调度器 | 说明 |
|---|---|
StepLR(optimizer, step_size, gamma) | 每 N 步乘以 gamma |
CosineAnnealingLR(optimizer, T_max) | 余弦退火 |
ReduceLROnPlateau(optimizer) | 指标不提升时降低 |
OneCycleLR(optimizer, max_lr) | 一周期策略 |
| 变换 | 说明 |
|---|---|
transforms.RandomResizedCrop(size) | 随机裁剪并缩放 |
transforms.RandomHorizontalFlip() | 随机水平翻转 |
transforms.ColorJitter(...) | 颜色抖动 |
transforms.RandomRotation(degrees) | 随机旋转 |
transforms.Normalize(mean, std) | 标准化 |
torch.cuda.empty_cache()
data = data.to(device)
loss.sum().backward() 或传入 gradient 参数
labels = labels.float() 或检查损失函数
| GPU | 显存 | CUDA 核心 | 适合场景 | 价格区间 |
|---|---|---|---|---|
| RTX 4060 | 8 GB | 3072 | 学习、小项目、BERT-small 微调 | ¥2000-3000 |
| RTX 4060 Ti | 16 GB | 4352 | 中型模型、ResNet/ViT 微调 | ¥3500-4500 |
| RTX 4070 Ti Super | 16 GB | 8448 | 中大型项目、7B LLM 微调(QLoRA) | ¥5500-6500 |
| RTX 4090 | 24 GB | 16384 | 大型模型、Stable Diffusion、13B 微调 | ¥12000-15000 |
| 平台 | GPU 选项 | 价格 | 适合场景 |
|---|---|---|---|
| Google Colab | T4(免费)/ A100(Pro) | 免费 / $10/月 | 学习、小实验 |
| Kaggle | T4 × 2(免费) | 免费 | 竞赛、中等实验 |
| AutoDL | RTX 4090 / A100 | ¥2-8/小时 | 国内用户首选 |
| AWS | V100 / A100 | $1-4/小时 | 企业级、分布式训练 |
推理时显存约为训练时的 1/3 到 1/2(不需要保存梯度和优化器状态)。
del x; torch.cuda.empty_cache()# 显存监控代码
import torch
def print_gpu_memory():
"""打印当前 GPU 显存使用情况"""
if not torch.cuda.is_available():
print("没有 GPU")
return
allocated = torch.cuda.memory_allocated() / 1024**3
cached = torch.cuda.memory_reserved() / 1024**3
total = torch.cuda.get_device_properties(0).total_mem / 1024**3
print(f"已分配: {allocated:.2f} GB | 缓存: {cached:.2f} GB | 总显存: {total:.2f} GB")
# 训练时定期检查
print_gpu_memory() # 已分配: 0.52 GB | 缓存: 0.68 GB | 总显存: 24.00 GB
# 查看每层的显存占用
def print_model_memory(model, input_size=(1, 3, 224, 224)):
"""估算模型的显存占用"""
params = sum(p.numel() for p in model.parameters())
buffers = sum(b.numel() for b in model.buffers())
param_size = params * 4 / 1024**2 # float32 = 4 bytes
print(f"参数量: {params:,} | 模型大小: {param_size:.1f} MB")
torch.mm、@ 和 torch.einsum 的速度。torch.compile 加速模型,对比速度。