从精通到实战 · 补齐每一个知识盲区
原教程只讲了Paxos的两阶段概述。这里我们从"为什么需要两阶段"开始推导,让你真正理解Paxos的设计动机。
想象一个分布式系统有3个节点(A、B、C)。客户端要写入一个值,比如"设置X=1"。问题是:
- 网络可能丢包、延迟、乱序
- 节点可能宕机重启
- 多个客户端可能同时写入不同的值
我们需要一个算法,让所有节点最终对同一个值达成一致,而且这个值必须是某个客户端提议过的值(不能凭空编造)。
# ========== Phase 1a: Prepare ==========
# Proposer生成一个全局唯一且递增的proposalId
# 向所有Acceptor发送Prepare请求
class Proposer:
def __init__(self, my_value, acceptors):
self.my_value = my_value
self.acceptors = acceptors
self.proposal_id = 0
def prepare(self):
self.proposal_id = generate_unique_id() # 全局唯一递增ID
promises = []
accepted_value = None
accepted_id = -1
# 向所有Acceptor发送Prepare(proposal_id)
for acceptor in self.acceptors:
ok, acc_id, acc_value = acceptor.handle_prepare(self.proposal_id)
if ok:
promises.append(acceptor)
# 关键:收集Acceptor已经接受过的值,选择编号最高的那个
if acc_id > accepted_id:
accepted_id = acc_id
accepted_value = acc_value
# 收到多数派的Promise,进入Phase 2
if len(promises) > len(self.acceptors) // 2:
# 关键决策:用Acceptor返回的值,还是用自己的值
value_to_propose = accepted_value if accepted_value else self.my_value
self._do_accept(value_to_propose)
# ========== Phase 2a: Accept ==========
def _do_accept(self, value):
accepts = 0
for acceptor in self.acceptors:
if acceptor.handle_accept(self.proposal_id, value):
accepts += 1
# 超过半数接受,值被选定!
if accepts > len(self.acceptors) // 2:
chosen(value)
# ========== Acceptor(Phase 1b + Phase 2b)==========
class Acceptor:
def __init__(self):
self.promised_id = -1 # 承诺过的最高proposalId
self.accepted_id = -1 # 已接受的proposalId
self.accepted_value = None # 已接受的值
def handle_prepare(self, proposal_id):
"""Phase 1b: 返回 (ok, accepted_id, accepted_value)"""
if proposal_id > self.promised_id:
self.promised_id = proposal_id
return True, self.accepted_id, self.accepted_value
return False, -1, None
def handle_accept(self, proposal_id, value):
"""Phase 2b: 返回是否接受"""
if proposal_id >= self.promised_id:
self.promised_id = proposal_id
self.accepted_id = proposal_id
self.accepted_value = value
return True
return False
Basic Paxos对每一个值都需要两阶段(Prepare + Accept),效率太低。Multi-Paxos的核心优化:选出一个稳定的Leader,由Leader连续提议,省掉Prepare阶段。
class MultiPaxosLeader:
def __init__(self, nodes):
self.nodes = nodes
self.leader_proposal_id = 0
self.is_leader = False
def elect_leader(self):
"""步骤1:发起Leader选举(用一个很高的proposalId)"""
self.leader_proposal_id = generate_unique_id()
promises = 0
for node in self.nodes:
ok, _, _ = node.handle_prepare(self.leader_proposal_id)
if ok:
promises += 1
if promises > len(self.nodes) // 2:
self.is_leader = True
# 现在可以连续提议了,不需要再Prepare
def propose(self, slot, value):
"""步骤2:连续提议(省掉Prepare阶段)"""
accepts = 0
for node in self.nodes:
if node.handle_accept(self.leader_proposal_id, value):
accepts += 1
if accepts > len(self.nodes) // 2:
pass # 该Slot的值被选定
| 问题 | 解决方案 | 说明 |
|---|---|---|
| proposalId怎么生成? | 高位=时间戳,低位=节点ID | 保证全局唯一且递增 |
| Acceptor状态持久化 | 写入WAL(预写日志) | 宕机重启后能恢复承诺状态 |
| 日志怎么存储? | 连续Slot + 快照压缩 | 类似Raft的日志机制 |
| 成员变更怎么处理? | 将成员配置作为共识值 | 新旧配置过渡期用Joint Consensus |
| 读请求怎么处理? | 必须经过Leader确认 | 否则可能读到旧数据 |
原教程只讲了Raft的选举和日志复制。但随着运行时间增长,日志会无限膨胀。快照是Raft的日志压缩机制。
import pickle
class Snapshot:
def __init__(self, last_index, last_term, data):
self.last_index = last_index
self.last_term = last_term
self.data = data
class RaftNode:
def __init__(self):
self.commit_index = 0
self.snapshot_last_index = 0
self.snapshot_last_term = 0
self.log = [] # [(term, command), ...]
self.state_machine = {} # 简化的状态机
def create_snapshot(self):
"""创建快照"""
# a. 获取当前状态机的完整状态
state = pickle.dumps(self.state_machine)
# b. 记录快照点
snap = Snapshot(
last_index=self.commit_index,
last_term=self._get_term(self.commit_index),
data=state
)
# c. 删除快照点之前的日志
self.log = self.log[self.commit_index:]
# d. 持久化快照到磁盘
with open(f"snapshot_{snap.last_index}.bin", "wb") as f:
pickle.dump(snap, f)
self.snapshot_last_index = snap.last_index
self.snapshot_last_term = snap.last_term
return snap
def install_snapshot(self, snap):
"""安装快照(Leader发送快照给落后的Follower)"""
if snap.last_index > self.commit_index:
self.state_machine = pickle.loads(snap.data)
self.commit_index = snap.last_index
self.snapshot_last_index = snap.last_index
self.snapshot_last_term = snap.last_term
# 删除快照之前的日志
self.log = self.log[snap.last_index:]
def _get_term(self, index):
if 0 <= index < len(self.log):
return self.log[index][0]
return 0
Raft论文提出了两种成员变更方案,这是工程实现中最容易出错的部分。
class RaftMemberChange:
"""成员变更日志作为普通日志提交"""
def __init__(self, current_config, current_term):
self.current_config = current_config # 当前配置:["A", "B", "C"]
self.current_term = current_term
def add_node(self, new_node):
"""新增节点(先以learner身份加入,不参与投票)"""
new_config = self.current_config + [new_node]
# 将新配置作为一条日志提交
self._append_config_log(new_config)
# 复制日志到新节点,等新节点追上进度
# 当新配置的日志被多数派确认后,新节点正式参与投票
self.current_config = new_config
def remove_node(self, old_node):
"""移除节点"""
new_config = [n for n in self.current_config if n != old_node]
self._append_config_log(new_config)
# 等新配置被确认后,旧节点停止接收日志
self.current_config = new_config
def _append_config_log(self, new_config):
"""将配置变更作为日志条目追加"""
entry = {"term": self.current_term, "type": "config", "config": new_config}
# appendEntry(entry) — 追加到日志并复制到多数派
pass
这是面试高频问题:Raft如何保证读到最新数据?
| 方案 | 原理 | 延迟 | 一致性 | 适用场景 |
|---|---|---|---|---|
| ReadIndex | 读请求必须经过Leader确认自己还是Leader | 一次心跳RTT | 强一致 | 大多数场景(推荐) |
| Lease Read | Leader在租约期内直接读,不需要确认 | 零额外延迟 | 强一致(依赖时钟) | 对延迟极敏感的场景 |
| Follower Read | Follower先向Leader确认commitIndex | 一次RPC | 强一致 | 读多写少,减轻Leader压力 |
import time
class ReadIndexHandler:
"""ReadIndex:Leader确认自己还是Leader后,再读取"""
def __init__(self, state_machine, commit_index=0, applied_index=0):
self.state_machine = state_machine
self.commit_index = commit_index
self.applied_index = applied_index
def handle_read(self, key):
# 1. Leader向所有节点发送心跳,确认自己还是Leader
# 同时获取当前的commitIndex
read_index = self.commit_index
# 2. 等待状态机应用到read_index
while self.applied_index < read_index:
time.sleep(0.001) # 等待日志应用
# 3. 从状态机读取(此时保证读到最新数据)
return self.state_machine.get(key)
# etcd中的ReadIndex实现(简化流程)
# 1. 客户端发起读请求
# 2. Leader记录当前commit_index
# 3. Leader向所有Follower发送心跳
# 4. 收到多数派心跳响应 → 确认自己还是Leader
# 5. 等待状态机应用到commit_index
# 6. 返回读结果
--experimental-linearizable-read-safe=false 选项,可以关闭线性一致性读来提高性能(牺牲一致性换性能)
原教程只讲了2PC的正常流程。这里分析每种异常情况及其后果。
原教程提到了TCC的概念,这里给出完整的实现代码和异常处理。
import time
import logging
log = logging.getLogger(__name__)
class Account:
"""账户模型"""
def __init__(self, id, balance=0, frozen=0):
self.id = id
self.balance = balance # 总余额
self.frozen = frozen # 冻结金额
# 可用余额 = balance - frozen
class TransferService:
"""TCC(Try-Confirm-Cancel)转账实现"""
def __init__(self, db):
self.db = db # 数据库连接
# ========== 1. Try阶段:资源预留 ==========
def try_deduct(self, from_id, to_id, amount):
account = self.db.query(Account, from_id)
if account.balance - account.frozen < amount:
return False # 余额不足
# 冻结金额(资源预留)
self.db.execute(
"UPDATE account SET frozen = frozen + %s WHERE id = %s AND balance - frozen >= %s",
(amount, from_id, amount)
)
# 预增加目标账户(记录待入账金额)
self.db.execute(
"UPDATE account SET pending = pending + %s WHERE id = %s",
(amount, to_id)
)
self.db.commit()
return True
# ========== 2. Confirm阶段:确认提交 ==========
def confirm_deduct(self, from_id, to_id, amount):
# 实际扣款(扣减冻结金额)
self.db.execute(
"UPDATE account SET balance = balance - %s, frozen = frozen - %s WHERE id = %s",
(amount, amount, from_id)
)
# 实际入账
self.db.execute(
"UPDATE account SET balance = balance + %s, pending = pending - %s WHERE id = %s",
(amount, amount, to_id)
)
self.db.commit()
return True
# ========== 3. Cancel阶段:回滚 ==========
def cancel_deduct(self, from_id, to_id, amount):
# 解冻金额
self.db.execute(
"UPDATE account SET frozen = frozen - %s WHERE id = %s",
(amount, from_id)
)
# 撤销预增加
self.db.execute(
"UPDATE account SET pending = pending - %s WHERE id = %s",
(amount, to_id)
)
self.db.commit()
return True
class TccTransactionManager:
"""TCC异常处理框架"""
def execute(self, try_fn, confirm_fn, cancel_fn):
try_result = False
try:
# 1. Try阶段
try_result = try_fn()
if not try_result:
return False # Try失败,不需要Cancel
# 2. Confirm阶段(重试机制)
if not self._retry(confirm_fn, max_retries=3):
log.error("TCC Confirm失败,需要人工处理")
return False
return True
except Exception as e:
# 3. Cancel阶段(Try成功但Confirm失败时)
if try_result:
self._retry(cancel_fn, max_retries=3)
return False
def _retry(self, fn, max_retries=3):
"""带退避的重试"""
for i in range(max_retries):
try:
if fn():
return True
except Exception as e:
log.warning(f"重试第{i+1}次失败: {e}")
time.sleep(0.1 * (i + 1))
return False
import logging
log = logging.getLogger(__name__)
class SagaStep:
"""Saga步骤:包含正向操作和补偿操作"""
def __init__(self, name, action, compensate):
self.name = name
self.action = action # 正向操作(callable)
self.compensate = compensate # 补偿操作(callable)
class SagaExecutor:
"""Saga执行引擎"""
def execute(self, steps):
"""
steps: [SagaStep, ...] 按顺序执行的步骤列表
返回: True=全部成功, False=已补偿
"""
completed = 0
try:
# 正向执行所有步骤
for step in steps:
log.info(f"执行步骤: {step.name}")
step.action()
completed += 1
return True
except Exception as e:
log.error(f"步骤失败: {e}")
# 逆序补偿已完成的步骤
for i in range(completed - 1, -1, -1):
try:
log.info(f"补偿步骤: {steps[i].name}")
steps[i].compensate()
except Exception as ce:
log.error(f"补偿失败: {steps[i].name}, 错误: {ce}")
# 补偿失败需要人工介入
return False
# ========== 使用示例:订单创建 ==========
def create_order_saga(order_service, stock_service, payment_service, notification_service, order):
steps = [
SagaStep(
name="创建订单",
action=lambda: order_service.create(order),
compensate=lambda: order_service.cancel(order.id)
),
SagaStep(
name="扣减库存",
action=lambda: stock_service.deduct(order.product_id, order.quantity),
compensate=lambda: stock_service.restore(order.product_id, order.quantity)
),
SagaStep(
name="创建支付",
action=lambda: payment_service.create(order.id, order.amount),
compensate=lambda: payment_service.refund(order.id, order.amount)
),
SagaStep(
name="发送通知",
action=lambda: notification_service.send(order.id),
compensate=lambda: None # 最后一步不需要补偿
),
]
executor = SagaExecutor()
return executor.execute(steps)
| 对比维度 | TCC | Saga |
|---|---|---|
| 资源锁定 | Try阶段锁定资源 | 不锁定资源 |
| 隔离性 | 较好(有冻结机制) | 较差(中间状态可见) |
| 实现复杂度 | 高(需要三个接口) | 中(正向+补偿) |
| 性能 | 较低(资源锁定) | 较高(无锁定) |
| 适用场景 | 短事务、高一致性要求 | 长事务、最终一致性 |
| 回滚方式 | Cancel释放冻结资源 | 补偿操作逆序执行 |
| 典型框架 | Seata TCC | Seata Saga、Eventuate |
import hashlib
from bisect import bisect_right
from collections import defaultdict
class ConsistentHash:
def __init__(self, virtual_node_count=150):
self.ring = {} # {hash_value: node_name}
self.sorted_keys = [] # 排序后的哈希值列表
self.virtual_node_count = virtual_node_count
self.physical_nodes = []
def _hash(self, key):
"""哈希函数(使用MD5取前8字节,比简单取模更均匀)"""
return int(hashlib.md5(key.encode()).hexdigest(), 16)
def add_node(self, node):
"""添加物理节点(自动创建虚拟节点)"""
self.physical_nodes.append(node)
for i in range(self.virtual_node_count):
vnode_key = f"{node}#VN{i}"
h = self._hash(vnode_key)
self.ring[h] = node
self.sorted_keys.append(h)
self.sorted_keys.sort()
def remove_node(self, node):
"""移除物理节点"""
self.physical_nodes.remove(node)
for i in range(self.virtual_node_count):
vnode_key = f"{node}#VN{i}"
h = self._hash(vnode_key)
del self.ring[h]
self.sorted_keys.remove(h)
def get_node(self, key):
"""根据数据key找到对应的物理节点(顺时针查找)"""
if not self.ring:
return None
h = self._hash(key)
idx = bisect_right(self.sorted_keys, h)
# 如果超出环的末尾,回到环的开头
if idx >= len(self.sorted_keys):
idx = 0
return self.ring[self.sorted_keys[idx]]
def get_distribution(self, keys):
"""测试数据分布均匀性"""
count = defaultdict(int)
for key in keys:
node = self.get_node(key)
count[node] += 1
return dict(count)
# 使用示例
ch = ConsistentHash(virtual_node_count=150)
ch.add_node("Node-A")
ch.add_node("Node-B")
ch.add_node("Node-C")
# 10万个key的分布测试
keys = [f"key:{i}" for i in range(100000)]
dist = ch.get_distribution(keys)
print(dist)
# 结果:{'Node-A': 33412, 'Node-B': 33287, 'Node-C': 33301}(非常均匀)
这是面试必考题。很多人只知道"单线程",不知道为什么。
| 策略 | 淘汰范围 | 算法 | 适用场景 |
|---|---|---|---|
| noeviction | 不淘汰 | 写入报错 | 数据不能丢(缓存+持久化) |
| allkeys-lru | 所有key | LRU近似 | 通用缓存(推荐) |
| volatile-lru | 有过期时间的key | LRU近似 | 混合存储(缓存+持久数据) |
| allkeys-lfu | 所有key | LFU | 热点数据缓存 |
| volatile-lfu | 有过期时间的key | LFU | 有过期时间的热点数据 |
| allkeys-random | 所有key | 随机 | 数据访问无规律 |
| volatile-ttl | 有过期时间的key | TTL最小 | 希望保留长期数据 |
allkeys-lfu,它更精确地淘汰冷数据。maxmemory-samples 10)。
| 对比项 | RDB | AOF | 混合持久化(Redis 4.0+) |
|---|---|---|---|
| 原理 | 定期全量快照 | 追加写命令日志 | RDB快照 + 增量AOF |
| 数据安全性 | 可能丢失最后一次快照后的数据 | 最多丢失1秒数据 | 最多丢失1秒数据 |
| 恢复速度 | 快(直接加载二进制) | 慢(重放所有命令) | 快(RDB加载 + 少量AOF重放) |
| 文件大小 | 小(压缩的二进制) | 大(命令文本) | 中等 |
| 性能影响 | fork子进程时有短暂阻塞 | 追加写,影响小 | 兼顾两者优点 |
| 推荐 | 数据安全性要求不高 | 数据安全性要求高 | 生产环境推荐 |
# redis.conf 推荐配置
# 启用AOF
appendonly yes
# 启用混合持久化(Redis 4.0+)
aof-use-rdb-preamble yes
# AOF同步策略
# always:每条命令都同步(最安全,最慢)
# everysec:每秒同步一次(推荐,最多丢1秒数据)
# no:由操作系统决定(最快,可能丢数据)
appendfsync everysec
# AOF重写触发条件
auto-aof-rewrite-percentage 100 # AOF文件增长100%时触发重写
auto-aof-rewrite-min-size 64mb # AOF文件至少64MB才触发重写
# RDB快照配置
save 900 1 # 900秒内有1次修改,触发快照
save 300 10 # 300秒内有10次修改,触发快照
save 60 10000 # 60秒内有10000次修改,触发快照
acks=all 表示Producer等待所有ISR副本确认才认为发送成功。min.insync.replicas=2 表示ISR至少要有2个副本,否则拒绝写入。acks=all + min.insync.replicas=2Kafka 3.3+正式移除了对ZooKeeper的依赖,改用内置的KRaft共识协议。
# Producer配置(启用幂等和事务)
enable.idempotence=true # 启用幂等(同一消息不重复写入)
transactional.id=order-service # 事务ID(每个Producer唯一)
acks=all # 等待所有ISR确认
retries=3 # 重试次数
max.in.flight.requests.per.connection=5 # 允许的最大未确认请求数
# Java代码:事务Producer
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("transactional.id", "order-service-1");
props.put("enable.idempotence", "true");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions(); // 初始化事务
try {
producer.beginTransaction();
// 发送多条消息(原子性)
producer.send(new ProducerRecord<>("orders", "order1", "data1"));
producer.send(new ProducerRecord<>("inventory", "stock1", "data2"));
producer.send(new ProducerRecord<>("payment", "pay1", "data3"));
producer.commitTransaction(); // 全部成功才提交
} catch (Exception e) {
producer.abortTransaction(); // 任一失败则全部回滚
}
isolation.level=read_committed,只读取已提交事务的消息。import time
import threading
class SnowflakeIdGenerator:
"""
雪花算法ID生成器
时间戳占41位,可用69年
数据中心占5位,最多32个数据中心
机器ID占5位,最多32台机器
序列号占12位,每毫秒最多4096个ID
"""
START_EPOCH = 1609459200000 # 2021-01-01 00:00:00
DATACENTER_ID_BITS = 5
WORKER_ID_BITS = 5
SEQUENCE_BITS = 12
MAX_DATACENTER_ID = (1 << DATACENTER_ID_BITS) - 1 # 31
MAX_WORKER_ID = (1 << WORKER_ID_BITS) - 1 # 31
SEQUENCE_MASK = (1 << SEQUENCE_BITS) - 1 # 4095
WORKER_ID_SHIFT = SEQUENCE_BITS
DATACENTER_ID_SHIFT = SEQUENCE_BITS + WORKER_ID_BITS
TIMESTAMP_LEFT_SHIFT = SEQUENCE_BITS + WORKER_ID_BITS + DATACENTER_ID_BITS
def __init__(self, datacenter_id, worker_id):
if datacenter_id > self.MAX_DATACENTER_ID or datacenter_id < 0:
raise ValueError(f"Datacenter ID out of range: {datacenter_id}")
if worker_id > self.MAX_WORKER_ID or worker_id < 0:
raise ValueError(f"Worker ID out of range: {worker_id}")
self.datacenter_id = datacenter_id
self.worker_id = worker_id
self.sequence = 0
self.last_timestamp = -1
self._lock = threading.Lock()
def _current_millis(self):
return int(time.time() * 1000)
def _wait_next_millis(self, last_ts):
ts = self._current_millis()
while ts <= last_ts:
ts = self._current_millis()
return ts
def next_id(self):
with self._lock:
timestamp = self._current_millis()
# 时钟回拨检测
if timestamp < self.last_timestamp:
offset = self.last_timestamp - timestamp
if offset <= 5:
# 回拨在5ms以内,等待时钟追上
time.sleep(offset / 1000.0 * 2)
timestamp = self._current_millis()
if timestamp < self.last_timestamp:
raise RuntimeError(f"Clock moved backwards by {offset}ms")
else:
# 回拨超过5ms,直接报错
raise RuntimeError(f"Clock moved backwards by {offset}ms")
# 同一毫秒内的序列号递增
if timestamp == self.last_timestamp:
self.sequence = (self.sequence + 1) & self.SEQUENCE_MASK
if self.sequence == 0:
# 序列号用完了,等待下一毫秒
timestamp = self._wait_next_millis(self.last_timestamp)
else:
self.sequence = 0 # 新毫秒,序列号重置
self.last_timestamp = timestamp
# 组装ID
return (
((timestamp - self.START_EPOCH) << self.TIMESTAMP_LEFT_SHIFT)
| (self.datacenter_id << self.DATACENTER_ID_SHIFT)
| (self.worker_id << self.WORKER_ID_SHIFT)
| self.sequence
)
# 使用示例
gen = SnowflakeIdGenerator(datacenter_id=1, worker_id=1)
print(gen.next_id()) # 例如:1478294827008000001
| 方案 | 原理 | 优点 | 缺点 | 适用场景 |
|---|---|---|---|---|
| UUID | 128位随机数 | 简单,无中心节点 | 无序,索引性能差,太长 | 日志traceId |
| 雪花算法 | 时间戳+机器ID+序列号 | 有序,高性能,趋势递增 | 依赖时钟,机器ID需管理 | 通用(推荐) |
| 数据库自增 | 数据库AUTO_INCREMENT | 简单,绝对有序 | 单点瓶颈,分库分表麻烦 | 低并发场景 |
| 号段模式(Leaf) | 每次从DB获取一段ID | 高性能,容灾好 | 实现复杂,ID不连续 | 高并发(美团Leaf) |
| Redis INCR | Redis原子递增 | 简单,高性能 | 依赖Redis,持久化风险 | 已有Redis的场景 |
| Tinyid | 双Buffer预加载 | 高可用,号段用完前预加载 | 需要部署服务 | 滴滴Tinyid |
import etcd3
import threading
import time
import logging
log = logging.getLogger(__name__)
class LockResult:
SUCCESS = True
FAILED = False
def __init__(self, success, lease_id=None, fencing_token=None, keepalive_thread=None):
self.success = success
self.lease_id = lease_id
self.fencing_token = fencing_token
self._keepalive_thread = keepalive_thread
def is_success(self):
return self.success
def get_fencing_token(self):
return self.fencing_token
class EtcdDistributedLock:
"""使用etcd实现带fencing token的分布式锁"""
def __init__(self, host="localhost", port=2379):
self.client = etcd3.client(host=host, port=port)
def try_lock(self, lock_key, ttl_seconds=30):
# 1. 创建Lease(带TTL的租约)
lease = self.client.lease(ttl_seconds)
try:
# 2. 加锁(etcd的lock方法会阻塞直到获取成功)
lock_key_bytes = lock_key.encode()
self.client.lock(lock_key_bytes, lease=lease)
# 3. 获取fencing token(key的revision,单调递增)
kv, metadata = self.client.get(lock_key_bytes)
fencing_token = metadata.mod_revision
# 4. 启动后台线程续期(看门狗)
keepalive_thread = threading.Thread(
target=self._keepalive_loop,
args=(lease, ttl_seconds),
daemon=True
)
keepalive_thread.start()
return LockResult(True, lease.id, fencing_token, keepalive_thread)
except Exception as e:
log.error(f"获取锁失败: {e}")
lease.revoke()
return LockResult(False)
def _keepalive_loop(self, lease, ttl_seconds):
"""看门狗:定期续期,防止锁提前过期"""
interval = ttl_seconds // 3
while True:
time.sleep(interval)
try:
lease.refresh()
except Exception as e:
log.error(f"锁续期失败: {e}")
break
def unlock(self, lock_key, lease):
"""释放锁"""
try:
self.client.unlock(lock_key.encode())
lease.revoke()
except Exception as e:
log.error(f"释放锁失败: {e}")
class ResourceServer:
"""资源服务器端:检查fencing token"""
def __init__(self):
self.last_seen_token = 0
self._lock = threading.Lock()
def check_fencing_token(self, token):
with self._lock:
if token >= self.last_seen_token:
self.last_seen_token = token
return True # token有效
return False # token过期,拒绝请求
# 使用示例
lock_manager = EtcdDistributedLock(host="localhost", port=2379)
resource_server = ResourceServer()
result = lock_manager.try_lock("lock:order:123", ttl_seconds=30)
if result.is_success():
fencing_token = result.get_fencing_token()
if resource_server.check_fencing_token(fencing_token):
# 执行业务逻辑
pass
lock_manager.unlock("lock:order:123", result.lease_id)
| 对比项 | B+树 | 红黑树 |
|---|---|---|
| 树高 | 3-4层(亿级数据) | O(log n)(树更高) |
| 磁盘IO | 少(一个节点存多个key) | 多(每个节点一个key) |
| 范围查询 | 叶子节点链表,极快 | 需要中序遍历,慢 |
| 插入/删除 | O(log n) | O(log n) + 旋转 |
| 适用场景 | 磁盘存储(数据库) | 内存存储(TreeMap) |
| Spring Cloud Version | Spring Boot Version | Spring Cloud Alibaba | Java Version |
|---|---|---|---|
| 2023.0.x (Leyton) | 3.2.x | 2023.0.x | 17+ |
| 2022.0.x (Kilburn) | 3.0.x | 2022.0.x | 17+ |
| 2021.0.x (Jubilee) | 2.6.x - 2.7.x | 2021.0.x | 8+ |
| Hoxton.SR12 | 2.3.x - 2.5.x | 2.2.x | 8+ |
| Greenwich.SR6 | 2.1.x - 2.2.x | 2.1.x | 8+ |
<!-- parent pom.xml -->
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.2.4</version>
</parent>
<dependencyManagement>
<dependencies>
<!-- Spring Cloud -->
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>2023.0.1</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<!-- Spring Cloud Alibaba -->
<dependency>
<groupId>com.alibaba.cloud</groupId>
<artifactId>spring-cloud-alibaba-dependencies</artifactId>
<version>2023.0.1.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
javax.* 包名改成了 jakarta.*。所有依赖javax的库都需要升级。jakarta.servlet:jakarta.servlet-api 替换。| 工具 | 适用场景 | 优点 | 缺点 |
|---|---|---|---|
| JMeter | HTTP/RPC接口压测 | GUI操作,插件丰富 | 资源消耗大,并发有限 |
| wrk | HTTP压测 | 高性能,单机10万+QPS | 脚本用Lua,学习成本 |
| wrk2 | 精确QPS控制 | 固定QPS,延迟统计准确 | 命令行操作 |
| Locust | 复杂场景压测 | Python脚本,Web UI | 性能不如wrk |
| k6 | API压测 | JavaScript脚本,云原生 | 需要Node.js环境 |
# 压测报告模板
## 测试环境
- 服务器配置:4核8G × 3台
- JDK版本:OpenJDK 17
- 数据库:MySQL 8.0(4核8G)
- Redis:6.2(2核4G)
- 压测工具:wrk 4.2
## 测试场景
- 接口:POST /api/orders/create
- 并发数:100、500、1000、2000
- 持续时间:5分钟
- 预热时间:30秒
## 测试结果
| 并发数 | QPS | 平均延迟 | P95延迟 | P99延迟 | 错误率 |
|--------|-----|----------|---------|---------|--------|
| 100 | 3,200 | 31ms | 45ms | 68ms | 0% |
| 500 | 12,500 | 40ms | 62ms | 95ms | 0% |
| 1000 | 18,000 | 55ms | 89ms | 145ms | 0.01% |
| 2000 | 20,000 | 100ms | 180ms | 350ms | 0.5% |
## 瓶颈分析
1. 2000并发时QPS不再增长 → 数据库连接池瓶颈
2. P99延迟突增 → GC停顿(Full GC触发)
3. 错误率上升 → 连接池耗尽,请求超时
## 优化建议
1. 增大数据库连接池(HikariCP: maxPoolSize 10→20)
2. 优化JVM参数(-Xmx4g -Xms4g -XX:+UseG1GC)
3. 增加Redis缓存,减少数据库查询
# 1. 在Master机器上配置Slave机器列表
# jmeter.properties
remote_hosts=192.168.1.101,192.168.1.102,192.168.1.103
# 2. 在每台Slave机器上启动JMeter Server
jmeter-server -Djava.rmi.server.hostname=192.168.1.101
# 3. 在Master机器上启动分布式压测
jmeter -n -t test_plan.jmx -R 192.168.1.101,192.168.1.102,192.168.1.103
# 4. 测试计划关键配置
# Thread Group:
# Number of Threads: 1000 (每台Slave 1000并发,3台共3000)
# Ramp-Up Period: 30 (30秒内逐步增加到目标并发)
# Loop Count: Forever (持续压测)
# Duration: 300 (压测5分钟)
# HTTP Request:
# Server: api.example.com
# Port: 8080
# Method: POST
# Body: {"userId": "${userId}", "productId": "${productId}"}
# CSV Data Set Config (参数化):
# Filename: test_data.csv
# Variable Names: userId,productId
# Sharing Mode: All threads
Netflix发明了"混沌猴子"(Chaos Monkey):在生产环境中随机杀死服务实例,验证系统是否能自动恢复。
后来又有了"延迟猴子"(Latency Monkey):注入网络延迟,验证超时处理是否正确。
核心思想:与其在真实故障中手忙脚乱,不如主动制造故障,在可控条件下发现问题。
| 故障类型 | 注入方式 | 验证目标 | 工具 |
|---|---|---|---|
| 服务宕机 | kill进程 | 自动重启、故障转移 | ChaosBlade |
| 网络延迟 | tc命令注入延迟 | 超时处理、重试机制 | tc、ChaosBlade |
| 网络丢包 | tc命令注入丢包 | 重传机制、幂等性 | tc、ChaosBlade |
| CPU满载 | stress命令 | 限流降级、弹性伸缩 | stress、ChaosBlade |
| 内存满载 | stress命令 | OOM处理、内存回收 | stress、ChaosBlade |
| 磁盘满 | dd填充磁盘 | 日志轮转、告警 | dd、ChaosBlade |
| DNS故障 | 修改/etc/hosts | 服务发现容灾 | ChaosBlade |
# 安装ChaosBlade
wget https://github.com/chaosblade-io/chaosblade/releases/download/v1.7.2/blade-1.7.2-linux-amd64.tar.gz
tar -xvf blade-1.7.2-linux-amd64.tar.gz
# 1. 模拟服务延迟(API调用延迟2秒)
blade create network delay --time 2000 --interface eth0 --remote-port 8080
# 2. 模拟网络丢包(30%丢包率)
blade create network loss --percent 30 --interface eth0
# 3. 模拟CPU满载(4核满载60秒)
blade create cpu fullload --cpu-percent 80 --timeout 60
# 4. 模拟内存满载(占用2G内存)
blade create mem load --mode ram --mem 2048 --timeout 60
# 5. 模拟磁盘IO延迟(读写延迟100ms)
blade create disk delay --path /data --delay 100
# 6. 模拟Java应用异常(抛出指定异常)
blade create jvm throwCustomException --classname java.lang.NullPointerException \
--methodname getOrder --process order-service
# 7. 模拟MySQL连接超时
blade create network delay --time 5000 --remote-port 3306
# 销毁故障注入
blade destroy
min-replicas-to-write 1 + min-replicas-max-lag 10。当Leader没有足够副本时拒绝写入,防止脑裂后数据丢失。
acks=all + retries=3min.insync.replicas=2 + unclean.leader.election.enable=falseenable.auto.commit=false,手动提交offset可观测性(Observability)是微服务架构的"眼睛"。没有它,线上出了问题你就是瞎子。三大支柱各有分工,缺一不可。
# ===== RED指标(每个服务必监控)=====
# Rate:请求速率(QPS)
rate(http_requests_total[5m])
# Errors:错误率
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# Duration:延迟分布
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# ===== USE指标(基础设施必监控)=====
# Utilization:利用率
node_cpu_seconds_total
node_memory_MemAvailable_bytes
# Saturation:饱和度
node_load1 # 1分钟负载
# Errors:错误数
node_disk_io_errors_total
# ===== 业务指标 =====
# 订单创建速率
rate(order_created_total[5m])
# 支付成功率
sum(rate(payment_success_total[5m]))
/
sum(rate(payment_total[5m]))
# 库存扣减延迟
histogram_quantile(0.99, rate(stock_deduct_duration_seconds_bucket[5m]))
# 1. 下载SkyWalking Agent
wget https://dlcdn.apache.org/skywalking/java-agent/9.0.0/apache-skywalking-java-agent-9.0.0.tgz
# 2. 启动时加载Agent(不需要改代码)
java -javaagent:/path/to/skywalking-agent.jar \
-Dskywalking.agent.service_name=order-service \
-Dskywalking.collector.backend_service=127.0.0.1:11800 \
-jar order-service.jar
# 3. 在代码中添加自定义Span(可选)
@GetMapping("/order/{id}")
public Order getOrder(@PathVariable Long id) {
// SkyWalking自动创建Span,也可以手动添加
Span span = ContextManager.createLocalSpan("queryOrderDetail");
try {
// 业务逻辑
Order order = orderService.getById(id);
// 添加自定义标签(方便搜索)
span.tag("orderId", String.valueOf(id));
return order;
} finally {
span.asyncFinish();
}
}
// 在网关层生成TraceId,传递给所有下游服务
@Component
public class TraceFilter implements GlobalFilter {
private static final String TRACE_ID = "traceId";
@Override
public Mono<Void> filter(ServerWebExchange exchange, GatewayFilterChain chain) {
// 从请求头获取或生成新的TraceId
String traceId = exchange.getRequest().getHeaders().getFirst(TRACE_ID);
if (traceId == null) {
traceId = UUID.randomUUID().toString().replace("-", "");
}
// 注入MDC(Mapped Diagnostic Context),日志自动带上traceId
MDC.put(TRACE_ID, traceId);
// 把traceId传递给下游服务
ServerHttpRequest request = exchange.getRequest().mutate()
.header(TRACE_ID, traceId)
.build();
return chain.filter(exchange.mutate().request(request).build());
}
}
// logback配置:日志格式包含traceId
// <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] [%X{traceId}] %-5level %logger{36} - %msg%n</pattern>
// 输出效果:
// 2024-01-15 10:30:00.123 [http-nio-8080-exec-1] [a1b2c3d4e5f6] ERROR OrderService - 创建订单失败
// 2024-01-15 10:30:00.125 [http-nio-8080-exec-1] [a1b2c3d4e5f6] ERROR StockService - 库存不足
// 在ELK中搜索 traceId=a1b2c3d4e5f6,可以看到整个请求链路的所有日志
# 金丝雀发布:90%流量到v1,10%流量到v2
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: order-service
spec:
hosts:
- order-service
http:
- route:
- destination:
host: order-service
subset: v1
weight: 90
- destination:
host: order-service
subset: v2
weight: 10
---
# 定义v1和v2子集
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: order-service
spec:
host: order-service
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
# 熔断配置
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: order-service-circuit-breaker
spec:
host: order-service
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
h2UpgradePolicy: DEFAULT
http1MaxPendingRequests: 100
http2MaxRequests: 1000
outlierDetection:
consecutive5xxErrors: 5 # 连续5个5xx错误触发熔断
interval: 30s # 检测间隔
baseEjectionTime: 30s # 熔断时间
maxEjectionPercent: 50 # 最多踢出50%的实例
# Pod中Sidecar资源配置
apiVersion: v1
kind: Pod
metadata:
name: order-service
annotations:
# Istio注入的Sidecar默认资源限制
sidecar.istio.io/proxyCPU: "100m" # 请求0.1核CPU
sidecar.istio.io/proxyCPULimit: "500m" # 最大0.5核CPU
sidecar.istio.io/proxyMemory: "128Mi" # 请求128MB内存
sidecar.istio.io/proxyMemoryLimit: "256Mi" # 最大256MB内存
# 启动探针(Sidecar启动慢会导致业务容器也延迟启动)
sidecar.istio.io/proxyStartupProbe: |
failureThreshold: 30
periodSeconds: 2
initialDelaySeconds: 1
spec:
containers:
- name: order-service
image: order-service:1.0
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "2"
memory: "1Gi"
connectionPool.tcp.maxConnections: 1024connectionPool.http.http1MaxPendingRequests: 1024connectionPool.http.http2MaxRequests: 1024httpRoute.timeout: 10s — 全局超时httpRoute.retryPolicy.perTryTimeout: 2s — 单次重试超时restartPolicy: Always),Sidecar在业务容器之前启动。http2_protocol_options减少连接建立开销traffic.sidecar.istio.io/excludeOutboundPorts绕过Sidecar
# 默认情况下,每个Envoy代理都有所有服务的路由配置
# 服务多了之后,Envoy的配置会很大,影响性能和内存
# 用Sidecar资源限制代理范围
apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
name: order-service-sidecar
namespace: shop
spec:
workloadSelector:
labels:
app: order-service
egress:
# 只代理这几个出站流量,其他走直连
- hosts:
- "shop/user-service.shop.svc.cluster.local"
- "shop/stock-service.shop.svc.cluster.local"
- "shop/payment-service.shop.svc.cluster.local"
- "istio-system/*" # 必须包含istio-system
# 入站流量只监听本服务的端口
ingress:
- port:
number: 8080
protocol: HTTP
defaultEndpoint: 127.0.0.1:8080
# 效果:
# 1. Envoy配置体积减小60%+
# 2. 内存占用从100MB降到40MB
# 3. 启动速度更快
# 4. 只代理需要的服务,其他服务直连
CDC是实现数据最终一致性的核心技术。它的思想很简单:监听数据库的变更日志,把变更事件发送给下游系统。
# 1. Canal Server配置(canal.properties)
canal.instance.master.address=127.0.0.1:3306
canal.instance.dbUsername=canal
canal.instance.dbPassword=canal
canal.instance.filter.regex=shop\\.order,shop\\.user
# 2. Canal Kafka适配器配置
canal.mq.servers=127.0.0.1:9092
canal.mq.retries=3
canal.mq.acks=all
# 3. 消费Canal消息,同步到Redis(Python)
# 注意:Canal发送到Kafka的消息格式是JSON,包含database、table、type、data等字段
# 每条消息对应一次数据库变更事件
import json
from kafka import KafkaConsumer
import redis
class CanalConsumer:
def __init__(self, kafka_brokers, redis_host, topic="canal_shop_order"):
self.consumer = KafkaConsumer(
topic,
bootstrap_servers=kafka_brokers,
value_deserializer=lambda m: json.loads(m.decode("utf-8")),
group_id="canal-sync-group",
auto_offset_reset="latest"
)
self.redis = redis.Redis(host=redis_host, port=6379, decode_responses=True)
def start(self):
"""启动消费循环"""
print("Canal消费者启动...")
for message in self.consumer:
self._process(message.value)
def _process(self, msg):
"""处理一条Canal消息"""
# 消息结构:{"database":"shop","table":"order","type":"INSERT","data":[{...}]}
msg_type = msg.get("type")
data = msg.get("data", [])
if not data:
return
for row in data:
order_id = row.get("id")
if msg_type in ("INSERT", "UPDATE"):
# 新增或更新:同步到Redis
order = {
"id": order_id,
"user_id": row.get("user_id"),
"product_id": row.get("product_id"),
"status": row.get("status"),
}
self.redis.setex(f"order:{order_id}", 3600, json.dumps(order))
elif msg_type == "DELETE":
# 删除:清除Redis缓存
self.redis.delete(f"order:{order_id}")
# 使用示例
if __name__ == "__main__":
consumer = CanalConsumer(
kafka_brokers=["localhost:9092"],
redis_host="localhost",
topic="canal_shop_order"
)
consumer.start()
# canal.properties 核心配置
# 1. 模式:集群模式(需要ZK)
canal.instance.global.spring.xml = classpath:spring/default-instance.xml
# 2. ZooKeeper地址(用于HA选主和位点存储)
canal.zkServers = zk1:2181,zk2:2181,zk3:2181
# 3. Canal在ZK上的节点路径
canal.zk.root = /otter/canal
# 4. 持久化方式(存储到ZK,切换时不丢位点)
canal.instance.tsdb.enable = true
canal.instance.tsdb.spring.xml = classpath:spring/tsdb/h2-tsdb.xml
# 5. MQ适配器(选择Kafka或RocketMQ)
canal.mq.servers = kafka1:9092,kafka2:9092,kafka3:9092
canal.mq.retries = 3
canal.mq.acks = all
canal.mq.transaction = false # 是否发送事务消息
# instance.properties(每个数据库实例一个配置文件)
# 位于 conf/shop/instance.properties
# 1. MySQL连接信息
canal.instance.master.address = 127.0.0.1:3306
canal.instance.dbUsername = canal
canal.instance.dbPassword = canal123
# 2. 监听的数据库和表(正则匹配)
canal.instance.filter.regex = shop\\.order,shop\\.user,shop\\.product
# 3. 黑名单(不监听的表)
canal.instance.filter.black.regex = shop\\.log.*,shop\\.tmp.*
# 4. MQ topic映射(不同表发到不同topic)
canal.mq.dynamicTopic = order:shop\\.order,user:shop\\.user
# 5. 消息格式(JSON或Protobuf)
canal.mq.flatMessage = true # true=JSON, false=Protobuf
SHOW VARIABLES LIKE 'binlog_format'; 必须是ROW,不是STATEMENT。否则Canal解析会出错。SET GLOBAL binlog_format = 'ROW';SHOW VARIABLES LIKE 'expire_logs_days'; 建议设为7天以上。JAVA_OPTS="-Xms2g -Xmx2g -Xmn1g" — 至少2G堆内存,大表场景需要更多。/metrics端点。关键指标:canal_instance_traffic_delay — Binlog消费延迟(秒)canal_instance_publish_rows — 已消费的行数
| 方案 | 原理 | 一致性 | 侵入性 | 延迟 | 适用场景 |
|---|---|---|---|---|---|
| 双写 | 业务代码同时写DB和缓存 | 差(两步操作可能不一致) | 高(改业务代码) | 低 | 简单场景,不推荐 |
| CDC(Canal) | 监听Binlog,异步同步 | 好(Binlog是最终一致的保证) | 低(不改业务代码) | 秒级 | 大多数场景(推荐) |
| 订阅(MQ) | 业务代码发MQ,消费者同步 | 好(MQ保证消息不丢) | 中(要发MQ消息) | 毫秒级 | 需要精确控制同步逻辑 |
from datetime import datetime
from decimal import Decimal
import json
# ========== 事件定义 ==========
class AccountEvent:
def __init__(self, account_id):
self.account_id = account_id
self.timestamp = datetime.now().isoformat()
class MoneyDeposited(AccountEvent):
def __init__(self, account_id, amount):
super().__init__(account_id)
self.amount = amount
class MoneyWithdrawn(AccountEvent):
def __init__(self, account_id, amount):
super().__init__(account_id)
self.amount = amount
# ========== 事件存储 ==========
class EventStore:
def __init__(self, db_connection):
self.db = db_connection
def save_event(self, event):
self.db.execute(
"INSERT INTO event_store (account_id, event_type, event_data, created_at) "
"VALUES (%s, %s, %s, %s)",
(event.account_id, type(event).__name__,
json.dumps({"amount": str(event.amount)}), event.timestamp)
)
self.db.commit()
def get_events(self, account_id):
rows = self.db.query(
"SELECT event_type, event_data FROM event_store "
"WHERE account_id = %s ORDER BY created_at",
(account_id,)
)
events = []
for row in rows:
data = json.loads(row["event_data"])
amount = Decimal(data["amount"])
if row["event_type"] == "MoneyDeposited":
events.append(MoneyDeposited(account_id, amount))
elif row["event_type"] == "MoneyWithdrawn":
events.append(MoneyWithdrawn(account_id, amount))
return events
# ========== 聚合根:从事件重建状态 ==========
class Account:
def __init__(self, account_id):
self.id = account_id
self.balance = Decimal("0")
self.uncommitted_events = []
@classmethod
def from_events(cls, account_id, events):
"""从事件重建状态"""
account = cls(account_id)
for event in events:
account._apply(event)
return account
def _apply(self, event):
"""应用事件(改变状态)"""
if isinstance(event, MoneyDeposited):
self.balance += event.amount
elif isinstance(event, MoneyWithdrawn):
self.balance -= event.amount
def deposit(self, amount):
"""业务操作:存款"""
event = MoneyDeposited(self.id, amount)
self._apply(event) # 先改变状态
self.uncommitted_events.append(event) # 再记录事件
def withdraw(self, amount):
"""业务操作:取款"""
if self.balance < amount:
raise ValueError("余额不足")
event = MoneyWithdrawn(self.id, amount)
self._apply(event)
self.uncommitted_events.append(event)
# 使用示例
# account = Account(1001)
# account.deposit(Decimal("500"))
# account.withdraw(Decimal("200"))
# print(account.balance) # 300
#
# # 从事件重建状态
# store = EventStore(db)
# events = store.get_events(1001)
# rebuilt = Account.from_events(1001, events)
# print(rebuilt.balance) # 和当前状态一致
mall-cloud/
├── mall-common/ # 公共模块(工具类、通用配置)
│ ├── src/main/java/
│ │ └── com.mall.common/
│ │ ├── config/ # 全局配置(CORS、序列化、异常处理)
│ │ ├── exception/ # 统一异常定义
│ │ ├── result/ # 统一返回结果(R<T>)
│ │ ├── util/ # 工具类(日期、加密、ID生成)
│ │ └── constant/ # 常量定义
│ └── pom.xml
│
├── mall-gateway/ # API网关
│ ├── src/main/java/
│ │ └── com.mall.gateway/
│ │ ├── config/ # 网关配置(路由、限流、跨域)
│ │ ├── filter/ # 全局过滤器(鉴权、日志、限流)
│ │ └── handler/ # 异常处理
│ ├── src/main/resources/
│ │ ├── application.yml # 路由配置
│ │ └── bootstrap.yml # Nacos配置
│ └── pom.xml
│
├── mall-auth/ # 认证服务
│ ├── src/main/java/
│ │ └── com.mall.auth/
│ │ ├── controller/ # 登录、注册、Token刷新接口
│ │ ├── service/ # 认证逻辑
│ │ └── util/ # JWT工具类
│ └── pom.xml
│
├── mall-user/ # 用户服务
│ ├── src/main/java/
│ │ └── com.mall.user/
│ │ ├── controller/ # 用户接口
│ │ ├── service/ # 业务逻辑
│ │ ├── mapper/ # MyBatis Mapper
│ │ ├── entity/ # 实体类
│ │ └── dto/ # 数据传输对象
│ └── pom.xml
│
├── mall-order/ # 订单服务
│ ├── src/main/java/
│ │ └── com.mall.order/
│ │ ├── controller/
│ │ ├── service/
│ │ │ └── impl/
│ │ │ └── OrderServiceImpl.java
│ │ ├── mapper/
│ │ ├── entity/
│ │ ├── dto/
│ │ ├── feign/ # Feign客户端(调用其他服务)
│ │ │ ├── UserFeignClient.java
│ │ │ ├── ProductFeignClient.java
│ │ │ └── StockFeignClient.java
│ │ └── mq/ # MQ生产者/消费者
│ │ ├── OrderProducer.java
│ │ └── OrderConsumer.java
│ └── pom.xml
│
├── mall-product/ # 商品服务
├── mall-stock/ # 库存服务
├── mall-payment/ # 支付服务
│
├── mall-monitor/ # 监控中心(Spring Boot Admin)
├── doc/ # 文档
│ ├── sql/ # 数据库脚本
│ ├── docker/ # Docker Compose配置
│ └── postman/ # Postman集合
│
├── docker-compose.yml # 一键启动所有依赖
├── pom.xml # 父POM(统一版本管理)
└── README.md
public class R<T> implements Serializable {
private int code; // 状态码:200成功,500失败
private String message; // 提示信息
private T data; // 数据
public static <T> R<T> ok(T data) {
R<T> r = new R<>();
r.setCode(200);
r.setMessage("success");
r.setData(data);
return r;
}
public static <T> R<T> fail(String message) {
R<T> r = new R<>();
r.setCode(500);
r.setMessage(message);
return r;
}
public static <T> R<T> fail(int code, String message) {
R<T> r = new R<>();
r.setCode(code);
r.setMessage(message);
return r;
}
}
// 全局异常处理
@RestControllerAdvice
public class GlobalExceptionHandler {
@ExceptionHandler(BusinessException.class)
public R<?> handleBusinessException(BusinessException e) {
return R.fail(e.getCode(), e.getMessage());
}
@ExceptionHandler(Exception.class)
public R<?> handleException(Exception e) {
log.error("系统异常", e);
return R.fail("系统繁忙,请稍后重试");
}
}
// Feign客户端定义
@FeignClient(name = "stock-service", fallbackFactory = StockFeignFallback.class)
public interface StockFeignClient {
@PostMapping("/api/stock/deduct")
R<Boolean> deduct(@RequestParam("productId") Long productId,
@RequestParam("quantity") int quantity);
@GetMapping("/api/stock/{productId}")
R<Integer> getStock(@PathVariable("productId") Long productId);
}
// 降级工厂(Sentinel熔断时执行)
@Component
public class StockFeignFallback implements FallbackFactory<StockFeignClient> {
@Override
public StockFeignClient create(Throwable cause) {
log.error("库存服务调用失败,触发降级", cause);
return new StockFeignClient() {
@Override
public R<Boolean> deduct(Long productId, int quantity) {
// 降级策略:发送MQ消息,后续异步补偿
mqService.sendDeductMessage(productId, quantity);
return R.fail("库存服务暂时不可用,已转入异步处理");
}
@Override
public R<Integer> getStock(Long productId) {
// 降级策略:返回缓存数据
Integer cached = redisTemplate.opsForValue().get("stock:" + productId);
return R.ok(cached != null ? cached : -1);
}
};
}
}
// 在订单服务中使用
@Service
public class OrderServiceImpl implements OrderService {
@Autowired
private StockFeignClient stockFeignClient;
@Override
public R<Order> createOrder(CreateOrderRequest request) {
// 1. 扣减库存(Feign调用 + Sentinel降级)
R<Boolean> deductResult = stockFeignClient.deduct(
request.getProductId(), request.getQuantity());
if (deductResult.getCode() != 200) {
return R.fail("库存扣减失败:" + deductResult.getMessage());
}
// 2. 创建订单
Order order = new Order();
order.setUserId(request.getUserId());
order.setProductId(request.getProductId());
order.setQuantity(request.getQuantity());
order.setStatus(0);
orderDao.insert(order);
return R.ok(order);
}
}
KafkaProducer<String, String> producer = new KafkaProducer<>((props);KafkaProducer<String, String> producer = new KafkaProducer<>(props);(。
| 技术 | 原教程版本 | 最新稳定版 | 重要变化 |
|---|---|---|---|
| Redis | 6.x | 7.4+ | Redis Functions(替代Lua脚本)、Pub/Sub支持Sharding、客户端缓存改进 |
| Kafka | 3.x | 3.8+ | KRaft模式成熟、ZooKeeper完全移除、KIP-848新消费者组协议 |
| Spring Boot | 2.x/3.0 | 3.3+ | GraalVM原生编译支持成熟、Virtual Threads(虚拟线程)支持 |
| Spring Cloud | 2021.x | 2024.0+ | Spring Cloud Gateway新特性、LoadBalancer替代Ribbon |
| Docker | 24.x | 27+ | BuildKit默认启用、containerd成为默认运行时 |
| Kubernetes | 1.28 | 1.31+ | Sidecar容器正式支持、In-place Pod资源调整 |
| Nacos | 2.x | 2.4+ | gRPC通信稳定、集群Raft协议优化 |
| Sentinel | 1.8 | 1.8+ | 版本变化不大,注意与Spring Boot 3.x的兼容性 |
Java 21正式引入虚拟线程,这是Java并发编程的重大变革。它对高并发架构有深远影响。
import asyncio
import aiohttp
import time
# Python的asyncio和Java虚拟线程解决同一个问题:
# 大量IO密集型任务时,传统线程太重
# 1. 基本用法:async/await
async def fetch_url(session, url):
async with session.get(url) as resp:
return await resp.text()
# 2. 并发执行10万个IO任务(类似Java虚拟线程池)
async def main():
urls = [f"https://httpbin.org/get?id={i}" for i in range(1000)]
# 用Semaphore控制并发数(避免打爆目标服务)
semaphore = asyncio.Semaphore(100)
async def limited_fetch(session, url):
async with semaphore:
return await fetch_url(session, url)
async with aiohttp.ClientSession() as session:
# asyncio.gather并发执行所有任务
# 每个任务是一个协程(类似虚拟线程,内存开销极小)
tasks = [limited_fetch(session, url) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
print(f"完成 {len(results)} 个请求")
# 3. 运行
asyncio.run(main())
# 4. Python vs Java 对比
# Python asyncio = 协程(用户态调度,IO时自动切换)
# Java Virtual Thread = 虚拟线程(JVM调度,IO时自动卸载)
# 两者解决同一个问题:大量IO任务时,线程太重
# Python更成熟(asyncio从3.4就有),Java虚拟线程是21才引入
# 核心思想一样:IO等待时不占操作系统线程,让其他任务执行