Actor 指标

总Policy Loss

参考笔记

• PPO Policy Loss 数学公式笔记
• Verl Policy Loss 代码实现笔记

核心计算逻辑

• Policy loss = PG损失 - 熵奖励 + KL 惩罚
• Policy_loss = pg_loss - entropy_coeff * entropy_loss + kl_loss_coef * kl_loss

策略梯度loss (pg_loss)

参考笔记

• PPO PG loss 数学公式笔记
• Verl PG loss 代码实现笔记

PGLoss 核心计算逻辑

Seq-Level PG Loss

$${\mathcal{L}}_{\text{ppo}}(\theta )=-\underset{\mathrm{样}\mathrm{本}\mathrm{间}\mathrm{平}\mathrm{均}}{\underbrace{\frac{1}{G}\sum _{i=1}^G}}\underset{\text{序列内平均}}{\underbrace{\frac{1}{|o_i|}\sum _{t=1}^{|o_t|}}}min(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})}\cdot {\hat{A}}_{i,t},\text{clip}(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})},1-ϵ,1+ϵ)\cdot {\hat{A}}_{i,t})$$

Token-Level PG Loss

$${\mathcal{L}}_{\text{DAPO}}(\theta )=-\underset{\text{所有Token直接做平均}}{\underbrace{\frac{1}{\sum _{i=1}^G|o_i|}\sum _{i=1}^G\sum _{t=1}^{|o_t|}}}min(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})}\cdot {\hat{A}}_{i,t},\text{clip}(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})},1-ϵ,1+ϵ)\cdot {\hat{A}}_{i,t})$$