Actor 指标

汇总

监控指标

pg_loss

• IS权重*优势+PPO-Clip+PPO-Dual-Clip

PPO-Clip

$${\mathcal{L}}_{\text{DAPO}}(\theta )=-\underset{\text{所有Token直接做平均}}{\underbrace{\frac{1}{\sum _{i=1}^G|o_i|}\sum _{i=1}^G\sum _{t=1}^{|o_t|}}}min(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})}\cdot {\hat{A}}_{i,t},\text{clip}(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})},1-{ϵ}_{\text{low}},1+{ϵ}_{\text{high}})\cdot {\hat{A}}_{i,t})$$

PPO-Dual-Clip

• 对 优势为负 + IS权重大 的 token 做 额外clip，
• 该tokenloss，最大不超过优势绝对值*clip_ratio_c(3)

$$L_t^{\text{Dual-Clip}}=\{\begin{array}{lll}& -min(r_t{A}_t,\text{clip}(r_t,1-ϵ,1+ϵ)A_t)& A_t\ge 0\\ & min(-r_t{A}_t,-c{A}_t)& A_t<0\end{array}$$

ppo_kl

• 采样策略${\pi }_{old}$和学习策略${\pi }_{\theta }$之间的KL距离。$$D_{KL}({\pi }_{old}||{\pi }_{\theta })=E_{x\sim {\pi }_{old}}[\log \frac{{\pi }_{old}(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}]$$

pg_clipfrac

• IS权重 被PPO-Clip区间 $[1-{ϵ}_{\text{low}},1+{ϵ}_{\text{high}}]$ 裁剪的比例

pg_clipfrac_lower

• PPO-Clip后，被PPO-Dual-Clip针对负优势，裁剪的比例。$$L_t^{\text{Dual-Clip}}=\{\begin{array}{lll}& -min(r_t{A}_t,\text{clip}(r_t,1-ϵ,1+ϵ)A_t)& A_t\ge 0\\ & min(-r_t{A}_t,-c{A}_t)& A_t<0\end{array}$$

PG Loss 相关

总 Policy Loss

参考笔记

理论笔记

• PPO-Actor 笔记 ，KL 笔记

• PPO Policy Loss 数学公式笔记

代码笔记

• Verl Policy Loss 代码实现笔记

Policy Loss 核心计算逻辑

PG Loss

• 优势*重要性权重 结合 PPO-Clip，具体见下文
• Policy loss = PG损失 - 熵奖励 + KL 惩罚
• Policy_loss = pg_loss - entropy_coeff * entropy_loss + kl_loss_coef * kl_loss

熵奖励

• Policy Loss = Policy Loss - entropy_coeff * entropy_loss

KL惩罚${\pi }_{\theta }$和${\pi }_{ref}$

• Policy Loss = Policy Loss + kl_loss_coef * kl_loss

Policy Loss 具体计算过程

数据读取

• 从output读取当前log_probs和entropy
• 从data读取old_log_probs、advantages
• 从config里读取相关参数，entropy_coeff, kl_loss_coef, loss_agg_mode等。

计算 PGLoss

• 根据config获得policy_loss_fn，默认是compute_policy_loss_vanilla
• 入参：old_log_prob、log_prob、advantages、response_mask、loss_agg_mode等
• 出参：pg_loss, pg_metrics
• 具体见下文。

计算 熵奖励

• 根据loss_agg_mode，去计算response位置的熵loss
• 根据entropy_coeff 熵系数，总loss- 熵loss，因为探索需要奖励，所以是减去熵loss

计算 KL惩罚

• 根据loss_agg_mode, log_prob和ref_log_prob，去计算response位置的KLloss
• 根据kl_loss_coef，总loss + klloss。因为偏离需要惩罚，所以是加上KLLoss。

PG loss

Verl PGLoss 核心公式

Verl PG Loss

• 优势*权重+PPO-Clip+PPO-Dual-Clip后的最终pg_loss。

Seq-Level PG Loss

$${\mathcal{L}}_{\text{ppo}}(\theta )=-\underset{\mathrm{样}\mathrm{本}\mathrm{间}\mathrm{平}\mathrm{均}}{\underbrace{\frac{1}{G}\sum _{i=1}^G}}\underset{\text{序列内平均}}{\underbrace{\frac{1}{|o_i|}\sum _{t=1}^{|o_t|}}}min(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})}\cdot {\hat{A}}_{i,t},\text{clip}(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})},1-{ϵ}_{\text{low}},1+{ϵ}_{\text{high}})\cdot {\hat{A}}_{i,t})$$

Token-Level PG Loss

$${\mathcal{L}}_{\text{DAPO}}(\theta )=-\underset{\text{所有Token直接做平均}}{\underbrace{\frac{1}{\sum _{i=1}^G|o_i|}\sum _{i=1}^G\sum _{t=1}^{|o_t|}}}min(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})}\cdot {\hat{A}}_{i,t},\text{clip}(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})},1-{ϵ}_{\text{low}},1+{ϵ}_{\text{high}})\cdot {\hat{A}}_{i,t})$$

Dual-Clip Loss

• 在优势$A_t<0$时，额外增加一个裁剪
• 负优势+IS权重偏差很大，限制惩罚力度

$$L_t^{\text{Dual-Clip}}=\{\begin{array}{lll}& -min(r_t{A}_t,\text{clip}(r_t,1-ϵ,1+ϵ)A_t)& A_t\ge 0\\ & min(-r_t{A}_t,-c{A}_t)& A_t<0\end{array}$$

@register_policy_loss("vanilla")  # type: ignore[arg-type]
def compute_policy_loss_vanilla(
    old_log_prob: torch.Tensor,
    log_prob: torch.Tensor,
    advantages: torch.Tensor,
    response_mask: torch.Tensor,
    loss_agg_mode: str = "token-mean",
    config: Optional[DictConfig | AlgoConfig] = None,
    rollout_is_weights: torch.Tensor | None = None,
) -> tuple[torch.Tensor, dict[str, Any]]:

    assert config is not None
    assert not isinstance(config, AlgoConfig)
    clip_ratio = config.clip_ratio  # Clipping parameter ε for standard PPO. See https://arxiv.org/abs/1707.06347.
    clip_ratio_low = config.clip_ratio_low if config.clip_ratio_low is not None else clip_ratio
    clip_ratio_high = config.clip_ratio_high if config.clip_ratio_high is not None else clip_ratio
    clip_ratio_c = config.get(  # Lower bound of the ratio for dual-clip PPO. See https://arxiv.org/pdf/1912.09729.
        "clip_ratio_c", 3.0
    )

    cliprange = clip_ratio
    cliprange_low = clip_ratio_low
    cliprange_high = clip_ratio_high

    assert clip_ratio_c > 1.0, (
        "The lower bound of the clip_ratio_c for dual-clip PPO should be greater than 1.0,"
        + f" but get the value: {clip_ratio_c}."
    )
    negative_approx_kl = log_prob - old_log_prob
    # KL做CLIP
    negative_approx_kl = torch.clamp(negative_approx_kl, min=-20.0, max=20.0)
    # IS 重要性权重
    ratio = torch.exp(negative_approx_kl) 
    # old_logprob-logprob，old-采样，logprob-学习，PPO_KL近似KL散度,监控指标
    ppo_kl = verl_F.masked_mean(-negative_approx_kl, response_mask)
    # 朴素无clip的pg_loss，-优势*重要性权重
    pg_losses1 = -advantages * ratio  
    if cliprange_low is None:
        cliprange_low = cliprange
    if cliprange_high is None:
        cliprange_high = cliprange
    # ppo-clip 第二项，对ratio做clip，乘以advantages
    pg_losses2 = -advantages * torch.clamp(ratio, 1 - cliprange_low, 1 + cliprange_high)  # - clip(ratio, 1-cliprange, 1+cliprange) * A  #
    # max(-ratio * A, -clip(ratio, 1-cliprange, 1+cliprange) * A)
    # ppo clip loss
    clip_pg_losses1 = torch.maximum(pg_losses1, pg_losses2)    
    # ppo clip 比例，有多少loss2 > loss1的
    pg_clipfrac = verl_F.masked_mean(torch.gt(pg_losses2, pg_losses1).float(), response_mask)
    
    # dual ppo clip，在优势小于0时，额外增加clip
    pg_losses3 = -advantages * clip_ratio_c 
    clip_pg_losses2 = torch.min(pg_losses3, clip_pg_losses1) 
    # dual ppo clip 比例
    pg_clipfrac_lower = verl_F.masked_mean(
        torch.gt(clip_pg_losses1, pg_losses3) * (advantages < 0).float(), response_mask
    )
    # dual ppo clip，在优势小于0时，额外增加一个裁剪
    pg_losses = torch.where(advantages < 0, clip_pg_losses2, clip_pg_losses1)  

    # Apply rollout correction weights if provided
    if rollout_is_weights is not None:
        pg_losses = pg_losses * rollout_is_weights

    pg_loss = agg_loss(loss_mat=pg_losses, loss_mask=response_mask, loss_agg_mode=loss_agg_mode)  

    pg_metrics = {
        "actor/pg_clipfrac": pg_clipfrac.detach().item(),
        "actor/ppo_kl": ppo_kl.detach().item(),
        "actor/pg_clipfrac_lower": pg_clipfrac_lower.detach().item(),
    }
    return pg_loss, pg_metrics

pg_clipfrac

pg_clipfrac

pg_clipfrac

• IS权重 被PPO-Clip区间 $[1-{ϵ}_{\text{low}},1+{ϵ}_{\text{high}}]$ 裁剪的比例

$${\mathcal{L}}_{\text{DAPO}}(\theta )=-\underset{\text{所有Token直接做平均}}{\underbrace{\frac{1}{\sum _{i=1}^G|o_i|}\sum _{i=1}^G\sum _{t=1}^{|o_t|}}}min(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})}\cdot {\hat{A}}_{i,t},\text{clip}(\frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{{\theta }_{old}}(o_{i,t}\mid q,o_{i,<t})},1-{ϵ}_{\text{low}},1+{ϵ}_{\text{high}})\cdot {\hat{A}}_{i,t})$$

@register_policy_loss("vanilla")  # type: ignore[arg-type]
def compute_policy_loss_vanilla():
  	# ...
    # ...
  	# ppo-clip 第2项，对ratio做clip，乘以advantages
    pg_losses2 = -advantages * torch.clamp(ratio, 1 - cliprange_low, 1 + cliprange_high)  		# - clip(ratio, 1-cliprange, 1+cliprange) * A 
    # max(-ratio * A, -clip(ratio, 1-cliprange, 1+cliprange) * A)
    # ppo clip loss
    clip_pg_losses1 = torch.maximum(pg_losses1, pg_losses2)    
    # ppo clip 比例，有多少loss2 > loss1的
    pg_clipfrac = verl_F.masked_mean(torch.gt(pg_losses2, pg_losses1).float(), response_mask)
    # ....

pg_clipfrac_lower

pg_clipfrac_lower

pg_clipfrac_lower

• PPO-Clip后，被PPO-Dual-Clip针对负优势，裁剪的比例。

$$L_t^{\text{Dual-Clip}}=\{\begin{array}{lll}& {\text{L}}_{\mathrm{t}}^{\mathrm{P}\mathrm{P}\mathrm{O}-\mathrm{s}\mathrm{t}\mathrm{d}}& A_t\ge 0\\ & min({\text{L}}_{\mathrm{t}}^{\mathrm{P}\mathrm{P}\mathrm{O}-\mathrm{s}\mathrm{t}\mathrm{d}},-c\cdot A_t)& A_t<0\end{array}$$

@register_policy_loss("vanilla")  # type: ignore[arg-type]
def compute_policy_loss_vanilla():
  	# ...
    # ...
  	# dual ppo clip，在优势小于0时，额外增加clip
    pg_losses3 = -advantages * clip_ratio_c 
    clip_pg_losses2 = torch.min(pg_losses3, clip_pg_losses1) 
    # dual ppo clip 比例
    pg_clipfrac_lower = verl_F.masked_mean(
        torch.gt(clip_pg_losses1, pg_losses3) * (advantages < 0).float(), response_mask
    )  
    # ....

KL 相关

PPO_KL

PPO_KL 监测指标

ppo_kl

• 采样策略${\pi }_{old}$和学习策略${\pi }_{\theta }$之间的KL距离。$$D_{KL}({\pi }_{old}||{\pi }_{\theta })=E_{x\sim {\pi }_{old}}[\log \frac{{\pi }_{old}(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}]$$

@register_policy_loss("vanilla")  # type: ignore[arg-type]
def compute_policy_loss_vanilla():
  	# ...
    # ...
  	negative_approx_kl = log_prob - old_log_prob
    # KL做CLIP
    negative_approx_kl = torch.clamp(negative_approx_kl, min=-20.0, max=20.0)
    # IS 重要性权重
    ratio = torch.exp(negative_approx_kl)
    # old_logprob-logprob，old-采样，logprob-学习，PPO_KL近似KL散度,监控指标
    ppo_kl = verl_F.masked_mean(-negative_approx_kl, response_mask)  
    # ....

KL Loss

KL Loss 惩罚

KL 笔记

• KLLoss 理论与实现笔记

KL惩罚 (${\pi }_{\theta }$ vs ${\pi }_{ref}$)

• Policy Loss = Policy Loss + kl_loss_coef * kl_loss

$${\text{KL}}_{\text{loss}}=-\beta \cdot D_{KL}({\pi }_{\theta },{\pi }_{{\theta }_{ref}})$$

K1

• K1笔记

$${D_{KL}}_t({\pi }_{\theta },{\pi }_{{\theta }_{ref}})=\log \frac{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{ref}(o_{i,t}\mid q,o_{i,<t})}$$

K3

• K3笔记

$${D_{KL}}_t({\pi }_{\theta },{\pi }_{{\theta }_{ref}})=\frac{{\pi }_{ref}(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}-\log \frac{{\pi }_{ref}(o_{i,t}\mid q,o_{i,<t})}{{\pi }_{\theta }(o_{i,t}\mid q,o_{i,<t})}-1$$

if self.config.use_kl_loss:
    ref_log_prob = data["ref_log_prob"]
    # compute kl loss
    kld = kl_penalty(logprob=log_prob, ref_logprob=ref_log_prob, kl_penalty=self.config.kl_loss_type) 
    kl_loss = agg_loss(loss_mat=kld, loss_mask=response_mask, loss_agg_mode=self.config.loss_agg_mode) 
		# 核心：将KL损失（乘以一个系数）加到总损失上
    policy_loss = policy_loss + kl_loss * self.config.kl_loss_coef 

def kl_penalty(logprob: torch.FloatTensor, ref_logprob: torch.FloatTensor, kl_penalty) -> torch.FloatTensor:
    """Compute KL divergence given logprob and ref_logprob. Optionally using straight through to bind k2 on other
    kl penalty compute method for unbiased KL gradient estimation.
    See more description in http://joschu.net/blog/kl-approx.html

    Args:
        logprob:
        ref_logprob:

    Returns:
        kl_estimate
    """
    forward_score = kl_penalty_forward(logprob, ref_logprob, kl_penalty) 
    if not kl_penalty.endswith("+") or kl_penalty in ("mse", "k2"):
        return forward_score

    """
    The expectation of k1 and k3 estimator is the expected value of KL, but the expected gradient of k1 and k3
    estimator is not the expected gradient of KL. On the other hand k2 estimator gives right gradient estimator, 
    so we use a straight through trick here if the kl_penalty method ends with '+', e.g., k3+. 
    """
    backward_score = 0.5 * (logprob - ref_logprob).square()

    return backward_score - backward_score.detach() + forward_score.detach()

def kl_penalty_forward(logprob: torch.FloatTensor, ref_logprob: torch.FloatTensor, kl_penalty) -> torch.FloatTensor:
    """Compute KL divergence given logprob and ref_logprob.
    Copied from https://github.com/huggingface/trl/blob/main/trl/trainer/ppo_trainer.py#L1104
    See more description in http://joschu.net/blog/kl-approx.html

    Args:
        logprob:
        ref_logprob:

    Returns:
        kl_estimate
    """
    if kl_penalty in ("kl", "k1"):
        return logprob - ref_logprob  

    if kl_penalty == "abs":
        return (logprob - ref_logprob).abs()

    if kl_penalty in ("mse", "k2"):
        return 0.5 * (logprob - ref_logprob).square()  

    # J. Schulman. Approximating kl divergence, 2020.
    # # URL http://joschu.net/blog/kl-approx.html.
    if kl_penalty in ("low_var_kl", "k3"):
        kl = ref_logprob - logprob 
        # For numerical stability
        kl = torch.clamp(kl, min=-20, max=20)
        ratio = torch.exp(kl)
        kld = (ratio - kl - 1).contiguous()
        return torch.clamp(kld, min=-10, max=10)

    if kl_penalty == "full":
        # so, here logprob and ref_logprob should contain the logits for every token in vocabulary
        raise NotImplementedError

    raise NotImplementedError

熵

计算熵

数学公式

• 熵计算公式推导和实现(Verl+FSDP) 笔记

• 定义公式

  $$H(X)=-\sum _{i=1}^{n}p(x_i)\cdot \text{log}(p(x_i))$$

• 推导结果 (推导过程详见笔记)

  $$H(X)=-\sum _{i=1}p(x_i)z_{x_i}+\log (\sum _j{e}^{z_{x_j}})$$

代码实现-VocabParallelEntropy 根据logits计算熵

• 计算exp_logits：logits求指数
• 计算sum_exp_logits：exp_logits 求和，用于计算概率
• 计算softmax_logits：每个logit的概率，exp_logits/sum_exp_logits
• 计算sum_softmax_times_logits：logits概率 * logits
• 计算最终entropy：log logits求和 - logits概率 * logits

class _VocabParallelEntropy(torch.autograd.Function):
  def forward(ctx, vocab_parallel_logits: torch.Tensor) -> torch.Tensor:
      def mul_reduce(a, b):
          return (a * b).sum(dim=-1, keepdim=True)
      # 稳定性操作，避免溢出，减去最大值
      logits_max = vocab_parallel_logits.max(dim=-1, keepdim=True).values
      normalized_vocab_parallel_logits = vocab_parallel_logits - logits_max
      # exp_logits
      normalized_exp_logits = normalized_vocab_parallel_logits.exp_() 
      # sum_exp_logits
      normalized_sum_exp_logits = normalized_exp_logits.sum(dim=-1, keepdim=True) 
      # 计算每个logit概率
      softmax_logits = normalized_exp_logits.div_(normalized_sum_exp_logits) 
      # p_i * logits_i
      sum_softmax_times_logits = mul_reduce(softmax_logits, vocab_parallel_logits)  
      # 最终的熵，log_sum_exp_logits - sum_softmax_times_logits + logits_max
      entropy = logits_max + normalized_sum_exp_logits.log() - sum_softmax_times_logits 
      ctx.save_for_backward(vocab_parallel_logits, softmax_logits, sum_softmax_times_logits)
      return entropy.squeeze(dim=-1)

  def backward(ctx, grad_output: torch.Tensor) -> torch.Tensor:
      vocab_parallel_logits, softmax_logits, sum_softmax_times_logits = ctx.saved_tensors
      # reuse softmax_logits as grad
      vocab_parallel_logits.sub_(sum_softmax_times_logits)
      softmax_logits.mul_(vocab_parallel_logits)
      softmax_logits.mul_(grad_output.unsqueeze(dim=-1))
      # recover vocab_parallel_logits
      vocab_parallel_logits.add_(sum_softmax_times_logits)
      softmax_logits.mul_(-1)
      return softmax_logits

grad_norm

GradNorm 计算

计算全局梯度的L2范数

• L2范数：平方之和再开方
• total_norm = sqrt(sum(p.grad.data.norm(2)^2 for p in parameters))

$$\text{grad\_norm}=||g|{|}_2=(\sum _i|g_i{|}^2{)}^{\frac{1}{2}}$$

计算裁剪系数

• clip_coef = max_norm / total_norm
• 如果 clip_coef < 1，说明梯度过大，需要裁剪。
• 如果 clip_coef >=1，梯度在合理区间，不做操作。

根据裁剪系数裁剪梯度

• 乘以裁剪系数，所有梯度分量按比例缩放
• for p in parameters: p.grad.data.mul_(clip_coef)

max_norm的作用

• N维向量/移动距离，无论山坡多陡，要求一步更新最多只能是学习率*max_norm

max_norm/grad_norm 为了整体稳定性

• 计算裁剪系数，是不除以参数数量的。
  • 系数变小，单参数更新确实变小，但众人拾材火焰高，整体模型更新并不小。
  • 参数越大的模型，单个参数更新不应该单次更新太大。
• 限制的是模型全局移动距离，而不是单个参数的步长。

verl 配置 max_norm

# max_norm
actor_rollout_ref.actor.optim.clip_grad=1
critic.optim.clip_grad=1

计算total_grad_norm、裁剪系数，进行裁剪

total_norm = sqrt(sum(p.grad.data.norm(2)^2 for p in parameters)) 
clip_coef = max_norm / (total_norm + 1e-6)
if clip_coef < 1:
  for p in parameters:
    p.grad.data.mul_(clip_coef)  

示例

假设 max_norm = 1.0，模型只有两个参数，原始梯度为
p1.grad = [3.0, 4.0]    # L2 norm = 5                   
p2.grad = [0.0, 0.0]    # L2 norm = 0                                                 

Step 1: 计算总范数                                                                       
total_norm = sqrt(3^2 + 4^2 + 0^2 + 0^2) = sqrt(25) = 5.0                                                                             
Step 2: 计算裁剪系数
clip_coef = 1.0 / 5.0 = 0.2

Step 3: 所有梯度分量统一乘以 0.2
p1.grad = [3.0 * 0.2, 4.0 * 0.2] = [0.6, 0.8]                               
p2.grad = [0.0 * 0.2, 0.0 * 0.2] = [0.0, 0.0]                                                                                  
new_total_norm = sqrt(0.6^2 + 0.8^2) = 1.0

lr

loss_func 函数代码

def loss_func(output, data, meta_info):
    # For memory efficiency
    # We move calculation of entropy to compute_log_probs, forward_only == True
    log_probs = None
    entropy = None
    if isinstance(output, dict):
        log_probs = output["log_probs"] 
        if "entropy" in output:
            entropy = output["entropy"] 
    else:
        assert isinstance(output, torch.Tensor)
        log_probs = output

    device = log_probs.device
    metrics = {}  
    if forward_only:
        if post_process_fn is None:
            pass
            # metrics["logits"] = output
        else:
            stats = post_process_fn(output, data)
            metrics.update(stats)
        if not calculate_entropy:
            return torch.tensor(1.0, device=device), metrics

    responses = data["responses"]
    response_length = responses.size(1)
    response_mask = data["response_mask"].to(bool)
    loss_agg_mode = self.config.loss_agg_mode
    # compute policy loss
    log_prob = log_probs[:, -response_length - 1 : -1].contiguous() 
    ret_entropy = None
    stats = {}
    if not forward_only:
        old_log_prob = data["old_log_probs"] 
        advantages = data["advantages"] 

        entropy_coeff = self.config.entropy_coeff
        loss_agg_mode = self.config.loss_agg_mode
        # 调用core_algos.py里的policy_loss_fn，具体见下文
        loss_mode = self.config.policy_loss.get("loss_mode", "vanilla") 
        policy_loss_fn = get_policy_loss_fn(loss_mode) 

        # Extract pre-computed rollout correction weights if present
        # Weights are computed centrally in trainer and added when algorithm.rollout_is=True
        rollout_is_weights = data.get("rollout_is_weights", None)
        pg_loss, pg_metrics = policy_loss_fn( 
            old_log_prob=old_log_prob, 
            log_prob=log_prob, 
            advantages=advantages, 
            response_mask=response_mask, 
            loss_agg_mode=loss_agg_mode, 
            config=self.config, 
            rollout_is_weights=rollout_is_weights, 
        ) 
        stats.update(pg_metrics)

        # Skip if using pure rollout correction mode (metrics already in pg_metrics)
        rollout_log_prob = data.get("rollout_log_probs", None)
        if loss_mode != "rollout_correction" and rollout_log_prob is not None:
            # Compute metrics using CURRENT policy π_θ vs π_rollout
            # Tracks evolving off-policy gap as π_θ updates during mini-batch training
            from verl.trainer.ppo.rollout_corr_helper import compute_rollout_corr_metrics_from_logprobs

            rollout_corr_metrics = compute_rollout_corr_metrics_from_logprobs(
                log_prob=log_prob,
                rollout_log_prob=rollout_log_prob,
                response_mask=response_mask,
            )
            stats.update(rollout_corr_metrics)

        stats["actor/pg_loss"] = pg_loss.detach().item()
        policy_loss = pg_loss

    if calculate_entropy:
      	# 熵奖励
        entropy = output["entropy"][:, -response_length - 1 : -1].contiguous() 
        if not forward_only: 
            entropy_loss = agg_loss(loss_mat=entropy, loss_mask=response_mask, loss_agg_mode=loss_agg_mode) 
            entropy_coeff = meta_info["entropy_coeff"] 
            policy_loss = pg_loss - entropy_coeff * entropy_loss 
        else:
            ret_entropy = entropy

    if forward_only:
        policy_loss = torch.tensor(1.0, device=device)
    else:
        if self.config.use_kl_loss: 
          	# KL 惩罚
            ref_log_prob = data["ref_log_prob"] 
            # compute kl loss
            kld = kl_penalty(logprob=log_prob, ref_logprob=ref_log_prob, kl_penalty=self.config.kl_loss_type) 
            kl_loss = agg_loss(loss_mat=kld, loss_mask=response_mask, loss_agg_mode=self.config.loss_agg_mode) 

            policy_loss = policy_loss + kl_loss * self.config.kl_loss_coef 
            metrics["actor/kl_loss"] = kl_loss.detach().item() 
            metrics["actor/kl_coef"] = self.config.kl_loss_coef 

        # return loss and stats

    append_to_dict(metrics, stats)
    return policy_loss, [metrics, ret_entropy]

Critic 指标

训练Reward

需要稳定上涨

训练Advantages

小心

相关笔记

• GRPO 优势 理论笔记
• PPO Critic、PPO Reward Model

GRPO 组相对优势

优势计算

分组采样

• 对每个query，采样1组输出，RewardModel 为每个输出打分，给出奖励。

$$q\to \mathbf{o}=\{o_1,o_2,\cdots ,o_G\}\to \mathbf{r}=\{r_1,r_2,\cdots ,r_G\}$$

• 组内基线计算：组内平均奖励/奖励标准差

$$\text{mean}(\mathbf{r}),\text{std}(\mathbf{r})$$

相对组优势计算

• 为每个输出$o_i$，计算组内的相对得分，作为组内的相对优势

$${\hat{r}}_i=\frac{r_i-\text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}\to {\hat{A}}_i={\hat{r}}_i$$

• ${\hat{A}}_{i,t}$ 组内相对优势
  • 不同于PPO仅最后时刻token有奖励信号。
  • 所有时刻的token $o_{t\le T}$，都使用同一个 组内相对得分，作为t时刻优势信号

$${\hat{A}}_{i,t}={\hat{A}}_i={\hat{r}}_i=\frac{r_i-\text{mean}(\mathbf{r})}{\text{std}(\mathbf{r})}$$

# NOTE(sgm): this implementation only consider outcome supervision, where the reward is a scalar.
@register_adv_est(AdvantageEstimator.GRPO)  # or simply: @register_adv_est("grpo")
def compute_grpo_outcome_advantage(
    token_level_rewards: torch.Tensor,
    response_mask: torch.Tensor,
    index: np.ndarray,
    epsilon: float = 1e-6,
    norm_adv_by_std_in_grpo: bool = True,
    config: Optional[AlgoConfig] = None,
) -> tuple[torch.Tensor, torch.Tensor]:
    """
    Compute advantage for GRPO, operating only on Outcome reward
    (with only one scalar reward for each response).

    Args:
        token_level_rewards: `(torch.Tensor)`
            shape is (bs, response_length)
        response_mask: `(torch.Tensor)`
            shape is (bs, response_length)
        index: `(np.ndarray)`
            index array for grouping
        epsilon: `(float)`
            small value to avoid division by zero
        norm_adv_by_std_in_grpo: `(bool)`
            whether to scale the GRPO advantage
        config: `(Optional[AlgoConfig])`
            algorithm configuration object

    Note:
        If norm_adv_by_std_in_grpo is True, the advantage is scaled by the std, as in the original GRPO.
        If False, the advantage is not scaled, as in Dr.GRPO (https://arxiv.org/abs/2503.20783).

    Returns:
        advantages: `(torch.Tensor)`
            shape is (bs, response_length)
        Returns: `(torch.Tensor)`
            shape is (bs, response_length)
    """
    scores = token_level_rewards.sum(dim=-1)

    id2score = defaultdict(list)
    id2mean = {}
    id2std = {}

    with torch.no_grad():
        bsz = scores.shape[0]
        for i in range(bsz):
            id2score[index[i]].append(scores[i])
        for idx in id2score:
            if len(id2score[idx]) == 1:
                id2mean[idx] = torch.tensor(0.0)
                id2std[idx] = torch.tensor(1.0)
            elif len(id2score[idx]) > 1:
                scores_tensor = torch.stack(id2score[idx])
                id2mean[idx] = torch.mean(scores_tensor)
                id2std[idx] = torch.std(scores_tensor)
            else:
                raise ValueError(f"no score in prompt index: {idx}")
        for i in range(bsz):
            if norm_adv_by_std_in_grpo:
                scores[i] = (scores[i] - id2mean[index[i]]) / (id2std[index[i]] + epsilon)
            else:
                scores[i] = scores[i] - id2mean[index[i]]
        scores = scores.unsqueeze(-1) * response_mask

    return scores, scores

训练Returns

长度轮数指标

response_length/clip_ratio

提示

response_length

• 回复长度

response_length/clip_ratio

• 有多少比例达到max_response_length
• 如果值过高，说明max_response_length 可能设的过短

response_length_non_aborted

• 排除掉response为0的样本，即fake_data，response_mask全为0
• 即全部有效样本

num_turns

测试集评测指标

val@mean@n

val@best@n