Qwen3.5 RL 环境踩坑
📅 发表于 2026/04/01
🔄 更新于 2026/08/05
👁️ — 次访问
📝 1752 字
⏳ 8 分钟
qwen3.5
#pytorch
#flash-attention
#transformer-engine
#megatron-lm
#sglang
#mbridge
#verl
#GDN
#PackedSequence
#CP
基础环境升级
Pytorch
python
# vllm 需要2.10.0
pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 --index-url https://download.pytorch.org/whl/cu128
# sglang 需要2.9.1
pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu128
# 升级：最新sglang 需要torch 2.11
# 如果锁住版本，请修改 /etc/pip/constraint.txt
pip install torch==2.11.0 torchvision==0.26.0 torchaudio==2.11.0 --index-url https://download.pytorch.org/whl/cu128
pip install --force-reinstall torch==2.11.0 torchvision==0.26.0 torchaudio==2.11.0 --index-url https://download.pytorch.org/whl/cu128
pip install --force-reinstall torch==2.11.0+cu128 torchvision==0.26.0+cu128 torchaudio==2.11.0+cu128 --index-url https://download.pytorch.org/whl/cu128必须测试有效性
bash
python -c "import torch, torchvision; print(torch.__version__, torch.version.cuda, torchvision.__version__)"其他
bash
# torchao
python -m pip install --pre --no-deps torchao \
--index-url https://download.pytorch.org/whl/nightly/cu128
python - <<'PY'
import torch, torchao
print(torch.__version__, torch.version.cuda, torch.cuda.is_available())
print(torchao.__version__)
PY部分前置安装依赖
bash
# 部分前置依赖
pip install packaging
pip install ninja
pip install pybind11FlashAttention
笔记
- 查看版本：flash_attn/init.py
- 需要先确定torch版本
- 注意和transformer_engine的兼容性，目前仅支持到
2.8.3
bash
# 下载
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
# check版本 我是安装的2.8.3
git checkout v2.8.3
# 拉取依赖
git submodule update --init --recursive
# 从源码编译，
MAX_JOBS=8 python setup.py install
# flash attention 3安装
cd hopper
MAX_JOBS=8 python setup.py install
# 完整四条
python_path=python -c "import site; print(site.getsitepackages()[0])"
mkdir -p $python_path/flash_attn_3
cp flash_attn_interface.py $python_path/flash_attn_3/
# 验证
python3 -c "from flash_attn_3.flash_attn_interface import flash_attn_func; print('ok')"
python3 -c "from verl.workers.engine.megatron import MegatronEngine; print('ok')"
# 看版本
vim flash-attention/flash_attn/__init__.py
# 测试
cd flash-attention
pytest -q -s tests/test_flash_attn.pyTransformerEngine
TransformerEngine
bash
git clone https://github.com/NVIDIA/TransformerEngine.git
cd /root/app/TransformerEngine
# 安装对应版本 release_v2.12
git fetch origin
git checkout release_v2.12
git checkout release_v2.14
git checkout release_v2.16
git submodule update --init --recursive
# 编译
NVTE_FRAMEWORK=pytorch pip install . --no-build-isolation
# 如果重新编译，可以删除之前的一些内容
# 删除旧的构建目录
rm -rf build
rm -rf transformer_engine/common/build/
rm -rf transformer_engine.egg-info
rm -f /root/app/TransformerEngine/transformer_engine_torch.cpython-312-x86_64-linux-gnu.so
rm -f /root/app/TransformerEngine/transformer_engine/transformer_engine_torch.cpython-312-x86_64-linux-gnu.so
# 测试
python -c "import transformer_engine; print(transformer_engine.__version__)"
python -c "import transformer_engine.pytorch; print('Transformer Engine PyTorch backend imported successfully!')"Megatron-LM
bash
# 拉取源码，升级到dev版本
git clone https://github.com/NVIDIA/Megatron-LM.git
# git fetch --all --tags
git fetch origin
# 0.16.0 不支持Qwen3.5-GDN CP
# git checkout core_r0.16.0
git checkout core_r0.18.0
pip install -e .SGLang
Sglang
- sglang 依赖版本：注意
cuda-python等版本，如果driver是535等，仅支持12.x，则不支持13.x，会失败的。 - 新sglang
需要安装 rust：使用阿里云镜像加速Rust与Cargo安装及更新 - 最终还是使用0.5.9版本，0.5.11后面的，sglang官方都使用cuda13.x，dsw机器不支持。
sglang 安装
bash
# 0.5.9 pytorch2.9.1
git clone https://github.com/sgl-project/sglang.git
# 0.5.14 pytorch 2.11.0
cd sglang
git checkout v0.5.14
pip install -e "python"先安装和配置rust/cargo
bash
# 临时设置环境变量以替换默认更新源和分发服务器
export RUSTUP_UPDATE_ROOT=https://mirrors.aliyun.com/rustup/rustup
export RUSTUP_DIST_SERVER=https://mirrors.aliyun.com/rustup
# 安装脚本
curl --proto '=https' --tlsv1.2 -sSf https://mirrors.aliyun.com/repo/rust/rustup-init.sh | sh
配置文件：~/.cargo/config.toml，注意不要使用阿里云文档里的 [registry]，已经废弃了。
[source.crates-io]
replace-with = 'aliyun'
[source.aliyun]
registry = "sparse+https://mirrors.aliyun.com/crates.io-index/"mbridge
bash
# 需要从git库安装，pip安装的代码比较旧。
git clone https://github.com/ISEEKYAN/mbridge.git
pip uninstall mbridge && pip install --no-deps /path/mbridgeVerl
bash
git clone https://github.com/verl-project/verl.git
cd verl
# 特定版本
git checkout release/v0.8.0
git submodule update --init --recursive recipe
pip install -e .
# 新版本需要安装，--no-deps 只安装这个包，不知道安装或升级依赖包
pip install TransferQueue==0.1.6 --no-deps后置依赖
bash
# 最后再次更新transformers，支持Qwen3.5
pip install transformers==5.3.0
pip install flash-linear-attention
# 避免报错：
pip install peft==0.15.0megatron-mbridge
由于升级了Megatron-LM==0.18.0，之前旧的mbridge已经不支持，经常会出现模型加载不进来、特别慢最终崩溃掉等问题。经排查，需要使用新版megatron-mbridge
bash
# 可使用的版本
pip install --no-deps \
megatron-bridge==0.4.2 \
nvidia-modelopt==0.42.0 \
pulp==3.3.2
# 评测命令
python -c "from megatron.bridge import AutoBridge; print('Megatron-Bridge OK', AutoBridge)"
python -c "from verl.models.mcore.bridge import AutoBridge; print('verl Bridge OK', AutoBridge)"
# 训练脚本替换，不再使用vanilla mbridge
actor_rollout_ref.actor.megatron.vanilla_mbridge=False \
actor_rollout_ref.ref.megatron.vanilla_mbridge=False \Qwen3.5 踩坑
Sglang Rollout 死循环/卡住
Sglang 卡住
Issue
修正方案
- agent_loop代码
AgentLoopWorker.generate_sequences里：新增stop_token_ids - main_ppo_sync：
AgentLoopWorkerTQ.generate_sequences：新增stop_token_ids
python
# manfeng update start
if "stop_token_ids" not in sampling_params and self.tokenizer is not None:
# Check for chat-specific stop tokens
im_end_id = self.tokenizer.convert_tokens_to_ids("<|im_end|>")
if im_end_id is not None:
sampling_params["stop_token_ids"] = [im_end_id]
# manfeng update endTP>1 Sglang 卡住
TP>1 Sglang 卡住
tp = 1没问题。
tp > 1 有问题。请拉取
最新的sglang代码，本地安装相关链接：
python
from sglang.srt.entrypoints.http_server import (ServerArgs, _GlobalState,
app,
set_global_state)
from sglang.srt.managers.io_struct import (ContinueGenerationReqInput,
GenerateReqInput,
PauseGenerationReqInput,
ReleaseMemoryOccupationReqInput,
ResumeMemoryOccupationReqInput)
from sglang.srt.entrypoints.engine import Engine as SGLangEngine
from sglang.srt.managers.tokenizer_manager import ServerStatus
async def launch_server(self, master_address: str = None, master_port: int = None):
...
# --- mf修改开始 ---
(
self.tokenizer_manager,
self.template_manager,
port_args, # 端口参数，set_global_state 不需要它
scheduler_init_result, # 包含 scheduler_infos
) = SGLangEngine._launch_subprocesses(
server_args=server_args,
init_tokenizer_manager_func=sglang.srt.entrypoints.engine.init_tokenizer_manager,
run_scheduler_process_func=sglang.srt.entrypoints.engine.run_scheduler_process,
run_detokenizer_process_func=sglang.srt.entrypoints.engine.run_detokenizer_process,
)
# scheduler_info 需要从 scheduler_init_result.scheduler_infos 取出
# 对应原来 launch_server 返回值的第三个元素
self.scheduler_info = scheduler_init_result.scheduler_infosTransformer apply_chat_template导致的Bug
新版transformers apply_chat_template
问题
- input_ids should be a list of lists for batch processing sglang
原因和解决
- 旧版：默认return_dict=False
- 新版：默认return_dict=True，导致问题
- https://github.com/sgl-project/sglang/issues/19435
反向传播Size错误
bash
torch._dynamo.exc.TorchRuntimeError: Dynamo failed to run FX node with fake tensors: call_method view(*(FakeTensor(..., device='cuda:0', size=(9056, 4, 4, 256), dtype=torch.bfloat16), 9056, 4, 512), **{}): got RuntimeError("shape '[9056, 4, 512]' is invalid for input of size 37093376")bash
(3) verl patch in verl/models/mcore/patch.py: applies the gate-slicing fix when
num_query_groups < tp_size (same as Megatron-LM PR #3529), plus the existing
mbridge compatibility patch. Without the gate patch, training fails with a
shape mismatch in _apply_output_gate.
This part of the code modification was written by me and hasn’t been submitted to VERL. It specifically targets the case where kv_head < tp in Qwen-27B.不支持 GDN PackedSequence
GDN PackedSequence
核心
- 这个待Megatron-LM 支持，已提交PR，等待代码中，似乎仍有nan error.
暂时解法
- 需要remove_padding=False
- 需要use_fused_kernels=False
不支持 CP
信息
问题
- Gated delta net does not support context parallel for now
解法
- 暂时 CP = 1
- 拉取 dev megatron-LM，重新安装，但是不支持GDN，仍然无法使用
bash
cd /root/app/Megatron-LM
git pull
git checkout dev
pip install -e .Megarton-LM-Dev和mbridge 不兼容报错
bash
TypeError: Qwen3_5VLTransformerConfig.__init__() got an unexpected keyword argument 'async_tensor_model_parallel_allreduce'async_tensor_model_parallel_allreduce
问题
- 最新版本的mbridge已经无此问题了！！！
- megatron model_parallel_config.py 已删除 async_tensor_model_parallel_allreduce 参数
- mbrigde 还保留该参数：qwen3_5_vl_bridge.py
解决方法
- 修改qwen3_5_vl_bridge，注释掉async_tensor_model_parallel_allreduce参数即可
- 修改llm_bridge.py，手动pop，但这个会影响其他的。不建议。
bash
cp ~/pro/verl/recipe/swe/updated_mbridge/qwen3_5_vl_bridge.py /root/app/mbridge/mbridge/models/qwen3_5
pip uninstall mbridge -y && pip install --no-deps /root/app/mbridge