环境相关

隐藏Git Log

避免git reward hacking

目的

• 防止LLM查看git log历史，偷看答案

血的教训

• 表现
  • SWE-rebench作为训练集，训练reward一直上升。
  • SWE-V作为测试集，评估reward先涨后跌，直到0。
• 原因
  • SWE-rebench 镜像
    • Reward Hacking: 没删git history，模型利用git log命令，查看答案，获取奖励。
  • SWE-verified 镜像
    • 社区已删除相关镜像的git log，镜像是干净的。(2509月之后)
    • 模型用训练的git-log模式，去swe-v做评估，但镜像本无log，导致无效0奖励

社区脚本

• 脚本1，来自社区，抹除未来的轨迹。
• 但是git gc --prune=now --aggressive 会比较耗时，
• 可以考虑去掉--aggressive参数，
  • 常规的 git gc：只是简单地把这些不可达的垃圾对象删掉，速度相对较快。
  • 带有 --aggressive 参数的 git gc：这会让 Git 彻底重新计算和压缩整个仓库的所有对象（寻找最优的 Delta 压缩率）。对于大型项目，这相当于一次深度的全盘重组，耗时几分钟甚至十几分钟都不稀奇，远远超过了你设置的 60 秒 Timeout。

#!/bin/bash

# Default value
REMOVE_TAG=true

# Parse parameters
while [[ $# -gt 0 ]]; do
    case $1 in
        --remove_tag)
            if [ "$2" = "true" ] || [ "$2" = "false" ]; then
                REMOVE_TAG=$2
                shift 2
            else
                echo "Error: --remove_tag parameter must be true or false"
                exit 1
            fi
            ;;
        *)
            TARGET_COMMIT=$1
            shift
            ;;
    esac
done

# Check if commit_id is provided
if [ -z "$TARGET_COMMIT" ]; then
    echo "Usage: $0 <commit_id> [--remove_tag true|false]"
    echo "  --remove_tag: Whether to delete tags (Default: true)"
    exit 1
fi

echo "--- 1. Protecting local modifications (Stashing) ---"
# --include-untracked will stash untracked files together
# If there are no modifications, stash will return failure, so add a check
STASH_RESULT=$(git stash push --include-untracked -m "Pre-sanitization backup")

echo "--- 2. Starting to clean up future information ---"
# Force reset to the target commit
# git reset --hard $TARGET_COMMIT

# Remove remote repository and other branches
git remote remove origin 2>/dev/null
git branch | grep -v "^\*" | xargs -r git branch -D

# Determine whether to delete tags based on parameters
if [ "$REMOVE_TAG" = "true" ]; then
    echo "Deleting tags..."
    git tag | xargs -r git tag -d
else
    echo "Skipping tag deletion (--remove_tag=false)"
fi

# Completely clean up reflog and objects to prevent leakage
git reflog expire --expire=now --all
git gc --prune=now --aggressive

echo "--- 3. Restoring local modifications (Unstashing) ---"
# If stash was successful before, restore it now
if [[ "$STASH_RESULT" != "No local changes to save" ]]; then
    git stash pop
    echo "Local modifications have been restored."
else
    echo "No modifications requiring preservation were detected earlier."
fi

echo "--- Cleanup completed! Locked to $TARGET_COMMIT ---"

脚本2，暴力删除所有内容。

# 确保在 /testbed 或项目根目录下执行
rm -rf .git
git init
git add .
git commit -m "Initial commit for agent task"

在启动完成容器后，进行reset

async def reset_git_log(self, cwd="/testbed", timeout=60, log_prefix="") -> EnvStepResponse:
    # 1. 将 Bash 脚本内容写入容器的 /tmp 目录
    remote_path = "/tmp/clean_git_log.sh"
    await self.write_content_to_container(RESET_GIT_LOG_COMMAND, remote_path, log_prefix=log_prefix)  

    # 2. 根据 repo 动态决定是否保留 Git Tags
    # 注意：传给 Bash 脚本的 true/false 必须全小写
    if "pytest" in self.test_spec.repo:
        cmd = f"/bin/bash {remote_path} HEAD --remove_tag false"
    else:
        cmd = f"/bin/bash {remote_path} HEAD --remove_tag true"

    # 3. 在目标仓库目录 (默认 /testbed) 执行清理脚本
    resp = await self.execute_command(cmd, cwd=cwd, timeout=timeout, log_prefix=log_prefix)  

    # 4. 阅后即焚，清理战场，不给 Agent 留痕迹
    rm_cmd = f"rm -rf {remote_path}"
    await self.execute_command(rm_cmd, cwd=cwd, timeout=timeout, log_prefix=log_prefix)
    return resp

评估需用脚本

评估需执行脚本

• 由于评估脚本较长，因此不能直接execute_cmd (eval_script)
• 需要先把脚本保存到镜像，再去运行那个脚本

async def run_eval_script(self, eval_script: str, cwd="", timeout=360, log_prefix="") -> EnvStepResponse:
    remote_path = "/tmp/eval.sh"
    await self.write_content_to_container(eval_script, remote_path, log_prefix=log_prefix)  
    cmd = f"/bin/bash {remote_path}"
    resp = await self.execute_command(cmd, cwd=cwd, timeout=timeout, log_prefix=log_prefix)
    return resp