Add eval_kit evaluation suite with WebRTC visualization
- eval_kit/core: 零框架依赖的指标内核（Trace/metrics/aggregation/terrain_spec/report/naming） - eval_kit/isaaclab: 适配层（env_cfg 改造、rollout 采集、checkpoint 加载） - eval_kit/profiles.py / protocols.py: 机器人形态描述与评估口径预设 - eval_kit/tests: 24 项内核自检 + 21 项命令行自检（都不需要仿真） - eval_suite.py: 编排入口，每个（阶段, 种子）一个子进程；与 eval.py 指标口径一致 - WebRTC 可视化：--livestream 透传、视口相机跟随/自由（--view-env / --free-camera）、 连接前停顿（--pause-s）、自动放开客户端改画幅、打印连接地址 - 诊断与替代路径：rollout 进度打印、--record-frames 抓帧
This commit is contained in:
syfffffjj
@@ -0,0 +1,561 @@
|
||||
# eval_kit —— 可搬运的机器人性能评价套件
|
||||
|
||||
**这份套件与 `scripts/reinforcement_learning/rsl_rl/eval.py` 并存，不导入也不修改它。**
|
||||
两者的指标名完全一致、取点一致、公式一致，所以输出可以直接互相对照。
|
||||
|
||||
## 为什么要另写一套
|
||||
|
||||
`eval.py` 是一个能跑通的评估脚本，但它把三件事混在一起：读仿真环境、算指标、格式化输出。
|
||||
其中「读环境」必须知道具体机器人长什么样——它写死了 `.*_wheel` 和 `["base_link", ".*_knee"]`。
|
||||
于是它只能评估火蚁；而且因为 IsaacLab 的 `find_bodies` 对匹配不到的正则会直接抛
|
||||
`ValueError`，换一个没有轮子的机器人跑会在进循环之前就崩，不是给出一个错的分数。
|
||||
|
||||
本套件把三层切开，让不依赖框架的部分可以单独测试、单独搬运：
|
||||
|
||||
```
|
||||
eval_kit/
|
||||
├── core/ ★ 零框架依赖（只用 numpy + 标准库），可整块拷到别的项目
|
||||
│ ├── schema.py │ RobotProfile / EvalProtocol / Trace 三个数据契约
|
||||
│ ├── metrics.py │ 指标纯函数（唯一算数的地方）
|
||||
│ ├── aggregation.py │ 上限网格、score、跨种子 mean±std
|
||||
│ ├── terrain_spec.py │ 地形等级规格（纯数据）
|
||||
│ ├── report.py │ 格式化与指标筛选
|
||||
│ └── naming.py │ 容错版的名字/正则解析
|
||||
├── isaaclab/ × 依赖 isaaclab，唯一接触仿真内部的地方
|
||||
│ ├── env_setup.py │ 把协议与地形规格翻译成 env_cfg 改动
|
||||
│ ├── recorder.py │ 跑 rollout 并采集成 Trace
|
||||
│ └── policy.py │ checkpoint 解析与策略加载
|
||||
├── profiles.py ★ 机器人形态描述：加机器人 = 加一条记录
|
||||
├── protocols.py ★ 评估协议预设：改口径 = 改这里
|
||||
└── tests/
|
||||
├── test_core.py ★ 合成轨迹测试指标（不需要仿真）
|
||||
└── test_cli.py ★ 命令行参数与口径覆盖自检（不需要仿真）
|
||||
```
|
||||
|
||||
打 ★ 的是「可套用」的部分：`core/`、`profiles.py`、`protocols.py` 与两个测试文件
|
||||
可以在没有 Isaac Sim、没有 GPU 的机器上运行。将来要接别的框架（legged_gym、mujoco），
|
||||
只需要再写一个适配层产出 `Trace`，`core/` 一行都不用改。
|
||||
|
||||
## 用法
|
||||
|
||||
### 1. 激活火蚁项目的虚拟环境
|
||||
|
||||
```bash
|
||||
cd ~/syf/IsaacLab-2.3.2
|
||||
source env_isaaclab_py311/bin/activate
|
||||
cd ~/syf/huoyi_training # 评估脚本在仓库根目录跑
|
||||
```
|
||||
|
||||
验证环境是否正确：`python -c "import isaaclab"` 应当无报错，且 `isaaclab.__file__`
|
||||
指向 `IsaacLab-2.3.2/source/isaaclab/isaaclab/__init__.py`（可编辑安装）。
|
||||
|
||||
注意有两个容易踩空的近邻环境：系统 `python3`（3.10）和 conda 的
|
||||
`env_rl_training` 都**不能**跑——前者没有 isaaclab/isaacsim，后者有 isaacsim 5.1
|
||||
但没有 isaaclab。只有 `env_isaaclab_py311` 是完整的。仓库根目录是必须的，
|
||||
因为 `logs/` 查找用的是相对 cwd 的路径。
|
||||
|
||||
### 2. 让 IsaacLab 的日志目录可写
|
||||
|
||||
`/tmp/isaaclab` 属于机器上的其它用户，默认会 `PermissionError`。
|
||||
`configure_logging` 用的是 `tempfile.gettempdir()/isaaclab/logs`，而 `gettempdir()`
|
||||
尊重 `TMPDIR`，所以设一下即可（不必去动别人的目录）：
|
||||
|
||||
```bash
|
||||
export TMPDIR=/home/suyufei/tmp
|
||||
```
|
||||
|
||||
### 3. 跑评估
|
||||
|
||||
两条硬性约束，先看：
|
||||
|
||||
- **必须传 `--checkpoint`**（绝对路径最省事）。编排进程不启动仿真，因此不加载
|
||||
hydra，无法按 `--load-run` 用实验名查找权重。
|
||||
- **每个种子一个子进程**。IsaacLab 在本配置下**一个进程只能创建一个环境**：
|
||||
第一个环境建好后，`ROUGH_TERRAINS_CFG` 这类模块级单例会被运行时长出自引用，
|
||||
之后任何 cfg 构造（新实例、开场存的 pristine 深拷贝、重新深拷贝）都会在
|
||||
`cfg.validate()` 里 RecursionError —— 三种补救方式都实测失败。所以
|
||||
`eval_suite.py` 的编排进程自己不建环境，而是为每个（阶段, 种子）fork 一个子进程。
|
||||
代价是每个子进程要重新启动一次 Isaac Sim（约 15 s），所以
|
||||
`--protocol standard_v1_multiseed`（5 个种子）大约要 5~10 分钟。
|
||||
|
||||
```bash
|
||||
# 标准评估（默认用 solenopsis 描述 + standard_v1 协议）
|
||||
python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 \
|
||||
--checkpoint /绝对路径/model_19999.pt \
|
||||
--headless
|
||||
|
||||
# 上限评估 + 5 个随机种子 + 存轨迹
|
||||
python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 \
|
||||
--checkpoint /绝对路径/model_19999.pt \
|
||||
--protocol standard_v1_multiseed --save-trace /tmp/traces --headless
|
||||
|
||||
# 抗扰评估（每 2.01 s 施加一次水平扰动）
|
||||
python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 \
|
||||
--checkpoint /绝对路径/model_19999.pt \
|
||||
--eval-type robustness --headless
|
||||
|
||||
# 只看某一种地形的上限
|
||||
python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 \
|
||||
--checkpoint /绝对路径/model_19999.pt \
|
||||
--eval-type upper_limits --terrain-type stairs_up --headless
|
||||
|
||||
# 推荐用于机器人对比：放大地砖，消除越界终止项的误判（见下一节）
|
||||
python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 \
|
||||
--checkpoint /绝对路径/model_19999.pt \
|
||||
--protocol standard_v1_clean --headless
|
||||
|
||||
# 或者直接关掉那个终止项
|
||||
python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 \
|
||||
--checkpoint /绝对路径/model_19999.pt \
|
||||
--no-terrain-bounds --headless
|
||||
|
||||
# 冒烟测试（小规模，只为确认流水线通）
|
||||
python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 \
|
||||
--checkpoint /绝对路径/model_19999.pt \
|
||||
--protocol smoke --headless
|
||||
```
|
||||
|
||||
输出默认写到 `<checkpoint 目录>/eval_kit/<eval_type>_<时间戳>.json`——刻意与
|
||||
`eval.py` 的 `<checkpoint 目录>/eval/` 分开，避免两套结果混在一个目录里。
|
||||
|
||||
快速自检（不需要仿真）：
|
||||
|
||||
```bash
|
||||
python3 eval_kit/tests/test_core.py # 24 项：指标、聚合、名字解析、轨迹存取
|
||||
python3 eval_kit/tests/test_cli.py # 19 项：参数解析、口径覆盖、路径规则、推流开关
|
||||
```
|
||||
|
||||
### 4. 用 WebRTC 看画面（可视化）
|
||||
|
||||
机器上没有显示器时，用 IsaacLab 的推流模式看评估过程：加 `--livestream` 即可
|
||||
（这个参数由 `AppLauncher` 提供，`eval_suite.py` 自己没有定义）。
|
||||
|
||||
```bash
|
||||
# 局域网直连（推荐）：客户端和服务器在同一网段
|
||||
python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 \
|
||||
--checkpoint /绝对路径/model_18200.pt \
|
||||
--protocol smoke --terrain-type stairs_up \
|
||||
--livestream 2 --device cuda:6
|
||||
```
|
||||
|
||||
- `--livestream 2`：局域网/私有网络 WebRTC，客户端直连 `<本机 IP>:49100`。
|
||||
- `--livestream 1`：公网 WebRTC，走 NVIDIA NVCF 信令，需要 `PUBLIC_IP` 且能出公网。
|
||||
- 两个值都会让 `AppLauncher` 自动进入 headless（推流由服务端渲染），
|
||||
**不要再传 `--headless`**（传了也无害，只是多余）。
|
||||
|
||||
客户端用 NVIDIA 的 **Isaac Sim WebRTC Streaming Client**（随 Isaac Sim 一起提供，
|
||||
Windows/Linux 都有），填入服务器地址即可；端口默认 49100，脚本启动时会把实际
|
||||
地址打印出来：
|
||||
|
||||
```
|
||||
[eval_kit] WebRTC 推流已启用（--livestream 2，局域网直连）
|
||||
[eval_kit] 客户端连接地址：10.62.25.137:49100 192.168.77.26:49100
|
||||
```
|
||||
|
||||
脚本为推流额外做三件事：
|
||||
|
||||
1. **视口相机默认跟着被评估的机器人**。`--view-env N` 选第 N 个环境（默认 env 0），
|
||||
上限评估时会打印这个编号对应哪种地形、哪个难度等级。不设的话推流画面停在
|
||||
默认相机上，看不到机器人。
|
||||
想自己控制镜头就加 **`--free-camera`**：改为不跟随任何物体，起始摆在俯视全场的
|
||||
位置（按实际地形范围算中心与半径），之后在客户端里用鼠标/键盘自由移动 ——
|
||||
Isaac Lab 的 `ViewportCameraController` 只在 `origin_type="asset_root"` 时每帧
|
||||
改写相机，设成 `world` 就只在开头写一次，不会把你的操作抢回去。同时给了
|
||||
`--view-env` 时以 `--free-camera` 为准。
|
||||
2. **开始 rollout 前留出连接时间**：`--pause-s`（秒），推流时默认 30 s，
|
||||
期间**物理被冻结**、画面照常刷新，在终端按回车可提前开始。不推流时默认 0。
|
||||
已验证：停顿与否跑出来的指标逐位相同（同一 seed）。
|
||||
注意这一窗口里机器人是**悬空静止**的：它停在协议规定的起始位姿上，而
|
||||
`standard_v1_clean` 的 `start_z_offset_m=0.5` 会把它抬高 0.5 m 再落下，
|
||||
所以「离地没落下、也不动」正是正常现象；`sim.play()` 之后才开始动。
|
||||
启动时会打印倒计时，别把它当成卡死。
|
||||
3. **打印客户端连接地址**（从 carb 设置里读实际端口，不是写死的）。
|
||||
|
||||
**想看清机器人走路**：`smoke`/`standard_v1` 一类协议一个 episode 只有 2~10 s 仿真时间，
|
||||
扣掉 30 s 等待后留给你看的时间很短。只为观察的话把 episode 拉长（这会改变 `protocol`
|
||||
指纹，**别拿这些数字做对比**）：
|
||||
|
||||
```bash
|
||||
# 只看画面：60 s 的 episode、60 个机器人，混合地形上一直走
|
||||
python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 --checkpoint /绝对路径/model_18200.pt \
|
||||
--protocol standard_v1_clean --eval-type velocity_error \
|
||||
--episode-length-s 60 --num-envs 60 \
|
||||
--livestream 2 --view-env 0 --pause-s 30 --device cuda:0
|
||||
```
|
||||
|
||||
**每个（阶段, 种子）是一个独立子进程**，各自开一次推流；子进程退出时画面中断，
|
||||
下一个阶段会自动重新推流，客户端重连即可。所以 `--protocol standard_v1_multiseed`
|
||||
这种 8 个种子的协议会看到 8 次「断线—重连」。想连续观察就用单种子的协议
|
||||
（`standard_v1` / `smoke`），或者按地形单独跑。
|
||||
|
||||
**这块机器上跑推流要先挑一张空闲卡**。渲染 + 编码要占显存，而渲染路径（MGPU/
|
||||
流编码器）会在 GPU 0 上申请资源，GPU 0 正被别人的训练占满时会刷大量
|
||||
`Out of GPU memory` / `Buffer creation failed for the device: 0`，严重时
|
||||
`CUDA error: an illegal memory access` 直接把评估打断。处置：
|
||||
|
||||
```bash
|
||||
nvidia-smi --query-gpu=index,memory.used --format=csv # 挑一张空卡
|
||||
--device cuda:N # 渲染与物理都钉到 N
|
||||
--rendering_mode performance # 可选：低画质换显存与延迟
|
||||
--kit_args=--/renderer/multiGpu/maxGpuCount=1 # 可选：限制渲染只用一张卡
|
||||
```
|
||||
|
||||
`--device` 现在同时决定**策略网络**放在哪张卡：以前策略只认 agent 配置里的
|
||||
`device`（默认 `cuda:0`），在别人占着 0 号卡时会莫名其妙 OOM。
|
||||
|
||||
**读数提醒（未解）**：同一 checkpoint、同一协议、同一 seed，**启动方式不同会得到不同的
|
||||
数值**。实测（`--protocol smoke --eval-type velocity_error --num-envs 64`，同一个
|
||||
`model_18200.pt`）：
|
||||
|
||||
| 启动方式 | mean_lin_vel_error | mean_yaw_vel_error | 中途早退 |
|
||||
| --- | --- | --- | --- |
|
||||
| `--headless --device cuda:6`（三次独立重跑） | 0.8159 | 0.1480 | 10 |
|
||||
| `CUDA_VISIBLE_DEVICES=6 --headless --device cuda:0` | 0.8507 | 0.3334 | 17 |
|
||||
| `--headless --enable_cameras`（渲染开，无窗口） | 0.8507 | 0.3334 | 17 |
|
||||
| `--livestream 2 --device cuda:6`（推流，暂停 0~30 s 都试过） | 0.8507 | 0.3334 | 17 |
|
||||
|
||||
每一组内部都是**逐位可复现**的（同组三次重跑完全一致），跨组则有稳定差异；`--pause-s`
|
||||
已验证与差异无关（headless 加暂停与不加完全相同；推流下暂停 0 s 与 30 s 也相同）。
|
||||
两档之间的成因尚未定位（与「是否渲染」和「`--device` 写法/可见卡数」都相关，未能归因到单一因素）。
|
||||
**在这个问题查清之前：拿数字做对比时固定一种启动方式**，不要把可视化那次的数值与历史
|
||||
headless 结果混在一个表里报。看画面用推流没有影响，只是它的数字属于另一档。
|
||||
|
||||
**画面卡住时先看终端，不要猜**。rollout 期间每 10% 打印一行进度：
|
||||
|
||||
```
|
||||
[eval_kit] 停顿 5.0 s 结束，开始 rollout（画面从这里才动）
|
||||
[eval_kit] rollout 100/1000 步，用时 7.0 s
|
||||
[eval_kit] rollout 200/1000 步，用时 13.6 s
|
||||
...
|
||||
[eval_kit] 结果已保存：.../upper_limits_2026-10-19_17-16-19.json
|
||||
```
|
||||
|
||||
- **有 `rollout` 打印在涨** → 仿真在正常推进，卡的是画面（渲染/编码侧）。按优先级试：
|
||||
换到空闲卡（`--device cuda:N`，N 用空闲卡的编号）、
|
||||
`--rendering_mode performance`、确认客户端没断线、机器人数量少一点（`--num-envs`）。
|
||||
已知一种具体成因已经默认修掉：客户端改窗口大小/全屏会发 `onCustomResizeMessage`，
|
||||
服务端默认拒绝并让推流的 `shared swapchain buffer` 进入坏状态，此后画面不再更新
|
||||
（仿真还在跑）。脚本现在会给推流自动追加
|
||||
`--/app/livestream/allowResize=true --/app/livestream/allowDynamicResize=true`。
|
||||
如果日志里仍出现 `onCustomResizeMessage ... must both be positive`，**别缩放客户端窗口**，
|
||||
并把那段日志发出来。
|
||||
- **`rollout` 打印不再涨** → 是仿真停了，通常是在客户端 UI 里动了时间轴/仿真控件：
|
||||
视口下方的时间轴 Play/Pause/Stop 按钮、IsaacLab 面板里的 `Record Animation`
|
||||
（START 之后默认 10 s 结束会**直接关掉应用**）、`Reset Simulation on Stop`。
|
||||
看画面时别点它们；真被误触了就 Ctrl-C 重跑。
|
||||
- **进程已经结束**（终端出现 `结果已保存`，或被 Ctrl-C）→ 客户端只会把最后一帧留在屏幕上，
|
||||
看起来就是「画面定住」，这既不是卡住也没有在算，重跑即可。
|
||||
- 一轮跑完的标志只有两个：终端出现 `结果已保存：<路径>.json`，以及
|
||||
`<checkpoint 目录>/eval_kit/` 下多出一个 JSON。
|
||||
|
||||
### 已知问题：WebRTC 画面在客户端接上后停止更新
|
||||
|
||||
实测到过这个现象：仿真、渲染、相机都正常（`--record-frames` 抓出的帧显示机器人在动、
|
||||
相机跟着走），**只有推流给客户端的画面不再刷新**。已排除的原因：物理冻结（终端
|
||||
`rollout` 进度一直在涨）、暂停窗口（停顿结束后仍不动）、客户端改画幅被拒（已默认打开
|
||||
`allowResize`/`allowDynamicResize`）、时间轴被暂停（那样进度会停）。
|
||||
|
||||
推流器默认把日志静音（`app.livestream.logLevel = "silence"`），所以要先让它说话：
|
||||
|
||||
```bash
|
||||
--kit_args=--/app/livestream/logLevel=debug
|
||||
# 另外可以试：--kit_args=--/renderer/multiGpu/maxGpuCount=1
|
||||
# （IsaacLab 自己的 isaaclab.python.rendering.kit 就是这么设的，多卡机器上更保险）
|
||||
```
|
||||
|
||||
再不行就用 `--record-frames` 那条路径看评估过程（功能上等价，只是不能实时移动相机）。
|
||||
|
||||
**推流不可用时的替代路径：抓帧成视频。** `--record-frames DIR` 会在 rollout 期间按
|
||||
间隔把视口画面存成 PNG，跑完打印合成命令：
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=7 python scripts/reinforcement_learning/rsl_rl/eval_suite.py \
|
||||
--task Rough-Deeprobotics-M20-v0 --checkpoint /绝对路径/model_18200.pt \
|
||||
--protocol standard_v1_clean --eval-type velocity_error \
|
||||
--episode-length-s 30 --num-envs 16 \
|
||||
--livestream 2 --pause-s 0 --view-env 0 --device cuda:0 \
|
||||
--record-frames /tmp/eval_frames --record-every 5
|
||||
```
|
||||
|
||||
抓帧依赖视口渲染，所以**必须开着渲染**：推流（`--livestream 2`）或本地 GUI 都行，
|
||||
纯 headless（`--headless` 且不推流）没有视口，抓不到帧。跑完合成
|
||||
（`--record-every 5` 时每帧 0.1 s 仿真时间，10 fps 即按仿真速度播放）：
|
||||
|
||||
```bash
|
||||
cd /tmp/eval_frames && ffmpeg -y -framerate 10 -pattern_type glob -i '*.png' \
|
||||
-pix_fmt yuv420p -c:v libx264 -vf 'pad=ceil(iw/2)*2:ceil(ih/2)*2' /tmp/eval.mp4
|
||||
```
|
||||
|
||||
已实测：600 步抓到 102 张 1280×720 PNG，合出的 mp4 能正常播放，画面里机器人从
|
||||
起始高度落地并在地形上移动、相机跟着它走 —— 这条路径绕开了推流传输层，
|
||||
在推流卡住时仍然能看评估过程。
|
||||
|
||||
## 三个对象
|
||||
|
||||
- **`RobotProfile`**（`profiles.py`）：这个机器人谁是足端、谁是支撑体、哪些关节耗电、多重。
|
||||
评价代码里没有任何 link/joint 名字或质量常数。
|
||||
- **`EvalProtocol`**（`protocols.py`）：一次评估的全部口径——指令、时长、成功距离、
|
||||
倾角阈值、加速度阈值、能耗约定、地形规格、种子。决定数值的字段会被哈希成
|
||||
`protocol_fingerprint` 写进输出 JSON。
|
||||
- **`Trace`**（`core/schema.py`）：一次 rollout 的逐帧记录（机身位姿/速度、关节力矩与速度、
|
||||
足端位置、支撑接触力与时长）。第 0 帧是动作前的起始状态，之后每步一帧。
|
||||
|
||||
流水线是单向的：适配器产 `Trace` → 内核算指标 → 报告层格式化。
|
||||
`Trace` 可以存成 `.npz`，事后加新指标不用重跑仿真。
|
||||
|
||||
## 加一个机器人
|
||||
|
||||
在 `profiles.py` 加一条记录即可，评估代码不动：
|
||||
|
||||
```python
|
||||
MY_ROBOT = RobotProfile(
|
||||
name="my_robot",
|
||||
mass_kg=None, # None = 运行时从资产读真实质量
|
||||
feet_body_patterns=(".*_foot",),
|
||||
support_body_patterns=(), # 没有支撑体语义 → 相关指标返回 nan
|
||||
actuated_joint_patterns=(".*_joint",),
|
||||
support_force_threshold_n=15.0,
|
||||
body_length_m=0.5,
|
||||
)
|
||||
```
|
||||
|
||||
`test_core.py::test_naming_does_not_crash_when_morphology_is_missing` 就是在验证这件事：
|
||||
用火蚁的描述去解析一个没有轮子的机器人，得到的是「未命中」记录 + 跳过对应指标，而不是崩溃。
|
||||
|
||||
## 相对 `eval.py` 的三处有意差异
|
||||
|
||||
指标名、取点、公式全部保持一致，只有三处刻意不同：
|
||||
|
||||
| 项 | `eval.py` | 本套件 | 原因 |
|
||||
|---|---|---|---|
|
||||
| COT 的质量 | `--robot_mass` 默认 25.482 | 默认从 `root_physx_view.get_masses()` 读（`Solenopsis.urdf` 实为 26.581 kg），命令行可覆盖 | 旧默认值是更早的 M20 简化模型名义质量，会让 COT 偏低约 4% |
|
||||
| 加速度阈值 | `--acc_threshold` 2.0 m/s²（绝对值） | `acc_threshold_g` 重力倍数（默认仍是 2.0/9.81） | 绝对值对不同量级的机器人不公平 |
|
||||
| 缺失形态的指标 | 返回 0.0 | 返回 `nan`，并在 JSON 里记录 `stability_margin_method="unavailable"` | 0.0 看起来像测出来的值，实际是没测 |
|
||||
|
||||
另外新增了两项检查：输出 JSON 里带 `protocol_fingerprint`（防止不同口径的结果被悄悄比较），
|
||||
以及 `early_resets`（环境在 episode 中途被重置时会警告，因为那会让路程/进度类指标混入两次 rollout）。
|
||||
|
||||
## 口径与 `criterion.py` 的对照（备忘）
|
||||
|
||||
`criterion.py`（来自 legged_gym 的山猫项目）与 `eval.py` 在 7 处口径上不一致，
|
||||
移植/对照时要注意。已经核对过的结论：
|
||||
|
||||
| 项 | `criterion.py` | `eval.py` / 本套件 |
|
||||
|---|---|---|
|
||||
| 能耗积分 | `Σ(τ·q̇)·dt`（有符号） | `Σ\|τ·q̇\|·dt`（绝对值，本套件可切换但默认绝对值） |
|
||||
| COT 质量 | 写死 27.78822 | 自动读取（26.581 kg） |
|
||||
| 倾角阈值 | π/6 ≈ 0.524 rad | 0.70 rad |
|
||||
| 稳定裕度 | ZMP 到足端凸包距离（scipy + shapely） | 机身到足端包围盒距离 |
|
||||
| 成功判定 | env 维护目标区域 | 沿指令方向前进 ≥ success_distance |
|
||||
| 时间效率基准 | 走过半块地形的时间 | 走完 success_distance 的时间 |
|
||||
| 上限 score | 累积成功率加权 | **代数等价，已验算一致** ✓ |
|
||||
|
||||
其中「能耗积分」这一条最要紧：有符号口径下负功会抵消正功，同一段运动算出的能耗可以
|
||||
差一倍以上，所以跨机器人比较时必须在协议里钉死一种。
|
||||
|
||||
## ⚠️ 发现的问题：越界终止项让约 20% 的环境从第 0 步就被重置
|
||||
|
||||
这是实跑冒烟测试时发现的，**是 `eval.py` 评估设置里既有的问题，不是本套件引入的**。
|
||||
|
||||
IsaacLab 的 `terrain_out_of_bounds` 终止项（`isaaclab_tasks/.../mdp/terminations.py`）
|
||||
比较的是机器人的**世界坐标**与**整张地图**的范围，而不是机器人自己那块地砖：
|
||||
|
||||
```python
|
||||
map_width = n_rows * tile + 2 * border_width
|
||||
x_out_of_bounds = |root_pos_w[:, 0]| > 0.5 * map_width - distance_buffer
|
||||
```
|
||||
|
||||
在 `eval.py` 的默认评估配置下（地砖 2.0 m、`border_width` 2.0、`distance_buffer` 3.0、
|
||||
起始位姿 x = −0.9），网格最外侧格子中心在 ±(rows−1)/2 × 2.0 处，机器人起始位置因此
|
||||
越过阈值。实测数据（8 难度 × 2 地形 × 每格 2 个 = 32 个环境）：
|
||||
|
||||
- 9 行格子中有 2 个整格越界（origin x = −7.0 → 起始 x = −7.9 > 阈值 7.0）
|
||||
- **首步就有 6/32 = 19% 的环境被判越界并重置**
|
||||
- 5×5 网格下同理：最左一整列（20%）从第 0 步起每步都被重置
|
||||
|
||||
还有第二个机制：`|y|` 恰好**等于**阈值的那些列（y = ±4.0 对阈值 4.0）处于浮点边界上，
|
||||
取决于最后一个 bit 是否触发。
|
||||
|
||||
两个后果，都影响可比性：
|
||||
|
||||
1. 这些环境永远没有机会尝试地形，却全部被记成失败 → 成功率和 score 系统性偏低。
|
||||
2. 地形行序是**打乱**的（哪一行落在最外侧每次都不同）→ **被清零的是哪个难度等级
|
||||
跨种子都不稳定**，所以逐等级成功率和 score 连自己跟自己比都不可靠。
|
||||
|
||||
### 三种处置（按推荐顺序）
|
||||
|
||||
| 方案 | 做法 | 效果 |
|
||||
|---|---|---|
|
||||
| 1 | `--terrain-size 4.0` 或 `--protocol standard_v1_clean` | 保留终止项，消除误判。阈值由 `minimum_terrain_size_for_bounds()` 给出：`tile >= 2*(0.9 + 3.0 − 2.0) = 3.8 m` |
|
||||
| 2 | `--no-terrain-bounds` 或 `--protocol standard_v1_no_bounds` | 直接去掉该终止项。成功判定本就按前进距离算，走出地砖不构成失败 |
|
||||
| 3 | 保持默认（2.0 m 地砖 + 启用终止项） | 与旧 `eval.py` 数值可比，但带着上述偏差；只建议用于对照 |
|
||||
|
||||
### 因果验证
|
||||
|
||||
同一配置（5×5 网格、10 个环境、8 步、零动作策略），只切换该终止项：
|
||||
|
||||
| `enforce_terrain_bounds` | 各环境中途重置次数 | 结论 |
|
||||
|---|---|---|
|
||||
| `True`（默认，与 eval.py 一致） | `[7, 1, 0, 0, 0, 0, 0, 7, 7, 0]` | 4/10 环境受污染 |
|
||||
| `False` | `[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]` | 无早退 |
|
||||
|
||||
本套件的 `early_resets` 检查就是为发现这类问题加的，输出 JSON 里一直带着它。
|
||||
|
||||
## ⚠️ 发现的问题二、三：地形参数与生成位置
|
||||
|
||||
下面两处同样是实跑时发现的，也都继承自 ``eval.py`` 的评估设置。
|
||||
|
||||
### 问题二：机器人在倒金字塔台阶地形里被生成到几何体内部
|
||||
|
||||
旧写法把重置高度钉在瓦片中心的 ``env_origin.z`` 上（``pose_range.z = (0, 0)``），
|
||||
这只对「瓦片中心是最高点」的地形成立。
|
||||
|
||||
``stairs_down`` 用的 ``MeshInvertedPyramidStairsTerrainCfg`` **中心是最低点**，
|
||||
再叠加 ``platform_width`` 被从 3.0 m 压到 0.4 m（``min(0.4, tile*0.2)``），
|
||||
起始点 x = −0.9 m 就落在平台之外且低于该处地表 —— 机器人被生成在几何体内部、卡住。
|
||||
|
||||
结果是 ``stairs_down`` **在全部 8 个难度上都是 0.00**，连只有 4 cm 台阶的 L1 也是，
|
||||
看上去像「这机器人不会下台阶」，实际是 harness 的问题。
|
||||
|
||||
**修复**：``EvalProtocol.start_z_offset_m``（CLI ``--start-z-offset``）。
|
||||
实测对照（同一 checkpoint、只跑 ``--terrain-type stairs_down``）：
|
||||
|
||||
| 起始高度 | score | 成功率 |
|
||||
|---|---|---|
|
||||
| 0.0（旧 eval.py） | 0.00 / 8 | 0.05 |
|
||||
| 0.3 m | 3.64 / 8 | 0.47 |
|
||||
| **0.5 m** | **4.53 / 8** | — |
|
||||
|
||||
取 0.5 m 是因为嵌入深度随难度递增（L1 约 0.11 m、L8 约 0.50 m），0.5 刚好覆盖最深一级。
|
||||
``standard_v1_clean`` 已内置该值；``standard_v1`` 保持 0.0 以维持与旧 eval.py 的可比性。
|
||||
代价是机器人在平台上方落下会弹一下，``stairs_up`` 略降，但整体是净收益。
|
||||
|
||||
### 问题三：粗糙地形的噪声参数被当成起伏幅度
|
||||
|
||||
``rough_slope`` 用的 ``HfRandomUniformTerrainCfg`` 有两个容易误解的点，IsaacLab 源码里
|
||||
写得很清楚但很容易忽略：
|
||||
|
||||
1. **``difficulty`` 参数被忽略**（docstring 原文：*"The difficulty parameter is ignored
|
||||
for this terrain"*）。所以那一列**没有难度梯度**，8 行是同一分布的重复采样 ——
|
||||
这也正是它能被看出来的原因：它的成功率不单调（0.12/0.00/0.38/0.25/0.62/0.75/0.12/0.00）。
|
||||
2. **``noise_range`` 是绝对高度范围，不是起伏幅度**。旧 eval.py 设成 ``(0.05, 0.40)``，
|
||||
而 ``noise_step`` 又被改成 0.01 —— 在 1 cm 水平网格上放 40 cm 高的随机起伏，
|
||||
等于一堵堵随机尖刺墙，轮半径只有 0.096 m 的机器人无法通过。
|
||||
|
||||
**修复**：``noise_step`` 回到上游的 0.02，``noise_range`` 的上限改为 0.10（与训练地形
|
||||
``noise_range=(0.02, 0.10)`` 同量级）。实测：
|
||||
|
||||
| ``noise_range`` | score | 成功率 |
|
||||
|---|---|---|
|
||||
| (0.05, 0.40)（旧 eval.py）+ noise_step 0.01 | 0.12 / 8 | 0.05 |
|
||||
| (0.02, 0.10) + noise_step 0.02 | **6.38 / 8** | 0.86 |
|
||||
|
||||
同时给规格加了 ``isaaclab_extra_params``（写与等级无关的固定字段），并把裸 ``setattr``
|
||||
换成带 ``hasattr`` 校验的 ``_set_sub_terrain_field``：字段名写错现在会直接报错，
|
||||
不会再静默造一个假属性、让地形保持默认难度（那看起来只是「很难」）。
|
||||
|
||||
``rough_slope`` 的 ``unit`` 里也写明了「此地形忽略难度参数，等级排名无意义」，
|
||||
所以这条限制会随输出 JSON 一起保存。
|
||||
|
||||
### 三处修正的合计效果
|
||||
|
||||
同一 checkpoint（``model_19999.pt``、seed 0），只切换协议预设：
|
||||
|
||||
| 地形 | 旧 eval.py 口径 | ``standard_v1_clean`` |
|
||||
|---|---|---|
|
||||
| stairs_up | 2.88 | 3.03 |
|
||||
| stairs_down | 0.00 | **4.21** |
|
||||
| smooth_slope | 6.50 | 6.50 |
|
||||
| rough_slope | 0.12 | **6.03** |
|
||||
| boxes | 4.28 | 4.60 |
|
||||
| **总成功率** | 0.434 | **0.659** |
|
||||
| **平均前进距离** | 3.59 m | **5.26 m** |
|
||||
| **COT** | 4.50 | **2.45** |
|
||||
| **score 总和** | 13.78 | **24.38** |
|
||||
|
||||
## ⚠️ 尚存的未解异常：所有地形的 L8 都是 0.00
|
||||
|
||||
修正后在完整评估里观察到：**五个地形在最高难度 L8 上的成功率全部是 0.00**。
|
||||
|
||||
对 ``stairs_up`` / ``boxes`` 这类台阶高度确实随难度递增的地形，L8 失败是合理的；
|
||||
但 ``rough_slope`` 忽略难度参数，L8 的统计分布与 L1–L7 完全相同（那 7 行是 0.75–1.00），
|
||||
却精确地得到 0.00 —— 这在统计上不可能是能力问题，指向**最后一行的地形原点或生成高度有问题**
|
||||
（最可能是 ``terrain_origins`` 最后一行的 z 或布局异常，使机器人被放在地形之外/之下）。
|
||||
|
||||
影响：每个地形的 ``score`` 末项 ``acc[-1] * num_levels`` 因 ``acc[-1] = 0`` 而丢失，
|
||||
score 被低估约 1 分（以 smooth_slope 为例，6.50 → 若 L8 正常约 7.40）。
|
||||
|
||||
**这一条尚未定位，也没有修。** 在有明确结论之前，请把矩阵最后一行的数据当作不可用，
|
||||
score 当作保守下界。
|
||||
|
||||
## 验证状态（请如实看待）
|
||||
|
||||
**已在真机（`IsaacLab-2.3.2/env_isaaclab_py311` + RTX 4090）上实跑验证**：
|
||||
|
||||
- `eval_kit/isaaclab/` 三个模块已端到端跑通，三步冒烟全部 PASS：
|
||||
- `names` —— 形态解析在真实 body 名上命中：足端 4 个
|
||||
`['fl_wheel','fr_wheel','hl_wheel','hr_wheel']`、支撑体 5 个
|
||||
`['base_link','fl_knee',...,'hr_knee']`、驱动关节 8 个，全部与描述文件一致
|
||||
- `plain` —— 配置 → 采集 → 指标全链路：11 帧 × 4 环境，指标齐出
|
||||
（`margin=0.1667 m`、`max_tilt=0.051 rad`、`energy=8.0572 J`、`COT=2.1570`）
|
||||
- `upper` —— 上限网格链路：8 个环境按 2 类型 × 2 等级分格，矩阵与 score 正确折叠
|
||||
- **整机质量从资产读出为 26.581 kg**，与 `docs/solenopsis_urdf_comparison.md` 的记录一致；
|
||||
这确认了 `eval.py` 的 `--robot_mass` 默认值 25.482 确实偏低约 4%，本套件改为自动读取
|
||||
是有必要的。
|
||||
- 越界终止项的偏差已定位、量化并做过因果验证（见上一节）。
|
||||
- `eval_kit/core/` 的指标用合成轨迹验证，期望值手算：22 项测试通过。
|
||||
- `eval_suite.py` 的参数解析与口径覆盖用桩替换 `isaaclab.app` 后测试：11 项通过。
|
||||
|
||||
**仍未验证**：
|
||||
|
||||
- 没有用**训练到收敛的** checkpoint 跑过，所以指标数值本身没有和 `eval.py` 做过逐项对照。
|
||||
现有的 `model_0.pt` 是 iteration 0（基本等于随机权重），跑出来的 `success_rate` 恒为 0、
|
||||
`time_consumption` 恒为 nan，拿这种退化值对照没有信息量——**这时"通过"是弱证据，
|
||||
"报错"才是强证据**，而报错部分已经全部排除。
|
||||
- 因此**第一次拿到训练好的 checkpoint 时仍建议做等价性验证**：同一 checkpoint、
|
||||
同一组参数分别跑 `eval.py` 与 `eval_suite.py --protocol standard_v1`，除「相对 eval.py
|
||||
的三处有意差异」一节列出的三项外应逐位相同。注意 `eval.py` 目前**无法启动**（见下）。
|
||||
- `profiles.py` 里的 `lite3` 记录没有在仿真里跑过。
|
||||
|
||||
## ⚠️ `eval.py` 目前无法启动
|
||||
|
||||
实跑时顺带发现的：`eval.py` 自己也加了一个 `--verbose`，而
|
||||
`AppLauncher.add_app_launcher_args()` 会无条件添加同名参数，argparse 因此在**构造
|
||||
parser 阶段**就抛错，脚本连 `--help` 都跑不出来：
|
||||
|
||||
```
|
||||
argparse.ArgumentError: argument --verbose: conflicting option string: --verbose
|
||||
```
|
||||
|
||||
`--device` 和 `--verbose` 是两次踩到的坑，但只有 `device / headless / enable_cameras /
|
||||
experience / livestream / rendering_mode / xr` 这 7 个会被 AppLauncher 自己的显式检查
|
||||
拦下并给出可读报错；`--verbose / --info / --cpu` 等其余 7 个由 argparse 拦。
|
||||
本套件把 14 个名字都记在 `tests/test_cli.py::APPLAUNCHER_ARG_NAMES` 里，并在
|
||||
`test_parser_compatible_with_app_launcher` 中复刻检查，所以这类冲突不必启动仿真就能发现。
|
||||
|
||||
修 `eval.py` 只需把那行 `--verbose` 改名（例如 `--verbose-terrain`）并同步
|
||||
`args_cli.verbose` 的用法。**我没有改动 `eval.py`**——按你的要求新写一套、不碰原文件。
|
||||
|
||||
## 已知限制
|
||||
|
||||
- **一个进程只能建一个环境**（IsaacLab 在本配置下的限制，实测确认）。因此
|
||||
`eval_suite.py` 用「编排进程 + 每（阶段, 种子）一个子进程」的结构，每个子进程
|
||||
重新启动一次 Isaac Sim。这不是本套件的设计偏好，是绕不过去的约束。
|
||||
- **地形规格是唯一真正不可移植的部分。** 规格数值（等级 × 类型 × 绝对高度）是纯数据，
|
||||
但「把规格翻译成地形生成器」的代码是 IsaacLab 专用（`isaaclab/env_setup.py`）。
|
||||
换框架需要另写这个 builder。
|
||||
- **稳定裕度口径较粗。** 当前是「机身到足端轴对齐包围盒的距离」，比 ZMP/凸包版本粗糙，
|
||||
但不会在退化多边形上失败、也不需要 scipy。算法名记录在输出 JSON 里。
|
||||
- **重复试验只覆盖随机种子。** `aggregate_runs` 给的是种子间的 mean±std，
|
||||
不含 checkpoint 之间的方差（同一实验不同迭代的权重差异）——那个要靠多取几个 checkpoint 评估。
|
||||
- **单次 rollout 的噪声很大。** 比较两个策略/机器人时请用 `--seeds` 给多个种子，
|
||||
看 mean±std，不要看单次数值。
|
||||
@@ -0,0 +1,35 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""可搬运的机器人性能评价套件。
|
||||
|
||||
分层：
|
||||
|
||||
- :mod:`eval_kit.core` —— 零框架依赖（只用 numpy + 标准库），可整体拷到别的项目
|
||||
- :mod:`eval_kit.isaaclab` —— IsaacLab 适配层，唯一接触仿真环境内部的地方
|
||||
- :mod:`eval_kit.profiles` —— 机器人形态描述（加机器人 = 加一条记录）
|
||||
- :mod:`eval_kit.protocols` —— 评估协议预设
|
||||
|
||||
命令行入口：``scripts/reinforcement_learning/rsl_rl/eval_suite.py``。
|
||||
"""
|
||||
|
||||
from eval_kit.core.aggregation import aggregate_runs, build_upper_limit_matrix
|
||||
from eval_kit.core.metrics import compute_metrics
|
||||
from eval_kit.core.schema import EvalProtocol, RobotProfile, Trace
|
||||
from eval_kit.core.terrain_spec import get_terrain_spec
|
||||
from eval_kit.profiles import PROFILES, get_profile
|
||||
from eval_kit.protocols import PROTOCOLS, get_protocol
|
||||
|
||||
__all__ = [
|
||||
"EvalProtocol",
|
||||
"PROFILES",
|
||||
"PROTOCOLS",
|
||||
"RobotProfile",
|
||||
"Trace",
|
||||
"aggregate_runs",
|
||||
"build_upper_limit_matrix",
|
||||
"compute_metrics",
|
||||
"get_profile",
|
||||
"get_protocol",
|
||||
"get_terrain_spec",
|
||||
]
|
||||
@@ -0,0 +1,26 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""评价内核：不依赖 isaaclab / isaacsim / torch / GPU，只依赖 numpy 与标准库。
|
||||
|
||||
这里的模块可以在任何机器上 import 和测试，包括没有仿真环境的机器。
|
||||
"""
|
||||
|
||||
from eval_kit.core.aggregation import aggregate_runs, build_upper_limit_matrix
|
||||
from eval_kit.core.metrics import compute_metrics
|
||||
from eval_kit.core.naming import NameResolution, resolve_matching_indices
|
||||
from eval_kit.core.schema import EvalProtocol, RobotProfile, Trace
|
||||
from eval_kit.core.terrain_spec import TERRAIN_SPECS, get_terrain_spec
|
||||
|
||||
__all__ = [
|
||||
"TERRAIN_SPECS",
|
||||
"EvalProtocol",
|
||||
"NameResolution",
|
||||
"RobotProfile",
|
||||
"Trace",
|
||||
"aggregate_runs",
|
||||
"build_upper_limit_matrix",
|
||||
"compute_metrics",
|
||||
"get_terrain_spec",
|
||||
"resolve_matching_indices",
|
||||
]
|
||||
@@ -0,0 +1,212 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""上限评估的网格汇总：把逐环境结果折叠成「难度等级 × 地形类型」的成功率矩阵。
|
||||
|
||||
纯数据变换，只依赖 numpy，与仿真框架无关。
|
||||
|
||||
``score`` 的公式与 ``criterion.py`` / 旧 ``eval.py`` 完全一致（三者代数等价）：
|
||||
对每个地形类型，先算逐级累积成功率 ``acc[i] = Π_{j<=i} rate[j]``，
|
||||
再算 ``score = Σ_i acc[i] * (i+1) * (1 - rate[i+1])``，最后一级的 ``(1 - rate)`` 取 1。
|
||||
含义是「在爬到第 i+1 级之前失败」的累积期望，值越大说明越能爬到高难度。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Sequence
|
||||
|
||||
import numpy as np
|
||||
|
||||
from eval_kit.core.terrain_spec import TerrainSpec
|
||||
|
||||
|
||||
def accumulated_success(rates: Sequence[float]) -> list[float]:
|
||||
"""逐级累积成功率。"""
|
||||
accumulated: list[float] = []
|
||||
running = 1.0
|
||||
for rate in rates:
|
||||
running *= float(rate)
|
||||
accumulated.append(running)
|
||||
return accumulated
|
||||
|
||||
|
||||
def upper_limit_score(rates: Sequence[float]) -> float:
|
||||
"""由某一地形类型各难度等级的成功率算 score。"""
|
||||
rates = [float(rate) for rate in rates]
|
||||
accumulated = accumulated_success(rates)
|
||||
score = 0.0
|
||||
for index, acc_rate in enumerate(accumulated):
|
||||
next_failure = 1.0 - rates[index + 1] if index + 1 < len(rates) else 1.0
|
||||
score += acc_rate * (index + 1) * next_failure
|
||||
return score
|
||||
|
||||
|
||||
def aggregate_runs(runs: Sequence[tuple[int, dict[str, Any]]]) -> dict[str, Any]:
|
||||
"""把多次（不同随机种子）的指标聚合成 mean/std。
|
||||
|
||||
单次 rollout 的噪声足以淹没两个策略之间的真实差异，所以做对比时应该跑多个
|
||||
种子并看分布，而不是看单个数。
|
||||
|
||||
Args:
|
||||
runs: ``(seed, metrics)`` 列表，``metrics`` 来自
|
||||
:func:`eval_kit.core.metrics.compute_metrics`。
|
||||
"""
|
||||
if not runs:
|
||||
raise ValueError("没有任何一次 run 可以聚合。")
|
||||
|
||||
# 逐环境的私有字段不参与聚合
|
||||
metric_keys: list[str] = []
|
||||
for _, metrics in runs:
|
||||
for key, value in metrics.items():
|
||||
if key.startswith("_"):
|
||||
continue
|
||||
if isinstance(value, (int, float)) and key not in metric_keys:
|
||||
metric_keys.append(key)
|
||||
|
||||
summary: dict[str, Any] = {}
|
||||
for key in metric_keys:
|
||||
values = np.array([float(metrics.get(key, float("nan"))) for _, metrics in runs], dtype=np.float64)
|
||||
finite = values[np.isfinite(values)]
|
||||
if finite.size == 0:
|
||||
summary[key] = {
|
||||
"mean": float("nan"),
|
||||
"std": float("nan"),
|
||||
"min": float("nan"),
|
||||
"max": float("nan"),
|
||||
"values": values.tolist(),
|
||||
"finite_runs": 0,
|
||||
}
|
||||
continue
|
||||
summary[key] = {
|
||||
"mean": float(np.mean(finite)),
|
||||
"std": float(np.std(finite, ddof=1)) if finite.size > 1 else 0.0,
|
||||
"min": float(np.min(finite)),
|
||||
"max": float(np.max(finite)),
|
||||
"values": values.tolist(),
|
||||
"finite_runs": int(finite.size),
|
||||
}
|
||||
|
||||
consistent: dict[str, Any] = {}
|
||||
for key in runs[0][1]:
|
||||
if not key.startswith("_") and isinstance(runs[0][1][key], str):
|
||||
values = {metrics.get(key) for _, metrics in runs}
|
||||
consistent[key] = runs[0][1][key]
|
||||
if len(values) > 1:
|
||||
consistent[f"{key}_INCONSISTENT"] = sorted(str(value) for value in values)
|
||||
|
||||
return {
|
||||
"num_runs": len(runs),
|
||||
"seeds": [int(seed) for seed, _ in runs],
|
||||
"summary": summary,
|
||||
"constant": consistent,
|
||||
"per_run": [{"seed": int(seed), "metrics": metrics} for seed, metrics in runs],
|
||||
}
|
||||
|
||||
|
||||
def build_upper_limit_matrix(
|
||||
per_env: dict[str, list[float]],
|
||||
level_ids: np.ndarray,
|
||||
type_ids: np.ndarray,
|
||||
terrain_types: Sequence[str],
|
||||
spec: TerrainSpec,
|
||||
num_levels: int,
|
||||
) -> dict[str, Any]:
|
||||
"""把逐环境结果汇总成网格。
|
||||
|
||||
Args:
|
||||
per_env: 含 ``success`` / ``max_progress`` / ``max_tilt`` 的逐环境列表，
|
||||
由 :func:`eval_kit.core.metrics.compute_metrics` 在
|
||||
``include_per_env=True`` 时给出。
|
||||
level_ids: ``(N,)`` 每个环境所属的难度等级（0 起）。
|
||||
type_ids: ``(N,)`` 每个环境所属的地形类型序号（对应 ``terrain_types``）。
|
||||
terrain_types: 地形类型名，同时决定矩阵列顺序。
|
||||
spec: 地形规格，用于取每级的难度值。
|
||||
num_levels: 难度等级数（矩阵行数）。
|
||||
"""
|
||||
success = np.asarray(per_env["success"], dtype=np.float64)
|
||||
max_progress = np.asarray(per_env["max_progress"], dtype=np.float64)
|
||||
max_tilt = np.asarray(per_env["max_tilt"], dtype=np.float64)
|
||||
level_ids = np.asarray(level_ids, dtype=np.int64)
|
||||
type_ids = np.asarray(type_ids, dtype=np.int64)
|
||||
|
||||
if not (success.shape == max_progress.shape == max_tilt.shape == level_ids.shape == type_ids.shape):
|
||||
raise ValueError("逐环境数组与网格编号数组长度不一致。")
|
||||
|
||||
per_type: dict[str, Any] = {}
|
||||
for type_index, terrain_type in enumerate(terrain_types):
|
||||
level_values = spec.levels_for(terrain_type, num_levels)
|
||||
rates: list[float] = []
|
||||
level_metrics: list[dict[str, Any]] = []
|
||||
for level_index, level_value in enumerate(level_values):
|
||||
mask = (type_ids == type_index) & (level_ids == level_index)
|
||||
total = int(np.count_nonzero(mask))
|
||||
if total == 0:
|
||||
rates.append(float("nan"))
|
||||
level_metrics.append(
|
||||
{
|
||||
"level": level_index + 1,
|
||||
"difficulty": float(level_value),
|
||||
"success_rate": float("nan"),
|
||||
"success_count": 0.0,
|
||||
"num_envs": 0,
|
||||
"avg_progress_m": float("nan"),
|
||||
"avg_max_tilt_rad": float("nan"),
|
||||
}
|
||||
)
|
||||
continue
|
||||
count = float(np.count_nonzero(success[mask] > 0.5))
|
||||
rates.append(count / total)
|
||||
level_metrics.append(
|
||||
{
|
||||
"level": level_index + 1,
|
||||
"difficulty": float(level_value),
|
||||
"success_rate": count / total,
|
||||
"success_count": count,
|
||||
"num_envs": total,
|
||||
"avg_progress_m": float(np.mean(max_progress[mask])),
|
||||
"avg_max_tilt_rad": float(np.mean(max_tilt[mask])),
|
||||
}
|
||||
)
|
||||
|
||||
per_type[terrain_type] = {
|
||||
"levels": level_metrics,
|
||||
"success_rate": rates,
|
||||
"success_rate_accumulated": accumulated_success(rates),
|
||||
"score": upper_limit_score(rates),
|
||||
"max_level": len(level_values),
|
||||
"difficulty_unit": spec.unit.get(terrain_type, ""),
|
||||
}
|
||||
|
||||
# 打成便于打印与比较的矩阵：行 = 难度等级，列 = 地形类型
|
||||
success_matrix: list[list[float]] = []
|
||||
accumulated_matrix: list[list[float]] = []
|
||||
difficulty_matrix: list[list[float]] = []
|
||||
for level_index in range(num_levels):
|
||||
success_row: list[float] = []
|
||||
accumulated_row: list[float] = []
|
||||
difficulty_row: list[float] = []
|
||||
for terrain_type in terrain_types:
|
||||
entry = per_type[terrain_type]
|
||||
rates = entry["success_rate"]
|
||||
accumulated = entry["success_rate_accumulated"]
|
||||
levels = entry["levels"]
|
||||
success_row.append(rates[level_index] if level_index < len(rates) else float("nan"))
|
||||
accumulated_row.append(accumulated[level_index] if level_index < len(accumulated) else float("nan"))
|
||||
difficulty_row.append(levels[level_index]["difficulty"] if level_index < len(levels) else float("nan"))
|
||||
success_matrix.append(success_row)
|
||||
accumulated_matrix.append(accumulated_row)
|
||||
difficulty_matrix.append(difficulty_row)
|
||||
|
||||
return {
|
||||
"benchmark": spec.name,
|
||||
"terrain_types": list(terrain_types),
|
||||
"num_levels": num_levels,
|
||||
"difficulty_unit": {terrain_type: spec.unit.get(terrain_type, "") for terrain_type in terrain_types},
|
||||
"per_type": per_type,
|
||||
"scores": {terrain_type: per_type[terrain_type]["score"] for terrain_type in terrain_types},
|
||||
"matrix": {
|
||||
"success_rate": success_matrix,
|
||||
"success_rate_accumulated": accumulated_matrix,
|
||||
"difficulty": difficulty_matrix,
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,262 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""指标计算：纯函数，只依赖 numpy。
|
||||
|
||||
输入是 :class:`~eval_kit.core.schema.Trace`、:class:`~eval_kit.core.schema.RobotProfile`
|
||||
和 :class:`~eval_kit.core.schema.EvalProtocol`，输出是一个扁平的指标字典。
|
||||
|
||||
指标口径与 ``scripts/reinforcement_learning/rsl_rl/eval.py`` 保持一致（同样的取点、
|
||||
同样的公式），因此默认协议下两套评估的数值可以直接对照；差别只在三处，且都是
|
||||
有意为之：
|
||||
|
||||
1. **质量**不再写死在命令行默认值里，而是从 ``Trace.mass_kg`` 取（适配层从资产读），
|
||||
传入的 ``profile.mass_kg`` 只作兜底。
|
||||
2. **加速度阈值**以重力倍数表达，换算成 m/s² 后参与计算，使阈值可跨量级机器人复用。
|
||||
3. **缺失的形态**返回 ``nan`` 而不是 0。旧实现里没有足端的机器人会得到
|
||||
「支撑裕度 = 0.00」，看起来像测出来的值，实际是没测。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
from eval_kit.core.schema import EvalProtocol, RobotProfile, Trace
|
||||
|
||||
#: 全部帧，用于成功判定与最大进度（含起始帧，与旧实现取点一致）。
|
||||
_ALL_FRAMES = slice(None)
|
||||
#: 动作执行之后的帧，用于逐帧累积类指标。
|
||||
_STEP_FRAMES = slice(1, None)
|
||||
|
||||
_GRAVITY = 9.81
|
||||
|
||||
|
||||
def tilt_from_projected_gravity(projected_gravity_b: np.ndarray) -> np.ndarray:
|
||||
"""由机体系投影重力算倾角 [rad]。
|
||||
|
||||
``projected_gravity_b`` 为 ``(T, N, 3)``，返回 ``(T, N)``。
|
||||
"""
|
||||
z_axis = np.clip(-projected_gravity_b[..., 2], -1.0, 1.0)
|
||||
return np.arccos(z_axis)
|
||||
|
||||
|
||||
def support_margin_xy(root_xy: np.ndarray, foot_xy: np.ndarray) -> np.ndarray:
|
||||
"""机身投影点到足端轴对齐包围盒的距离 [m]，在盒内为正、盒外为负。
|
||||
|
||||
``root_xy`` 为 ``(N, 2)``，``foot_xy`` 为 ``(N, F, 2)``，返回 ``(N,)``。
|
||||
|
||||
这是当前实现采用的口径（与旧 ``eval.py`` 相同），记录在
|
||||
``EvalProtocol.stability_margin_method`` 里。它比凸包/ZMP 版本粗糙，
|
||||
但不会在退化多边形上失败，也不需要 scipy。
|
||||
"""
|
||||
min_xy = np.min(foot_xy, axis=1)
|
||||
max_xy = np.max(foot_xy, axis=1)
|
||||
|
||||
inside_x = (root_xy[:, 0] >= min_xy[:, 0]) & (root_xy[:, 0] <= max_xy[:, 0])
|
||||
inside_y = (root_xy[:, 1] >= min_xy[:, 1]) & (root_xy[:, 1] <= max_xy[:, 1])
|
||||
inside = inside_x & inside_y
|
||||
|
||||
inside_dist = np.minimum(
|
||||
np.minimum(root_xy[:, 0] - min_xy[:, 0], max_xy[:, 0] - root_xy[:, 0]),
|
||||
np.minimum(root_xy[:, 1] - min_xy[:, 1], max_xy[:, 1] - root_xy[:, 1]),
|
||||
)
|
||||
outside_dx = np.clip(np.maximum(min_xy[:, 0] - root_xy[:, 0], root_xy[:, 0] - max_xy[:, 0]), 0.0, None)
|
||||
outside_dy = np.clip(np.maximum(min_xy[:, 1] - root_xy[:, 1], root_xy[:, 1] - max_xy[:, 1]), 0.0, None)
|
||||
outside_dist = np.sqrt(outside_dx**2 + outside_dy**2)
|
||||
|
||||
return np.where(inside, inside_dist, -outside_dist)
|
||||
|
||||
|
||||
def _nanmean_or_nan(values: np.ndarray) -> float:
|
||||
"""均值；空数组返回 nan（而不是 0，也不是 numpy 的警告）。"""
|
||||
if values.size == 0:
|
||||
return float("nan")
|
||||
return float(np.mean(values))
|
||||
|
||||
|
||||
def compute_metrics(
|
||||
trace: Trace,
|
||||
profile: RobotProfile,
|
||||
protocol: EvalProtocol,
|
||||
*,
|
||||
include_per_env: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""由一次 rollout 的轨迹算出全部指标。
|
||||
|
||||
指标名的命名与旧 ``eval.py`` 完全一致，便于新旧结果直接对照。
|
||||
"""
|
||||
num_steps = trace.num_steps
|
||||
num_envs = trace.num_envs
|
||||
dt = float(trace.dt)
|
||||
|
||||
if abs(protocol.dt - dt) > 0.01 * dt:
|
||||
raise ValueError(
|
||||
f"协议 dt={protocol.dt} 与轨迹 dt={dt} 不一致。"
|
||||
"通常意味着协议里的步长与任务的 decimation 不匹配。"
|
||||
)
|
||||
|
||||
command = np.asarray(trace.command, dtype=np.float64)
|
||||
command_xy = command[:2]
|
||||
command_yaw = float(command[2])
|
||||
|
||||
command_dir = command_xy.copy()
|
||||
norm = float(np.linalg.norm(command_dir))
|
||||
command_dir = command_dir / norm if norm > 1e-6 else np.array([1.0, 0.0])
|
||||
|
||||
# ------------------------------------------------------------------ 成功判定
|
||||
base_xy = np.asarray(trace.base_pos_w, dtype=np.float64)[..., :2]
|
||||
delta_xy = base_xy - base_xy[0][None, :, :]
|
||||
progress = np.einsum("tnc,c->tn", delta_xy[_ALL_FRAMES], command_dir)
|
||||
max_progress = progress.max(axis=0)
|
||||
reached = max_progress >= protocol.success_distance_m
|
||||
|
||||
tilt = tilt_from_projected_gravity(np.asarray(trace.projected_gravity_b, dtype=np.float64))
|
||||
tilt_steps = tilt[_STEP_FRAMES]
|
||||
max_tilt = tilt_steps.max(axis=0)
|
||||
ever_fallen = max_tilt > protocol.tilt_threshold_rad
|
||||
|
||||
success = reached & ~ever_fallen
|
||||
success_count = int(np.count_nonzero(success))
|
||||
|
||||
# 首次达成成功的时刻：第 i 帧（i>=1）达成则记 i*dt。
|
||||
crossed = progress[_STEP_FRAMES] >= protocol.success_distance_m
|
||||
complete_time = np.full((num_envs,), np.nan, dtype=np.float64)
|
||||
for env_index in range(num_envs):
|
||||
hit = np.flatnonzero(crossed[:, env_index])
|
||||
if hit.size:
|
||||
complete_time[env_index] = float(hit[0] + 1) * dt
|
||||
|
||||
successful_time = complete_time[success & np.isfinite(complete_time)]
|
||||
avg_time = _nanmean_or_nan(successful_time)
|
||||
|
||||
# ------------------------------------------------------------- 速度跟踪误差
|
||||
lin_vel_b = np.asarray(trace.base_lin_vel_b, dtype=np.float64)[_STEP_FRAMES]
|
||||
ang_vel_b = np.asarray(trace.base_ang_vel_b, dtype=np.float64)[_STEP_FRAMES]
|
||||
lin_error = np.linalg.norm(lin_vel_b[..., :2] - command_xy[None, None, :], axis=-1)
|
||||
yaw_error = np.abs(ang_vel_b[..., 2] - command_yaw)
|
||||
|
||||
# -------------------------------------------------------------- 能耗与路程
|
||||
joint_torque = np.asarray(trace.joint_torque, dtype=np.float64)[_STEP_FRAMES]
|
||||
joint_vel = np.asarray(trace.joint_vel, dtype=np.float64)[_STEP_FRAMES]
|
||||
mechanical_power = joint_torque * joint_vel
|
||||
if protocol.energy_convention == "absolute":
|
||||
integrand = np.abs(mechanical_power)
|
||||
elif protocol.energy_convention == "signed":
|
||||
integrand = mechanical_power
|
||||
else:
|
||||
raise ValueError(
|
||||
f"未知能耗约定 {protocol.energy_convention!r}；可选 'absolute' 或 'signed'。"
|
||||
)
|
||||
energy = integrand.sum(axis=(0, 2)) * dt
|
||||
|
||||
lin_vel_w = np.asarray(trace.base_lin_vel_w, dtype=np.float64)
|
||||
route = np.linalg.norm(lin_vel_w[_STEP_FRAMES][..., :2], axis=-1).sum(axis=0) * dt
|
||||
|
||||
# ------------------------------------------------------------ 运动不稳定性
|
||||
accel = np.linalg.norm(np.diff(lin_vel_w, axis=0), axis=-1) / dt
|
||||
instability_count = (accel > protocol.acc_threshold_mps2).sum(axis=0)
|
||||
|
||||
stable_time = (tilt_steps < protocol.tilt_threshold_rad).sum(axis=0) * dt
|
||||
|
||||
# ------------------------------------------------------------- 支撑多边形
|
||||
feet_pos_w = np.asarray(trace.feet_pos_w, dtype=np.float64)[_STEP_FRAMES]
|
||||
num_feet = int(feet_pos_w.shape[2])
|
||||
if num_feet > 0:
|
||||
margins = support_margin_xy(base_xy[_STEP_FRAMES][:, :, :].reshape(-1, 2), feet_pos_w.reshape(-1, num_feet, 2))
|
||||
margins = margins.reshape(feet_pos_w.shape[0], num_envs)
|
||||
avg_margin = _nanmean_or_nan(margins)
|
||||
margin_method = protocol.stability_margin_method
|
||||
else:
|
||||
avg_margin = float("nan")
|
||||
margin_method = "unavailable"
|
||||
|
||||
# --------------------------------------------------------------- 支撑接触
|
||||
support_force = np.asarray(trace.support_force_n, dtype=np.float64)
|
||||
support_contact_time = np.asarray(trace.support_contact_time_s, dtype=np.float64)
|
||||
num_support_bodies = int(support_force.shape[2])
|
||||
support_threshold = (
|
||||
profile.support_force_threshold_n
|
||||
if protocol.support_force_threshold_n is None
|
||||
else protocol.support_force_threshold_n
|
||||
)
|
||||
if num_support_bodies > 0:
|
||||
active = np.any(support_force[_STEP_FRAMES] > support_threshold, axis=2) # (T-1, N)
|
||||
support_steps = active.sum(axis=0)
|
||||
# 起始帧之前视为「未接触」，因此首帧接触也算一次事件（与旧实现一致）。
|
||||
active_padded = np.concatenate([np.zeros((1, num_envs), dtype=bool), active], axis=0)
|
||||
support_events = (active_padded[1:] & ~active_padded[:-1]).sum(axis=0)
|
||||
max_support_time = support_contact_time[_STEP_FRAMES].max(axis=(0, 2))
|
||||
support_contact_ratio = _nanmean_or_nan(support_steps / max(num_steps, 1))
|
||||
avg_support_events = float(np.mean(support_events))
|
||||
avg_max_support_time = float(np.mean(max_support_time))
|
||||
if success_count > 0:
|
||||
success_support_ratio = float(np.mean((support_events[success] > 0).astype(np.float64)))
|
||||
else:
|
||||
success_support_ratio = float("nan")
|
||||
else:
|
||||
support_contact_ratio = float("nan")
|
||||
avg_support_events = float("nan")
|
||||
avg_max_support_time = float("nan")
|
||||
success_support_ratio = float("nan")
|
||||
|
||||
# --------------------------------------------------------- 质量与 COT
|
||||
mass = trace.mass_kg if trace.mass_kg is not None else profile.mass_kg
|
||||
if mass is None or mass <= 0.0:
|
||||
cost_of_transport = np.full((num_envs,), np.nan)
|
||||
mass_used = float("nan")
|
||||
else:
|
||||
cost_of_transport = energy / ((route + 1e-6) * mass * _GRAVITY)
|
||||
mass_used = float(mass)
|
||||
|
||||
metrics: dict[str, Any] = {
|
||||
"profile": profile.name,
|
||||
"protocol_fingerprint": protocol.fingerprint(),
|
||||
"num_envs": float(num_envs),
|
||||
"num_steps": float(num_steps),
|
||||
"episode_length_s": float(num_steps * dt),
|
||||
"terrain_size_m": float(protocol.terrain_size_m),
|
||||
"success_distance_m": float(protocol.success_distance_m),
|
||||
"robot_mass_kg": mass_used,
|
||||
"energy_convention": protocol.energy_convention,
|
||||
"stability_margin_method": margin_method,
|
||||
"tilt_threshold_rad": float(protocol.tilt_threshold_rad),
|
||||
"acc_threshold_mps2": float(protocol.acc_threshold_mps2),
|
||||
"support_force_threshold_n": float(support_threshold) if num_support_bodies > 0 else float("nan"),
|
||||
# --- 成功率与进度
|
||||
"success_count": float(success_count),
|
||||
"success_rate": float(success_count / max(num_envs, 1)),
|
||||
"avg_final_progress_m": float(np.mean(max_progress)),
|
||||
# --- 时间
|
||||
"avg_time_consumption_s": avg_time,
|
||||
"time_efficiency": float(
|
||||
avg_time / (protocol.success_distance_m / max(abs(float(command[0])), 1e-6))
|
||||
)
|
||||
if np.isfinite(avg_time)
|
||||
else float("nan"),
|
||||
# --- 跟踪误差
|
||||
"mean_lin_vel_error": float(np.mean(lin_error)),
|
||||
"mean_yaw_vel_error": float(np.mean(yaw_error)),
|
||||
# --- 能耗
|
||||
"avg_energy_j": float(np.mean(energy)),
|
||||
"avg_cost_of_transport": _nanmean_or_nan(cost_of_transport),
|
||||
# --- 稳定性
|
||||
"unstability_rate": float(np.mean(instability_count / max(num_steps, 1))),
|
||||
"avg_stability_margin_m": avg_margin,
|
||||
"avg_max_tilt_rad": float(np.mean(max_tilt)),
|
||||
"avg_stable_time_s": float(np.mean(stable_time)),
|
||||
# --- 支撑接触
|
||||
"avg_support_contact_ratio": support_contact_ratio,
|
||||
"avg_support_events": avg_support_events,
|
||||
"avg_max_support_contact_time_s": avg_max_support_time,
|
||||
"successful_runs_with_support_ratio": success_support_ratio,
|
||||
}
|
||||
|
||||
if include_per_env:
|
||||
metrics["_per_env"] = {
|
||||
"success": success.astype(float).tolist(),
|
||||
"max_progress": max_progress.tolist(),
|
||||
"max_tilt": max_tilt.tolist(),
|
||||
}
|
||||
return metrics
|
||||
@@ -0,0 +1,94 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""按正则列表解析名字，与 IsaacLab 的 ``resolve_matching_names`` 语义一致，
|
||||
但**匹配不到不抛异常**。
|
||||
|
||||
IsaacLab 的实现用 ``re.fullmatch`` 逐个比对，并且：某个名字同时命中多个正则会报错，
|
||||
某个正则一个都没命中也报错。对环境配置来说这种严格是有用的（写错了应该立刻发现），
|
||||
但评价代码不该因为「这个机器人没有轮子」就崩掉——那时正确行为是跳过依赖轮子的指标。
|
||||
|
||||
这个模块是纯逻辑，不依赖 isaaclab，因此可以单独测试。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Sequence
|
||||
|
||||
|
||||
class NameResolution:
|
||||
"""一次名字解析的结果。"""
|
||||
|
||||
__slots__ = ("indices", "names", "unmatched_patterns", "ambiguous_names")
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
indices: list[int],
|
||||
names: list[str],
|
||||
unmatched_patterns: list[str],
|
||||
ambiguous_names: list[tuple[str, list[str]]],
|
||||
) -> None:
|
||||
self.indices = indices
|
||||
self.names = names
|
||||
#: 一个都没命中的正则。
|
||||
self.unmatched_patterns = unmatched_patterns
|
||||
#: 同时命中多个正则的名字，以及命中的正则。
|
||||
self.ambiguous_names = ambiguous_names
|
||||
|
||||
def __bool__(self) -> bool:
|
||||
return bool(self.indices)
|
||||
|
||||
def __len__(self) -> int:
|
||||
return len(self.indices)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return (
|
||||
f"NameResolution({len(self.indices)} 项, "
|
||||
f"未命中={self.unmatched_patterns}, 多重命中={self.ambiguous_names})"
|
||||
)
|
||||
|
||||
|
||||
def resolve_matching_indices(patterns: Sequence[str], names: Sequence[str]) -> NameResolution:
|
||||
"""把正则列表解析成 ``names`` 中的下标。
|
||||
|
||||
行为与 IsaacLab 的 ``resolve_matching_names`` 一致之处：用 ``re.fullmatch``；
|
||||
返回顺序遵循 ``patterns`` 的顺序。
|
||||
不同之处：不抛异常，而是把「未命中」和「多重命中」记在返回值里，由调用方决定
|
||||
是警告还是忽略。
|
||||
|
||||
Args:
|
||||
patterns: 正则列表，例如 ``("base_link", ".*_knee")``。
|
||||
names: 候选名字，例如 ``articulation.body_names``。
|
||||
"""
|
||||
hits: dict[int, list[str]] = {}
|
||||
pattern_hits: dict[str, int] = {pattern: 0 for pattern in patterns}
|
||||
|
||||
for index, name in enumerate(names):
|
||||
for pattern in patterns:
|
||||
try:
|
||||
matched = re.fullmatch(pattern, name) is not None
|
||||
except re.error as exc: # 正则本身写错，属于配置错误，应当立刻暴露
|
||||
raise ValueError(f"机器人描述里的正则 {pattern!r} 无法编译：{exc}") from exc
|
||||
if matched:
|
||||
hits.setdefault(index, []).append(pattern)
|
||||
pattern_hits[pattern] += 1
|
||||
|
||||
indices: list[int] = []
|
||||
resolved_names: list[str] = []
|
||||
ambiguous: list[tuple[str, list[str]]] = []
|
||||
taken: set[int] = set()
|
||||
|
||||
# 按 patterns 顺序输出，跳过已被更早的正则取走的下标
|
||||
for pattern in patterns:
|
||||
for index, matched_patterns in hits.items():
|
||||
if pattern not in matched_patterns or index in taken:
|
||||
continue
|
||||
taken.add(index)
|
||||
indices.append(index)
|
||||
resolved_names.append(names[index])
|
||||
if len(matched_patterns) > 1:
|
||||
ambiguous.append((names[index], list(matched_patterns)))
|
||||
|
||||
unmatched = [pattern for pattern, count in pattern_hits.items() if count == 0]
|
||||
return NameResolution(indices, resolved_names, unmatched, ambiguous)
|
||||
@@ -0,0 +1,191 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""结果的格式化与筛选：纯函数，只依赖 numpy 与标准库。
|
||||
|
||||
输出格式与旧 ``eval.py`` 保持一致，便于对照。``filter_metrics`` 决定每种
|
||||
``eval_type`` 关心哪些指标。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any, Sequence
|
||||
|
||||
import numpy as np
|
||||
|
||||
#: 支持的评估类型。
|
||||
EVAL_TYPES: tuple[str, ...] = (
|
||||
"all",
|
||||
"upper_limits",
|
||||
"velocity_error",
|
||||
"access_rate",
|
||||
"time_consumption",
|
||||
"energy_consumption",
|
||||
"motion_stability",
|
||||
"stability_margin",
|
||||
"robustness",
|
||||
)
|
||||
|
||||
#: 每种评估类型关心哪些指标。
|
||||
METRIC_SELECTION: dict[str, tuple[str, ...]] = {
|
||||
"velocity_error": ("mean_lin_vel_error", "mean_yaw_vel_error"),
|
||||
"access_rate": ("success_count", "success_rate", "avg_final_progress_m"),
|
||||
"time_consumption": ("success_count", "avg_time_consumption_s", "time_efficiency"),
|
||||
"energy_consumption": ("avg_energy_j", "avg_cost_of_transport"),
|
||||
"motion_stability": ("unstability_rate",),
|
||||
"stability_margin": ("avg_stability_margin_m",),
|
||||
"robustness": ("success_rate", "avg_max_tilt_rad", "avg_stable_time_s"),
|
||||
}
|
||||
|
||||
|
||||
def filter_metrics(metrics: dict[str, Any], eval_type: str) -> dict[str, Any]:
|
||||
"""按评估类型裁剪指标字典；``all`` 返回全部。"""
|
||||
if eval_type == "all":
|
||||
return metrics
|
||||
if eval_type not in METRIC_SELECTION:
|
||||
raise KeyError(f"未知评估类型 {eval_type!r}；可选：{sorted(EVAL_TYPES)}")
|
||||
return {key: metrics[key] for key in METRIC_SELECTION[eval_type] if key in metrics}
|
||||
|
||||
|
||||
def _format_cell(value: float, width: int = 14) -> str:
|
||||
if value is None or not np.isfinite(value):
|
||||
return " -".rjust(width)
|
||||
return f"{value:.2f}".rjust(width)
|
||||
|
||||
|
||||
def format_matrix(matrix: Sequence[Sequence[float]], terrain_types: Sequence[str], *, title: str) -> str:
|
||||
"""把「行=等级、列=地形」的矩阵格式化成字符串。"""
|
||||
width = 14
|
||||
header = "level".ljust(8) + "".join(terrain_type[: width - 1].rjust(width) for terrain_type in terrain_types)
|
||||
lines = [title, header]
|
||||
for level_index, row in enumerate(matrix, start=1):
|
||||
cells = "".join(_format_cell(float(value), width) for value in row)
|
||||
lines.append(f"{level_index:<8}{cells}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_numpy_matrix(matrix: Sequence[Sequence[float]], *, title: str) -> str:
|
||||
"""紧凑的 numpy 风格矩阵，便于贴进报告；nan 显示成 ``-``。
|
||||
|
||||
用 ``formatter`` 而不是 ``nanstr``——后者是 ``np.set_printoptions`` 的参数，
|
||||
不是 ``np.array2string`` 的关键字（numpy 1.21 与 1.26 都会对前者报 TypeError）。
|
||||
"""
|
||||
array = np.round(np.array(matrix, dtype=np.float32), 2)
|
||||
|
||||
def _cell(value: float) -> str:
|
||||
return " -" if not np.isfinite(value) else f"{value:.2f}"
|
||||
|
||||
body = np.array2string(array, formatter={"float_kind": _cell})
|
||||
return f"{title}:\n{body}"
|
||||
|
||||
|
||||
def format_upper_limit_summary(result: dict[str, Any]) -> str:
|
||||
"""上限评估的汇总输出：成功率矩阵、累积成功率矩阵、各类型 score。"""
|
||||
terrain_types = result["terrain_types"]
|
||||
matrix = result["matrix"]
|
||||
scores = np.array([result["scores"][terrain_type] for terrain_type in terrain_types], dtype=np.float32)
|
||||
lines = [
|
||||
format_numpy_matrix(matrix["success_rate"], title="success rate"),
|
||||
format_numpy_matrix(matrix["success_rate_accumulated"], title="success rate accumulated"),
|
||||
f"score: {np.array2string(scores, precision=2, suppress_small=True)} of {result['num_levels']}",
|
||||
]
|
||||
units = {unit for unit in result.get("difficulty_unit", {}).values() if unit}
|
||||
if units:
|
||||
lines.append(f"difficulty units: {'; '.join(sorted(units))}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_rollout_summary(metrics: dict[str, Any]) -> str:
|
||||
"""混合地形汇总跑的全指标输出。"""
|
||||
|
||||
def fmt(key: str, spec: str = "{:.2f}") -> str:
|
||||
value = float(metrics.get(key, float("nan")))
|
||||
return "nan" if not np.isfinite(value) else spec.format(value)
|
||||
|
||||
success_ratio = float(metrics.get("successful_runs_with_support_ratio", float("nan")))
|
||||
success_ratio_text = "nan" if not np.isfinite(success_ratio) else f"{success_ratio * 100:.1f}%"
|
||||
support_ratio = float(metrics.get("avg_support_contact_ratio", float("nan")))
|
||||
support_ratio_text = "nan" if not np.isfinite(support_ratio) else f"{support_ratio * 100:.1f}%"
|
||||
mass = float(metrics.get("robot_mass_kg", float("nan")))
|
||||
mass_text = "nan" if not np.isfinite(mass) else f"{mass:.3f}"
|
||||
|
||||
return "\n".join(
|
||||
[
|
||||
f"success rate: {fmt('success_rate')}",
|
||||
f"success count: {int(round(float(metrics.get('success_count', 0.0))))} "
|
||||
f"time consumption: {fmt('avg_time_consumption_s')}",
|
||||
f"time efficiency: {fmt('time_efficiency')}",
|
||||
f"mean lin vel error: {fmt('mean_lin_vel_error')} "
|
||||
f"mean yaw vel error: {fmt('mean_yaw_vel_error')}",
|
||||
f"average energy: {fmt('avg_energy_j')} J "
|
||||
f"average cost of transport: {fmt('avg_cost_of_transport')}",
|
||||
f"unstability rate: {fmt('unstability_rate', '{:.3f}')} "
|
||||
f"average stability margin: {fmt('avg_stability_margin_m')}",
|
||||
f"average max tilt: {fmt('avg_max_tilt_rad')} rad "
|
||||
f"average stable time: {fmt('avg_stable_time_s')} s",
|
||||
f"support contact ratio: {support_ratio_text} "
|
||||
f"events/episode: {fmt('avg_support_events')} "
|
||||
f"max duration: {fmt('avg_max_support_contact_time_s')} s "
|
||||
f"successful runs using support: {success_ratio_text}",
|
||||
f"[protocol] mass={mass_text} kg energy={metrics.get('energy_convention')} "
|
||||
f"margin={metrics.get('stability_margin_method')} "
|
||||
f"tilt>={fmt('tilt_threshold_rad')} acc>={fmt('acc_threshold_mps2')} "
|
||||
f"fingerprint={metrics.get('protocol_fingerprint')}",
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def format_aggregate_summary(aggregate: dict[str, Any], keys: Sequence[str] | None = None) -> str:
|
||||
"""多次试验的 ``mean ± std`` 汇总。
|
||||
|
||||
做机器人对比时应看这个而不是单次数值。
|
||||
"""
|
||||
summary = aggregate.get("summary", {})
|
||||
selected = list(keys) if keys is not None else list(summary.keys())
|
||||
lines = [f"runs = {aggregate.get('num_runs')} seeds = {aggregate.get('seeds')}"]
|
||||
for key in selected:
|
||||
entry = summary.get(key)
|
||||
if entry is None:
|
||||
continue
|
||||
mean = float(entry["mean"])
|
||||
std = float(entry["std"])
|
||||
if not np.isfinite(mean):
|
||||
lines.append(f" {key:<38} nan（{entry.get('finite_runs', 0)}/{aggregate.get('num_runs')} 次有效）")
|
||||
else:
|
||||
lines.append(f" {key:<38} {mean:>10.4f} ± {std:<8.4f} [{entry['min']:.4f}, {entry['max']:.4f}]")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def format_metric_result(eval_type: str, metrics: dict[str, Any]) -> str:
|
||||
"""单一评估类型的精简输出。"""
|
||||
|
||||
def fmt(key: str) -> str:
|
||||
value = float(metrics.get(key, float("nan")))
|
||||
return "nan" if not np.isfinite(value) else f"{value:.2f}"
|
||||
|
||||
if eval_type == "velocity_error":
|
||||
return f"mean lin vel error: {fmt('mean_lin_vel_error')}\nmean yaw vel error: {fmt('mean_yaw_vel_error')}"
|
||||
if eval_type == "access_rate":
|
||||
return f"success rate: {fmt('success_rate')}\navg final progress: {fmt('avg_final_progress_m')} m"
|
||||
if eval_type == "time_consumption":
|
||||
return (
|
||||
f"success count: {int(round(float(metrics.get('success_count', 0.0))))} "
|
||||
f"time consumption: {fmt('avg_time_consumption_s')}\n"
|
||||
f"time efficiency: {fmt('time_efficiency')}"
|
||||
)
|
||||
if eval_type == "energy_consumption":
|
||||
return f"average energy: {fmt('avg_energy_j')} J\naverage cost of transport: {fmt('avg_cost_of_transport')}"
|
||||
if eval_type == "motion_stability":
|
||||
return f"unstability rate: {fmt('unstability_rate')}"
|
||||
if eval_type == "stability_margin":
|
||||
return (
|
||||
f"average stability margin: {fmt('avg_stability_margin_m')} m "
|
||||
f"(method: {metrics.get('stability_margin_method')})"
|
||||
)
|
||||
if eval_type == "robustness":
|
||||
return (
|
||||
f"success rate: {fmt('success_rate')}\n"
|
||||
f"average max tilt: {fmt('avg_max_tilt_rad')} rad\n"
|
||||
f"average stable time: {fmt('avg_stable_time_s')} s"
|
||||
)
|
||||
return format_rollout_summary(metrics)
|
||||
@@ -0,0 +1,392 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""评价套件的数据契约。
|
||||
|
||||
三个对象构成整条流水线：
|
||||
|
||||
``RobotProfile`` 机器人形态（谁是足端、谁是支撑体、质量多少）
|
||||
``EvalProtocol`` 评估协议（指令、时长、各种阈值、能耗约定）
|
||||
``Trace`` 一次 rollout 的标准化逐帧记录
|
||||
|
||||
上游（任何框架的适配器）负责产出 ``Trace``，下游（:mod:`eval_kit.core.metrics`）
|
||||
只从 ``Trace`` 计算指标。指标代码因此不接触任何仿真对象。
|
||||
|
||||
这个模块只依赖 numpy 与标准库。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from dataclasses import asdict, dataclass
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
#: 帧索引约定：``Trace`` 的第 0 帧是执行动作之前的起始状态，
|
||||
#: 第 ``i`` 帧（``i >= 1``）是第 ``i`` 步动作之后的状态。
|
||||
#: 因此一次 ``steps`` 步的 rollout 会产生 ``steps + 1`` 帧。
|
||||
START_FRAME = 0
|
||||
|
||||
#: 成功判定与最大进度使用全部帧（含起始帧），与 legged_gym 侧
|
||||
#: ``criterion.py`` 及现有 ``eval.py`` 的取点一致。
|
||||
PROGRESS_FRAMES = slice(None)
|
||||
|
||||
#: 逐帧累积类指标只使用动作之后的帧。
|
||||
STEP_FRAMES = slice(1, None)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RobotProfile:
|
||||
"""机器人形态描述。
|
||||
|
||||
评价代码里不应出现具体的 link/joint 名字或质量常数，全部集中在这里。
|
||||
新增一个机器人 = 新增一条 ``RobotProfile`` 记录，不改评价代码。
|
||||
"""
|
||||
|
||||
name: str
|
||||
"""机器人标识，同时也是输出文件名的一部分。"""
|
||||
|
||||
mass_kg: float | None
|
||||
"""整机质量 [kg]；``None`` 表示运行时从仿真资产读取（推荐）。"""
|
||||
|
||||
feet_body_patterns: tuple[str, ...] = ()
|
||||
"""支撑多边形的接触点，例如火蚁的四个轮。空元组表示该机器人不计算支撑裕度。"""
|
||||
|
||||
support_body_patterns: tuple[str, ...] = ()
|
||||
"""允许作为「短暂支撑」的机体，例如火蚁的机身与膝部。空元组表示不做支撑统计。"""
|
||||
|
||||
actuated_joint_patterns: tuple[str, ...] = ()
|
||||
"""参与能耗统计的关节。"""
|
||||
|
||||
support_force_threshold_n: float = 15.0
|
||||
"""判定「支撑接触」的法向力阈值 [N]。"""
|
||||
|
||||
body_length_m: float = 0.0
|
||||
"""机身长度 [m]，仅用于校验地形瓦片尺寸是否够大。"""
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return asdict(self)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EvalProtocol:
|
||||
"""评估协议：一次评估的全部「口径」。
|
||||
|
||||
这些字段决定数值大小，因此都会被写进输出 JSON 的 ``fingerprint``，
|
||||
避免不同设置下的结果被悄悄拿来比较。
|
||||
|
||||
默认值与现有 ``scripts/reinforcement_learning/rsl_rl/eval.py`` 的默认
|
||||
命令行参数一致（能耗用绝对值口径、倾角阈值 0.70 rad、加速度阈值 2.0 m/s²
|
||||
折算成重力倍数），因此默认协议下的新结果可以直接和旧评估结果对照。
|
||||
"""
|
||||
|
||||
steps: int = 500
|
||||
"""每个 episode 的控制步数。dt=0.02 时 500 步 = 10 s。"""
|
||||
|
||||
dt: float = 0.02
|
||||
"""控制步长 [s]，即 ``decimation * sim.dt``。"""
|
||||
|
||||
command: tuple[float, float, float] = (1.0, 0.0, 0.0)
|
||||
"""固定速度指令 ``(lin_vel_x, lin_vel_y, ang_vel_z)``。"""
|
||||
|
||||
start_pose: tuple[float, float, float] = (-0.9, 0.0, 0.0)
|
||||
"""起始位姿 ``(x, y, yaw)``，相对地形瓦片中心。"""
|
||||
|
||||
start_z_offset_m: float = 0.0
|
||||
"""起始高度相对地形瓦片中心高度的额外抬升 [m]。
|
||||
|
||||
默认 0.0 表示把机器人直接放在 ``env_origin.z`` 上，与旧 ``eval.py`` 的写法一致。
|
||||
但那样只对「瓦片中心是最高点」的地形成立：像倒金字塔台阶（``pyramid_stairs_inv``，
|
||||
中心是最低点）这类地形，机器人在 ``|x| > platform`` 处会被直接生成到地形几何体
|
||||
内部，卡住无法移动，导致该地形在所有难度上都得到 0 成功率。
|
||||
|
||||
给一个正值（例如 0.3 m）让机器人从地形表面上方落下，可以适配任意地形形状。
|
||||
改这个值会改变 ``protocol_fingerprint``。
|
||||
"""
|
||||
|
||||
success_distance_m: float = 2.0
|
||||
"""沿指令方向前进多少米算成功。"""
|
||||
|
||||
tilt_threshold_rad: float = 0.70
|
||||
"""倾角超过该值 [rad] 视为摔倒。0.70 rad ≈ 40°。"""
|
||||
|
||||
acc_threshold_g: float = 2.0 / 9.81
|
||||
"""机身加速度超过该值（单位 g）计为一次不稳定。用重力倍数而非绝对 m/s²，
|
||||
使该阈值对不同量级的机器人保持可比。默认值 2.0 m/s² 折算而来。"""
|
||||
|
||||
energy_convention: str = "absolute"
|
||||
"""能耗积分约定，``"absolute"`` 表示 ``Σ|τ·q̇|·dt``，
|
||||
``"signed"`` 表示 ``Σ(τ·q̇)·dt``。两者数值差异可达一倍以上，
|
||||
全局必须一致；该值会写进输出 JSON。"""
|
||||
|
||||
stability_margin_method: str = "foot_bbox"
|
||||
"""支撑裕度算法。``"foot_bbox"`` = 机体到足端轴对齐包围盒的距离（当前实现）。"""
|
||||
|
||||
support_force_threshold_n: float | None = 15.0
|
||||
"""判定支撑接触的法向力阈值 [N]；``None`` 时改用机器人描述里的值。"""
|
||||
|
||||
enable_interval_pushes: bool = False
|
||||
"""是否启用周期性外力扰动（``robustness`` 评估用）。"""
|
||||
|
||||
enforce_terrain_bounds: bool = True
|
||||
"""是否保留 IsaacLab 的 ``terrain_out_of_bounds`` 终止项。
|
||||
|
||||
**默认 True 是为了与旧 ``eval.py`` 的结果可比，但已知它有系统性偏差**：
|
||||
该终止项比较的是机器人的世界坐标与整张地图的范围，而不是它自己那块地砖。
|
||||
在地砖 2 m、网格 5x5 或 8xN 的评估配置下，最外侧的格子会被误判为越界，
|
||||
机器人从第 0 步起每步都被重置，永远没机会尝试地形；而 ``|y|`` 恰好等于阈值
|
||||
的那两列则处于浮点边界上。实测：8x2 网格下首步就有 6/32 个环境被误判。
|
||||
|
||||
三种处置方式（建议前两种，按顺序）：
|
||||
|
||||
1. 把 :attr:`terrain_size_m` 提到 **≥ 3.8**，即可让所有起始位置落进阈值内，
|
||||
同时保留这个终止项；
|
||||
2. 置为 ``False``，评估时去掉这个终止项（成功判定本来就是按前进距离算的，
|
||||
走出地砖不构成失败）；
|
||||
3. 保持 ``True`` 且用小地砖 —— 数值会偏低，且被影响的难度等级因行序打乱而
|
||||
每次不同，不建议用来做对比。
|
||||
"""
|
||||
|
||||
push_interval_s: tuple[float, float] = (2.0, 2.01)
|
||||
"""扰动间隔 [s]，取一个极窄区间让扰动近似等周期但可比较。"""
|
||||
|
||||
push_velocity_range: tuple[float, float] = (-0.5, 0.5)
|
||||
"""每个水平方向上的扰动速度范围 [m/s]。"""
|
||||
|
||||
terrain_spec: str = "standard_v1"
|
||||
"""地形等级规格名，见 :mod:`eval_kit.core.terrain_spec`。"""
|
||||
|
||||
terrain_size_m: float = 2.0
|
||||
"""单块地形瓦片边长 [m]。"""
|
||||
|
||||
num_levels: int = 8
|
||||
"""上限评估的难度等级数（网格行数）。"""
|
||||
|
||||
envs_per_cell: int = 1
|
||||
"""上限评估每个「地形类型 × 难度」格子里放几个机器人。
|
||||
|
||||
默认 1。**注意：由于起始位姿被钉死，同一格的多个机器人起始坐标完全相同**，
|
||||
它们不是独立样本，差异只来自物理求解器的非确定性。所以靠「一格放多个」来提高
|
||||
样本量是假精度；正确做法是 ``envs_per_cell=1`` + 多个 :attr:`seeds`
|
||||
（不同种子会生成不同的地形实例，因此跨种子才是独立样本）。
|
||||
|
||||
代价：``envs_per_cell=1`` 时单次评估里每格成功率只能是 0 或 1，
|
||||
必须配合足够多的种子才能得到有意义的分辨率。经验值：种子数 ≥ 8 时
|
||||
每格分辨率达到 1/8 ≈ 12.5%。
|
||||
"""
|
||||
|
||||
seeds: tuple[int, ...] = (0,)
|
||||
"""重复试验的随机种子。单次 rollout 噪声很大，比较机器人时应跑多个种子。
|
||||
|
||||
每个种子会重新生成地形实例（``terrain_generator.seed = seed``），所以跨种子
|
||||
的样本是独立的；这是提高统计可靠性的正确途径，而不是加大 ``envs_per_cell``。
|
||||
"""
|
||||
|
||||
@property
|
||||
def acc_threshold_mps2(self) -> float:
|
||||
"""加速度阈值的 m/s² 形式，供指标计算使用。"""
|
||||
return self.acc_threshold_g * 9.81
|
||||
|
||||
def minimum_terrain_size_for_bounds(self, *, border_width: float = 2.0, distance_buffer: float = 3.0) -> float:
|
||||
"""保留 ``terrain_out_of_bounds`` 终止项且不误判所需的最小地砖边长 [m]。
|
||||
|
||||
这个终止项判定 ``|origin + start| <= 0.5 * 地图边长 - distance_buffer``，
|
||||
而最外侧格子中心在 ``(rows-1)/2 * tile``，地图边长 ``= rows * tile + 2 * border``。
|
||||
两边的 ``rows`` 项相消，于是结论与网格行数无关：
|
||||
|
||||
``tile >= 2 * (|start_x| + distance_buffer - border_width)``
|
||||
|
||||
默认的 ``border_width=2.0`` / ``distance_buffer=3.0`` 与
|
||||
:func:`eval_kit.isaaclab.env_setup.apply_eval_terrain_baseline` 及 IsaacLab 的
|
||||
默认值一致；取 ``start_x = -0.9`` 得 ``tile >= 3.8 m``。
|
||||
|
||||
放在内核里（而不是适配层）是为了让编排进程也能用——它不启动仿真，
|
||||
无法 import 任何依赖 isaaclab 的模块。
|
||||
"""
|
||||
return max(0.0, 2.0 * (abs(self.start_pose[0]) + distance_buffer - border_width))
|
||||
|
||||
def fingerprint(self) -> str:
|
||||
"""协议指纹：决定数值的字段的稳定哈希，写进输出 JSON。"""
|
||||
payload = {
|
||||
"steps": self.steps,
|
||||
"dt": self.dt,
|
||||
"command": list(self.command),
|
||||
"start_pose": list(self.start_pose),
|
||||
"start_z_offset_m": self.start_z_offset_m,
|
||||
"success_distance_m": self.success_distance_m,
|
||||
"tilt_threshold_rad": self.tilt_threshold_rad,
|
||||
"acc_threshold_g": self.acc_threshold_g,
|
||||
"energy_convention": self.energy_convention,
|
||||
"stability_margin_method": self.stability_margin_method,
|
||||
"support_force_threshold_n": self.support_force_threshold_n,
|
||||
"enable_interval_pushes": self.enable_interval_pushes,
|
||||
"enforce_terrain_bounds": self.enforce_terrain_bounds,
|
||||
"push_interval_s": list(self.push_interval_s),
|
||||
"push_velocity_range": list(self.push_velocity_range),
|
||||
"terrain_spec": self.terrain_spec,
|
||||
"terrain_size_m": self.terrain_size_m,
|
||||
"num_levels": self.num_levels,
|
||||
"envs_per_cell": self.envs_per_cell,
|
||||
}
|
||||
blob = json.dumps(payload, sort_keys=True).encode("utf-8")
|
||||
return hashlib.sha256(blob).hexdigest()[:12]
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
data = asdict(self)
|
||||
data["acc_threshold_mps2"] = self.acc_threshold_mps2
|
||||
data["fingerprint"] = self.fingerprint()
|
||||
return data
|
||||
|
||||
|
||||
@dataclass
|
||||
class Trace:
|
||||
"""一次 rollout 的标准化逐帧记录。
|
||||
|
||||
所有数组的第一维都是帧、第二维都是环境。第 0 帧是动作执行之前的起始状态。
|
||||
|
||||
``feet_pos_w`` / ``support_force_n`` / ``support_contact_time_s`` 的第二维
|
||||
可能是 0（该机器人没有足端或没有支撑体语义），指标层会跳过对应指标。
|
||||
"""
|
||||
|
||||
dt: float
|
||||
command: np.ndarray
|
||||
"""``(3,)`` 本次 rollout 使用的固定指令。"""
|
||||
|
||||
base_pos_w: np.ndarray
|
||||
"""``(T, N, 3)`` 机身世界位置。"""
|
||||
|
||||
base_lin_vel_w: np.ndarray
|
||||
"""``(T, N, 3)`` 机身世界系线速度，用于路程与加速度（不稳定性）。"""
|
||||
|
||||
base_lin_vel_b: np.ndarray
|
||||
"""``(T, N, 3)`` 机体系线速度，用于速度跟踪误差。"""
|
||||
|
||||
base_ang_vel_b: np.ndarray
|
||||
"""``(T, N, 3)`` 机体系角速度，用第 2 个分量做 yaw 跟踪误差。"""
|
||||
|
||||
projected_gravity_b: np.ndarray
|
||||
"""``(T, N, 3)`` 机体系投影重力，用于倾角。"""
|
||||
|
||||
joint_torque: np.ndarray
|
||||
"""``(T, N, J)`` 实际施加的关节力矩。"""
|
||||
|
||||
joint_vel: np.ndarray
|
||||
"""``(T, N, J)`` 关节速度。"""
|
||||
|
||||
feet_pos_w: np.ndarray
|
||||
"""``(T, N, F, 2)`` 足端/轮心的世界 xy，用于支撑多边形。"""
|
||||
|
||||
support_force_n: np.ndarray
|
||||
"""``(T, N, S)`` 支撑体的法向合力。"""
|
||||
|
||||
support_contact_time_s: np.ndarray
|
||||
"""``(T, N, S)`` 支撑体的连续接触时长。"""
|
||||
|
||||
done: np.ndarray
|
||||
"""``(T, N)`` 每步之后的终止标志（1.0 表示该步结束后被重置）。"""
|
||||
|
||||
mass_kg: float | None = None
|
||||
"""实际使用的整机质量 [kg]。``None`` 表示上游未能从资产读到。"""
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
self.validate()
|
||||
|
||||
@property
|
||||
def num_frames(self) -> int:
|
||||
return int(self.base_pos_w.shape[0])
|
||||
|
||||
@property
|
||||
def num_steps(self) -> int:
|
||||
"""实际执行的控制步数（帧数减一）。"""
|
||||
return self.num_frames - 1
|
||||
|
||||
@property
|
||||
def num_envs(self) -> int:
|
||||
return int(self.base_pos_w.shape[1])
|
||||
|
||||
@property
|
||||
def num_joints(self) -> int:
|
||||
return int(self.joint_torque.shape[2])
|
||||
|
||||
def validate(self) -> None:
|
||||
"""形状一致性检查。早失败好过算出一堆形状广播出来的错数字。"""
|
||||
frames, envs = self.num_frames, self.num_envs
|
||||
if frames < 2:
|
||||
raise ValueError(f"Trace 至少需要 2 帧（起始帧 + 至少一步），收到 {frames} 帧。")
|
||||
|
||||
expected: dict[str, tuple[int, ...]] = {
|
||||
"base_pos_w": (frames, envs, 3),
|
||||
"base_lin_vel_w": (frames, envs, 3),
|
||||
"base_lin_vel_b": (frames, envs, 3),
|
||||
"base_ang_vel_b": (frames, envs, 3),
|
||||
"projected_gravity_b": (frames, envs, 3),
|
||||
"done": (frames, envs),
|
||||
}
|
||||
for name, shape in expected.items():
|
||||
array = np.asarray(getattr(self, name))
|
||||
if array.shape != shape:
|
||||
raise ValueError(f"{name} 形状应为 {shape}，实际 {array.shape}。")
|
||||
|
||||
joints = self.num_joints
|
||||
for name in ("joint_torque", "joint_vel"):
|
||||
if np.asarray(getattr(self, name)).shape != (frames, envs, joints):
|
||||
raise ValueError(
|
||||
f"{name} 形状应为 {(frames, envs, joints)}，实际 {np.asarray(getattr(self, name)).shape}。"
|
||||
)
|
||||
|
||||
feet = np.asarray(self.feet_pos_w)
|
||||
if feet.shape[:2] != (frames, envs) or feet.ndim != 4 or feet.shape[3] != 2:
|
||||
raise ValueError(f"feet_pos_w 形状应为 {(frames, envs, feet.shape[2] if feet.ndim > 2 else 0, 2)}，"
|
||||
f"实际 {feet.shape}。")
|
||||
|
||||
for name in ("support_force_n", "support_contact_time_s"):
|
||||
array = np.asarray(getattr(self, name))
|
||||
if array.ndim != 3 or array.shape[:2] != (frames, envs):
|
||||
raise ValueError(f"{name} 形状应为 {(frames, envs, array.shape[2] if array.ndim > 2 else 0)}，"
|
||||
f"实际 {array.shape}。")
|
||||
|
||||
if np.asarray(self.command).shape != (3,):
|
||||
raise ValueError(f"command 应为 (3,)，实际 {np.asarray(self.command).shape}。")
|
||||
|
||||
def save(self, path: str) -> None:
|
||||
"""把轨迹存成 ``.npz``，便于事后用新指标重新分析，无需重跑仿真。"""
|
||||
np.savez_compressed(
|
||||
path,
|
||||
dt=np.asarray(self.dt),
|
||||
command=self.command,
|
||||
base_pos_w=self.base_pos_w,
|
||||
base_lin_vel_w=self.base_lin_vel_w,
|
||||
base_lin_vel_b=self.base_lin_vel_b,
|
||||
base_ang_vel_b=self.base_ang_vel_b,
|
||||
projected_gravity_b=self.projected_gravity_b,
|
||||
joint_torque=self.joint_torque,
|
||||
joint_vel=self.joint_vel,
|
||||
feet_pos_w=self.feet_pos_w,
|
||||
support_force_n=self.support_force_n,
|
||||
support_contact_time_s=self.support_contact_time_s,
|
||||
done=self.done,
|
||||
mass_kg=np.asarray(-1.0 if self.mass_kg is None else self.mass_kg),
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def load(cls, path: str) -> Trace:
|
||||
with np.load(path) as data:
|
||||
mass = float(data["mass_kg"])
|
||||
return cls(
|
||||
dt=float(data["dt"]),
|
||||
command=data["command"],
|
||||
base_pos_w=data["base_pos_w"],
|
||||
base_lin_vel_w=data["base_lin_vel_w"],
|
||||
base_lin_vel_b=data["base_lin_vel_b"],
|
||||
base_ang_vel_b=data["base_ang_vel_b"],
|
||||
projected_gravity_b=data["projected_gravity_b"],
|
||||
joint_torque=data["joint_torque"],
|
||||
joint_vel=data["joint_vel"],
|
||||
feet_pos_w=data["feet_pos_w"],
|
||||
support_force_n=data["support_force_n"],
|
||||
support_contact_time_s=data["support_contact_time_s"],
|
||||
done=data["done"],
|
||||
mass_kg=None if mass < 0.0 else mass,
|
||||
)
|
||||
@@ -0,0 +1,157 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""地形等级规格：纯数据，与仿真框架无关。
|
||||
|
||||
规格描述「评估用哪几种地形、每种地形分几级、每级多高」，用绝对高度表示，
|
||||
因为机器人越障能力本来就是由绝对高度决定的。
|
||||
|
||||
规格本身不含任何框架信息；「把规格翻译成本框架的地形生成器参数」这件事
|
||||
由各框架的 builder 负责（当前是 :mod:`eval_kit.isaaclab.env_setup`）。
|
||||
换框架时改 builder，不改这里的数值。
|
||||
|
||||
数值与 ``scripts/reinforcement_learning/rsl_rl/eval.py`` 的 ``TERRAIN_LEVELS``
|
||||
保持一致，因此两套评估所用地形严格相同。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Mapping
|
||||
|
||||
#: 无量纲坡度：高 / 水平距离。0.40 相当于 21.8°。
|
||||
_UNIT_SLOPE = "slope (rise/run, 0.40 ≈ 21.8°)"
|
||||
#: 台阶高度，米。
|
||||
_UNIT_METER = "m"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TerrainSpec:
|
||||
"""一套地形评估规格。"""
|
||||
|
||||
name: str
|
||||
"""规格名，写进输出 JSON。"""
|
||||
|
||||
types: tuple[str, ...]
|
||||
"""参与评估的地形类型，同时也是上限矩阵的列顺序。"""
|
||||
|
||||
levels: Mapping[str, tuple[float, ...]]
|
||||
"""每种地形的难度等级值（绝对量，单位见 :attr:`unit`），由易到难。"""
|
||||
|
||||
unit: Mapping[str, str]
|
||||
"""每种地形等级值的单位说明。"""
|
||||
|
||||
isaaclab_subterrain: Mapping[str, str]
|
||||
"""地形类型 → IsaacLab ``ROUGH_TERRAINS_CFG`` 里的子地形名。"""
|
||||
|
||||
isaaclab_param: Mapping[str, str]
|
||||
"""地形类型 → 需要在子地形配置里写入的字段名。"""
|
||||
|
||||
feature_scaling: Mapping[str, tuple[float, float]] = field(default_factory=dict)
|
||||
"""瓦片尺寸缩小时的特征尺寸上限：``字段名 -> (相对瓦片的比例, 绝对上限)``。
|
||||
|
||||
例如 ``{"step_width": (0.125, 0.25)}`` 表示 ``step_width = min(0.25, tile * 0.125)``。
|
||||
这样同一套规格可以用在不同大小的地形瓦片上。
|
||||
"""
|
||||
|
||||
grid_rows: int = 5
|
||||
grid_cols: int = 5
|
||||
"""非上限评估时地形生成器的网格规模。"""
|
||||
|
||||
isaaclab_extra_params: Mapping[str, Mapping[str, float]] = field(default_factory=dict)
|
||||
"""地形类型 → 需要额外写入的固定字段，例如 ``{"rough_slope": {"noise_step": 0.02}}``。
|
||||
|
||||
与 :attr:`isaaclab_param` 的区别：后者写的是随等级变化的首尾值，这里写的是与等级
|
||||
无关的固定量。写入时会校验字段是否存在，写错会直接报错而不是静默造一个假属性。
|
||||
"""
|
||||
|
||||
def levels_for(self, terrain_type: str, num_levels: int | None = None) -> tuple[float, ...]:
|
||||
"""取某种地形的等级值，``num_levels`` 为 None 时取全部。"""
|
||||
if terrain_type not in self.levels:
|
||||
raise KeyError(f"地形规格 {self.name!r} 里没有 {terrain_type!r}；可选：{sorted(self.levels)}")
|
||||
values = tuple(self.levels[terrain_type])
|
||||
if num_levels is None:
|
||||
return values
|
||||
if num_levels > len(values):
|
||||
raise ValueError(
|
||||
f"地形 {terrain_type!r} 只定义了 {len(values)} 级难度，请求了 {num_levels} 级。"
|
||||
)
|
||||
return values[:num_levels]
|
||||
|
||||
def scaling_for(self, field_name: str, tile_size_m: float) -> float | None:
|
||||
"""按瓦片尺寸算某个特征字段的上限值，未定义则返回 None。"""
|
||||
rule = self.feature_scaling.get(field_name)
|
||||
if rule is None:
|
||||
return None
|
||||
ratio, cap = rule
|
||||
return min(cap, tile_size_m * ratio)
|
||||
|
||||
def to_dict(self) -> dict[str, object]:
|
||||
return {
|
||||
"name": self.name,
|
||||
"types": list(self.types),
|
||||
"levels": {key: list(value) for key, value in self.levels.items()},
|
||||
"unit": dict(self.unit),
|
||||
}
|
||||
|
||||
|
||||
STANDARD_V1 = TerrainSpec(
|
||||
name="standard_v1",
|
||||
types=("stairs_up", "stairs_down", "smooth_slope", "rough_slope", "boxes"),
|
||||
levels={
|
||||
"stairs_up": (0.04, 0.06, 0.08, 0.10, 0.12, 0.14, 0.16, 0.18),
|
||||
"stairs_down": (0.04, 0.06, 0.08, 0.10, 0.12, 0.14, 0.16, 0.18),
|
||||
"smooth_slope": (0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40),
|
||||
# 注意：random_rough 用的是 HfRandomUniformTerrainCfg，IsaacLab 明确写了
|
||||
# 「difficulty 参数被忽略」，所以这里只有首尾值真正起作用 —— 写成
|
||||
# noise_range=(levels[0], levels[-1])，是**绝对高度范围**而非起伏幅度。
|
||||
# 旧 eval.py 把它设成 (0.05, 0.40)，在 2 cm 水平网格上等于 40 cm 高的随机尖刺，
|
||||
# 轮半径 0.096 m 的机器人无法通过，那一列几乎恒为 0（实测只从 0.12 升到 0.45）。
|
||||
# 这里回到与训练地形同量级的范围（上游训练用 noise_range=(0.02, 0.10)）。
|
||||
"rough_slope": (0.02, 0.03, 0.04, 0.05, 0.06, 0.08, 0.09, 0.10),
|
||||
"boxes": (0.04, 0.06, 0.08, 0.10, 0.12, 0.14, 0.16, 0.18),
|
||||
},
|
||||
unit={
|
||||
"stairs_up": _UNIT_METER,
|
||||
"stairs_down": _UNIT_METER,
|
||||
"smooth_slope": _UNIT_SLOPE,
|
||||
"rough_slope": _UNIT_METER + "（高度范围；此地形忽略难度参数，8 行同分布，等级排名无意义）",
|
||||
"boxes": _UNIT_METER,
|
||||
},
|
||||
isaaclab_subterrain={
|
||||
"stairs_up": "pyramid_stairs",
|
||||
"stairs_down": "pyramid_stairs_inv",
|
||||
"smooth_slope": "hf_pyramid_slope",
|
||||
"rough_slope": "random_rough",
|
||||
"boxes": "boxes",
|
||||
},
|
||||
isaaclab_param={
|
||||
"stairs_up": "step_height_range",
|
||||
"stairs_down": "step_height_range",
|
||||
"smooth_slope": "slope_range",
|
||||
"rough_slope": "noise_range",
|
||||
"boxes": "grid_height_range",
|
||||
},
|
||||
feature_scaling={
|
||||
"platform_width": (0.2, 0.4),
|
||||
"border_width": (0.05, 0.1),
|
||||
"step_width": (0.125, 0.25),
|
||||
"grid_width": (0.15, 0.3),
|
||||
},
|
||||
# 上游 ROUGH_TERRAINS_CFG 用 noise_step=0.02；旧 eval.py 把它改成 0.01 后又叠加
|
||||
# noise_range 上限 0.40 m，相当于 1 cm 水平网格上放 40 cm 起伏 —— 极端尖锐的尖刺，
|
||||
# 轮半径只有 0.096 m 的机器人基本无法通过。这里回到上游的 0.02。
|
||||
isaaclab_extra_params={
|
||||
"rough_slope": {"noise_step": 0.02},
|
||||
},
|
||||
grid_rows=5,
|
||||
grid_cols=5,
|
||||
)
|
||||
|
||||
TERRAIN_SPECS: dict[str, TerrainSpec] = {STANDARD_V1.name: STANDARD_V1}
|
||||
|
||||
|
||||
def get_terrain_spec(name: str) -> TerrainSpec:
|
||||
if name not in TERRAIN_SPECS:
|
||||
raise KeyError(f"未知地形规格 {name!r}；可选：{sorted(TERRAIN_SPECS)}")
|
||||
return TERRAIN_SPECS[name]
|
||||
@@ -0,0 +1,18 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""IsaacLab 适配层：唯一接触仿真环境内部的地方。
|
||||
|
||||
这一层**依赖** isaaclab / isaacsim，因此必须在 ``AppLauncher`` 启动之后才能 import。
|
||||
上面的 :mod:`eval_kit.core` 与 :mod:`eval_kit.profiles` 不依赖它，可以单独使用。
|
||||
|
||||
职责边界：
|
||||
|
||||
- :mod:`eval_kit.isaaclab.env_setup` —— 把协议与地形规格翻译成 env_cfg 的改动
|
||||
- :mod:`eval_kit.isaaclab.recorder` —— 跑 rollout 并采集成 :class:`~eval_kit.core.schema.Trace`
|
||||
- :mod:`eval_kit.isaaclab.policy` —— checkpoint 解析与策略加载
|
||||
|
||||
这里没有任何指标计算逻辑，指标全在内核里。
|
||||
"""
|
||||
|
||||
__all__ = ["env_setup", "policy", "recorder"]
|
||||
@@ -0,0 +1,307 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""把评估协议与地形规格翻译成 IsaacLab 的 ``env_cfg`` 改动。
|
||||
|
||||
这些函数对应旧 ``eval.py`` 里的 ``apply_eval_terrain_baseline`` /
|
||||
``configure_eval_env`` / ``configure_upper_limit_env`` / ``assign_upper_limit_origins``，
|
||||
区别是：
|
||||
|
||||
- 数值全部来自 :class:`~eval_kit.core.terrain_spec.TerrainSpec` 与
|
||||
:class:`~eval_kit.core.schema.EvalProtocol`，没有散落的常数
|
||||
- 需要关掉的随机化事件分「通用」与「机器人相关」两组，机器人相关的那组用
|
||||
``hasattr`` 守卫，因此换机器人不会因为字段不存在而报错
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
from typing import Any, Sequence
|
||||
|
||||
import torch
|
||||
|
||||
from isaaclab.terrains.config.rough import ROUGH_TERRAINS_CFG
|
||||
|
||||
from eval_kit.core.schema import EvalProtocol
|
||||
from eval_kit.core.terrain_spec import TerrainSpec
|
||||
|
||||
#: 所有任务都需要的通用随机化项。
|
||||
COMMON_RANDOMIZATION_TERMS: tuple[str, ...] = (
|
||||
"randomize_rigid_body_material",
|
||||
"randomize_rigid_body_mass",
|
||||
"randomize_rigid_body_mass_base",
|
||||
"randomize_rigid_body_inertia",
|
||||
"randomize_com_positions",
|
||||
"randomize_apply_external_force_torque",
|
||||
"randomize_actuator_gains",
|
||||
)
|
||||
|
||||
#: 火蚁专属的随机化项（机身/膝部与轮胎分开做材质随机化）。
|
||||
ROBOT_SPECIFIC_RANDOMIZATION_TERMS: tuple[str, ...] = (
|
||||
"randomize_wheel_material",
|
||||
"randomize_support_material",
|
||||
)
|
||||
|
||||
|
||||
def set_fixed_command(env_cfg: Any, protocol: EvalProtocol) -> None:
|
||||
"""把速度指令钉成一个恒定值，并在整个 episode 内不重采样。"""
|
||||
command_cfg = env_cfg.commands.base_velocity
|
||||
command_x, command_y, command_yaw = protocol.command
|
||||
command_cfg.heading_command = False
|
||||
command_cfg.resampling_time_range = (
|
||||
protocol.steps * protocol.dt + 0.01,
|
||||
protocol.steps * protocol.dt + 0.01,
|
||||
)
|
||||
command_cfg.ranges.lin_vel_x = (command_x, command_x)
|
||||
command_cfg.ranges.lin_vel_y = (command_y, command_y)
|
||||
command_cfg.ranges.ang_vel_z = (command_yaw, command_yaw)
|
||||
command_cfg.debug_vis = False
|
||||
|
||||
|
||||
def apply_eval_terrain_baseline(env_cfg: Any, spec: TerrainSpec, protocol: EvalProtocol) -> None:
|
||||
"""所有任务统一用同一套评估地形，保证不同机器人/不同实验面对的是同一块地。"""
|
||||
if env_cfg.scene.terrain.terrain_type == "plane":
|
||||
env_cfg.scene.terrain.terrain_type = "generator"
|
||||
env_cfg.scene.terrain.terrain_generator = copy.deepcopy(ROUGH_TERRAINS_CFG)
|
||||
|
||||
generator = env_cfg.scene.terrain.terrain_generator
|
||||
generator.size = (protocol.terrain_size_m, protocol.terrain_size_m)
|
||||
generator.border_width = 2.0
|
||||
generator.curriculum = False
|
||||
generator.num_rows = spec.grid_rows
|
||||
generator.num_cols = spec.grid_cols
|
||||
generator.use_cache = False
|
||||
env_cfg.scene.terrain.max_init_terrain_level = None
|
||||
|
||||
# 瓦片变小的时候，台阶宽度/平台宽度等特征也要跟着收，否则一块地里塞不下
|
||||
for sub_cfg in generator.sub_terrains.values():
|
||||
for field_name in spec.feature_scaling:
|
||||
if hasattr(sub_cfg, field_name):
|
||||
setattr(sub_cfg, field_name, spec.scaling_for(field_name, protocol.terrain_size_m))
|
||||
|
||||
|
||||
def _build_push_event(protocol: EvalProtocol):
|
||||
"""构造周期性扰动事件。"""
|
||||
from isaaclab.managers import EventTermCfg as EventTerm
|
||||
|
||||
try:
|
||||
# 与旧 eval.py 同源，保证扰动行为完全一致
|
||||
import rl_training.tasks.manager_based.locomotion.velocity.mdp as mdp
|
||||
except ImportError:
|
||||
from isaaclab_tasks.manager_based.locomotion.velocity import mdp
|
||||
|
||||
return EventTerm(
|
||||
func=mdp.push_by_setting_velocity,
|
||||
mode="interval",
|
||||
interval_range_s=(protocol.push_interval_s[0], protocol.push_interval_s[1]),
|
||||
params={
|
||||
"velocity_range": {
|
||||
"x": protocol.push_velocity_range,
|
||||
"y": protocol.push_velocity_range,
|
||||
}
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
def _configure_viewer(env_cfg: Any, viewer_env_index: int | None, free_camera: bool) -> None:
|
||||
"""设置视口相机：跟随某个机器人，或交给客户端自己控制。
|
||||
|
||||
``ViewportCameraController`` 只在 ``origin_type`` 是 ``asset_root`` /
|
||||
``asset_body`` 时每帧改写相机位姿；设成 ``world`` 只在初始化时写一次，
|
||||
之后客户端的操作不会被抢回去 —— 这就是「自由视角」。起始视角由子进程按
|
||||
实际地形范围另设（见 ``eval_suite._set_free_camera_overview``）。
|
||||
"""
|
||||
viewer = getattr(env_cfg, "viewer", None)
|
||||
if viewer is None:
|
||||
return
|
||||
if free_camera:
|
||||
viewer.origin_type = "world"
|
||||
# 仅作初始化时的临时值，正式视角在环境建好后按地形范围重设
|
||||
viewer.eye = (0.0, -20.0, 15.0)
|
||||
viewer.lookat = (0.0, 0.0, 0.0)
|
||||
return
|
||||
if viewer_env_index is None:
|
||||
return
|
||||
viewer.origin_type = "asset_root"
|
||||
viewer.asset_name = "robot"
|
||||
viewer.env_index = int(viewer_env_index)
|
||||
# 相对机器人的偏移：斜前方 2.5 m、抬高 1.6 m，看向机身略前方
|
||||
viewer.eye = (2.5, -2.5, 1.6)
|
||||
viewer.lookat = (0.4, 0.0, 0.1)
|
||||
|
||||
|
||||
def configure_eval_env(
|
||||
env_cfg: Any,
|
||||
spec: TerrainSpec,
|
||||
protocol: EvalProtocol,
|
||||
*,
|
||||
num_envs: int | None = None,
|
||||
device: str | None = None,
|
||||
seed: int | None = None,
|
||||
viewer_env_index: int | None = None,
|
||||
free_camera: bool = False,
|
||||
) -> None:
|
||||
"""把训练用配置改造成单次评估用配置。
|
||||
|
||||
关掉观测噪声、课程、随机化，把重置位姿钉死，指令固定，地形统一。
|
||||
|
||||
``viewer_env_index`` 不为 None 时，把视口相机改成跟随该环境的机器人；
|
||||
``free_camera`` 为 True 时改成不跟随任何物体、由客户端自己移动（两者互斥，
|
||||
前者优先）。这两种都只在有画面时（本地 GUI 或推流）才有意义 —— headless
|
||||
且不推流时 IsaacLab 根本不创建相机控制器，设了也无害。
|
||||
"""
|
||||
env_cfg.scene.num_envs = num_envs if num_envs is not None else env_cfg.scene.num_envs
|
||||
env_cfg.episode_length_s = protocol.steps * protocol.dt
|
||||
if seed is not None:
|
||||
env_cfg.seed = seed
|
||||
if device is not None:
|
||||
env_cfg.sim.device = device
|
||||
|
||||
_configure_viewer(env_cfg, viewer_env_index, free_camera)
|
||||
|
||||
env_cfg.observations.policy.enable_corruption = False
|
||||
set_fixed_command(env_cfg, protocol)
|
||||
apply_eval_terrain_baseline(env_cfg, spec, protocol)
|
||||
|
||||
# 课程必须关掉，否则难度/指令会在评估过程中自己变
|
||||
for term in ("terrain_levels", "command_levels"):
|
||||
if getattr(env_cfg.curriculum, term, None) is not None:
|
||||
setattr(env_cfg.curriculum, term, None)
|
||||
|
||||
for term in (*COMMON_RANDOMIZATION_TERMS, *ROBOT_SPECIFIC_RANDOMIZATION_TERMS):
|
||||
if hasattr(env_cfg.events, term):
|
||||
setattr(env_cfg.events, term, None)
|
||||
|
||||
# 重置位姿钉死：同一批机器人从同一个位置、同一姿态、零速度出发
|
||||
reset_base = getattr(env_cfg.events, "randomize_reset_base", None)
|
||||
if reset_base is not None:
|
||||
start_x, start_y, start_yaw = protocol.start_pose
|
||||
# z 是相对地形瓦片中心高度的抬升。默认 0.0 与旧 eval.py 一致，但只对
|
||||
# 「瓦片中心最高」的地形成立；倒金字塔台阶这类中心最低的地形需要正值，
|
||||
# 否则机器人在 |x|>platform 处会被生成到地形几何体内部而卡住。
|
||||
reset_base.params = {
|
||||
"pose_range": {
|
||||
"x": (start_x, start_x),
|
||||
"y": (start_y, start_y),
|
||||
"z": (protocol.start_z_offset_m, protocol.start_z_offset_m),
|
||||
"roll": (0.0, 0.0),
|
||||
"pitch": (0.0, 0.0),
|
||||
"yaw": (start_yaw, start_yaw),
|
||||
},
|
||||
"velocity_range": {axis: (0.0, 0.0) for axis in ("x", "y", "z", "roll", "pitch", "yaw")},
|
||||
}
|
||||
|
||||
if hasattr(env_cfg.events, "randomize_push_robot"):
|
||||
env_cfg.events.randomize_push_robot = _build_push_event(protocol) if protocol.enable_interval_pushes else None
|
||||
|
||||
# 越界终止项：它比较的是世界坐标与整张地图的范围，不是机器人自己那块地砖，
|
||||
# 小地砖 + 大网格时会把最外侧格子判成越界（详见 EvalProtocol 的说明）。
|
||||
if not protocol.enforce_terrain_bounds and getattr(env_cfg.terminations, "terrain_out_of_bounds", None) is not None:
|
||||
env_cfg.terminations.terrain_out_of_bounds = None
|
||||
|
||||
|
||||
# 说明：判定地砖是否够大的逻辑已移到 eval_kit.core.schema.EvalProtocol.
|
||||
# minimum_terrain_size_for_bounds()。它只用到协议自己的 start_pose，放在适配层会
|
||||
# 让不启动仿真的编排进程被迫 import isaaclab（进而因缺 pxr 而失败）。
|
||||
|
||||
|
||||
def configure_upper_limit_env(
|
||||
env_cfg: Any,
|
||||
spec: TerrainSpec,
|
||||
protocol: EvalProtocol,
|
||||
terrain_types: Sequence[str],
|
||||
*,
|
||||
seed: int | None = None,
|
||||
device: str | None = None,
|
||||
viewer_env_index: int | None = None,
|
||||
free_camera: bool = False,
|
||||
) -> None:
|
||||
"""配置上限评估的固定网格：行 = 难度等级，列 = 地形类型。
|
||||
|
||||
``seed`` 同时决定地形生成与重置采样。显式传入是为了让地形本身在不同种子间
|
||||
保持一致（同一块地），而机器人初始状态可以随种子变化。
|
||||
|
||||
``viewer_env_index`` / ``free_camera`` 透传给 :func:`configure_eval_env`，
|
||||
用于让视口相机跟随某个环境的机器人，或者干脆交给客户端自由控制。
|
||||
"""
|
||||
configure_eval_env(
|
||||
env_cfg, spec, protocol, seed=seed, device=device,
|
||||
viewer_env_index=viewer_env_index, free_camera=free_camera,
|
||||
)
|
||||
|
||||
generator = env_cfg.scene.terrain.terrain_generator
|
||||
generator.curriculum = True
|
||||
generator.num_rows = protocol.num_levels
|
||||
generator.num_cols = len(terrain_types)
|
||||
generator.difficulty_range = (0.0, 1.0)
|
||||
if hasattr(generator, "seed"):
|
||||
generator.seed = int(seed) if seed is not None else int(env_cfg.seed or 0)
|
||||
|
||||
# 每个格子里放 envs_per_cell 个机器人
|
||||
env_cfg.scene.num_envs = protocol.num_levels * len(terrain_types) * protocol.envs_per_cell
|
||||
env_cfg.scene.terrain.max_init_terrain_level = protocol.num_levels - 1
|
||||
|
||||
# 只保留这次评估要用到的子地形，且按被选中的次数分配占比
|
||||
isaaclab_names = [spec.isaaclab_subterrain[terrain_type] for terrain_type in terrain_types]
|
||||
original = generator.sub_terrains
|
||||
generator.sub_terrains = {name: copy.deepcopy(original[name]) for name in isaaclab_names}
|
||||
for name, sub_cfg in generator.sub_terrains.items():
|
||||
sub_cfg.proportion = isaaclab_names.count(name) / len(isaaclab_names)
|
||||
|
||||
# 把等级表里的首尾值写进子地形的难度区间。
|
||||
# 用 _set_sub_terrain_field 而不是裸 setattr：字段名写错要立刻报错，
|
||||
# 静默造一个假属性会让地形保持默认难度，而结果看起来「只是很难」。
|
||||
for terrain_type in terrain_types:
|
||||
sub_cfg = generator.sub_terrains[spec.isaaclab_subterrain[terrain_type]]
|
||||
levels = spec.levels_for(terrain_type, protocol.num_levels)
|
||||
_set_sub_terrain_field(sub_cfg, spec.isaaclab_param[terrain_type], (levels[0], levels[-1]), terrain_type)
|
||||
|
||||
# 与等级无关的固定字段（例如粗糙地形的噪声步长）
|
||||
for terrain_type, extra in spec.isaaclab_extra_params.items():
|
||||
if terrain_type not in terrain_types:
|
||||
continue
|
||||
sub_cfg = generator.sub_terrains[spec.isaaclab_subterrain[terrain_type]]
|
||||
for field_name, value in extra.items():
|
||||
_set_sub_terrain_field(sub_cfg, field_name, value, terrain_type)
|
||||
|
||||
|
||||
def _set_sub_terrain_field(sub_cfg: Any, field_name: str, value: Any, terrain_type: str) -> None:
|
||||
"""写入子地形字段，字段不存在时报错而不是静默创建一个假属性。"""
|
||||
if not hasattr(sub_cfg, field_name):
|
||||
raise AttributeError(
|
||||
f"地形规格为 {terrain_type!r} 指定的字段 {field_name!r} 在 "
|
||||
f"{type(sub_cfg).__name__} 上不存在。裸 setattr 会静默造一个假属性、"
|
||||
f"使地形保持默认难度，而那看起来只是「很难」。可用字段："
|
||||
f"{sorted(k for k in vars(sub_cfg) if not k.startswith('_'))}"
|
||||
)
|
||||
setattr(sub_cfg, field_name, value)
|
||||
|
||||
|
||||
def assign_upper_limit_origins(
|
||||
env: Any,
|
||||
spec: TerrainSpec,
|
||||
protocol: EvalProtocol,
|
||||
terrain_types: Sequence[str],
|
||||
) -> tuple[torch.Tensor, torch.Tensor]:
|
||||
"""给每个格子平均分配机器人，并直接把地形等级/类型/起点写进地形对象。
|
||||
|
||||
返回 ``(level_ids, type_ids)``，用于事后把逐环境结果折叠成矩阵。
|
||||
"""
|
||||
base_env = env.unwrapped
|
||||
terrain = base_env.scene.terrain
|
||||
device = base_env.device
|
||||
|
||||
levels: list[int] = []
|
||||
types: list[int] = []
|
||||
for type_index in range(len(terrain_types)):
|
||||
for level_index in range(protocol.num_levels):
|
||||
levels.extend([level_index] * protocol.envs_per_cell)
|
||||
types.extend([type_index] * protocol.envs_per_cell)
|
||||
|
||||
level_ids = torch.tensor(levels, dtype=torch.long, device=device)
|
||||
type_ids = torch.tensor(types, dtype=torch.long, device=device)
|
||||
terrain.terrain_levels = level_ids
|
||||
terrain.terrain_types = type_ids
|
||||
terrain.env_origins = terrain.terrain_origins[level_ids, type_ids]
|
||||
env.reset()
|
||||
return level_ids, type_ids
|
||||
@@ -0,0 +1,71 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""checkpoint 解析与策略加载（IsaacLab + RSL-RL）。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
from typing import Any
|
||||
|
||||
import gymnasium as gym
|
||||
|
||||
from isaaclab.envs import (
|
||||
DirectMARLEnv,
|
||||
DirectMARLEnvCfg,
|
||||
DirectRLEnvCfg,
|
||||
ManagerBasedRLEnvCfg,
|
||||
multi_agent_to_single_agent,
|
||||
)
|
||||
from isaaclab_tasks.utils import get_checkpoint_path
|
||||
from isaaclab_rl.rsl_rl import RslRlVecEnvWrapper
|
||||
|
||||
|
||||
def resolve_checkpoint(
|
||||
log_root_path: str,
|
||||
load_run: str | None = None,
|
||||
checkpoint: str | None = None,
|
||||
) -> str:
|
||||
"""解析出 checkpoint 的绝对路径。
|
||||
|
||||
允许 ``load_run`` / ``checkpoint`` 传绝对路径或带目录的相对路径，从而跨实验目录
|
||||
取权重（例如从 Stage 1 的目录里取权重来评估），否则回退到 IsaacLab 的按名查找。
|
||||
"""
|
||||
if checkpoint is not None:
|
||||
checkpoint_path = os.path.expanduser(checkpoint)
|
||||
if os.path.isabs(checkpoint_path) or os.path.dirname(checkpoint_path):
|
||||
checkpoint_path = os.path.abspath(checkpoint_path)
|
||||
if not os.path.isfile(checkpoint_path):
|
||||
raise FileNotFoundError(f"找不到 checkpoint：{checkpoint_path}")
|
||||
return checkpoint_path
|
||||
|
||||
if load_run is not None:
|
||||
run_path = os.path.expanduser(load_run)
|
||||
if os.path.isabs(run_path) or os.path.dirname(run_path):
|
||||
run_path = os.path.abspath(run_path)
|
||||
if not os.path.isdir(run_path):
|
||||
raise FileNotFoundError(f"找不到运行目录：{run_path}")
|
||||
return get_checkpoint_path(os.path.dirname(run_path), os.path.basename(run_path), checkpoint)
|
||||
|
||||
return get_checkpoint_path(log_root_path, load_run, checkpoint)
|
||||
|
||||
|
||||
def make_env(
|
||||
task: str,
|
||||
env_cfg: ManagerBasedRLEnvCfg | DirectRLEnvCfg | DirectMARLEnvCfg,
|
||||
clip_actions: float | None = None,
|
||||
):
|
||||
"""按已配置好的 ``env_cfg`` 建环境，并包成 RSL-RL 的向量环境。"""
|
||||
env = gym.make(task, cfg=env_cfg)
|
||||
if isinstance(env.unwrapped, DirectMARLEnv):
|
||||
env = multi_agent_to_single_agent(env)
|
||||
return RslRlVecEnvWrapper(env, clip_actions=clip_actions)
|
||||
|
||||
|
||||
def load_policy(env, agent_cfg: Any, resume_path: str, device: str | None = None):
|
||||
"""加载 checkpoint 并返回推理策略。"""
|
||||
from rsl_rl.runners import OnPolicyRunner
|
||||
|
||||
runner = OnPolicyRunner(env, agent_cfg.to_dict(), log_dir=None, device=device or agent_cfg.device)
|
||||
runner.load(resume_path)
|
||||
return runner.get_inference_policy(device=device or env.unwrapped.device)
|
||||
@@ -0,0 +1,257 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""跑 rollout 并把仿真信号采集成框架无关的 :class:`~eval_kit.core.schema.Trace`。
|
||||
|
||||
这是唯一读取 ``env`` 内部状态的地方。采集完之后环境就可以关掉了，后续所有
|
||||
指标计算都在内核里对纯数组进行。
|
||||
|
||||
形态相关的下标（谁是轮、谁是支撑体、哪些关节耗电）全部由
|
||||
:class:`~eval_kit.core.schema.RobotProfile` 的正则解析得到，代码里没有硬编码的
|
||||
link/joint 名字。解析不到时记录警告并跳过对应指标，而不是抛异常。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from eval_kit.core.naming import resolve_matching_indices
|
||||
from eval_kit.core.schema import EvalProtocol, RobotProfile, Trace
|
||||
|
||||
#: 默认的接触力传感器名字，与 ``velocity_env_cfg.py`` 的场景定义一致。
|
||||
DEFAULT_CONTACT_SENSOR = "contact_forces"
|
||||
|
||||
|
||||
@dataclass
|
||||
class RolloutRecorder:
|
||||
"""一次评估里固定不变的东西：形态下标与质量。建一次，跑多次。"""
|
||||
|
||||
feet_ids: list[int] = field(default_factory=list)
|
||||
feet_names: list[str] = field(default_factory=list)
|
||||
support_ids: list[int] = field(default_factory=list)
|
||||
support_names: list[str] = field(default_factory=list)
|
||||
joint_ids: list[int] = field(default_factory=list)
|
||||
joint_names: list[str] = field(default_factory=list)
|
||||
mass_kg: float | None = None
|
||||
contact_sensor_name: str = DEFAULT_CONTACT_SENSOR
|
||||
warnings: list[str] = field(default_factory=list)
|
||||
|
||||
def describe(self) -> str:
|
||||
lines = [
|
||||
f"足端 {len(self.feet_ids)} 个: {self.feet_names or '（无）'}",
|
||||
f"支撑体 {len(self.support_ids)} 个: {self.support_names or '（无）'}",
|
||||
f"驱动关节 {len(self.joint_ids)} 个: {self.joint_names or '（无）'}",
|
||||
f"整机质量: {'未知' if self.mass_kg is None else f'{self.mass_kg:.3f} kg'}",
|
||||
]
|
||||
for warning in self.warnings:
|
||||
lines.append(f"[警告] {warning}")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def read_total_mass(robot: Any) -> float | None:
|
||||
"""从仿真资产读整机质量。
|
||||
|
||||
优先读 PhysX 运行时的真实质量（会反映任何质量随机化），读不到再退回资产默认质量。
|
||||
"""
|
||||
try:
|
||||
masses = robot.root_physx_view.get_masses()
|
||||
return float(masses[0].sum().item())
|
||||
except Exception: # noqa: BLE001 - 不同版本 API 差异，读不到就退回
|
||||
pass
|
||||
try:
|
||||
return float(robot.data.default_mass[0].sum().item())
|
||||
except Exception: # noqa: BLE001
|
||||
return None
|
||||
|
||||
|
||||
def build_recorder(
|
||||
env: Any,
|
||||
profile: RobotProfile,
|
||||
*,
|
||||
contact_sensor_name: str = DEFAULT_CONTACT_SENSOR,
|
||||
) -> RolloutRecorder:
|
||||
"""按机器人描述解析出足端/支撑体/关节下标，并读一次质量。"""
|
||||
base_env = env.unwrapped
|
||||
robot = base_env.scene["robot"]
|
||||
recorder = RolloutRecorder(contact_sensor_name=contact_sensor_name)
|
||||
|
||||
feet = resolve_matching_indices(profile.feet_body_patterns, robot.body_names)
|
||||
recorder.feet_ids = feet.indices
|
||||
recorder.feet_names = feet.names
|
||||
for pattern in feet.unmatched_patterns:
|
||||
recorder.warnings.append(
|
||||
f"机器人 {profile.name!r} 的足端正则 {pattern!r} 在 body_names 里没有命中，"
|
||||
"支撑裕度指标将被跳过。"
|
||||
)
|
||||
|
||||
robot_bodies = set(robot.body_names)
|
||||
sensor = base_env.scene.sensors.get(contact_sensor_name) if base_env.scene.sensors else None
|
||||
if sensor is None:
|
||||
recorder.warnings.append(
|
||||
f"场景里没有接触力传感器 {contact_sensor_name!r}，支撑接触类指标将被跳过。"
|
||||
)
|
||||
else:
|
||||
support = resolve_matching_indices(profile.support_body_patterns, sensor.body_names)
|
||||
recorder.support_ids = support.indices
|
||||
recorder.support_names = support.names
|
||||
missing = [name for name in support.names if name not in robot_bodies]
|
||||
if missing:
|
||||
recorder.warnings.append(
|
||||
f"接触力传感器上的 {missing} 不在机器人的 body 列表里，指标可能对不上。"
|
||||
)
|
||||
for pattern in support.unmatched_patterns:
|
||||
if pattern in profile.support_body_patterns:
|
||||
recorder.warnings.append(
|
||||
f"机器人 {profile.name!r} 的支撑体正则 {pattern!r} 没有命中，"
|
||||
"支撑接触类指标将被跳过。"
|
||||
)
|
||||
|
||||
joints = resolve_matching_indices(profile.actuated_joint_patterns, robot.joint_names)
|
||||
recorder.joint_ids = joints.indices
|
||||
recorder.joint_names = joints.names
|
||||
if not joints.indices:
|
||||
recorder.warnings.append(
|
||||
"驱动关节正则一个都没命中，能耗与 COT 将无法计算（会返回 nan）。"
|
||||
)
|
||||
|
||||
recorder.mass_kg = read_total_mass(robot)
|
||||
if recorder.mass_kg is None and profile.mass_kg is None:
|
||||
recorder.warnings.append("既读不到资产质量、描述里也没写兜底值，COT 将返回 nan。")
|
||||
|
||||
return recorder
|
||||
|
||||
|
||||
class _FrameCollector:
|
||||
"""逐帧收集张量，最后一次性转成 numpy——每帧都转会明显变慢。"""
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.frames: dict[str, list[np.ndarray]] = {}
|
||||
|
||||
def add(self, **signals: np.ndarray) -> None:
|
||||
for key, value in signals.items():
|
||||
self.frames.setdefault(key, []).append(value)
|
||||
|
||||
def stack(self) -> dict[str, np.ndarray]:
|
||||
return {key: np.stack(values, axis=0) for key, values in self.frames.items()}
|
||||
|
||||
|
||||
def _capture(robot: Any, sensor: Any, recorder: RolloutRecorder, num_envs: int) -> dict[str, np.ndarray]:
|
||||
"""读一帧信号。所有张量都要 clone，否则会被后续仿真步就地覆写。"""
|
||||
|
||||
def to_numpy(tensor: torch.Tensor) -> np.ndarray:
|
||||
return tensor.detach().to("cpu", torch.float64).numpy()
|
||||
|
||||
signals = {
|
||||
"base_pos_w": to_numpy(robot.data.root_pos_w.clone()),
|
||||
"base_lin_vel_w": to_numpy(robot.data.root_lin_vel_w.clone()),
|
||||
"base_lin_vel_b": to_numpy(robot.data.root_lin_vel_b.clone()),
|
||||
"base_ang_vel_b": to_numpy(robot.data.root_ang_vel_b.clone()),
|
||||
"projected_gravity_b": to_numpy(robot.data.projected_gravity_b.clone()),
|
||||
"joint_torque": to_numpy(robot.data.applied_torque[:, recorder.joint_ids].clone())
|
||||
if recorder.joint_ids
|
||||
else np.zeros((num_envs, 0)),
|
||||
"joint_vel": to_numpy(robot.data.joint_vel[:, recorder.joint_ids].clone())
|
||||
if recorder.joint_ids
|
||||
else np.zeros((num_envs, 0)),
|
||||
"feet_pos_w": to_numpy(robot.data.body_pos_w[:, recorder.feet_ids, :2].clone())
|
||||
if recorder.feet_ids
|
||||
else np.zeros((num_envs, 0, 2)),
|
||||
}
|
||||
|
||||
if sensor is not None and recorder.support_ids:
|
||||
forces = sensor.data.net_forces_w
|
||||
contact_time = sensor.data.current_contact_time
|
||||
if forces is None:
|
||||
# 传感器还没更新过（第一帧），按无接触处理
|
||||
signals["support_force_n"] = np.zeros((num_envs, len(recorder.support_ids)))
|
||||
signals["support_contact_time_s"] = np.zeros((num_envs, len(recorder.support_ids)))
|
||||
else:
|
||||
selected = forces[:, recorder.support_ids, :]
|
||||
signals["support_force_n"] = to_numpy(torch.linalg.norm(selected, dim=-1).clone())
|
||||
signals["support_contact_time_s"] = to_numpy(
|
||||
(contact_time[:, recorder.support_ids].clone() if contact_time is not None else torch.zeros_like(selected[..., 0]))
|
||||
)
|
||||
else:
|
||||
signals["support_force_n"] = np.zeros((num_envs, 0))
|
||||
signals["support_contact_time_s"] = np.zeros((num_envs, 0))
|
||||
|
||||
return signals
|
||||
|
||||
|
||||
def record_rollout(
|
||||
env: Any,
|
||||
policy: Any,
|
||||
protocol: EvalProtocol,
|
||||
profile: RobotProfile,
|
||||
recorder: RolloutRecorder | None = None,
|
||||
) -> Trace:
|
||||
"""跑 ``protocol.steps`` 步，采集一条 :class:`Trace`。
|
||||
|
||||
第 0 帧记录动作执行之前的起始状态，之后每步记录一次动作之后的状态，
|
||||
因此帧数比步数多一。
|
||||
"""
|
||||
base_env = env.unwrapped
|
||||
robot = base_env.scene["robot"]
|
||||
if recorder is None:
|
||||
recorder = build_recorder(env, profile)
|
||||
sensors = base_env.scene.sensors
|
||||
sensor = sensors.get(recorder.contact_sensor_name) if sensors else None
|
||||
|
||||
num_envs = int(base_env.num_envs)
|
||||
collector = _FrameCollector()
|
||||
initial = _capture(robot, sensor, recorder, num_envs)
|
||||
# 起始帧还没有任何 episode 结束，置零以保证与后续帧形状一致
|
||||
initial["done"] = np.zeros(num_envs, dtype=np.float64)
|
||||
collector.add(**initial)
|
||||
|
||||
obs = env.get_observations()
|
||||
# 终端侧进度：可视化时推流画面可能因为渲染/编码出问题而冻住，这时只有终端能
|
||||
# 说明仿真到底有没有在推进（推进则打印继续，冻结则这里也停住）。每 10% 打一行。
|
||||
progress_every = max(1, protocol.steps // 10)
|
||||
started = time.monotonic()
|
||||
for step in range(protocol.steps):
|
||||
with torch.inference_mode():
|
||||
actions = policy(obs)
|
||||
if isinstance(actions, tuple):
|
||||
actions = actions[0]
|
||||
obs, _, dones, _ = env.step(actions)
|
||||
frame = _capture(robot, sensor, recorder, num_envs)
|
||||
frame["done"] = dones.detach().to("cpu", torch.float64).numpy().reshape(num_envs)
|
||||
collector.add(**frame)
|
||||
if (step + 1) % progress_every == 0 or step + 1 == protocol.steps:
|
||||
print(
|
||||
f"[eval_kit] rollout {step + 1}/{protocol.steps} 步，用时 {time.monotonic() - started:.1f} s",
|
||||
flush=True,
|
||||
)
|
||||
|
||||
arrays = collector.stack()
|
||||
return Trace(
|
||||
dt=float(base_env.step_dt),
|
||||
command=np.asarray(protocol.command, dtype=np.float64),
|
||||
mass_kg=recorder.mass_kg,
|
||||
**arrays,
|
||||
)
|
||||
|
||||
|
||||
def summarize_early_resets(trace: Trace) -> dict[str, Any]:
|
||||
"""统计 episode 中途被重置的环境数量。
|
||||
|
||||
中途重置会让后续帧来自新 episode，路程/进度类指标会被污染。旧
|
||||
``eval.py`` 没有检查这一点，所以出现过「指标看着正常其实混了两次 rollout」
|
||||
的情况不容易发现。
|
||||
"""
|
||||
# 最后一帧的 done 属于 episode 正常结束，不算早退
|
||||
early = trace.done[:-1] > 0.5
|
||||
per_env = early.sum(axis=0)
|
||||
affected = int(np.count_nonzero(per_env > 0))
|
||||
return {
|
||||
"envs_with_early_reset": affected,
|
||||
"total_early_resets": int(per_env.sum()),
|
||||
"max_resets_single_env": int(per_env.max()) if per_env.size else 0,
|
||||
"polluted": affected > 0,
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""机器人形态注册表。
|
||||
|
||||
**加一个机器人 = 在这里加一条记录**，评价代码不动。
|
||||
|
||||
数值来源：火蚁的 link/joint 取自
|
||||
``source/rl_training/rl_training/tasks/manager_based/locomotion/velocity/config/wheeled/deeprobotics_m20/rough_env_cfg.py``；
|
||||
机身尺寸取自 ``docs/solenopsis_urdf_comparison.md`` 里记录的碰撞体尺寸。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from eval_kit.core.schema import RobotProfile
|
||||
|
||||
SOLENOPSIS = RobotProfile(
|
||||
name="solenopsis",
|
||||
# None = 运行时从仿真资产读真实质量（Solenopsis.urdf 当前为 26.581 kg）。
|
||||
# 旧 eval.py 的命令行默认值是 25.482，那是更早的 M20 简化模型名义质量，
|
||||
# 会让 COT 偏低约 4%。
|
||||
mass_kg=None,
|
||||
# 四个轮就是支撑多边形的接触点。
|
||||
feet_body_patterns=(".*_wheel",),
|
||||
# 机身与膝部允许作为越障时的短暂支撑。
|
||||
support_body_patterns=("base_link", ".*_knee"),
|
||||
# 火蚁 8 个关节全部主动驱动；分两组写便于将来出现被动关节时收窄范围。
|
||||
actuated_joint_patterns=(".*knee_joint", ".*wheel_joint"),
|
||||
support_force_threshold_n=15.0,
|
||||
# 机身碰撞体 0.706 x 0.310 x 0.1714 m，取长边。
|
||||
body_length_m=0.706,
|
||||
)
|
||||
|
||||
#: 四足对照机，仅用于验证「换机器人只改数据」这一点，尚未在仿真里跑过。
|
||||
LITE3 = RobotProfile(
|
||||
name="lite3",
|
||||
mass_kg=None,
|
||||
feet_body_patterns=(".*_FOOT",),
|
||||
# 四足没有「机身支撑」这个语义，留空 → 支撑类指标返回 nan 而不是 0。
|
||||
support_body_patterns=(),
|
||||
actuated_joint_patterns=(".*_joint",),
|
||||
support_force_threshold_n=15.0,
|
||||
# 未知，仅用于地形瓦片尺寸校验。
|
||||
body_length_m=0.0,
|
||||
)
|
||||
|
||||
PROFILES: dict[str, RobotProfile] = {
|
||||
SOLENOPSIS.name: SOLENOPSIS,
|
||||
LITE3.name: LITE3,
|
||||
}
|
||||
|
||||
|
||||
def get_profile(name: str) -> RobotProfile:
|
||||
if name not in PROFILES:
|
||||
raise KeyError(f"未知机器人描述 {name!r}；可选：{sorted(PROFILES)}")
|
||||
return PROFILES[name]
|
||||
@@ -0,0 +1,102 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""评估协议预设。
|
||||
|
||||
协议就是一次评估的全部「口径」。默认预设 ``standard_v1`` 的数值与旧
|
||||
``scripts/reinforcement_learning/rsl_rl/eval.py`` 的命令行默认值一致，
|
||||
所以两套评估跑出来的数字可以直接对照。
|
||||
|
||||
要改口径，改预设或另加一个预设，不要在调用处临时传参——预设名会进
|
||||
``protocol_fingerprint``，写进输出 JSON。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import replace
|
||||
|
||||
from eval_kit.core.schema import EvalProtocol
|
||||
|
||||
#: 标准越障评估：10 s、前进 1 m/s、最多 2 m 算成功。与旧 eval.py 默认一致。
|
||||
#:
|
||||
#: **只建议用于与旧 ``eval.py`` 做对照**：它的地砖 2.0 m 会让越界终止项误判最外侧格子，
|
||||
#: 起始高度 0.0 m 会让机器人在倒金字塔台阶地形里被生成到几何体内部
|
||||
#: （``stairs_down`` 因此恒为 0）。要拿可用的数字请用 ``standard_v1_clean``。
|
||||
STANDARD_V1 = EvalProtocol(
|
||||
steps=500,
|
||||
dt=0.02,
|
||||
command=(1.0, 0.0, 0.0),
|
||||
start_pose=(-0.9, 0.0, 0.0),
|
||||
success_distance_m=2.0,
|
||||
tilt_threshold_rad=0.70,
|
||||
acc_threshold_g=2.0 / 9.81,
|
||||
energy_convention="absolute",
|
||||
stability_margin_method="foot_bbox",
|
||||
support_force_threshold_n=15.0,
|
||||
enable_interval_pushes=False,
|
||||
terrain_spec="standard_v1",
|
||||
terrain_size_m=2.0,
|
||||
num_levels=8,
|
||||
envs_per_cell=1,
|
||||
seeds=(0,),
|
||||
)
|
||||
|
||||
#: 抗扰评估：每 2.01 s 施加一次水平速度扰动，其余同 standard_v1。
|
||||
ROBUSTNESS_V1 = replace(
|
||||
STANDARD_V1,
|
||||
enable_interval_pushes=True,
|
||||
push_interval_s=(2.0, 2.01),
|
||||
push_velocity_range=(-0.5, 0.5),
|
||||
)
|
||||
|
||||
#: 冒烟测试：步数与网格都缩小，用于确认流水线能跑通，不要用它的数值下结论。
|
||||
#: 地形参数沿用 ``standard_v1_clean`` 的修正值，否则它会带上越界误判的假警告。
|
||||
SMOKE = replace(
|
||||
STANDARD_V1,
|
||||
steps=100,
|
||||
num_levels=2,
|
||||
envs_per_cell=1,
|
||||
terrain_size_m=4.0,
|
||||
start_z_offset_m=0.5,
|
||||
)
|
||||
|
||||
#: **推荐用于机器人对比**：修掉两处会让结果失真的问题，其余口径与 ``standard_v1`` 相同，
|
||||
#: 因此数值仍可与后者对照。
|
||||
#:
|
||||
#: 1. 地砖从 2.0 m 放大到 4.0 m，消除 IsaacLab ``terrain_out_of_bounds`` 的误判
|
||||
#: （阈值 3.8 m 由 ``EvalProtocol.minimum_terrain_size_for_bounds()`` 给出）。
|
||||
#: 2. 起始高度抬高 0.5 m，让机器人落到地形表面而不是被生成在地形内部。
|
||||
#: 实测证据：``--terrain-type stairs_down`` 单独跑，起始高度 0.0 m 时
|
||||
#: score=0.00 / 成功率 0.05，抬高后 score 显著恢复。取 0.5 是因为倒金字塔台阶的
|
||||
#: 嵌入深度随难度递增（L1 约 0.11 m、L8 约 0.50 m），0.5 m 刚好覆盖最深的一级。
|
||||
#: 旧 ``eval.py`` 用 0.0，所以它的倒金字塔台阶那一列恒为 0，是无效的。
|
||||
STANDARD_V1_CLEAN = replace(STANDARD_V1, terrain_size_m=4.0, start_z_offset_m=0.5)
|
||||
|
||||
#: 另一种处置方式：保留 2.0 m 地砖，直接去掉越界终止项。
|
||||
STANDARD_V1_NO_BOUNDS = replace(STANDARD_V1, enforce_terrain_bounds=False)
|
||||
|
||||
#: 多随机种子的标准评估 —— **对外给结论时用这个**。
|
||||
#:
|
||||
#: 单次 rollout 的噪声足以淹没机器人之间的差异，所以对比必须多种子；同时它基于
|
||||
#: ``STANDARD_V1_CLEAN``（地砖 4.0 m、起始高度 +0.5 m），不带已知的 harness 偏差。
|
||||
#:
|
||||
#: 默认 ``envs_per_cell = 1``（每格 1 个机器人），因为起始位姿钉死后同格多个机器人
|
||||
#: 起始坐标完全相同、不是独立样本。样本量靠种子数而非每格机器人数量来提升 ——
|
||||
#: 每个种子会重新生成地形实例，跨种子才是独立的。8 个种子给出每格 1/8 ≈ 12.5% 的
|
||||
#: 分辨率；想要更细就显式加 ``--seeds``。
|
||||
STANDARD_V1_MULTISEED = replace(STANDARD_V1_CLEAN, seeds=(0, 100, 200, 300, 400, 500, 600, 700))
|
||||
|
||||
PROTOCOLS: dict[str, EvalProtocol] = {
|
||||
"standard_v1": STANDARD_V1,
|
||||
"standard_v1_clean": STANDARD_V1_CLEAN,
|
||||
"standard_v1_no_bounds": STANDARD_V1_NO_BOUNDS,
|
||||
"robustness_v1": ROBUSTNESS_V1,
|
||||
"smoke": SMOKE,
|
||||
"standard_v1_multiseed": STANDARD_V1_MULTISEED,
|
||||
}
|
||||
|
||||
|
||||
def get_protocol(name: str) -> EvalProtocol:
|
||||
if name not in PROTOCOLS:
|
||||
raise KeyError(f"未知评估协议 {name!r}；可选：{sorted(PROTOCOLS)}")
|
||||
return PROTOCOLS[name]
|
||||
@@ -0,0 +1,544 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""命令行入口的静态自检：不需要 Isaac Sim，也不需要 GPU。
|
||||
|
||||
用桩替换 ``isaaclab.app``，从而可以单独加载 ``eval_suite.py``，检查参数解析与
|
||||
口径覆盖逻辑。仿真相关的部分（建环境、跑 rollout）不在覆盖范围内——那些必须真跑。
|
||||
|
||||
运行方式（在仓库根目录）::
|
||||
|
||||
python3 eval_kit/tests/test_cli.py
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import sys
|
||||
import types
|
||||
from pathlib import Path
|
||||
|
||||
if __package__ in (None, ""):
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
EVAL_SUITE_PATH = PROJECT_ROOT / "scripts" / "reinforcement_learning" / "rsl_rl" / "eval_suite.py"
|
||||
|
||||
#: ``AppLauncher.add_app_launcher_args()`` 会添加的参数名，抄自
|
||||
#: ``IsaacLab-2.3.2/source/isaaclab/isaaclab/app/app_launcher.py``。
|
||||
#: 自己的 parser 里出现任何一个都会让 argparse 抛
|
||||
#: ``ArgumentError: conflicting option string``，脚本连启动都做不到。
|
||||
#: （只有 device / headless / enable_cameras / experience / livestream /
|
||||
#: rendering_mode / xr 会被 AppLauncher 自己的显式检查拦下，--verbose / --info /
|
||||
#: --cpu 等则由 argparse 拦，所以必须把 14 个都列出来。）
|
||||
APPLAUNCHER_ARG_NAMES = (
|
||||
"--headless",
|
||||
"--livestream",
|
||||
"--enable_cameras",
|
||||
"--xr",
|
||||
"--device",
|
||||
"--cpu",
|
||||
"--verbose",
|
||||
"--info",
|
||||
"--experience",
|
||||
"--rendering_mode",
|
||||
"--kit_args",
|
||||
"--anim_recording_enabled",
|
||||
"--anim_recording_start_time",
|
||||
"--anim_recording_stop_time",
|
||||
)
|
||||
|
||||
|
||||
def _stub_isaaclab() -> None:
|
||||
"""装一个最小的 isaaclab.app 桩，只为让模块级 import 通过。
|
||||
|
||||
桩要复刻真实 ``AppLauncher`` 的行为，否则测试放过的正是真跑会崩的地方：
|
||||
|
||||
- ``add_app_launcher_args`` 会添加上面 14 个参数名
|
||||
- 如果 parser 里已经有同名参数，它**拒绝启动**而不是静默覆盖
|
||||
"""
|
||||
if "isaaclab.app" in sys.modules:
|
||||
return
|
||||
|
||||
class AppLauncher: # noqa: D401 - 桩
|
||||
def __init__(self, args):
|
||||
self.app = None
|
||||
|
||||
@staticmethod
|
||||
def add_app_launcher_args(parser):
|
||||
existing = {option for action in parser._actions for option in action.option_strings}
|
||||
conflict = sorted(existing & set(APPLAUNCHER_ARG_NAMES))
|
||||
if conflict:
|
||||
raise ValueError(
|
||||
f"argument {conflict[0]}: conflicting option string: {conflict[0]}"
|
||||
)
|
||||
parser.add_argument("--headless", action="store_true", default=False)
|
||||
parser.add_argument("--device", type=str, default=None)
|
||||
for name in APPLAUNCHER_ARG_NAMES:
|
||||
dest = name.lstrip("-").replace("-", "_")
|
||||
if dest in ("headless", "device"):
|
||||
continue
|
||||
parser.add_argument(name, dest=dest, nargs="?", const=True, default=None)
|
||||
return parser
|
||||
|
||||
isaaclab = types.ModuleType("isaaclab")
|
||||
app = types.ModuleType("isaaclab.app")
|
||||
app.AppLauncher = AppLauncher # type: ignore[attr-defined]
|
||||
isaaclab.app = app # type: ignore[attr-defined]
|
||||
sys.modules["isaaclab"] = isaaclab
|
||||
sys.modules["isaaclab.app"] = app
|
||||
|
||||
|
||||
def _load_eval_suite():
|
||||
_stub_isaaclab()
|
||||
specification = importlib.util.spec_from_file_location("eval_suite_under_test", EVAL_SUITE_PATH)
|
||||
if specification is None or specification.loader is None:
|
||||
raise RuntimeError(f"无法加载 {EVAL_SUITE_PATH}")
|
||||
module = importlib.util.module_from_spec(specification)
|
||||
specification.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def test_parser_defaults():
|
||||
"""默认值：协议预设名、机器人描述名、评估类型都在。"""
|
||||
module = _load_eval_suite()
|
||||
args = module._build_parser().parse_args([])
|
||||
|
||||
assert args.task == "Rough-Deeprobotics-M20-v0", args.task
|
||||
assert args.profile == "solenopsis"
|
||||
assert args.protocol == "standard_v1"
|
||||
assert args.eval_type == "all"
|
||||
assert args.num_envs == 250
|
||||
assert args.terrain_type == "all"
|
||||
# 口径覆盖项默认必须为 None，否则会静默覆盖协议预设
|
||||
for name in (
|
||||
"episode_length_s", "command_x", "command_y", "command_yaw", "start_x", "start_y",
|
||||
"terrain_size", "success_distance", "tilt_threshold", "acc_threshold_g",
|
||||
"energy_convention", "robot_mass", "num_levels", "envs_per_cell", "seeds",
|
||||
"enforce_terrain_bounds", "start_z_offset",
|
||||
):
|
||||
assert getattr(args, name) is None, f"{name} 默认应为 None，实际 {getattr(args, name)!r}"
|
||||
return "默认值正确，口径覆盖项默认全部为 None"
|
||||
|
||||
|
||||
def test_no_terrain_bounds_flag():
|
||||
"""--no-terrain-bounds 关闭越界终止项，并改变指纹。"""
|
||||
module = _load_eval_suite()
|
||||
from eval_kit.protocols import get_protocol
|
||||
|
||||
preset = get_protocol("standard_v1")
|
||||
assert preset.enforce_terrain_bounds is True, "standard_v1 应保持与 eval.py 一致（保留终止项）"
|
||||
|
||||
args = module._build_parser().parse_args(["--no-terrain-bounds"])
|
||||
protocol = module._apply_overrides(preset, args)
|
||||
assert protocol.enforce_terrain_bounds is False
|
||||
assert protocol.fingerprint() != preset.fingerprint(), "关闭终止项必须改变指纹"
|
||||
|
||||
# 不带该开关时不覆盖
|
||||
unchanged = module._apply_overrides(preset, module._build_parser().parse_args([]))
|
||||
assert unchanged.enforce_terrain_bounds is True
|
||||
return "开关生效且反映在指纹里"
|
||||
|
||||
|
||||
def test_clean_protocol_uses_big_tile():
|
||||
"""standard_v1_clean 通过放大地砖来消除越界误判，而不是关掉终止项。"""
|
||||
from eval_kit.protocols import get_protocol
|
||||
|
||||
clean = get_protocol("standard_v1_clean")
|
||||
parity = get_protocol("standard_v1")
|
||||
no_bounds = get_protocol("standard_v1_no_bounds")
|
||||
|
||||
assert clean.terrain_size_m >= 3.8, f"地砖 {clean.terrain_size_m} 不足以消除误判"
|
||||
assert clean.enforce_terrain_bounds is True, "clean 方案应保留终止项"
|
||||
assert no_bounds.enforce_terrain_bounds is False
|
||||
# 除地砖外其余口径与 parity 方案一致，数值仍可对照
|
||||
assert clean.command == parity.command
|
||||
assert clean.success_distance_m == parity.success_distance_m
|
||||
assert clean.num_levels == parity.num_levels
|
||||
return f"clean 地砖 {clean.terrain_size_m} m，保留终止项；no_bounds 关闭终止项"
|
||||
|
||||
|
||||
def test_overrides_are_opt_in():
|
||||
"""不传任何覆盖项时，协议必须与预设逐字段相同。"""
|
||||
module = _load_eval_suite()
|
||||
from eval_kit.protocols import get_protocol
|
||||
|
||||
args = module._build_parser().parse_args([])
|
||||
preset = get_protocol("standard_v1")
|
||||
protocol = module._apply_overrides(preset, args)
|
||||
|
||||
assert protocol == preset, "未传覆盖项却改变了协议"
|
||||
assert protocol.fingerprint() == preset.fingerprint()
|
||||
return "未传覆盖 → 协议与预设完全相同（含 fingerprint）"
|
||||
|
||||
|
||||
def test_overrides_apply_and_update_fingerprint():
|
||||
"""显式覆盖项生效，且指纹随之改变。"""
|
||||
module = _load_eval_suite()
|
||||
from eval_kit.protocols import get_protocol
|
||||
|
||||
args = module._build_parser().parse_args(
|
||||
[
|
||||
"--episode-length-s", "20",
|
||||
"--command-x", "0.8",
|
||||
"--command-yaw", "-0.3",
|
||||
"--success-distance", "3.5",
|
||||
"--tilt-threshold", "0.5",
|
||||
"--acc-threshold-g", "0.3",
|
||||
"--energy-convention", "signed",
|
||||
"--terrain-size", "1.5",
|
||||
"--num-levels", "4",
|
||||
"--envs-per-cell", "2",
|
||||
"--seeds", "7", "8",
|
||||
]
|
||||
)
|
||||
preset = get_protocol("standard_v1")
|
||||
protocol = module._apply_overrides(preset, args)
|
||||
|
||||
assert protocol.steps == 1000, protocol.steps # 20 s / 0.02 s
|
||||
assert protocol.command == (0.8, 0.0, -0.3), protocol.command
|
||||
assert protocol.success_distance_m == 3.5
|
||||
assert protocol.tilt_threshold_rad == 0.5
|
||||
assert abs(protocol.acc_threshold_g - 0.3) < 1e-12
|
||||
assert protocol.energy_convention == "signed"
|
||||
assert protocol.terrain_size_m == 1.5
|
||||
assert protocol.num_levels == 4
|
||||
assert protocol.envs_per_cell == 2
|
||||
assert protocol.seeds == (7, 8)
|
||||
assert protocol.fingerprint() != preset.fingerprint()
|
||||
# 没覆盖的字段保持预设
|
||||
assert protocol.start_pose == preset.start_pose
|
||||
assert protocol.dt == preset.dt
|
||||
return "11 项覆盖全部生效，未覆盖字段保持预设"
|
||||
|
||||
|
||||
def test_command_overrides_compose_independently():
|
||||
"""单独覆盖某一个速度分量时，另两个分量保持预设。"""
|
||||
module = _load_eval_suite()
|
||||
from eval_kit.protocols import get_protocol
|
||||
|
||||
preset = get_protocol("standard_v1") # command = (1.0, 0.0, 0.0)
|
||||
|
||||
only_y = module._apply_overrides(preset, module._build_parser().parse_args(["--command-y", "0.5"]))
|
||||
assert only_y.command == (1.0, 0.5, 0.0), only_y.command
|
||||
|
||||
only_yaw = module._apply_overrides(preset, module._build_parser().parse_args(["--command-yaw", "1.2"]))
|
||||
assert only_yaw.command == (1.0, 0.0, 1.2), only_yaw.command
|
||||
|
||||
both = module._apply_overrides(
|
||||
preset, module._build_parser().parse_args(["--command-y", "0.5", "--command-yaw", "1.2"])
|
||||
)
|
||||
assert both.command == (1.0, 0.5, 1.2), both.command
|
||||
|
||||
only_x = module._apply_overrides(preset, module._build_parser().parse_args(["--command-x", "2.0"]))
|
||||
assert only_x.command == (2.0, 0.0, 0.0), only_x.command
|
||||
return "command 三个分量可独立覆盖"
|
||||
|
||||
|
||||
def test_start_pose_overrides_compose():
|
||||
"""起始位姿同理。"""
|
||||
module = _load_eval_suite()
|
||||
from eval_kit.protocols import get_protocol
|
||||
|
||||
preset = get_protocol("standard_v1")
|
||||
protocol = module._apply_overrides(
|
||||
preset, module._build_parser().parse_args(["--start-x", "-1.2", "--start-y", "0.4"])
|
||||
)
|
||||
assert protocol.start_pose == (-1.2, 0.4, 0.0), protocol.start_pose
|
||||
return f"start_pose = {protocol.start_pose}"
|
||||
|
||||
|
||||
def test_start_z_offset_override():
|
||||
"""起始高度偏移：默认 0.0（与旧 eval.py 一致），显式给出时生效并改变指纹。"""
|
||||
module = _load_eval_suite()
|
||||
from eval_kit.protocols import get_protocol
|
||||
|
||||
preset = get_protocol("standard_v1")
|
||||
assert preset.start_z_offset_m == 0.0, "默认必须为 0.0，否则与 eval.py 不可比"
|
||||
|
||||
parser = module._build_parser()
|
||||
unchanged = module._apply_overrides(preset, parser.parse_args([]))
|
||||
assert unchanged.start_z_offset_m == 0.0
|
||||
assert unchanged.fingerprint() == preset.fingerprint()
|
||||
|
||||
lifted = module._apply_overrides(preset, parser.parse_args(["--start-z-offset", "0.3"]))
|
||||
assert lifted.start_z_offset_m == 0.3
|
||||
assert lifted.fingerprint() != preset.fingerprint(), "抬高起始高度必须改变指纹"
|
||||
# 只动高度，不动水平位姿
|
||||
assert lifted.start_pose == preset.start_pose
|
||||
return "默认 0.0，--start-z-offset 生效且反映在指纹里"
|
||||
|
||||
|
||||
def test_terrain_types_resolution():
|
||||
"""地形类型：all 展开成规格里的全部类型，单个类型要校验存在性。"""
|
||||
module = _load_eval_suite()
|
||||
from eval_kit.core.terrain_spec import get_terrain_spec
|
||||
|
||||
spec = get_terrain_spec("standard_v1")
|
||||
|
||||
all_args = module._build_parser().parse_args([])
|
||||
assert module._terrain_types_for(all_args, spec) == list(spec.types)
|
||||
|
||||
single_args = module._build_parser().parse_args(["--terrain-type", "boxes"])
|
||||
assert module._terrain_types_for(single_args, spec) == ["boxes"]
|
||||
|
||||
bad_args = module._build_parser().parse_args(["--terrain-type", "nonexistent"])
|
||||
try:
|
||||
module._terrain_types_for(bad_args, spec)
|
||||
except SystemExit as exc:
|
||||
assert "nonexistent" in str(exc)
|
||||
return "all / 单类型 / 非法类型校验都正确"
|
||||
raise AssertionError("非法地形类型没有报错")
|
||||
|
||||
|
||||
def test_all_profiles_and_protocols_are_loadable():
|
||||
"""每个机器人描述、每个协议预设都要能被命令行选中。"""
|
||||
module = _load_eval_suite()
|
||||
parser = module._build_parser()
|
||||
from eval_kit.profiles import PROFILES
|
||||
from eval_kit.protocols import PROTOCOLS
|
||||
|
||||
for name in PROFILES:
|
||||
args = parser.parse_args(["--profile", name])
|
||||
assert args.profile == name
|
||||
for name in PROTOCOLS:
|
||||
args = parser.parse_args(["--protocol", name])
|
||||
protocol = module._apply_overrides(PROTOCOLS[name], args)
|
||||
assert protocol.steps >= 1
|
||||
assert protocol.seeds, f"协议 {name} 没有种子"
|
||||
return f"{len(PROFILES)} 个描述、{len(PROTOCOLS)} 个协议预设都可加载"
|
||||
|
||||
|
||||
def test_eval_types_match_report_module():
|
||||
"""命令行的评估类型必须与指标筛选表一致，避免出现选了却没指标的项。"""
|
||||
module = _load_eval_suite()
|
||||
from eval_kit.core.report import EVAL_TYPES, METRIC_SELECTION
|
||||
|
||||
args = module._build_parser().parse_args([])
|
||||
choices = args.eval_type
|
||||
assert choices in EVAL_TYPES
|
||||
for eval_type in EVAL_TYPES:
|
||||
if eval_type in ("all", "upper_limits"):
|
||||
continue
|
||||
assert eval_type in METRIC_SELECTION, f"{eval_type} 没有对应的指标筛选规则"
|
||||
return f"{len(EVAL_TYPES)} 个评估类型，与 METRIC_SELECTION 对齐"
|
||||
|
||||
|
||||
def test_parser_compatible_with_app_launcher():
|
||||
"""自身的参数不能和 AppLauncher 冲突。
|
||||
|
||||
main() 的顺序是「先 _build_parser()，再 AppLauncher.add_app_launcher_args()」。
|
||||
这里复刻同样的顺序：任何冲突都会在这一步抛错，也就是脚本连启动都做不到。
|
||||
真跑时曾两次撞上这个坑（先是 --device、后是 --verbose），所以这个检查必须留下。
|
||||
"""
|
||||
module = _load_eval_suite()
|
||||
parser = module._build_parser()
|
||||
parser = module.AppLauncher.add_app_launcher_args(parser)
|
||||
args = parser.parse_args([])
|
||||
|
||||
# --device / --headless 由 AppLauncher 提供，自己的 parser 里不能再定义
|
||||
assert args.device is None
|
||||
assert args.headless is False
|
||||
assert args.task == "Rough-Deeprobotics-M20-v0"
|
||||
assert args.num_envs == 250
|
||||
return "与 AppLauncher 的 14 个参数名无冲突，--device/--headless 由其提供"
|
||||
|
||||
|
||||
def test_no_option_string_collisions_with_app_launcher():
|
||||
"""直接枚举检查：自身 parser 的任何 option string 都不能落在 AppLauncher 的名单里。"""
|
||||
module = _load_eval_suite()
|
||||
parser = module._build_parser()
|
||||
own = {option for action in parser._actions for option in action.option_strings}
|
||||
conflicts = sorted(own & set(APPLAUNCHER_ARG_NAMES))
|
||||
assert not conflicts, (
|
||||
f"与 AppLauncher 冲突: {conflicts} —— 真实启动时会抛 "
|
||||
"argparse.ArgumentError: conflicting option string"
|
||||
)
|
||||
return f"自身 {len(own)} 个 option string 与 AppLauncher 的 {len(APPLAUNCHER_ARG_NAMES)} 个均不冲突"
|
||||
|
||||
|
||||
def test_verbose_terrain_flag_replaces_verbose():
|
||||
"""--verbose 归 AppLauncher，自己的逐格调试用 --verbose-terrain。"""
|
||||
module = _load_eval_suite()
|
||||
parser = module._build_parser()
|
||||
args = parser.parse_args(["--verbose-terrain"])
|
||||
assert args.verbose_terrain is True
|
||||
plain = parser.parse_args([])
|
||||
assert plain.verbose_terrain is False
|
||||
return "--verbose-terrain 生效，未占用 --verbose"
|
||||
|
||||
|
||||
def test_output_path_defaults_next_to_checkpoint():
|
||||
"""默认输出落在 checkpoint 同目录的 eval_kit/ 下，不覆盖 eval.py 的 eval/。"""
|
||||
module = _load_eval_suite()
|
||||
import tempfile
|
||||
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
checkpoint = str(Path(directory) / "model_100.pt")
|
||||
Path(checkpoint).write_bytes(b"")
|
||||
args = module._build_parser().parse_args([])
|
||||
path = module._resolve_output_path(args, checkpoint, "all")
|
||||
|
||||
assert "/eval_kit/" in path, path
|
||||
assert path.endswith(".json")
|
||||
assert "all_" in Path(path).name
|
||||
return f"默认输出目录为 <checkpoint 目录>/eval_kit/"
|
||||
|
||||
|
||||
def test_livestream_value_falls_back_to_env_var():
|
||||
"""--livestream 默认 -1 表示「听 LIVESTREAM 环境变量」，不能当成 0。"""
|
||||
import os
|
||||
|
||||
module = _load_eval_suite()
|
||||
parser = module._build_parser()
|
||||
|
||||
args = parser.parse_args([])
|
||||
args.livestream = -1
|
||||
previous = os.environ.get("LIVESTREAM")
|
||||
os.environ.pop("LIVESTREAM", None)
|
||||
try:
|
||||
assert module._livestream_value(args) == 0, "没设环境变量时应视为关闭"
|
||||
os.environ["LIVESTREAM"] = "2"
|
||||
assert module._livestream_value(args) == 2, "应回退读 LIVESTREAM 环境变量"
|
||||
|
||||
explicit = parser.parse_args([])
|
||||
explicit.livestream = 1
|
||||
assert module._livestream_value(explicit) == 1, "命令行显式值优先于环境变量"
|
||||
finally:
|
||||
if previous is None:
|
||||
os.environ.pop("LIVESTREAM", None)
|
||||
else:
|
||||
os.environ["LIVESTREAM"] = previous
|
||||
return "命令行 > LIVESTREAM 环境变量 > 关闭"
|
||||
|
||||
|
||||
def test_viewer_env_index_covers_livestream():
|
||||
"""推流也要设视口相机；真 headless（且不推流）才返回 None。
|
||||
|
||||
这条决定推流画面能不能看到机器人：``AppLauncher`` 在开启推流时会把
|
||||
``headless`` 改写成 True，所以不能按 ``headless`` 判断「用户要不要看画面」。
|
||||
"""
|
||||
module = _load_eval_suite()
|
||||
# --headless 由 AppLauncher 提供，所以这里要按 main() 的顺序加上
|
||||
parser = module.AppLauncher.add_app_launcher_args(module._build_parser())
|
||||
|
||||
plain_headless = parser.parse_args(["--headless"])
|
||||
plain_headless.livestream = 0
|
||||
assert module._viewer_env_index(plain_headless) is None, "headless 且不推流时不应设视口相机"
|
||||
|
||||
streamed = parser.parse_args(["--headless", "--view-env", "7"])
|
||||
streamed.livestream = 2
|
||||
assert module._viewer_env_index(streamed) == 7, "推流时视口相机应生效并跟随 --view-env"
|
||||
|
||||
default_target = parser.parse_args([])
|
||||
default_target.livestream = 2
|
||||
assert module._viewer_env_index(default_target) == 0, "不传 --view-env 时默认跟随 env 0"
|
||||
|
||||
gui = parser.parse_args([])
|
||||
gui.livestream = 0
|
||||
assert module._viewer_env_index(gui) == 0, "本地 GUI（非 headless）仍然跟随 env 0"
|
||||
|
||||
# 自由视角：不跟随，且 --free-camera 优先于 --view-env
|
||||
free = parser.parse_args(["--headless", "--free-camera"])
|
||||
free.livestream = 2
|
||||
assert module._viewer_env_index(free) is None, "--free-camera 时不应跟随任何环境"
|
||||
both = parser.parse_args(["--headless", "--free-camera", "--view-env", "3"])
|
||||
both.livestream = 2
|
||||
assert module._viewer_env_index(both) is None, "--free-camera 与 --view-env 同时给出时以前者为准"
|
||||
return "headless→None；推流 / 本地 GUI→跟随 --view-env（默认 0）；--free-camera 优先于跟随"
|
||||
|
||||
|
||||
def test_free_camera_only_applies_with_a_picture():
|
||||
"""--free-camera 只在有画面时生效；给定 --view-env 也让位于它。"""
|
||||
module = _load_eval_suite()
|
||||
parser = module.AppLauncher.add_app_launcher_args(module._build_parser())
|
||||
|
||||
headless = parser.parse_args(["--headless", "--free-camera"])
|
||||
headless.livestream = 0
|
||||
assert module._display_enabled(headless) is False, "headless 且不推流时没有画面"
|
||||
assert module._viewer_env_index(headless) is None
|
||||
|
||||
streamed = parser.parse_args(["--free-camera"])
|
||||
streamed.livestream = 2
|
||||
assert module._display_enabled(streamed) is True, "推流算有画面"
|
||||
assert module._viewer_env_index(streamed) is None, "有画面 + --free-camera → 不跟随"
|
||||
|
||||
# 显式给出 --view-env 但没有 --free-camera：仍然跟随
|
||||
following = parser.parse_args(["--view-env", "2"])
|
||||
following.livestream = 2
|
||||
assert module._viewer_env_index(following) == 2
|
||||
return "无画面时 --free-camera 无效果；有画面时不跟随；未给该参数时行为不变"
|
||||
|
||||
|
||||
def test_pause_defaults_to_connection_window_only_when_streaming():
|
||||
"""--pause-s 是给推流客户端留连接时间的，默认值在 main() 里按是否推流决定。"""
|
||||
module = _load_eval_suite()
|
||||
# --livestream 由 AppLauncher 提供，所以这里要按 main() 的顺序加上
|
||||
parser = module.AppLauncher.add_app_launcher_args(module._build_parser())
|
||||
|
||||
assert parser.parse_args([]).pause_s is None, "默认必须是 None，否则区分不出用户显式传的 0"
|
||||
assert parser.parse_args(["--pause-s", "0"]).pause_s == 0.0
|
||||
assert parser.parse_args(["--pause-s", "12.5"]).pause_s == 12.5
|
||||
|
||||
def resolve(arguments):
|
||||
"""复刻 main() 里的取值规则。"""
|
||||
parsed = parser.parse_args(arguments)
|
||||
streaming = module._livestream_value(parsed) > 0
|
||||
return parsed.pause_s if parsed.pause_s is not None else (30.0 if streaming else 0.0)
|
||||
|
||||
assert resolve([]) == 0.0, "不推流时不该停顿"
|
||||
assert resolve(["--livestream", "2"]) == 30.0, "推流时默认留 30 s 连接时间"
|
||||
assert resolve(["--livestream", "2", "--pause-s", "0"]) == 0.0, "显式 0 可以关掉停顿"
|
||||
assert resolve(["--livestream", "1"]) == 30.0, "公网推流同样要留连接时间"
|
||||
return "不推流默认 0 s；推流默认 30 s；显式值优先"
|
||||
|
||||
|
||||
def test_local_ipv4_addresses_skips_loopback():
|
||||
"""打印给 WebRTC 客户端的地址里不能出现回环地址。"""
|
||||
module = _load_eval_suite()
|
||||
addresses = module._local_ipv4_addresses()
|
||||
assert all(not address.startswith("127.") for address in addresses), addresses
|
||||
assert all(address.count(".") == 3 for address in addresses), addresses
|
||||
return f"拿到 {len(addresses)} 个非回环地址：{addresses}"
|
||||
|
||||
|
||||
def test_record_frames_defaults_off():
|
||||
"""抓帧是可选功能：默认不抓，给了目录和间隔才生效。"""
|
||||
module = _load_eval_suite()
|
||||
parser = module._build_parser()
|
||||
|
||||
plain = parser.parse_args([])
|
||||
assert plain.record_frames is None, "默认不应抓帧"
|
||||
assert plain.record_every == 5, "默认每 5 步一张（0.1 s 仿真时间，合成 10 fps 视频）"
|
||||
|
||||
on = parser.parse_args(["--record-frames", "/tmp/frames", "--record-every", "1"])
|
||||
assert on.record_frames == "/tmp/frames"
|
||||
assert on.record_every == 1
|
||||
return "默认关闭；--record-frames 指定目录、--record-every 指定间隔"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
tests = [value for name, value in sorted(globals().items()) if name.startswith("test_")]
|
||||
failures: list[tuple[str, str]] = []
|
||||
for test in tests:
|
||||
name = test.__name__
|
||||
try:
|
||||
note = test()
|
||||
except (Exception, SystemExit) as exc: # noqa: BLE001
|
||||
# 也要接住 SystemExit：argparse 参数写错会直接 SystemExit(2)，
|
||||
# 不接的话整个测试文件当场退出，后面的测试一条都不跑、也没有汇总行。
|
||||
failures.append((name, f"{type(exc).__name__}: {exc}"))
|
||||
print(f"FAIL {name}\n {type(exc).__name__}: {exc}")
|
||||
else:
|
||||
print(f"ok {name}" + (f" — {note}" if note else ""))
|
||||
|
||||
print()
|
||||
if failures:
|
||||
print(f"{len(failures)}/{len(tests)} 个测试失败")
|
||||
return 1
|
||||
print(f"全部 {len(tests)} 个测试通过")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,645 @@
|
||||
# Copyright (c) 2025 Deep Robotics
|
||||
# SPDX-License-Identifier: BSD 3-Clause
|
||||
|
||||
"""评价内核的单元测试：用合成轨迹验证指标，不需要仿真环境。
|
||||
|
||||
运行方式（在仓库根目录）::
|
||||
|
||||
python3 eval_kit/tests/test_core.py
|
||||
|
||||
合成轨迹的价值在于：指标的期望值可以手算，因此能证明指标实现是对的，
|
||||
而不是「跑起来没报错」。这也是把指标做成纯函数的主要理由。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
import sys
|
||||
from dataclasses import replace
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
if __package__ in (None, ""):
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
||||
|
||||
from eval_kit.core.aggregation import (
|
||||
accumulated_success,
|
||||
aggregate_runs,
|
||||
build_upper_limit_matrix,
|
||||
upper_limit_score,
|
||||
)
|
||||
from eval_kit.core.metrics import compute_metrics, support_margin_xy, tilt_from_projected_gravity
|
||||
from eval_kit.core.naming import resolve_matching_indices
|
||||
from eval_kit.core.report import (
|
||||
filter_metrics,
|
||||
format_aggregate_summary,
|
||||
format_matrix,
|
||||
format_metric_result,
|
||||
format_numpy_matrix,
|
||||
format_rollout_summary,
|
||||
format_upper_limit_summary,
|
||||
)
|
||||
from eval_kit.core.schema import EvalProtocol, RobotProfile, Trace
|
||||
from eval_kit.core.terrain_spec import get_terrain_spec
|
||||
from eval_kit.profiles import get_profile
|
||||
from eval_kit.protocols import get_protocol
|
||||
|
||||
G = 9.81
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- 构造器
|
||||
|
||||
|
||||
def make_trace(
|
||||
*,
|
||||
num_envs: int = 3,
|
||||
steps: int = 10,
|
||||
dt: float = 0.02,
|
||||
speed: float = 1.0,
|
||||
tilt_rad: float = 0.0,
|
||||
torque: float = 1.0,
|
||||
joint_vel: float = 1.0,
|
||||
num_joints: int = 2,
|
||||
num_feet: int = 4,
|
||||
num_support: int = 2,
|
||||
support_force: float = 0.0,
|
||||
support_contact_time: float = 0.0,
|
||||
mass: float | None = 26.581,
|
||||
start_x: float = -0.9,
|
||||
command: tuple[float, float, float] = (1.0, 0.0, 0.0),
|
||||
accel_spike_at: int | None = None,
|
||||
accel_spike: float = 0.1,
|
||||
) -> Trace:
|
||||
"""造一条可控的合成轨迹。
|
||||
|
||||
机器人沿 +x 匀速前进；``accel_spike_at`` 指定在第几帧把速度永久抬高
|
||||
``accel_spike``，用于制造一次可预期的不稳定计数。
|
||||
"""
|
||||
frames = steps + 1
|
||||
time = np.arange(frames, dtype=np.float64) * dt
|
||||
|
||||
speed_profile = np.full(frames, float(speed), dtype=np.float64)
|
||||
if accel_spike_at is not None:
|
||||
speed_profile[accel_spike_at:] += accel_spike
|
||||
|
||||
x = start_x + np.concatenate([[0.0], np.cumsum(speed_profile[1:] * dt)])
|
||||
|
||||
base_pos_w = np.zeros((frames, num_envs, 3), dtype=np.float64)
|
||||
base_pos_w[:, :, 0] = x[:, None]
|
||||
|
||||
base_lin_vel_w = np.zeros((frames, num_envs, 3), dtype=np.float64)
|
||||
base_lin_vel_w[:, :, 0] = speed_profile[:, None]
|
||||
|
||||
projected_gravity_b = np.zeros((frames, num_envs, 3), dtype=np.float64)
|
||||
projected_gravity_b[:, :, 0] = -math.sin(tilt_rad)
|
||||
projected_gravity_b[:, :, 2] = -math.cos(tilt_rad)
|
||||
|
||||
joint_torque = np.full((frames, num_envs, num_joints), float(torque), dtype=np.float64)
|
||||
joint_vel_arr = np.full((frames, num_envs, num_joints), float(joint_vel), dtype=np.float64)
|
||||
|
||||
feet_pos_w = np.zeros((frames, num_envs, num_feet, 2), dtype=np.float64)
|
||||
if num_feet:
|
||||
offsets = np.linspace(-0.3, 0.3, num_feet)
|
||||
for foot_index, offset_x in enumerate(offsets):
|
||||
feet_pos_w[:, :, foot_index, 0] = x[:, None] + offset_x
|
||||
feet_pos_w[:, :, foot_index, 1] = 0.1 if foot_index % 2 else -0.1
|
||||
|
||||
support_force_n = np.full((frames, num_envs, num_support), float(support_force), dtype=np.float64)
|
||||
support_contact_time_s = np.full(
|
||||
(frames, num_envs, num_support), float(support_contact_time), dtype=np.float64
|
||||
)
|
||||
|
||||
return Trace(
|
||||
dt=dt,
|
||||
command=np.asarray(command, dtype=np.float64),
|
||||
base_pos_w=base_pos_w,
|
||||
base_lin_vel_w=base_lin_vel_w,
|
||||
base_lin_vel_b=base_lin_vel_w.copy(),
|
||||
base_ang_vel_b=np.zeros((frames, num_envs, 3), dtype=np.float64),
|
||||
projected_gravity_b=projected_gravity_b,
|
||||
joint_torque=joint_torque,
|
||||
joint_vel=joint_vel_arr,
|
||||
feet_pos_w=feet_pos_w,
|
||||
support_force_n=support_force_n,
|
||||
support_contact_time_s=support_contact_time_s,
|
||||
done=np.zeros((frames, num_envs), dtype=np.float64),
|
||||
mass_kg=mass,
|
||||
)
|
||||
|
||||
|
||||
def short_protocol(**overrides) -> EvalProtocol:
|
||||
"""时间很短、成功距离很小的协议，便于手算期望值。"""
|
||||
kwargs: dict[str, object] = {"steps": 10}
|
||||
kwargs.update(overrides)
|
||||
return replace(get_protocol("standard_v1"), **kwargs) # type: ignore[arg-type]
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------- 测试
|
||||
|
||||
|
||||
def test_perfect_tracker_succeeds():
|
||||
"""匀速前进、姿态水平：成功率 1，跟踪误差 0。"""
|
||||
protocol = short_protocol(success_distance_m=0.09)
|
||||
metrics = compute_metrics(make_trace(), get_profile("solenopsis"), protocol)
|
||||
|
||||
assert metrics["success_rate"] == 1.0, metrics["success_rate"]
|
||||
assert metrics["success_count"] == 3.0
|
||||
assert abs(metrics["mean_lin_vel_error"]) < 1e-9, metrics["mean_lin_vel_error"]
|
||||
assert abs(metrics["mean_yaw_vel_error"]) < 1e-9
|
||||
assert abs(metrics["avg_max_tilt_rad"]) < 1e-9
|
||||
assert metrics["unstability_rate"] == 0.0
|
||||
# speed=1 m/s 从 start_x=-0.9 出发：第 4 帧进度 0.08 < 0.09，第 5 帧 0.10 >= 0.09，
|
||||
# 因此首次越线在第 5 帧，耗时 5*0.02 = 0.10 s。
|
||||
assert abs(metrics["avg_time_consumption_s"] - 0.10) < 1e-9, metrics["avg_time_consumption_s"]
|
||||
return "success_rate=1.00, t=0.10 s, 跟踪误差 0"
|
||||
|
||||
|
||||
def test_stationary_robot_fails():
|
||||
"""原地不动：成功率 0，最终进度 0。"""
|
||||
protocol = short_protocol(success_distance_m=0.1)
|
||||
metrics = compute_metrics(make_trace(speed=0.0), get_profile("solenopsis"), protocol)
|
||||
|
||||
assert metrics["success_rate"] == 0.0
|
||||
assert abs(metrics["avg_final_progress_m"]) < 1e-9
|
||||
assert math.isnan(metrics["avg_time_consumption_s"]), metrics["avg_time_consumption_s"]
|
||||
assert math.isnan(metrics["time_efficiency"])
|
||||
return "success_rate=0.00, 进度 0, 耗时 nan"
|
||||
|
||||
|
||||
def test_faller_is_not_success():
|
||||
"""走够了距离但倾角超阈值：不算成功。"""
|
||||
protocol = short_protocol(success_distance_m=0.1, tilt_threshold_rad=0.70)
|
||||
# 倾角 1.0 rad > 0.70 rad
|
||||
metrics = compute_metrics(make_trace(tilt_rad=1.0), get_profile("solenopsis"), protocol)
|
||||
|
||||
assert abs(metrics["avg_final_progress_m"] - 0.2) < 1e-9
|
||||
assert metrics["success_rate"] == 0.0, "倾角超限却算成功了"
|
||||
assert abs(metrics["avg_max_tilt_rad"] - 1.0) < 1e-9
|
||||
assert metrics["avg_stable_time_s"] == 0.0
|
||||
return "进度达成但 success_rate=0.00，max_tilt=1.00 rad"
|
||||
|
||||
|
||||
def test_energy_and_cot_match_hand_calculation():
|
||||
"""能耗与 COT 与手算值一致。"""
|
||||
steps, dt, joints = 10, 0.02, 2
|
||||
mass = 26.581
|
||||
protocol = short_protocol(steps=steps)
|
||||
trace = make_trace(steps=steps, dt=dt, num_joints=joints, mass=mass)
|
||||
metrics = compute_metrics(trace, get_profile("solenopsis"), protocol)
|
||||
|
||||
expected_energy = 1.0 * 1.0 * joints * steps * dt # |tau*vel| * 关节数 * 步数 * dt
|
||||
expected_route = 1.0 * steps * dt
|
||||
# 分母带 1e-6 保护项，与旧 eval.py 的 `route + 1e-6` 一致
|
||||
expected_cot = expected_energy / ((expected_route + 1e-6) * mass * G)
|
||||
|
||||
assert abs(metrics["avg_energy_j"] - expected_energy) < 1e-12, (metrics["avg_energy_j"], expected_energy)
|
||||
assert abs(metrics["avg_cost_of_transport"] - expected_cot) < 1e-12, (
|
||||
metrics["avg_cost_of_transport"],
|
||||
expected_cot,
|
||||
)
|
||||
assert abs(metrics["robot_mass_kg"] - mass) < 1e-12
|
||||
return f"energy={expected_energy:.4f} J, COT={expected_cot:.5f}, mass={mass} kg（自动取自 Trace）"
|
||||
|
||||
|
||||
def test_signed_energy_convention_differs():
|
||||
"""有符号口径在存在负功时给出更小的能耗；两种约定都被记录。"""
|
||||
steps, dt = 10, 0.02
|
||||
trace = make_trace(steps=steps, dt=dt, num_joints=2, joint_vel=1.0)
|
||||
# 让后一半控制步的力矩反向，制造负功：帧 1..5 为 +1，帧 6..10 为 -1。
|
||||
half = steps // 2
|
||||
trace.joint_torque[1 + half :] = -1.0
|
||||
|
||||
absolute = compute_metrics(trace, get_profile("solenopsis"), short_protocol(energy_convention="absolute"))
|
||||
signed = compute_metrics(trace, get_profile("solenopsis"), short_protocol(energy_convention="signed"))
|
||||
|
||||
assert absolute["avg_energy_j"] > signed["avg_energy_j"], (
|
||||
absolute["avg_energy_j"],
|
||||
signed["avg_energy_j"],
|
||||
)
|
||||
assert abs(signed["avg_energy_j"]) < 1e-12, "正负功抵消后应为 0"
|
||||
assert absolute["energy_convention"] == "absolute"
|
||||
assert signed["energy_convention"] == "signed"
|
||||
return f"absolute={absolute['avg_energy_j']:.4f} J, signed={signed['avg_energy_j']:.4f} J"
|
||||
|
||||
|
||||
def test_instability_counts_acceleration_spikes():
|
||||
"""一次速度跳变恰好计一次不稳定。"""
|
||||
steps, dt = 10, 0.02
|
||||
protocol = replace(
|
||||
short_protocol(steps=steps), acc_threshold_g=2.0 / G
|
||||
)
|
||||
# 第 5 帧速度永久 +0.1 m/s → 该处加速度 0.1/0.02 = 5 m/s² > 2 m/s²
|
||||
metrics = compute_metrics(
|
||||
make_trace(steps=steps, dt=dt, accel_spike_at=5, accel_spike=0.1),
|
||||
get_profile("solenopsis"),
|
||||
protocol,
|
||||
)
|
||||
assert abs(metrics["unstability_rate"] - 1.0 / steps) < 1e-12, metrics["unstability_rate"]
|
||||
return f"unstability_rate={metrics['unstability_rate']:.3f} (= 1/{steps})"
|
||||
|
||||
|
||||
def test_cost_of_transport_uses_trace_mass_over_profile():
|
||||
"""Trace 里的质量优先于 profile 兜底值。"""
|
||||
profile = replace(get_profile("solenopsis"), mass_kg=999.0)
|
||||
metrics = compute_metrics(make_trace(mass=26.581), profile, short_protocol())
|
||||
assert abs(metrics["robot_mass_kg"] - 26.581) < 1e-12
|
||||
|
||||
trace_no_mass = make_trace(mass=None)
|
||||
metrics_fallback = compute_metrics(trace_no_mass, profile, short_protocol())
|
||||
assert abs(metrics_fallback["robot_mass_kg"] - 999.0) < 1e-12
|
||||
return "Trace 优先，profile 兜底"
|
||||
|
||||
|
||||
def test_report_formatters_run_on_real_shapes():
|
||||
"""报告层三个格式化函数都要能真跑——它们之前在没有任何测试覆盖的情况下
|
||||
把一个不存在的 numpy 参数留在里面，直到实跑才炸。"""
|
||||
spec = get_terrain_spec("standard_v1")
|
||||
terrain_types = ["stairs_up", "boxes"]
|
||||
level_ids = np.array([0, 0, 1, 1, 0, 0, 1, 1])
|
||||
type_ids = np.array([0, 0, 0, 0, 1, 1, 1, 1])
|
||||
per_env = {
|
||||
"success": [1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0],
|
||||
"max_progress": [3.0, 3.0, 1.0, 1.0, 2.5, 1.5, 3.0, 3.0],
|
||||
"max_tilt": [0.1] * 8,
|
||||
}
|
||||
matrix = build_upper_limit_matrix(per_env, level_ids, type_ids, terrain_types, spec, 2)
|
||||
|
||||
# 普通数值
|
||||
text = format_numpy_matrix(matrix["matrix"]["success_rate"], title="success rate")
|
||||
assert "success rate" in text and "1.00" in text, text
|
||||
|
||||
# 含 nan：把 title 里的 "nan" 排除在外，只检查矩阵主体
|
||||
empty = build_upper_limit_matrix(
|
||||
per_env, level_ids, type_ids, terrain_types, spec, 4
|
||||
)
|
||||
body_only = format_numpy_matrix(empty["matrix"]["success_rate"], title="unassigned levels").split("\n", 1)[1]
|
||||
assert "-" in body_only, f"nan 没有被替换成 -：{body_only}"
|
||||
assert "nan" not in body_only, f"主体里仍出现 nan：{body_only}"
|
||||
|
||||
# 上限汇总：走 build_upper_limit_matrix 的真实输出结构
|
||||
summary = format_upper_limit_summary(matrix)
|
||||
assert "success rate" in summary and "score" in summary, summary
|
||||
assert "difficulty units" in summary
|
||||
|
||||
# 逐等级矩阵（format_matrix）
|
||||
table = format_matrix(matrix["matrix"]["success_rate"], terrain_types, title="table")
|
||||
assert "level" in table and "stairs_up" in table, table
|
||||
|
||||
# 单指标与汇总
|
||||
metrics = compute_metrics(make_trace(), get_profile("solenopsis"), short_protocol(success_distance_m=0.05))
|
||||
assert "success rate" in format_metric_result("access_rate", metrics)
|
||||
assert "protocol" in format_rollout_summary(metrics)
|
||||
return "四个格式化函数在真实结构上都能跑，nan 正确显示为 -"
|
||||
|
||||
|
||||
def test_minimum_terrain_size_for_bounds():
|
||||
"""地砖下限公式：与网格行数无关，取默认值时是 3.8 m。
|
||||
|
||||
这个函数必须在纯内核里——编排进程不启动仿真，无法 import 任何依赖 isaaclab
|
||||
的模块（会因缺 pxr 失败）。之前它放在适配层，父进程一调用就崩。
|
||||
"""
|
||||
protocol = get_protocol("standard_v1")
|
||||
# start_x = -0.9, buffer = 3.0, border = 2.0 → 2*(0.9+3.0-2.0) = 3.8
|
||||
assert abs(protocol.minimum_terrain_size_for_bounds() - 3.8) < 1e-12
|
||||
|
||||
# clean 预设的地砖应当不小于该下限
|
||||
clean = get_protocol("standard_v1_clean")
|
||||
assert clean.terrain_size_m >= clean.minimum_terrain_size_for_bounds()
|
||||
|
||||
# 与网格行数无关
|
||||
for levels in (2, 5, 8, 16):
|
||||
assert abs(replace(protocol, num_levels=levels).minimum_terrain_size_for_bounds() - 3.8) < 1e-12
|
||||
|
||||
# border 与 buffer 可调，且不会返回负数
|
||||
assert abs(protocol.minimum_terrain_size_for_bounds(border_width=1.0, distance_buffer=1.0) - 1.8) < 1e-12
|
||||
assert protocol.minimum_terrain_size_for_bounds(border_width=10.0, distance_buffer=0.0) == 0.0
|
||||
|
||||
# 起始位置越靠外，要求越大
|
||||
far = replace(protocol, start_pose=(-1.5, 0.0, 0.0))
|
||||
assert far.minimum_terrain_size_for_bounds() > protocol.minimum_terrain_size_for_bounds()
|
||||
return "默认 3.8 m，与行数无关，参数可调且非负"
|
||||
|
||||
|
||||
def test_missing_morphology_returns_nan_not_zero():
|
||||
"""没有足端 / 没有支撑体的机器人应返回 nan，而不是看起来像测出来的 0。"""
|
||||
profile = RobotProfile(
|
||||
name="no_morphology",
|
||||
mass_kg=10.0,
|
||||
feet_body_patterns=(),
|
||||
support_body_patterns=(),
|
||||
actuated_joint_patterns=(".*",),
|
||||
)
|
||||
metrics = compute_metrics(make_trace(num_feet=0, num_support=0), profile, short_protocol())
|
||||
assert math.isnan(metrics["avg_stability_margin_m"]), metrics["avg_stability_margin_m"]
|
||||
assert math.isnan(metrics["avg_support_contact_ratio"])
|
||||
assert math.isnan(metrics["successful_runs_with_support_ratio"])
|
||||
assert metrics["stability_margin_method"] == "unavailable"
|
||||
return "缺失形态 → nan（旧实现给 0.00）"
|
||||
|
||||
|
||||
def test_support_contact_events_and_ratio():
|
||||
"""支撑接触比例与事件计数可手算。"""
|
||||
steps, dt = 10, 0.02
|
||||
trace = make_trace(steps=steps, dt=dt, support_force=0.0)
|
||||
# 连续 3 帧有超过阈值的支撑接触 → 1 次事件，占 3/10 步
|
||||
trace.support_force_n[2:5] = 100.0
|
||||
trace.support_contact_time_s[2:5] = 0.05
|
||||
|
||||
protocol = replace(short_protocol(steps=steps, success_distance_m=0.1))
|
||||
metrics = compute_metrics(trace, get_profile("solenopsis"), protocol)
|
||||
|
||||
assert abs(metrics["avg_support_contact_ratio"] - 3.0 / steps) < 1e-12, metrics["avg_support_contact_ratio"]
|
||||
assert abs(metrics["avg_support_events"] - 1.0) < 1e-12, metrics["avg_support_events"]
|
||||
assert abs(metrics["successful_runs_with_support_ratio"] - 1.0) < 1e-12
|
||||
return "contact_ratio=0.300, events=1.0, 成功率内用支撑=100%"
|
||||
|
||||
|
||||
def test_support_margin_sign_and_tilt_formula():
|
||||
"""支撑裕度在包围盒内外符号正确；倾角公式反解一致。"""
|
||||
root_inside = np.array([[0.0, 0.0]])
|
||||
feet = np.array([[[-1.0, -1.0], [1.0, -1.0], [1.0, 1.0], [-1.0, 1.0]]], dtype=np.float64)
|
||||
inside = support_margin_xy(root_inside, feet)
|
||||
assert abs(inside[0] - 1.0) < 1e-12, inside
|
||||
|
||||
root_outside = np.array([[2.0, 0.0]])
|
||||
outside = support_margin_xy(root_outside, feet)
|
||||
assert abs(outside[0] - (-1.0)) < 1e-12, outside
|
||||
|
||||
theta = 0.5
|
||||
gravity_b = np.array([[[-math.sin(theta), 0.0, -math.cos(theta)]]], dtype=np.float64)
|
||||
tilt = tilt_from_projected_gravity(gravity_b)
|
||||
assert abs(tilt[0, 0] - theta) < 1e-9
|
||||
return "盒内 +1.000，盒外 -1.000，倾角反解 0.500 rad"
|
||||
|
||||
|
||||
def test_accumulated_and_score_match_legged_gym_formula():
|
||||
"""累积成功率与 score 与 legged_gym criterion.py 的公式一致。"""
|
||||
rates = [1.0, 0.8, 0.5, 0.0]
|
||||
accumulated = accumulated_success(rates)
|
||||
assert np.allclose(accumulated, [1.0, 0.8, 0.4, 0.0]), accumulated
|
||||
|
||||
# 手算：i=0 → 1.0*1*(1-0.8)=0.2；i=1 → 0.8*2*(1-0.5)=0.8；
|
||||
# i=2 → 0.4*3*(1-0.0)=1.2；i=3 → 0.0*4*1=0.0
|
||||
expected = 0.2 + 0.8 + 1.2 + 0.0
|
||||
assert abs(upper_limit_score(rates) - expected) < 1e-12, (upper_limit_score(rates), expected)
|
||||
|
||||
# 全通过与全失败是两个极端
|
||||
assert abs(upper_limit_score([1.0] * 4) - 4.0) < 1e-12
|
||||
assert abs(upper_limit_score([0.0] * 4)) < 1e-12
|
||||
return f"score={expected:.2f}（手算），全通过=4.00，全失败=0.00"
|
||||
|
||||
|
||||
def test_upper_limit_matrix_cells():
|
||||
"""上限矩阵按「等级 × 地形」正确分格。"""
|
||||
spec = get_terrain_spec("standard_v1")
|
||||
terrain_types = ["stairs_up", "boxes"]
|
||||
num_levels = 2
|
||||
envs_per_cell = 2
|
||||
|
||||
level_ids = np.array([0, 0, 1, 1, 0, 0, 1, 1])
|
||||
type_ids = np.array([0, 0, 0, 0, 1, 1, 1, 1])
|
||||
# stairs_up: 等级0 两个都成功，等级1 两个都失败
|
||||
# boxes: 等级0 一个成功，等级1 两个都成功
|
||||
per_env = {
|
||||
"success": [1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0],
|
||||
"max_progress": [3.0, 3.0, 1.0, 1.0, 2.5, 1.5, 3.0, 3.0],
|
||||
"max_tilt": [0.1] * 8,
|
||||
}
|
||||
result = build_upper_limit_matrix(
|
||||
per_env, level_ids, type_ids, terrain_types, spec, num_levels
|
||||
)
|
||||
|
||||
stairs = result["per_type"]["stairs_up"]
|
||||
boxes = result["per_type"]["boxes"]
|
||||
assert stairs["success_rate"] == [1.0, 0.0], stairs["success_rate"]
|
||||
assert boxes["success_rate"] == [0.5, 1.0], boxes["success_rate"]
|
||||
assert np.allclose(stairs["success_rate_accumulated"], [1.0, 0.0])
|
||||
assert np.allclose(boxes["success_rate_accumulated"], [0.5, 0.5])
|
||||
# 行=等级、列=地形
|
||||
assert result["matrix"]["success_rate"] == [[1.0, 0.5], [0.0, 1.0]]
|
||||
# 难度值取自规格
|
||||
assert result["matrix"]["difficulty"][0] == [0.04, 0.04]
|
||||
assert result["matrix"]["difficulty"][1] == [0.06, 0.06]
|
||||
assert result["num_levels"] == num_levels
|
||||
assert result["scores"]["stairs_up"] == upper_limit_score([1.0, 0.0])
|
||||
return "矩阵 [[1.00,0.50],[0.00,1.00]]，难度 [[0.04,0.04],[0.06,0.06]]"
|
||||
|
||||
|
||||
def test_fingerprint_tracks_protocol_changes():
|
||||
"""改变口径会改变指纹；改变种子不会。"""
|
||||
base = get_protocol("standard_v1")
|
||||
same = get_protocol("standard_v1")
|
||||
assert base.fingerprint() == same.fingerprint()
|
||||
|
||||
changed_energy = replace(base, energy_convention="signed")
|
||||
changed_tilt = replace(base, tilt_threshold_rad=0.5)
|
||||
pushed = replace(base, enable_interval_pushes=True)
|
||||
reseeded = replace(base, seeds=(1, 2, 3))
|
||||
|
||||
assert base.fingerprint() != changed_energy.fingerprint()
|
||||
assert base.fingerprint() != changed_tilt.fingerprint()
|
||||
assert base.fingerprint() != pushed.fingerprint()
|
||||
assert base.fingerprint() == reseeded.fingerprint(), "种子不影响单次 rollout 的口径"
|
||||
return "能耗约定/倾角阈值/推力开关都改变指纹，种子不改变"
|
||||
|
||||
|
||||
def test_trace_validation_rejects_mismatched_shapes():
|
||||
"""形状不一致要早失败。"""
|
||||
trace = make_trace(num_envs=2, steps=5, num_joints=3)
|
||||
trace.joint_vel = np.zeros((6, 3, 3)) # 环境数写成 3，应为 2
|
||||
try:
|
||||
trace.validate()
|
||||
except ValueError as exc:
|
||||
assert "joint_vel" in str(exc), str(exc)
|
||||
return "捕获到形状错误"
|
||||
raise AssertionError("形状不一致却没有报错")
|
||||
|
||||
|
||||
def test_trace_roundtrip(tmp_path: Path | None = None):
|
||||
"""轨迹存盘再读回，指标不变——这是「事后重新分析」的前提。"""
|
||||
import tempfile
|
||||
|
||||
trace = make_trace(steps=8)
|
||||
protocol = short_protocol(steps=8, success_distance_m=0.05)
|
||||
profile = get_profile("solenopsis")
|
||||
before = compute_metrics(trace, profile, protocol)
|
||||
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
path = str(Path(directory) / "trace.npz")
|
||||
trace.save(path)
|
||||
reloaded = Trace.load(path)
|
||||
after = compute_metrics(reloaded, profile, protocol)
|
||||
|
||||
for key in before:
|
||||
if key.startswith("_"):
|
||||
continue
|
||||
left, right = before[key], after[key]
|
||||
if isinstance(left, float) and math.isnan(left):
|
||||
assert math.isnan(right), key
|
||||
else:
|
||||
assert left == right, (key, left, right)
|
||||
return f"存盘往返后 {len([k for k in before if not k.startswith('_')])} 个指标完全一致"
|
||||
|
||||
|
||||
def test_report_filters_and_formats():
|
||||
"""输出层按评估类型裁剪，且 nan 不炸格式串。"""
|
||||
metrics = compute_metrics(make_trace(num_feet=0, num_support=0), get_profile("solenopsis"), short_protocol())
|
||||
selected = filter_metrics(metrics, "stability_margin")
|
||||
assert list(selected.keys()) == ["avg_stability_margin_m"]
|
||||
|
||||
text = format_metric_result("stability_margin", metrics)
|
||||
assert "nan" in text, text
|
||||
text = format_metric_result("access_rate", metrics)
|
||||
assert "success rate" in text
|
||||
return "filter_metrics 裁剪正确，nan 格式化安全"
|
||||
|
||||
|
||||
def test_profile_registry():
|
||||
"""加机器人只改数据：火蚁有轮与支撑体，四足没有支撑体语义。"""
|
||||
solenopsis = get_profile("solenopsis")
|
||||
assert solenopsis.feet_body_patterns == (".*_wheel",)
|
||||
assert solenopsis.support_body_patterns == ("base_link", ".*_knee")
|
||||
|
||||
lite3 = get_profile("lite3")
|
||||
assert lite3.support_body_patterns == ()
|
||||
|
||||
try:
|
||||
get_profile("nonexistent")
|
||||
except KeyError:
|
||||
return "solenopsis / lite3 描述可用，未知名字报错"
|
||||
raise AssertionError("未知机器人没有报错")
|
||||
|
||||
|
||||
def test_naming_resolves_fire_ant_bodies():
|
||||
"""火蚁的 link/joint 名字能被描述里的正则解析出来。"""
|
||||
body_names = [
|
||||
"base_link",
|
||||
"fl_knee", "fr_knee", "hl_knee", "hr_knee",
|
||||
"fl_wheel", "fr_wheel", "hl_wheel", "hr_wheel",
|
||||
]
|
||||
resolution = resolve_matching_indices(get_profile("solenopsis").feet_body_patterns, body_names)
|
||||
assert resolution.names == ["fl_wheel", "fr_wheel", "hl_wheel", "hr_wheel"], resolution.names
|
||||
assert resolution.indices == [5, 6, 7, 8]
|
||||
assert not resolution.unmatched_patterns
|
||||
|
||||
support = resolve_matching_indices(get_profile("solenopsis").support_body_patterns, body_names)
|
||||
assert support.names == ["base_link", "fl_knee", "fr_knee", "hl_knee", "hr_knee"], support.names
|
||||
|
||||
joint_names = [
|
||||
"fl_knee_joint", "fr_knee_joint", "hl_knee_joint", "hr_knee_joint",
|
||||
"fl_wheel_joint", "fr_wheel_joint", "hl_wheel_joint", "hr_wheel_joint",
|
||||
]
|
||||
joints = resolve_matching_indices(get_profile("solenopsis").actuated_joint_patterns, joint_names)
|
||||
assert len(joints) == 8, joints.names
|
||||
return f"足端 4 个、支撑体 5 个、驱动关节 {len(joints)} 个"
|
||||
|
||||
|
||||
def test_naming_does_not_crash_when_morphology_is_missing():
|
||||
"""换一个没有轮子的机器人：未命中的正则被记录下来，而不是抛异常。"""
|
||||
quadruped_bodies = ["base", "FL_hip", "FL_thigh", "FL_shank", "FL_FOOT"]
|
||||
resolution = resolve_matching_indices(get_profile("solenopsis").feet_body_patterns, quadruped_bodies)
|
||||
assert len(resolution) == 0
|
||||
assert resolution.unmatched_patterns == [".*_wheel"]
|
||||
assert not resolution
|
||||
|
||||
# 四足自己的描述能解析
|
||||
lite3 = resolve_matching_indices(get_profile("lite3").feet_body_patterns, quadruped_bodies)
|
||||
assert lite3.names == ["FL_FOOT"]
|
||||
# 空描述合法，得到空结果
|
||||
empty = resolve_matching_indices(get_profile("lite3").support_body_patterns, quadruped_bodies)
|
||||
assert len(empty) == 0 and not empty.unmatched_patterns
|
||||
return "无轮机器人：未命中被记录，不崩溃"
|
||||
|
||||
|
||||
def test_naming_reports_ambiguous_and_bad_regex():
|
||||
"""一个名字命中多个正则会被记录；正则写错要立刻报错。"""
|
||||
resolution = resolve_matching_indices((".*_FOOT", "FL_.*"), ["FL_FOOT", "FR_FOOT"])
|
||||
assert resolution.names == ["FL_FOOT", "FR_FOOT"]
|
||||
assert resolution.ambiguous_names == [("FL_FOOT", [".*_FOOT", "FL_.*"])], resolution.ambiguous_names
|
||||
|
||||
try:
|
||||
resolve_matching_indices(("[unclosed",), ["anything"])
|
||||
except ValueError as exc:
|
||||
assert "无法编译" in str(exc)
|
||||
return "多重命中被记录，坏正则报错"
|
||||
raise AssertionError("坏正则没有报错")
|
||||
|
||||
|
||||
def test_aggregate_runs_mean_std_and_nan_handling():
|
||||
"""多次试验聚合：mean/std 正确，nan 不污染结果，非数值字段单独记录。"""
|
||||
runs = [
|
||||
(0, {"success_rate": 0.5, "avg_energy_j": 100.0, "profile": "solenopsis", "avg_cost_of_transport": float("nan")}),
|
||||
(1, {"success_rate": 1.0, "avg_energy_j": 200.0, "profile": "solenopsis", "avg_cost_of_transport": 0.4}),
|
||||
(2, {"success_rate": 0.75, "avg_energy_j": 150.0, "profile": "solenopsis", "avg_cost_of_transport": float("nan")}),
|
||||
]
|
||||
aggregate = aggregate_runs(runs)
|
||||
|
||||
assert aggregate["num_runs"] == 3
|
||||
assert aggregate["seeds"] == [0, 1, 2]
|
||||
|
||||
success = aggregate["summary"]["success_rate"]
|
||||
assert abs(success["mean"] - 0.75) < 1e-12, success
|
||||
assert abs(success["std"] - 0.25) < 1e-12, success
|
||||
assert success["min"] == 0.5 and success["max"] == 1.0
|
||||
assert success["values"] == [0.5, 1.0, 0.75]
|
||||
|
||||
# nan 被剔除，只剩 1 个有效值 → std 记 0
|
||||
cot = aggregate["summary"]["avg_cost_of_transport"]
|
||||
assert abs(cot["mean"] - 0.4) < 1e-12, cot
|
||||
assert cot["std"] == 0.0
|
||||
assert cot["finite_runs"] == 1
|
||||
|
||||
# 非数值字段单独放在 constant 里
|
||||
assert aggregate["constant"]["profile"] == "solenopsis"
|
||||
assert "profile_INCONSISTENT" not in aggregate["constant"]
|
||||
|
||||
# 全部无效 → mean 是 nan，且记录了有效次数
|
||||
all_nan = aggregate_runs([(0, {"x": float("nan")}), (1, {"x": float("nan")})])
|
||||
assert math.isnan(all_nan["summary"]["x"]["mean"])
|
||||
assert all_nan["summary"]["x"]["finite_runs"] == 0
|
||||
|
||||
text = format_aggregate_summary(aggregate)
|
||||
assert "success_rate" in text and "±" in text
|
||||
return "mean/std 正确，nan 被剔除，非数值字段单独登记"
|
||||
|
||||
|
||||
def test_aggregate_runs_flags_inconsistent_metadata():
|
||||
"""两次 run 的口径不一致时要显式暴露，而不是悄悄取第一个。"""
|
||||
aggregate = aggregate_runs(
|
||||
[
|
||||
(0, {"success_rate": 1.0, "energy_convention": "absolute"}),
|
||||
(1, {"success_rate": 1.0, "energy_convention": "signed"}),
|
||||
]
|
||||
)
|
||||
assert "energy_convention_INCONSISTENT" in aggregate["constant"], aggregate["constant"]
|
||||
return "口径不一致会显式标记"
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------- 运行器
|
||||
|
||||
|
||||
def main() -> int:
|
||||
tests = [value for name, value in sorted(globals().items()) if name.startswith("test_")]
|
||||
failures: list[tuple[str, str]] = []
|
||||
for test in tests:
|
||||
name = test.__name__
|
||||
try:
|
||||
note = test()
|
||||
except Exception as exc: # noqa: BLE001 - 测试运行器需要捕获全部异常
|
||||
failures.append((name, f"{type(exc).__name__}: {exc}"))
|
||||
print(f"FAIL {name}\n {type(exc).__name__}: {exc}")
|
||||
else:
|
||||
suffix = f" — {note}" if note else ""
|
||||
print(f"ok {name}{suffix}")
|
||||
|
||||
print()
|
||||
if failures:
|
||||
print(f"{len(failures)}/{len(tests)} 个测试失败")
|
||||
return 1
|
||||
print(f"全部 {len(tests)} 个测试通过")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
Reference in New Issue
Block a user