Luma

Evaluation

固定评估集上的策略对比

数字来自仓库内已提交报告。空单元格表示尚无真实测量，不填估算。每条结果带 provenance。

数据集与元数据

Recorded Run
Dataset
data/eval — stratified Livehouse archive sample + human labels
N
250 labeled images
Model / config
qwen2-vl (temp0; see stage3_v6_qwen2vl_temp0 report)
configs/eval_stage3.yaml (admission gates opened for full scoring)
Hardware
stamped in report.protocol.hardware when regenerated via scripts/eval/protocol.py

Quality = Spearman / MAE on overall vs human labels; Precision@K on human keepers. Eval config admits all frames to Stage3 (not production cull rates). New reports include protocol (seed/config/hardware/git).

reports/eval/stage3_v6_qwen2vl_temp0.json

策略总表

Recorded Run
策略质量VLM 调用延迟成本标签
全量 VLM（eval Stage3）Spearman 0.36 · MAE 6.09 · P@20 0.55100% of 250 eval images——Recorded Run
两阶段门控（生产路径）admitted Spearman 0.52 · MAE 4.48 · keeper coverage 0.0616 / 250 = 6.4%—~15× fewer VLM calls vs fullRecorded Run
  • 全量 VLM（eval Stage3）： Baseline calibration on fixed labels. Latency/cost not in this report. (reports/eval/stage3_v6_qwen2vl_temp0.json)
  • 两阶段门控（生产路径）： Offline replay of production apply_stage3_candidates_gating on the same 250 labels (not a second GPU pass). Gated Spearman is on the 16 admitted images only. (reports/eval/two_stage_gating.json)

Stage3 vs human

Recorded Run
Spearman
0.362
MAE
6.09
P@10
40%
P@20
55%

reports/eval/stage3_v6_qwen2vl_temp0.json

质量 · VLM 调用份额

Recorded Run

两阶段门控 vs 全量 Stage3：VLM 调用份额与质量对照（指标定义不同，只看形状）。

  • gated P@eff20VLM 6.4% · P 31%
  • Stage3 P@20 (full)VLM 100% · P 55%

Quantization example

Simulated

int4 vs fp16: ΔSpearman −0.009, est. $/1k −38%

Example / illustrative quant_compare payload — mark Simulated. Do not cite as measured production SLO.

reports/eval/quant_compare_example.json

尚未填写的对照

  • 偏好数据 → SFT/DPO

    data/eval/preferences/ 已导出 keep/reject pairs；训练与线上 reward 闭环尚未接入。

报告索引

  • Stage3 vs human · n=250

    reports/eval/stage3_v6_qwen2vl_temp0.json

    Recorded Run

  • Prod gating vs full-VLM · offline replay

    reports/eval/two_stage_gating.json

    Recorded Run

  • Provenance index for all eval showcase reports

    reports/eval/meta.json

    Recorded Run

  • Quantization example only

    reports/eval/quant_compare_example.json

    Simulated
Infra walkthrough首页Gallery