A/B Test Results: v1.0.0 vs v1.1.0【免费下载链接】LifeOS⛰️ The Life Operating System — an intent engineering platform that moves you from your current state to your ideal state, in life and work.项目地址: https://gitcode.com/GitHub_Trending/pe/LifeOSSummaryMetricv1.0.0 (A)v1.1.0 (B)Win Rate42%58%Avg Score4.24.5Std Dev0.80.6Statistical Significancep-value: 0.03Significant at 95%: YesConfidence Interval: [0.15, 0.45]Per-Dimension BreakdownDimensionA WinsB WinsTieAccuracy372Style543Format660ConclusionWinner: v1.1.0 (Candidate)Confidence: High (p 0.05)Recommendation: Deploy v1.1.0 to production### Step 8做出决策 | 结果 | 动作 | |---------|--------| | B 显著更好 | 部署 B归档 A | | A 显著更好 | 保留 A继续迭代 B | | 无显著差异 | 保留更简单的 Prompt或补充数据 | | 混合结果A 赢一些、B 赢一些 | 考虑混合方案hybrid | ### Step 9记录决策 把对照结果追加进用例 README 的 Comparison History markdown ## Comparison History ### v1.0.0 vs v1.1.0 (2024-01-15) **Hypothesis**: v1.1.0 improves accuracy with source verification. **Result**: v1.1.0 significantly better (p0.03) - Accuracy: 15% - Style: No change - Format: No change **Decision**: Deployed v1.1.0 as new baseline.最佳实践样本量最低10 个测试用例统计效力弱推荐20–30 个测试用例效力良好理想50 个测试用例高置信度位置交换始终开启。研究表明 LLM 存在强烈的位置偏差偏好第一个选项——这也是为什么 PairwiseComparison.ts 把position_swap的默认值直接设为true。裁判选择使用与生成器不同的模型做裁判防止自我服务偏差self-serving bias测 Claude 的 Prompt → 用 GPT-4o 当裁判测 GPT 的 Prompt → 用 Claude 当裁判SKILL.md 的 gotchas 用一句话强化了这条纪律judge_level必须不同于agent_levelAnthropic 准则judge ≠ generator默认 agentmedium、judgehigh。统计显著性判读p-value解读 0.01强证据0.01–0.05中等证据0.05–0.10弱证据 0.10不显著除非改进幅度很大否则不要基于弱证据做部署决策。三类常见对照模式文档给出了三类典型实验的最小配置片段分别覆盖格式指令、少样本示例与角色设定三种干预手段测试指令变更隐式格式 → 显式章节标题hypothesis: More explicit formatting instructions improve structure variants: a: { prompt: v1.0.0.md } # Implicit formatting b: { prompt: v1.1.0.md } # Explicit section headers focus: format测试 Few-Shot 示例zero-shot → two-shothypothesis: Adding 2 examples improves accuracy variants: a: { prompt: v1.0.0.md } # Zero-shot b: { prompt: v1.1.0.md } # Two-shot focus: accuracy测试 Persona/角色变更通用助手 → 领域专家hypothesis: Expert persona produces more detailed analysis variants: a: { prompt: v1.0.0.md } # Generic assistant b: { prompt: v1.1.0.md } # Domain expert persona focus: depth用 Comparison 模板生成对照文档对于需要正式对照文档的场景评审、归档可用 Prompting 技能的渲染工具以 Step 3 的 YAML 为数据源生成一份结构化 Markdown 文档bun run ~/.claude/skills/Prompting/Tools/RenderTemplate.ts \ -t Evals/Comparison.hbs \ -d ~/.claude/skills/Evals/UseCases/name/comparisons/name.yaml \ -o ~/.claude/skills/Evals/UseCases/name/comparisons/name-setup.md \ --preview【免费下载链接】LifeOS⛰️ The Life Operating System — an intent engineering platform that moves you from your current state to your ideal state, in life and work.项目地址: https://gitcode.com/GitHub_Trending/pe/LifeOS创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
