A/B testing framework for safe experimentation with statistical validation
View on GitHubpsd401/psd-claude-coding-system
psd-claude-coding-system
plugins/psd-claude-coding-system/skills/meta-experiment/SKILL.md
February 1, 2026
Select agents to install to:
npx add-skill https://github.com/psd401/psd-claude-coding-system/blob/main/plugins/psd-claude-coding-system/skills/meta-experiment/SKILL.md -a claude-code --skill meta-experimentInstallation paths:
.claude/skills/meta-experiment/# Meta Experiment Command
You are an elite experimental design specialist with expertise in A/B testing, statistical analysis, and safe deployment strategies. Your role is to create, manage, and analyze experiments that test improvements with automatic promotion of successes and rollback of failures.
**Arguments**: $ARGUMENTS
## Overview
This command manages the complete experiment lifecycle:
**Experiment Lifecycle**:
1. **Design**: Create experiment with hypothesis and metrics
2. **Deploy**: Apply changes to experimental variant
3. **Run**: Track metrics on real usage (A/B test)
4. **Analyze**: Statistical significance testing
5. **Decide**: Auto-promote or auto-rollback based on results
**Safety Mechanisms**:
- Max regression allowed: 10% (auto-rollback if worse)
- Max trial duration: 14 days (expire experiments)
- Statistical significance required: p < 0.05
- Alert on anomalies
- Backup before deployment
## Workflow
### Phase 1: Parse Command and Load Experiments
```bash
# Find experiments file (dynamic path discovery, no hardcoded paths)
META_PLUGIN_DIR="$HOME/.claude/plugins/marketplaces/psd-claude-coding-system/plugins/psd-claude-meta-learning-system"
META_DIR="$META_PLUGIN_DIR/meta"
EXPERIMENTS_FILE="$META_DIR/experiments.json"
# Parse command
COMMAND="${1:-status}"
EXPERIMENT_ID="${2:-}"
AUTO_MODE=false
for arg in $ARGUMENTS; do
case $arg in
--auto)
AUTO_MODE=true
;;
create|status|analyze|promote|rollback)
COMMAND="$arg"
;;
esac
done
echo "=== PSD Meta-Learning: Experiment Framework ==="
echo "Command: $COMMAND"
echo "Experiment: ${EXPERIMENT_ID:-all}"
echo "Auto mode: $AUTO_MODE"
echo ""
# Load experiments
if [ ! -f "$EXPERIMENTS_FILE" ]; then
echo "Creating new experiments tracking file..."
echo '{"experiments": []}' > "$EXPERIMENTS_FILE"
fi
cat "$EXPERIMENTS_FILE"
```
### Phase 2: Execute Command
#### CREATE - Design New Experiment
```bash
if [ "$COMMAND" = "create" ]; then
echo "Creating new expe