MCP For Claude Marketplace
A/B Test Significance Calculator

A/B Test Significance Calculator MCP Connector for Claude

A+

Calculate statistical significance, required sample sizes, and power for A/B tests.

4 tools Official Updated Oct 1, 2026 Official Vinkius Partner
More Details Connect to Claude

This MCP server provides a complete statistical engine for analyzing A/B test results. It allows you to determine if observed differences in conversion rates are statistically significant using the analyze_conversion_difference tool. You can plan future experiments by estimating necessary visitor counts with calculate_required_sample_size. Additionally, monitor your ongoing tests' strength via calculate_statistical_power and protect against false positives by using check_peeking_risk to detect the dangers of early data analysis.

ab-testingconversion-ratep-valuestatistical-significancesample-sizeexperimentation

4 tools expose this connector's capabilities to your AI agent.

calculate_required_sample_size

Calculate the required sample size per group for a new A/B test

analyze_conversion_rab_difference

Analyze the difference in conversion rates between two groups

calculate_statistical_power

Calculate the current power of an ongoing test

check_peeking_risk

Check the risk of peeking at A/B test results

See how to talk to your AI agent using A/B Test Significance Calculator.

Is the difference between 500 conversions in 10,000 visitors (Group A) and 550 conversions in 10,000 visitors (Group B) statistically significant at a 95% confidence level?

Yes, the p-value is approximately 0.032, which is below the 0.05 threshold, indicating statistical significance.

How many visitors do I need per group for a new test with a baseline conversion rate of 0.10, an MDE of 0.02, power of 0.80, and alpha of 0.05?

You will need approximately 3,975 visitors per group to achieve the desired statistical power.

I have checked my A/B test results 5 times so far. The current sample size is 1,200 and the planned sample size was 5,000. What is my risk level?

The risk level is high because you have performed multiple checks while only a small fraction of the planned sample size has been reached.

You can use the `analyze_conversion_difference` tool by providing the number of visitors and conversions for both your control and variant groups.

Related Connectors

Feature Flag Rollout Calculator

Feature Flag Rollout Calculator MCP

4 tools Official

Calculate deterministic user assignment, rollout projections, and statistical sample sizes for feature flags.

A+ View details →
test-regression-runner

test-regression-runner MCP

3 tools Official

Automated verification system that executes targeted test suites and detects regressions or flakiness.

A+ View details →
Mine Power Distribution Designer

Mine Power Distribution Designer MCP

4 tools Official

Design electrical distribution networks for mining environments, including transformer sizing and cable selection.

A+ View details →
Objection Handling Coverage Scorer

Objection Handling Coverage Scorer MCP

1 tools Official

Audit sales copy by measuring how effectively customer objections are addressed.

A+ View details →