Prepared by Nadali using AI.
An AI assistant rewrites a booking page. After the first few customers, the new version has more bookings. Is that enough to replace the old one? For a small business, the useful first question comes earlier: can the available traffic distinguish a worthwhile improvement from ordinary variation?
This guide develops a fictional repair workshop’s experiment. All quantities and outcomes are synthetic, not measured business results. The aim is a defensible decision plan, including permission to conclude that a live A/B test is impractical.
Define the change and the decision
The workshop wants to test a clearer explanation of its booking process. A is the existing explanation; B is an AI-assisted rewrite approved by a person. Prices, appointment availability, page layout and the booking form stay unchanged. This comparison estimates the effect of these two texts in this setting. It cannot establish that AI-written copy generally outperforms human writing.
Before launch, write down the decision: whether to adopt B, keep A or gather different evidence. Preserve both versions. Ask AI to identify assumptions and contradictions in the plan, but have someone who understands the service verify promises, prices and eligibility. A more persuasive false promise is not an improvement.
Randomise customers, not page views
In this example, eligible customers enter when they first reach the explanation. Randomly assign each customer to A or B with equal probability and retain that assignment on return visits. Run both variants over the same period. Monday versus Tuesday, or last month versus this month, confounds the text with time and demand.
Use an existing, authorised identifier where possible. A browser identifier is only a proxy for a person: cookie deletion, shared devices and switching devices can break consistency. Document this limitation rather than claiming perfect customer tracking. Do not collect extra personal data simply to make the experiment convenient.
One customer contributes one primary outcome. Ten visits from that customer are not ten independent observations. If the change affects a whole business account or staff team, assignment may need to occur at that group level. The analysis must then account for dependence within groups; the calculation below no longer applies. Microsoft’s methodological discussion explains why changing the randomisation unit changes variance and power.
Choose one primary metric and its denominator
Define success as at least one confirmed booking within seven days of first assignment. The primary rate is customers with that outcome divided by all eligible assigned customers whose seven-day observation window has finished. Multiple bookings still count once. Define a confirmed booking and the treatment of cancellations before collecting outcomes.
Retain customers in their assigned groups even if they did not click the button. Filtering to clickers would condition the comparison on behaviour the text might change. Record delivery failures separately instead of silently removing them.
Also monitor complaints about misleading information and technical failures. These are safeguards, with explicit escalation rules, not replacement success metrics chosen after seeing results. Exploratory breakdowns can suggest future questions; finding one attractive segment does not rescue an unsuccessful primary comparison.
Calculate whether the experiment fits the business
Suppose comparable historical records suggest a 10% booking rate. Treat that as a planning estimate, not a known constant. The workshop chooses a minimum detectable effect of 3 percentage points: 10% to 13%, a 30% relative increase. Smaller worthwhile effects would need a different, generally larger, sample.
Assume independent customers, equal allocation, a two-sided significance level of 5% and 80% power. Power means the planned test would detect the assumed effect in about 80% of repeated comparable experiments. It is not an 80% probability that B is better. NIST describes how baseline proportions, effect size, significance and power enter sample planning; its linked calculation is for one proportion, not a ready-made two-arm sample size.
For our two independent proportions, let pA = 0.10, pB = 0.13 and their mean m = 0.115. The normal-approximation planning calculation per arm is:
n ≈ [1.96 × √(2 × m × (1 − m)) + 0.8416 × √(pA × (1 − pA) + pB × (1 − pB))]² ÷ (pB − pA)².
This gives approximately 1,774 customers per arm. A working target of 1,800 per arm means approximately 3,600 customers. This is an approximation without continuity correction; confirm the design with a validated statistical implementation or a qualified analyst. The statsmodels methodological documentation describes normal-approximation sample planning for two independent proportions, including the approximation that ignores the far tail in a two-sided test.
At 100 eligible customers per week, 3,600 customers require about 36 weeks of enrolment, then seven days for the final outcomes. Random assignment does not guarantee exactly equal groups. Seasonal changes and the cost of holding a page constant may make this plan unsuitable. AI cannot solve a shortage of independent customers by generating synthetic visitors.
Set the stopping rule before seeing results
If the test is feasible, specify enrolment dates, expected sample size, full business cycles and the final observation window. Allow for ordinary weekly variation. A fixed end date does not guarantee adequate power: record in advance what happens if traffic falls short. An inconclusive result is allowed.
Do not end a conventional fixed-horizon test the first morning its result looks favourable, or repeatedly extend it until significance appears. Repeated testing requires a suitable sequential design established beforehand. Research guidance on monitoring experiments explicitly flags early peeking and multiple testing.
Operational monitoring continues. Broken bookings, misleading claims or privacy incidents justify stopping for safety, without declaring a statistical winner. Check that assignment, exposure and outcome logs join correctly. Investigate substantial unexplained allocation imbalance or missing outcomes before interpreting conversion rates. Do not let AI “repair” missing events by inventing them.
Report uncertainty, including an inconclusive ending
Imagine a separate, prematurely small dataset: A has 20 bookings among 200 customers; B has 26 among 200. Rates are 10% and 13%. The observed difference is 3 percentage points, but it is not proof of improvement.
An illustrative unpooled normal 95% interval uses a standard error of √(0.10 × 0.90 ÷ 200 + 0.13 × 0.87 ÷ 200) ≈ 0.03187. The difference plus or minus 1.96 standard errors gives roughly −3.25 to +9.25 percentage points. This approximate interval includes both harm and benefit; it is not a recommended small-sample default. Preselect an appropriate interval method for the real analysis.
“Inconclusive” does not mean equivalent. Report counts, denominators, the estimated difference, uncertainty, deviations and safeguards. Keep statistical evidence separate from the business judgment about whether an improvement covers implementation costs.
When traffic is too low
Use moderated usability sessions to find misunderstood wording, or a limited operational pilot to check that bookings and logs work. These can reveal problems without estimating a population conversion lift. A larger, genuinely meaningful change may be worth testing, but do not inflate the detectable effect merely to shrink the sample.
The final deliverable can be a decision not to launch: “Available traffic cannot resolve the improvement we care about within a stable period. We will first investigate comprehension.” AI helps organise the plan and check calculations. It does not supply missing evidence.
Sources checked 7 November 2026. The scenario, workflow and numerical examples are original editorial teaching material. No product comparison or experiment was conducted. Cover: an original conceptual editorial illustration generated with OpenAI for Nadali. It is not a documentary photograph, product interface or evidence of testing.
