Testing when media costs move every week: here is how long to let a variant run and when the data is reliable.
Short windows and fast decisions
Three benchmarks of duration.
A test lasts no more than a week: beyond that, the environment has changed too much for the comparison to hold.
Nor does it last less than three days: below that, you are reading noise, including when the figures look clear.
And it stops at a spend threshold, not at a number of days: a variant that has not reached a sufficient volume of purchases has demonstrated nothing.
One practical consequence: decide that threshold before launching, and write it down. The temptation to stop a test on the second day, when a variant seems to be losing, is the leading source of bad decisions.
Compare with the control, never with last month
Three reasons, all structural.
Media cost varies strongly from one week to the next, and it varies for everybody at once.
The displayed price of your product moves, which mechanically shifts the conversion rate.
And local seasonality is marked: pay periods, holidays and school terms change buying behaviour within days.
A correct test therefore isolates one single variable: the creative. Everything else, audience, budget, day, placement, has to be identical between control and variant.
Hooks, not new videos
Three reasons to test the first second rather than the whole video.
It is the part explaining most of the performance gaps.
It is the cheapest to produce: three hooks on the same video body cost an edit, not a shoot.
And it is the quickest to conclude: a hook gap shows in retention data before you even look at purchases.
Three variants per cycle are enough. Beyond that, the budget spreads out and no variant reaches the threshold that would make its measurement reliable.
What to measure when the basket changes
Three indicators, in this order of reliability.
The three-second retention rate, which depends neither on price nor on purchasing power: that is the most stable measure you have.
The click rate, which depends a little on context but stays comparable between two variants launched on the same day.
And the cost per purchase, the most useful commercially, but the most sensitive to price variations: it can only be compared within the same window.
One indicator to handle with care: revenue generated. It mixes the effect of the video and the effect of the price, and it becomes misleading as soon as a promotion runs.
Keep the lesson, not the video
Three consequences for what follows.
What is kept is the rule learned: which angle worked, which promise, which type of person.
What wears out is the video itself: a winning creative exhausts itself in a few weeks, whatever its initial success.
And what is replayed is the structure: reshooting the same angle with another creator gives a new video without starting from scratch.
One simple habit worth money: keep a test log, one line per variant, with the date, the angle, the result and the decision. Without that log, you buy the same lesson twice.
What is tested, and in what order
| Rank | Variable | Why first |
|---|---|---|
| 1 | The hook | It decides the three-second view |
| 2 | The argument | It decides the full view |
| 3 | The call to action | It decides the click |
What makes a test readable
One variable at a time, an identical budget on each side, and an equal run time. Without those three conditions, the result cannot be attributed to anything.
What is kept from each test
The winning file, the losing variant and the reason written in one line. Without that note, the same mistake recurs three months later.
The mistake that costs most
Three forms of the same mistake.
Changing two things at once, the video and the audience, which makes the result uninterpretable.
Stopping a variant before its spend threshold, believing you are reading a trend.
And concluding from a one-week test that the angle is bad, when it simply ran into a bad week.
One last marker, valid in an unstable market: when you doubt the data, increase the size of the test before increasing the number of tests. One reliable measurement on three variants is worth more than ten uncertain ones.
