What a sex toy survey has to do before anyone should quote it

Why this category is so badly measured

Sexual behavior is among the hardest things to survey honestly. People under report what they expect to be judged for and over report what they expect to be admired for, and the direction of that bias changes with the wording of the question, the country, and whether a human being is listening. A result produced without attention to all three is not a measurement of behavior: it is a measurement of what people are willing to say.

On top of that sits a commercial problem. Almost every widely quoted figure in this category originated with a company that sells the product, usually from a poll of its own customers or newsletter subscribers. Asking buyers how many people buy is circular by construction, and the result is always flattering. The figure then enters a news article, loses its method, and is cited thereafter as though it were population data.

That is the gap a properly run survey could fill, and it is the only reason to run one. A survey that cannot be held to a higher standard than the figures it is meant to replace is not worth the cost of its field work.

What a survey must publish to be citable

• Who was eligible, and how they were reached. The sampling frame, named. An online panel is a legitimate frame and a self selected website poll is not, and the difference has to be stated rather than implied.
• The provider and the panel. Which company ran the field work, and how its respondents were recruited and are compensated. A panel assembled by rewarding people for completing surveys has known biases, and acknowledging them is part of being usable.
• The sample size per market, not in total. A combined figure across five countries can conceal a cell of eighty people, and any national comparison drawn from such a cell is noise. Each market stands on its own count.
• The weighting. Which population figures the data was weighted to, from which statistical agency and which year, and on which variables. Unweighted panel data skews young, urban and online, and says so if it is honest.
• The exact question wording, in every language fielded. Translation changes answers in this subject more than in almost any other, so the wording belongs in the publication, not in a methodology available on request.
• The field dates and the completion rate. When it ran, how many started, how many finished, how many refused. A high refusal rate is itself a finding in a sensitive subject and hiding it misrepresents everything else.
• The raw tables. Counts, not only percentages, with margins of error stated per market. A figure whose underlying count is withheld cannot be checked, and this site does not publish figures that cannot be checked.

Pre registration, because the temptation is obvious

A survey commissioned by a publisher that earns affiliate commission has an interest in certain answers. Stating that is not enough on its own, because the pressure does not act at the moment of publication: it acts earlier, in which questions get asked, which cuts of the data get written up, and which results quietly do not.

The protection against that is to fix the questions, the planned analysis and the markets in writing before the field work starts, and to publish that document alongside the results, including the analyses that produced nothing. A survey where only the interesting findings appear is indistinguishable from a survey that was mined until something interesting turned up.

The same applies to the press release. If a survey is worth running, the findings get published in full on this site first and any summary sent to a journalist links to the full tables, not the other way round.