What "Validated" Actually Means: A Coach's Guide to Vetting Mental Assessments
"Validated" might be the hardest-working word in sport-tech marketing.
Sometimes it means an instrument with years of peer-reviewed research behind it. Sometimes it means someone's cousin liked the quiz. The brochure looks identical either way — and coaches, who were never trained in psychometrics and shouldn't have to be, are somehow expected to tell the difference before signing their athletes up.
That's not a fair position to put coaches in. So here's the working guide I wish every program had: what the key terms actually mean, and the six questions that separate credible tools from confident marketing.
One note on where I sit before we start. I'm a Certified Mental Performance Consultant® and the co-founder of a company that builds mental performance systems — which means I'm exactly the kind of person this article teaches you to interrogate. Good. Aim every one of these questions at people like me. Any credible provider will be glad you asked.
The terms, in plain language
Reliability: does it measure consistently? If the same athlete takes an assessment under similar conditions, the results should be similar. A bathroom scale that reads 180 on Tuesday and 195 on Wednesday isn't measuring your weight — it's generating numbers. Unreliable assessments do the same thing with your athletes' confidence and composure: they produce output that looks like data but can't be trusted from one sitting to the next.
Validity: does it measure what it claims to measure? This is different from reliability, and the difference matters. An instrument can be perfectly consistent and consistently wrong — reliably measuring something other than what's on the label. Validity is about whether "composure under pressure" on the report actually reflects composure under pressure in the athlete. It's the harder standard to meet, which is exactly why marketing language loves to blur it.
Peer review: has anyone independent checked? Peer-reviewed means researchers with no stake in the product examined the evidence — how the instrument was built, what it measures, how well — and the work was published where other scientists can challenge it. That is a categorically different claim than "research-backed," "science-based," or "developed with experts," none of which are verifiable and all of which can be printed on anything. Peer review isn't perfect. But it's the difference between evidence that survived scrutiny and evidence that never faced any.
Population: who was it built for? An instrument validated with corporate managers, or with the general public, was not built for the job of assessing developing athletes. That doesn't make it a bad instrument — it makes it the wrong tool for this job, the way an excellent road bike is the wrong choice for the playoffs. The question is always fit: was this developed and tested with athletes, and are your athletes' ages inside the range it was designed for?
The six questions
Ask these of any provider — in an email, in a demo, in the hallway at a conference. Write the answers down.
1. What instrument does this use, and where is the peer-reviewed evidence for it?
A credible answer names the underlying research and offers to share it. "It's a proprietary algorithm" is not an answer — it's a request to trust the brochure.
2. What exactly does it measure — and what does it not measure?
Real instruments measure specific, defined factors, and honest providers are quick to name the limits. Be cautious with anything that claims to capture everything about "the mental game" in one score. Precision is a feature; vagueness is a tell.
3. Who was it developed and validated with?
Athletes or office workers? What ages? A tool built for adults in the workplace was not designed for your under-16 room, no matter how good the dashboard looks.
4. Who interprets the results, and what are their qualifications?
Data does not interpret itself. Somewhere between the raw scores and the decisions made about your athletes, a human turns numbers into meaning — and that person's training matters as much as the instrument. If the answer is "the report explains itself," ask who's accountable when a fifteen-year-old misreads what their scores mean.
5. What happens with my athletes' data — and who sees individual results?
You should get a specific, written answer: who sees what, at what level of detail, stored where, for how long. This question matters enough that I'd walk away from any provider who hedges on it. (What good data ethics look like in development sport deserves its own article — it's coming.)
6. What happens after the assessment?
If the answer ends at "you receive a report," you're looking at a report mill, not a development tool. Assessment is a starting point. The value is in what it starts: interpretation, priorities, and a plan someone helps you act on.
The pattern to watch for
Here's what I've noticed over a decade of watching programs make these decisions: the providers with real answers enjoy these questions. They name the research. They tell you what their tool can't do. They put data commitments in writing without being pushed.
The ones who get defensive, or drown you in jargon, or pivot back to the demo — they've answered a different question, and it's the one that matters.
Your athletes will answer whatever assessment you choose honestly, because you asked them to. Their honesty deserves an instrument that earns it.
Hold every tool to that bar. Especially ours.
--
Joshua Hoetmer, MA, is a Certified Mental Performance Consultant® and Counselling Therapist with over a decade in high-performance sport across hockey, baseball, lacrosse, and basketball. He co-founded Mental Metrics Lab with Dr. Sharleen Hoar, PhD, Certified Mental Performance Consultant® and High Performance Sport Scientist, who has supported athletes and coaches across five Olympic and Paralympic cycles, Vancouver 2010 to Paris 2024. Mental Metrics Lab is a structured mental performance system for athlete development, built on a validated, peer-reviewed instrument. Learn more at mentalmetricslab.ca.
