How do I evaluate AI vendor claims on a product page?

Start with the exact words, not a paraphrase. For each claim, name the type (accuracy, compliance, automation, ROI, or vague AI label) and the specific words that create an evidence burden—a percentage, 'fully automated', 'compliant', or 'guaranteed'. Then name the evidence the claim would need to be verifiable in your environment: benchmark scope, audit coverage, baseline comparison, or test conditions. Finally, write one concrete question you can ask the vendor that names the claim and the evidence you need.

What makes an AI vendor claim credible versus unsupported?

A credible claim names the conditions under which it holds: what was tested, under what setup, with what sample, over what period, with what limits. An unsupported claim uses the same words but omits those conditions. 'Our AI achieves 95% accuracy on customer support requests, measured on 10,000 tickets across three industries with a 3% false-positive rate' is more credible than '95% accurate AI support.' The difference is not the number—it is the stated scope, sample, and limit that make the number verifiable.

How do I know if an AI vendor's marketing claims are accurate?

You cannot verify accuracy from the marketing page alone. What you can do is ask for the evidence: the benchmark or test that produced an accuracy figure; the audit scope that covers a compliance label; the baseline a cost-saving claim used. If the vendor cannot provide specific evidence for claims on their product page, that does not mean the product does not work—but it does mean the evidence burden has not been met, which is a signal to press further before relying on it.

What questions should I ask an AI vendor about their product claims?

For accuracy or performance claims: 'What test produced this number, and does it apply to our content type and volume?' For compliance claims: 'Which product features and data flows are in scope, and can we see the audit report?' For automation claims: 'Which steps run end to end, what triggers human review, and who is responsible for errors?' For ROI claims: 'What baseline and sample produced this figure, and does it apply to our industry and scale?' These questions can be asked in a sales call, a security questionnaire, or by running the claim through a structured review.

How is evaluating AI vendor claims different from evaluating regular software claims?

AI claims often carry a higher evidence burden because they involve statistical outputs, trained-model behavior, and compliance implications that traditional software does not. A spreadsheet either sums a column correctly or not. An AI model produces outputs that vary by input, training data, configuration, and context—so accuracy claims, automation claims, compliance claims, and ROI claims each require evidence about those specific conditions. The FTC has stated that AI performance and compliance claims should be backed by competent and reliable evidence before they are published.

What is the difference between AI hype and a real AI claim?

AI hype uses terms like 'AI-powered', 'AI-native', or 'intelligent automation' without explaining which workflow step the AI runs, what input it takes, or what output it produces. These are labels, not measurable claims. A real AI claim names a specific task, a tested outcome, stated test conditions, and limits. The test: if the word 'AI' is removed, does a specific capability remain? If yes, the claim is at least concrete enough to evaluate. If the claim collapses without 'AI', it is a label claim that needs to be pressed for specifics.

How to evaluate AI vendor claims before you buy

Last reviewed July 9, 2026

An AI vendor's product page, demo deck, or sales pitch makes specific claims about accuracy, automation, compliance, and ROI. This page gives buyers a structured process to turn those claims into evidence questions—before you demo, before you buy, or before you sign.

Check an AI vendor claim now How the evidence method works

Fastest path: copy one exact vendor sentence that matches this pattern, then open the checker. Add the public URL only if you want readable page context recorded alongside the wording. The result is an evidence-burden note you can reuse in vendor follow-up or internal review, not a verdict. Not sure what a result looks like? See a sample receipt.

What to verify before you rely on the claim

The exact claim text, not a paraphrase.
The claim type and the specific word that creates the evidence burden.
The evidence (benchmark, audit, baseline, test conditions) that would support the claim in your environment.

Sources behind How to evaluate AI vendor claims

FTC AI business guidance: Keep Your AI Claims in Check FTC guidance
· February 27, 2023
FTC guidance on the evidence standard for AI-related business claims, including the competent and reliable evidence requirement.
ASA AI as a marketing term report ASA/CAP guidance
· November 13, 2024
Official ASA/CAP report on how AI is used as a marketing descriptor, with evidence expectations for AI-related advertising claims.
FTC v. DoNotPay Inc. — AI Professional Replacement Claims FTC enforcement
· February 2025
Enforcement case showing that automation and professional-replacement AI claims require task scope, failure handling, and human-review disclosures.
FTC v. Workado — AI Content Detector Accuracy Claims FTC enforcement
· August 28, 2025
Enforcement case showing that AI accuracy claims require benchmark scope, content type coverage, and error rate disclosures.

Documented How to evaluate AI vendor claims examples

"AI-powered resolution rate of 95%"

Accuracy / Performance

Source and date: Common AI customer support marketing pattern · Pattern aligned with FTC AI enforcement standards
Evidence signal: Accuracy figure without a definition of 'resolved', denominator, escalation rate, or measurement period.
Evidence gap: Resolution definition, ticket categories, repeat-contact rate, escalation rate, human handoff count, and the period and product version measured.
Buyer question: For the 95% resolution rate claim, what counts as resolved, what is the repeat-contact rate, and how many tickets escalate to a human agent?

Load this sample in the checker

"Fully automated contract review with AI"

Automation / Replacement

Source and date: Common AI legal tool marketing pattern · Pattern aligned with FTC DoNotPay enforcement (February 2025)
Evidence signal: Full automation language without task scope, failure handling, or human-review boundary.
Evidence gap: Which contract types and clauses are in scope, what the AI misses, when a lawyer must still review, and what happens when the AI flags something incorrectly.
Buyer question: For the fully automated contract review claim, which clause types are in scope, which require a lawyer to confirm, and what does the AI do when it is uncertain?

Load this sample in the checker

"HIPAA-compliant AI for healthcare data"

Compliance / Safety

Source and date: Common AI compliance marketing pattern · Pattern aligned with HHS/OCR HIPAA AI marketing guidance
Evidence signal: Compliance label without audit scope, BAA terms, or the data flows the certification covers.
Evidence gap: Whether a BAA is available, which PHI data flows the product covers, what the audit or certification scope includes, and what falls outside the AI product's compliance boundary.
Buyer question: For the HIPAA-compliant claim, is a BAA available, which PHI data flows does the certification cover, and what is explicitly outside scope?

Load this sample in the checker

Steps for checking How to evaluate AI vendor claims

1. Write down the exact claim words Direct quote from the product page, demo, or email
What is the exact sentence—not a paraphrase—that the vendor is asking you to rely on?
2. Name the claim type Accuracy, compliance, automation, ROI, first/only/best, or vague AI label
Is this a measured result, a compliance scope, an automated task, a financial return, a competitive position, or just an AI label?
3. Identify the words that raise the evidence burden Percent figures, always/never, fully automated, compliant, guaranteed, first/only/best
Which specific words would require evidence if challenged—a number, a scope, an absolute, or a comparison?
4. Name the evidence the claim would need Benchmark, audit report, DPA, test conditions, sample size, baseline, or deployment scope
What document or data would support this claim in your specific environment, not just in a vendor-controlled demo?
5. Write one concrete question to ask the vendor Scoped question that names the claim and the evidence you need
Can you phrase this as a single sentence the vendor could answer with a specific document, number, or scope boundary?
6. Record the answer and what is still unresolved Vendor response, attachment, scope note, or outstanding gap
What did the vendor provide, and what evidence burden is still unmet after their response?

Evidence map for How to evaluate AI vendor claims

Claim pattern	Evidence needed	Buyer question
AI accuracy or performance figure ('95% accurate', 'human-level results')	Task definition, test dataset, model version, sample size, error rate, false-positive/negative rate, and conditions that match the buyer's use case.	What benchmark or test produced this number, and does the test setup reflect how we would use the product?
AI compliance or certification label ('HIPAA-compliant', 'SOC 2 certified', 'GDPR-ready')	Certification body or auditor, audit period, scope description, which product features are covered, and what the certification excludes.	Which product surfaces and data flows are in scope for this certification, and can we see the scope letter or report?
AI automation claim ('fully automated', 'no human review needed', 'end to end')	Task scope, failure handling, human handoff criteria, audit trail, and escalation path for edge cases.	Which tasks does the AI handle end to end, what triggers human review, and who is responsible when the AI produces an error?
AI ROI or outcome claim ('saves 40% of costs', 'increases revenue by X%')	Baseline definition, sample size, industry, measurement period, and whether the figure applies to the buyer's environment.	What baseline and sample produced this number, and does it apply to an organization of our size and type?
AI label claim ('AI-powered', 'AI-native', 'intelligent automation')	Which workflow step the AI runs, what input it takes, what output it produces, and what a human still reviews.	If the word AI is removed from this claim, what specific product capability remains?

Evidence buyers need for How to evaluate AI vendor claims

The exact claim text, not a paraphrase.
The claim type and the specific word that creates the evidence burden.
The evidence (benchmark, audit, baseline, test conditions) that would support the claim in your environment.
A concrete question that names the claim and the evidence you need.
A record of what the vendor provided and what remains unresolved.

Buyer questions for How to evaluate AI vendor claims

What is the exact claim text from the product page, demo, or email?
Is this an accuracy, compliance, automation, ROI, or vague AI label claim?
Which word creates the evidence burden: a percentage, 'fully', 'compliant', 'guaranteed', or 'first'?
What benchmark, audit, baseline, or test conditions would support this claim for our use case?
What one question can we ask the vendor that names the claim and the evidence we need?
What remains unresolved after the vendor responds?

Safer wording for How to evaluate AI vendor claims

Reported X% on a named benchmark with stated test conditions, for [content type/task/population].
Automates [named task types]; [other task types] remain under human review.
Covered by [named certification] for [product/feature scope]; excludes [named exceptions].
Based on a study of [sample size] customers in [industry] over [period], with [baseline] as the comparison.

How to evaluate AI vendor claims questions

How do I evaluate AI vendor claims on a product page?: Start with the exact words, not a paraphrase. For each claim, name the type (accuracy, compliance, automation, ROI, or vague AI label) and the specific words that create an evidence burden—a percentage, 'fully automated', 'compliant', or 'guaranteed'. Then name the evidence the claim would need to be verifiable in your environment: benchmark scope, audit coverage, baseline comparison, or test conditions. Finally, write one concrete question you can ask the vendor that names the claim and the evidence you need.
What makes an AI vendor claim credible versus unsupported?: A credible claim names the conditions under which it holds: what was tested, under what setup, with what sample, over what period, with what limits. An unsupported claim uses the same words but omits those conditions. 'Our AI achieves 95% accuracy on customer support requests, measured on 10,000 tickets across three industries with a 3% false-positive rate' is more credible than '95% accurate AI support.' The difference is not the number—it is the stated scope, sample, and limit that make the number verifiable.
How do I know if an AI vendor's marketing claims are accurate?: You cannot verify accuracy from the marketing page alone. What you can do is ask for the evidence: the benchmark or test that produced an accuracy figure; the audit scope that covers a compliance label; the baseline a cost-saving claim used. If the vendor cannot provide specific evidence for claims on their product page, that does not mean the product does not work—but it does mean the evidence burden has not been met, which is a signal to press further before relying on it.
What questions should I ask an AI vendor about their product claims?: For accuracy or performance claims: 'What test produced this number, and does it apply to our content type and volume?' For compliance claims: 'Which product features and data flows are in scope, and can we see the audit report?' For automation claims: 'Which steps run end to end, what triggers human review, and who is responsible for errors?' For ROI claims: 'What baseline and sample produced this figure, and does it apply to our industry and scale?' These questions can be asked in a sales call, a security questionnaire, or by running the claim through a structured review.
How is evaluating AI vendor claims different from evaluating regular software claims?: AI claims often carry a higher evidence burden because they involve statistical outputs, trained-model behavior, and compliance implications that traditional software does not. A spreadsheet either sums a column correctly or not. An AI model produces outputs that vary by input, training data, configuration, and context—so accuracy claims, automation claims, compliance claims, and ROI claims each require evidence about those specific conditions. The FTC has stated that AI performance and compliance claims should be backed by competent and reliable evidence before they are published.
What is the difference between AI hype and a real AI claim?: AI hype uses terms like 'AI-powered', 'AI-native', or 'intelligent automation' without explaining which workflow step the AI runs, what input it takes, or what output it produces. These are labels, not measurable claims. A real AI claim names a specific task, a tested outcome, stated test conditions, and limits. The test: if the word 'AI' is removed, does a specific capability remain? If yes, the claim is at least concrete enough to evaluate. If the claim collapses without 'AI', it is a label claim that needs to be pressed for specifics.

Method and limits

This guide walks buyers through a process for evaluating AI vendor claims. It is not a legal review, compliance certification, procurement approval, vendor ranking, or purchase recommendation.

Cite this page for source-backed evidence gaps and buyer questions, not as a truth finding, legal conclusion, compliance certification, company accusation, or company rating. If you have the exact vendor wording, Check an AI vendor claim now and paste one sentence first. If a source has changed, or you have supporting evidence or a company response, send a private correction or source note.