How to Choose a Conversational AI Partner: 10 Questions That Separate Builders from Resellers

The short version: the conversational-AI market is crowded with resellers wrapping the same APIs in different pricing pages. A capable partner is distinguishable by how they answer ten concrete questions about grounding, escalation, evaluation, data ownership and economics. Ask them all; the hand-waving starts quickly.

1. "How will the bot know our information — and only our information?"

The answer should include retrieval-augmented generation (RAG): your content indexed and retrieved per question, with the model constrained to answer from it. If you hear "we fine-tune ChatGPT on your data" with no mention of retrieval, grounding or citations, be cautious — that phrase is more often marketing than architecture.

2. "What happens when it doesn't know?"

You want confidence thresholds, honest "let me connect you" behaviour, and warm handover with transcript and summary. A vendor who claims their bot "handles everything" has never watched real traffic.

3. "Show us a failure."

Serious builders keep examples of hard cases — ambiguous questions, prompt-injection attempts, angry customers — and can show how the system behaved and what they changed. Polished demos are easy; failure handling is the product.

4. "How do you measure quality before and after launch?"

Listen for: evaluation sets built from your real transcripts, accuracy targets agreed before go-live, regression tests when prompts or models change, and weekly review of low-confidence conversations. "We monitor it" is not an answer.

5. "Who owns the data, the prompts and the accounts?"

The only acceptable answer: you do. Your CRM records, conversation logs, knowledge base, message templates, phone numbers and platform accounts should live in accounts you control, with the partner holding scoped access they can hand back. Anything else is a hostage situation with a monthly invoice.

6. "Which model providers do you use, and can we switch?"

Good partners are provider-agnostic — OpenAI, Anthropic, Google, open-weight — choose per task on benchmarks, and architect so the model layer can change without a rebuild. Model loyalty is a red flag; the leaderboard changes quarterly.

7. "How is our customers' data protected?"

Expect specifics: enterprise API tiers with no-training guarantees, PII redaction in logs, least-privilege credentials, encrypted transport, and willingness to sign NDAs/DPAs. For regulated industries, expect them to volunteer constraints before you raise them.

8. "What does the first month look like?"

A credible plan is phased: discovery on real transcripts, a scoped pilot, supervised launch, weekly tuning. Beware both extremes — "live tomorrow" (template reseller) and "six months of workshops" (consulting theatre).

9. "How does pricing behave when we grow?"

You want transparent build costs plus usage-based running costs you can model — and a partner who proactively engineers usage down (caching, routing smaller models, trimming context). If costs are opaque or 'per seat' for software no human sits in, keep walking.

10. "Who, exactly, will work on our account?"

Conversation design and AI engineering are crafts. Meet the people doing the work, not just the closer. Ask what they shipped last quarter and what broke.

A note on choosing us

We publish this list knowing competitors can read it — that is the point. These are the questions we would ask in your seat, and the standards we hold our own engagements to. If you put us through all ten, we will enjoy the meeting.