Stealth Technology Group Team
A marketing manager finds an AI meeting assistant that records calls, writes summaries, and drafts follow-up emails. She signs up with her work account in about two minutes. The tool asks for access to her calendar, her email, and her shared drive, and she clicks accept. It works well. Within a month, half the sales team is using it, and recordings of client calls, pricing discussions, and contract negotiations are sitting in a vendor’s cloud under terms nobody in the company has read.
Nothing dramatic has happened yet. But the company now has a new third party holding sensitive conversations, possibly using them to improve its models, possibly sharing them with subprocessors, and possibly retaining them long after anyone needs them. If a client later asks where their confidential discussion went, nobody can answer.
AI tools are being adopted faster than the review processes built to evaluate software vendors, and they carry risks that traditional vendor questionnaires do not cover. This guide sets out the questions that matter, how to scale the review to the risk, what the contract should say, and when the right answer is no.
Why AI Tools Need Their Own Vendor Review
Most organizations already review vendors that touch sensitive data. AI tools add several wrinkles that standard questionnaires miss. First, the data flows are unusual. An AI tool often ingests large volumes of unstructured content, such as documents, transcripts, emails, and chat history, rather than a defined set of fields. Anything a user can paste or connect may end up in the vendor’s environment.
Second, the vendor may use customer data in ways ordinary software does not. Some providers use inputs to train or refine models unless customers opt out. Others retain prompts for abuse monitoring or human review. The default setting differs between vendors and between plan tiers, and the difference between a consumer and an enterprise plan can be the difference between data being used for training and not.
Third, there is a supply chain behind the vendor. Many AI products are thin layers on top of a foundation model supplied by another company, which may in turn rely on cloud and infrastructure providers. Your data may pass through several hands, each with its own terms.
Fourth, the permissions are broad. AI assistants frequently request wide access to email, files, and calendars so they can be useful, and agents that can take actions on your behalf raise the stakes further. Finally, products change quickly. A feature added in a monthly update can alter the data handling that you approved last quarter, and AI features are increasingly embedded in tools you already use.
The NIST AI Risk Management Framework offers a structured way to think about these risks, covering governance, mapping context, measuring, and managing. You do not need to adopt it wholesale to benefit from its vocabulary, but it is a helpful reference when building your process.
Start by Finding Out What Is Already in Use
You cannot assess what you do not know about, and with AI the gap between sanctioned and actual use is usually wide. Employees try new tools constantly, often with free accounts and company email addresses.
Begin with discovery. Review identity logs and connected-application lists for AI tools that employees have signed in to or authorized. Check expense reports and corporate cards for subscriptions. Ask department heads what they use and why. The result is an inventory of tools, users, the data each touches, and the permissions each holds. Our articles on shadow AI and employee data exposure and SaaS sprawl describe how these tools accumulate and why the inventory is often a surprise.
Discovery also shows demand. If many people adopted the same meeting assistant, the underlying need is real, and the better response is to evaluate and sanction a tool that meets it rather than banning use and watching it continue unobserved. A clear AI usage policy makes the rules visible, and a sanctioned tool gives people a reason to follow them.
Questions About Data Handling
Data handling is the heart of the assessment. These are the questions that separate a safe arrangement from an unsafe one. Does the vendor use our inputs, outputs, or uploaded files to train or improve any model, theirs or a third party’s? Is that off by default, or do we have to opt out, and is the opt-out contractual or merely a setting? Do the answers differ by plan tier?
How long is data retained, and can we control it? Is deletion actually performed when we delete, and what happens in backups? Is there a zero-retention option for sensitive workloads? What happens to all of our data when the contract ends, and will the vendor certify deletion? Where is data processed and stored? Which regions, and can we choose? Is it logically separated from other customers’ data? Is it encrypted in transit and at rest, and who holds the keys?
Can vendor staff view our prompts or content? Under what circumstances, such as abuse monitoring or support, and with what access controls and logging? Is human review of content possible, and can it be disabled? Does the product learn from one customer and surface that knowledge to another? Cross-customer leakage through shared models or shared retrieval indexes is a real risk worth asking about directly.
Vendors that answer these precisely, in writing, and point to specific contract language are showing maturity. Vendors that answer in marketing terms, or say that data is “secure” without specifics, have not done the work.
Questions About the Model Supply Chain
Behind many AI products sit other companies, and your risk includes theirs. Ask which foundation models the product uses and who provides them. Ask whether the model provider receives your data and under what terms, and whether those terms match the ones you accepted. Ask for a list of subprocessors, including cloud, analytics, and support providers, and whether the vendor will notify you before adding new ones.
Ask how the vendor handles changes. If they switch model providers or update to a new model version, will you be told, and can you object? A change in the underlying model can alter data handling, output quality, and behavior, and a vendor that can change it silently is a vendor that can change your risk profile without your knowledge.
For public-sector and defense-adjacent work, also ask about authorizations. Controlled environments may require services that meet specific federal standards, and the FedRAMP program lists authorized cloud offerings. Whether a given AI service appears there, and at what impact level, can decide whether it can be used for certain data at all.
Questions About Security
Treat an AI vendor like any other vendor that holds sensitive data, and then add the AI-specific questions.
For the fundamentals, ask for independent evidence rather than assertions: a recent SOC 2 Type II report or ISO 27001 certification, the date and scope of penetration testing, and a summary of remediation. Ask about single sign-on and multi-factor authentication support, role-based access, audit logging you can export, encryption, vulnerability management, and a documented incident response process. Ask how quickly they would notify you of a breach affecting your data, and get the timeframe in the contract.
For AI-specific risk, ask about defenses against prompt injection, where malicious text in a document or web page manipulates the tool into ignoring instructions or leaking data. The OWASP Top 10 for Large Language Model Applications catalogs the common weaknesses, including insecure output handling, excessive agency, and sensitive information disclosure, and is a useful checklist for technical questions. Ask how the vendor tests for these, how outputs are filtered, and whether the product restricts what an agent can do without confirmation.
Ask about integrations. What permissions does the tool request when connected to your email, files, or calendar? Are they read-only or read-write? Can scopes be limited to specific folders or accounts? Can you revoke access centrally? Tools that insist on broad access for narrow functions deserve skepticism. This matters most for agent-style tools that take actions on your behalf, a distinction explained in our article on AI agents vs. traditional automation.
Questions About Accuracy, Accountability, and Governance
Security is not the only risk. AI tools produce confident output that can be wrong, and a vendor’s approach to that problem affects how safely you can use the tool.
Ask how the vendor addresses inaccurate output. Does the product cite sources, show confidence, or flag uncertainty? Can it be restricted to approved content? What evidence do they have of testing and evaluation, and can they share it? Ask how you can monitor quality over time, and whether version changes are documented.
Ask about accountability. Who owns the outputs, and who owns the inputs? What does the vendor say about intellectual property, and do they offer any indemnification if generated content infringes someone else’s rights? Understand what the vendor promises and, equally, what it disclaims. Many terms of service place all responsibility for outputs on the customer.
Ask about the human side. What controls support human review of outputs before they are used in client work or regulated activities? Can you configure approval steps, logging of who accepted what, and restrictions by role? Governance features such as these determine whether the tool can be used responsibly in a business setting. Our article on strategic intelligence and AI governance explores how to align these controls with existing security and compliance programs.
What the Contract Should Say
Answers given on a call or in a questionnaire are only as good as the contract that backs them. Before approving a higher-tier tool, review the agreement and data processing terms for the following points.
A clear statement that your data will not be used to train models, written as a commitment rather than a setting. Defined retention and deletion obligations, including at termination, with certification of deletion. A list of subprocessors with notice and the right to object to changes. A breach notification commitment with a specific timeframe. Security obligations that match what the vendor represented. Audit rights or access to independent assessment reports.
Reasonable limits on the vendor’s liability, especially for data exposure. Ownership of inputs and outputs. And notice before material changes to the service or terms. Regulated data adds requirements. Under HIPAA, an AI vendor that creates, receives, or maintains protected health information on your behalf must sign a business associate agreement, and the HHS guidance on business associates explains the obligation.
For defense contractors, an AI service that processes controlled unclassified information is part of the compliance environment and must meet the requirements in NIST SP 800-171, which is why many contractors prohibit consumer AI tools for that data entirely. Our CMMC compliance and compliance programs help organizations decide where AI tools fit and document the decision, and our guide on how third-party vendors affect CMMC compliance covers the broader vendor obligations.
Legal and professional services firms have their own duties. Legal practices must protect privileged and confidential client information, and finance, healthcare, and engineering organizations each handle data categories that restrict where it may go. The assessment should state which categories the tool may and may not touch.
Decide, Document, and Keep Watching
An assessment ends in a decision, and the decision should be recorded. The usual outcomes are approve, approve with conditions, or decline. Conditions are often the most practical result: approve the tool for specific teams and data types, require single sign-on and multi-factor authentication, disable training and retention features, restrict integrations to defined scopes, and require human review for certain outputs.
Record the decision, the reasoning, the data tier, the conditions, the owner, and the review date. Keep an approved tools list that employees can consult, so that the answer to “can I use this?” is easy to find. Without that list, people guess.
Then keep watching. AI products change quickly, so schedule reassessment at least annually and whenever the vendor announces a significant change: new model providers, new features, acquisition, new terms, or a security incident. Monitor connected applications and permissions on an ongoing basis, since scope creep is common. Our guide to AI readiness assessment can help you evaluate whether your organization has the data, governance, and infrastructure to use AI safely before you sign more contracts.
Red Flags
Some signals should slow or stop an approval. A vendor that will not state clearly whether it trains on your data. Terms that grant the vendor broad rights to use your content. No independent security evidence, or reluctance to share it. Requests for sweeping permissions that exceed the function. No clear deletion process. A marketing-heavy response to technical questions. An unwillingness to put commitments in the contract. A free or consumer-tier product proposed for sensitive data. And pressure to approve quickly because “everyone is already using it,” which describes a process failure rather than a reason.
A related warning sign is internal: if nobody owns the decision, tools are approved by default. Assign an owner for AI vendor reviews, with defined escalation to legal and security for higher tiers.
Who Should Run the Process
AI vendor risk assessment sits between IT, security, legal, compliance, and the business, and in many mid-market companies nobody owns all of it. The work is not complicated, but it is recurring, and it benefits from experience reading contracts and security evidence.
Stealth Technology Group’s AI Strategy & Governance service helps regulated organizations build the intake process, risk tiers, questionnaires, and approved-tools list described here, and our cybersecurity and managed IT services programs handle the technical controls such as identity, permissions, and monitoring. A vCIO can own the process for organizations without a dedicated security or compliance lead, and a co-managed IT arrangement lets internal staff keep day-to-day involvement. Organizations in Boston, Tampa, and Sarasota can read the client results from companies that built governance before scaling AI use.
Conclusion
AI tools can deliver real value, and most businesses will use many of them. The question is not whether to connect them to company data but on what terms. The meeting assistant in the opening story was probably a good product. The failure was that nobody asked what it did with the recordings before it was connected.
A workable process is not heavy. Find out what is already in use. Tier tools by data sensitivity. Ask specific questions about data handling, the model supply chain, security, and accountability. Put the commitments in the contract. Record the decision, and reassess when the product or the vendor changes. Do that consistently, and you can say yes to AI quickly where the risk is low, slow down where it is high, and explain your reasoning to a client, an auditor, or an insurer who asks.
If your organization is planning its CMMC compliance journey, contact Stealth Technology Group today at (617) 903-5559 or visit the website to learn how modern cybersecurity infrastructure can accelerate your path toward certification readiness.
