Must-Ask Questions for AI Vendors Before Giving Away Your Patient Data

An AI vendor promises to cut documentation time, reduce administrative work, and help clinicians spend more time with patients. The demo looks impressive.

The technology sounds promising. The sales team has answers to every question about productivity.

But before you upload a patient record, connect your electronic health record (EHR), or let an AI tool process clinical documentation, there is a more important question to ask:

What happens to your patient data after you hand it over?

For home health agencies, this question goes beyond cybersecurity. AI may interact with referral packets, clinical assessments, visit notes, patient conversations, insurance information, and billing workflows.

A mistake in how that information is handled can create privacy, compliance, operational, and clinical risks.

And with so many vendors claiming to be the next breakthrough in healthcare AI, a polished demo is not enough to establish that a product is safe, suitable, or effective.

Before trusting an AI vendor with patient information, your agency needs clear answers about data handling, security, clinical accuracy, home health expertise, and accountability.

Here are 12 questions to ask before moving forward.

What Should Healthcare Organizations Ask AI Vendors Before Sharing Patient Data?

Start with these 12 questions:

1. Will you sign a Business Associate Agreement (BAA)?
2. What exact patient data will your AI collect and process?
3. Will you use our patient data to train or improve AI models?
4. Where is our data stored and processed?
5. How is patient data protected in transit and at rest?
6. Who can access our patient data?
7. How long do you retain patient data, and how is it deleted?
8. What happens if there is a security incident or data breach?
9. Can you provide independent security assessments and supporting evidence?
10. How do you control AI errors, recording risks, and unauthorized actions?
11. Can you demonstrate real home health workflows and provide customer references?
12. Who reviews your compliance, and what happens to our data if we terminate the contract?

The answers should help your agency determine whether a vendor deserves further evaluation, what risks need to be addressed, and what safeguards should be required before implementation.

1. Will You Sign a Business Associate Agreement Before Accessing PHI?

A Business Associate Agreement, or BAA, is one of the first things to discuss when an AI vendor will create, receive, maintain, or transmit protected health information (PHI) on behalf of a HIPAA-regulated organization.

When a vendor qualifies as a business associate under HIPAA, the required agreement should establish permitted uses and disclosures of PHI, safeguard obligations, and relevant responsibilities between the parties. A general software contract or privacy policy is not a substitute for a required BAA.

Ask the vendor:

• Will you sign a BAA before accessing PHI when one is required?
• Does the agreement cover the specific AI product and services we intend to use?
• Which subcontractors or service providers may handle PHI, and how are their responsibilities addressed?
• What happens if PHI is accessed or disclosed improperly?
• What responsibilities do you accept for incident reporting and cooperation?

What should raise concern? A vendor that avoids the BAA discussion or expects your agency to upload PHI before the necessary agreements and reviews are complete.

A BAA is an important starting point, not proof that every aspect of a product or workflow is secure. Your agency must still assess its own risks, responsibilities, and implementation.

Reference: HHS guidance on business associates and BAAs.

2. What Exact Patient Data Will Your AI Collect and Process?

Not every AI feature needs access to every patient record.

A referral-processing tool may need referral documents, diagnoses, physician orders, and insurance information. A clinical documentation tool may process visit details, clinical observations, audio, or transcripts.

A patient engagement agent may handle phone numbers, messages, and responses.

The vendor should be able to explain precisely what information each feature needs and why.

Ask:

• What PHI does the product collect, receive, generate, or store?
• Does it process names, dates of birth, addresses, clinical notes, photographs, audio recordings, or insurance details?
• Does it require access to our entire EHR, or only selected information?
• Can we restrict access to the minimum data needed for the task?
• Is any information sent to external AI models, APIs, or other service providers?

For example, if an AI tool extracts information from a home health referral, does it need access to the entire patient record, or only the documents required for intake?

A trustworthy vendor should be able to map the data flow from input to output. If it cannot explain what enters its system, where the information goes, and what it generates, your agency cannot properly evaluate the risk.

3. Will You Use Our Patient Data to Train or Improve Your AI Models?

Do not assume that patient data is excluded from model training simply because a vendor sells its product to healthcare organizations.

Depending on the service, configuration, and contract, an AI provider may use submitted information to improve models or services. Other services may offer settings or contractual commitments that restrict such use.

Ask:

• Is our data used to train, fine-tune, or improve any AI model?
• Does that restriction apply to your own models and third-party models?
• Are prompts, uploaded files, transcripts, outputs, and feedback treated differently?
• Can our data be used for product analytics, evaluation, or human review?
• Are these restrictions documented in the contract and applicable product terms?

Look for explicit commitments, not vague assurances. If the vendor says patient data is never used for training, ask which data types, services, and subprocessors that commitment covers.

Also distinguish model training from data processing. A vendor may process information to deliver a service without using it to train a model, but that processing still needs appropriate safeguards and contractual authorization.

Contract language alone cannot authorize a use of PHI that HIPAA does not permit. Confirm that any proposed model-training or improvement use is legally permitted as well as contractually authorized.

See HHS guidance on permitted uses and cloud services.

4. Where Is Our Patient Data Stored and Processed?

Knowing where your data lives is part of understanding how it is protected.

An AI vendor may rely on cloud infrastructure, third-party model providers, transcription services, monitoring tools, or other subprocessors. The vendor's main application may not be the only place where data is handled.

Ask:

• Which cloud providers and AI service providers process our data?
• In which countries or regions is data stored and processed?
• Are backups, logs, transcripts, and temporary files stored separately?
• Can data be transferred across borders?
• Will we be notified if the vendor materially changes its subprocessors or data-processing arrangements?

For home health agencies, this matters when a workflow involves referral PDFs, voice recordings, clinical assessments, or patient messages. Each component may create a different copy or type of data.

The goal is to understand the actual architecture, responsibilities, and risks of the proposed service, rather than assume that a particular cloud configuration is automatically safer.

5. How Is Patient Data Protected in Transit and at Rest?

Ask the vendor to explain its technical safeguards in plain language and provide supporting documentation where appropriate.

Useful questions include:

• Is data encrypted while being transmitted and while stored?
• How are encryption keys managed and protected?
• Are strong authentication and role-based access controls available?
• Is multifactor authentication supported for privileged or administrative accounts?
• How are vulnerabilities identified, prioritized, and fixed?
• How are backups protected and recovered after an incident?

Encryption is important, but it is only one part of a broader security program. Access restrictions, monitoring, risk management, system maintenance, and incident response matter too.

Ask how these safeguards work in the actual product you are buying, rather than accepting a generic statement that the vendor uses “enterprise-grade security.”

6. Who Can Access Our Patient Data?

A vendor may have legitimate reasons for limited access, such as technical support, security investigations, or troubleshooting. The question is how that access is controlled.

Ask:

• Which employees, contractors, and subcontractors can access PHI?
• Is access granted only when required for an authorized purpose?
• Are privileged accounts restricted and monitored?
• Are support sessions logged?
• Can vendor personnel view patient data during troubleshooting?
• How quickly is access revoked when an employee changes roles or leaves?

Also ask whether the vendor can demonstrate who accessed particular records and when.

A strong answer describes specific controls, oversight, and accountability. “Only authorized people have access” is a starting statement, not a complete explanation.

7. How Long Do You Retain Patient Data, and How Is It Deleted?

Patient information can remain in more places than the main application: uploaded files, generated notes, audio recordings, transcripts, temporary processing areas, logs, and backups.

Ask:

• How long is each type of data retained?
• Can retention periods be configured?
• Are recordings and transcripts retained after processing is complete?
• What happens to backups and cached data when information is deleted?
• Can you confirm deletion when the contract ends?
• Are there legal, regulatory, or operational reasons that require certain records to be retained?

The vendor should explain the difference between deleting information from the active application and removing it from backups or other systems.

Your agency should confirm that deletion commitments are consistent with applicable record-retention obligations and the contract. Immediate deletion is not always appropriate if a legitimate retention requirement applies.

8. What Happens If There Is a Security Incident or Data Breach?

No vendor can credibly guarantee that an incident will never occur. What matters is whether it can detect problems, respond quickly, communicate clearly, and meet its obligations.

Ask:

• How do you detect and investigate security incidents?
• How quickly will you notify us after discovering an incident involving our data?
• Who is our designated incident-response contact?
• Will you provide information needed to assess the scope and impact?
• How do you preserve evidence and prevent further unauthorized access?
• What support will you provide during investigation, remediation, and required notifications?

Make sure the contract specifies relevant notification requirements and responsibilities. Do not rely on an informal promise to “let you know immediately.”

Your organization needs enough information to meet its own legal and operational obligations. A vendor's response process should make that possible.

9. Can You Provide Independent Security Assessments and Supporting Evidence?

Ask the vendor to demonstrate its security practices instead of relying only on claims in a sales presentation.

Depending on the service and its risk profile, useful evidence may include:

• Relevant SOC 2 reports or other independent assessments.
• Security policies and a summary of risk-management practices.
• Vulnerability-management and penetration-testing summaries.
• Incident-response and business-continuity documentation.
• Details of relevant certifications, their scope, and their validity.

Ask what systems, services, and time periods the evidence covers. A report covering one product or corporate environment does not automatically cover every AI feature or subprocessor.

Also, be cautious of the phrase “HIPAA certified.” HHS does not endorse, certify, or recommend specific technology products as HIPAA compliant. A vendor's use of compliance language should be supported by actual safeguards, contractual commitments, and implementation details.

Reference: HHS guidance on cloud services, product endorsements, and security reviews.

Will the Vendor Submit to Your Security Review?

Ask whether your IT, security, or privacy team can review the proposed architecture and data flows.

The vendor should be prepared to explain:

• How data moves between your systems, the AI product, and third parties.
• What access permissions and integration methods are used.
• Which subprocessors receive information.
• How logs, backups, and credentials are protected.
• What happens when a service or integration is changed.

A vendor that refuses reasonable questions about its architecture or cannot explain its data flows creates an avoidable obstacle to due diligence.

The goal is not to demand unrestricted access to proprietary source code. It is to obtain enough credible information to evaluate the security and privacy risks of the proposed service.

10. How Do You Control AI Errors, Recording Risks, and Unauthorized Actions?

Data security is only one part of healthcare AI risk. An AI system can protect information and still produce an incorrect summary, extract the wrong value, misinterpret a clinical statement, or trigger an inappropriate action.

Ask:

• How do you test accuracy for our intended clinical or administrative workflow?
• Can users trace AI-generated outputs back to the source information?
• What happens when information is missing, contradictory, or uncertain?
• Can the AI change records, send messages, or trigger actions without approval?
• Which decisions require a qualified human to review and approve the result?
• How are errors reported, investigated, and corrected?

Does the AI Record Patient Visits, and What Happens When a Patient Declines?

If an AI product records audio or captures information during a patient visit, understand exactly how that process works before using it in the field.

Ask:

• Does the product record continuously, record only after activation, or use another input method?
• How are patients informed, and how is any required consent obtained and documented?
• What happens when a patient declines recording?
• Is there a workable alternative that allows the clinician to complete the documentation?
• How are discrepancies between recordings, clinician observations, and the final note identified and resolved?
• Are recordings, transcripts, and derived outputs stored separately, and what are their retention periods?

Recording and transcription are not the same as completed clinical documentation. A transcript may contain errors, omit context, or fail to reflect a clinician's assessment.

The resulting documentation still needs appropriate clinical review.

Do not assume that deleting a recording automatically resolves every privacy or compliance concern. The vendor should explain its complete data-handling process, and your organization should assess the workflow against applicable legal, privacy, consent, and documentation requirements.

For example, if an AI tool drafts a clinical summary, your team should understand how a nurse verifies the content before it becomes part of the clinical record. Likewise, an AI agent processing referrals should make uncertainty visible rather than silently inventing missing information.

11. Can You Demonstrate Real Home Health Workflows and Provide Customer References?

A polished demo can make almost any technology look useful. The real test is whether it works in the conditions your clinicians and operations teams face every day.

Ask:

• Can you demonstrate the actual workflow from start to finish?
• Can we speak with at least three customers using the product in comparable settings?
• Can we speak with clinicians who use it, not just executive sponsors?
• Can we observe it being used in the field, with appropriate permissions and privacy safeguards?
• How does it handle incomplete information, exceptions, and unexpected situations?
• How much manual work remains after the AI completes its task?

A vendor should understand the home health environment, including clinical documentation, referral intake, operational coordination, and the relevant standards for its particular use case.

Does the Vendor Understand Home Health Compliance?

Ask the vendor to explain the requirements relevant to the workflow it supports.

For example:

• How does the product support applicable Medicare Conditions of Participation (CoPs) and documentation requirements?
• How does it distinguish information supported by the clinical record from missing or inferred information?
• What safeguards prevent unsupported diagnoses, clinical details, or coding information from being added?
• How are discrepancies and potentially inaccurate outputs flagged for review?
• Who reviews the workflow for compliance, and what qualifications do they bring?
• How are regulatory and product changes evaluated after implementation?

AI should not create clinical facts to complete a note or encourage documentation that is not supported by the record. The vendor should explain how it handles uncertainty, preserves source information, and supports appropriate human review.

Your agency should independently validate the workflow with qualified clinical and compliance professionals before relying on it.

Is the Product Actually Solving the Problem It Claims to Solve?

Ask the vendor to distinguish the functions performed by AI from those handled by traditional software, rules-based logic, or robotic process automation (RPA).

This is not about requiring an arbitrary percentage of a product to be AI. It is about understanding what the technology actually does and whether the claims match the product's capabilities.

Ask:

• Which tasks use AI, and which rely on rules or conventional automation?
• Why is AI needed for this particular workflow?
• How does the product perform when inputs are ambiguous or incomplete?
• What evidence supports the claimed improvements?
• What happens when the AI cannot confidently complete a task?

Can You Prove Your Performance Claims?

Claims about saving documentation time, reducing quality assurance work, or increasing patient visits can be useful, but the numbers need context.

Ask:

• What baseline was used to calculate the improvement?
• How many agencies and clinicians were included?
• Was the result measured in a pilot, a controlled evaluation, or routine production?
• Does the measurement include review, corrections, training, and ongoing support?
• Are results consistent across different agency sizes and workflows?
• Can a customer reference explain the results and the conditions under which they were achieved?

For example, claims such as 85% less charting time, 40% less QA time, or twice as many patient visits should be treated as claims to investigate, not expected outcomes for every agency.

Ask the vendor to show how the results were measured and whether they apply to your workflow. The most useful comparison is not the most impressive headline number.

It is the improvement your team can reasonably expect after accounting for the work that remains.

12. Who Reviews Your Compliance, and What Happens to Our Data If We Terminate the Contract?

Vendor accountability should extend beyond the sales team and the initial product launch.

Ask:

• Who formally reviewed the product's clinical and administrative workflows?
• What standards, policies, and requirements were considered?
• Does the review cover relevant documentation, coding integrity, and safeguards against unsupported upcoding?
• Who is responsible for reviewing changes to the model, product, or workflow?
• Can we export our data in a usable format?
• How long will access remain available after termination?
• When will the vendor delete data that it no longer needs to retain?
• How will deletion be confirmed?
• What happens if the vendor is acquired, discontinues the service, or becomes unable to operate?

A credible vendor should be able to explain who is accountable for compliance review, what was assessed, and how changes are governed. Your organization should also maintain its own approval process rather than assume the vendor's review replaces its responsibilities.

The contract should explain data return, transition access, retention, and deletion. Your agency should not be left unable to retrieve important information or uncertain about where patient data remains after the relationship ends.

This infographic summarizes suggested vendor-selection standards. A BAA is required when the vendor acts as a business associate; audit requirements should reflect the use case and risk assessment. A SOC 2 report is not HIPAA certification.

What Are the Biggest Red Flags When Evaluating a Healthcare AI Vendor?

A vendor does not need to have a perfect answer to every technical question during the first meeting. It should, however, be transparent about what it does, what it cannot yet demonstrate, and how it manages risk.

Watch for these warning signs:

• No clear BAA process: The vendor avoids discussing whether a BAA is required for the proposed service.
• Vague training policies: It cannot explain whether submitted data is used to train or improve models.
• Unclear data flows: It cannot identify key subprocessors or explain where data is processed.
• Security claims without evidence: It relies on marketing language but cannot provide meaningful supporting documentation.
• No clear deletion policy: It cannot explain retention, backups, or what happens when the contract ends.
• Limited incident transparency: It will not clarify how incidents are reported or investigated.
• Uncontrolled AI actions: The system can alter records or take consequential actions without appropriate permissions and oversight.
• Unsafe recording practices: The vendor cannot explain how patient preferences, applicable consent requirements, and alternatives are handled.
• No credible customer references: The vendor cannot connect you with customers who can discuss real-world use.
• Weak home health understanding: The vendor cannot explain how its product fits into the clinical or administrative workflow it claims to support.
• Unclear compliance ownership: No one can explain who reviewed the workflow or how unsupported documentation and coding are prevented.
• Pressure to upload real PHI immediately: The vendor pushes for live patient data before the necessary review and agreements are complete.

One warning sign alone does not establish that a vendor is unsafe. It does mean your team should ask follow-up questions and document its decision before proceeding.

How Can a Healthcare Organization Evaluate an AI Vendor Before Sharing PHI?

Use a consistent process rather than making the decision based on a compelling demo or a single stakeholder's recommendation.

Step 1: Define the use case. Identify the workflow, the problem you want to solve, and the minimum information the AI needs.

Step 2: Map the data. Document what enters the system, what is generated, which services receive the data, and where it is retained.

Step 3: Review agreements. Evaluate the BAA, privacy terms, security responsibilities, subcontractor provisions, incident notifications, and data-return and deletion clauses.

Step 4: Validate safeguards. Request relevant security documentation and assess access controls, encryption, audit logs, risk management, and incident response.

Step 5: Test the real workflow. Start with synthetic or appropriately de-identified data where feasible.

Test routine cases, edge cases, recording alternatives, errors, and human-review processes before moving to live PHI.

Step 6: Speak with customers. Request references from comparable organizations and, where feasible, observe the product in field use with appropriate permissions.

Step 7: Document approval. Involve the appropriate clinical, operational, privacy, security, and legal stakeholders.

Record the risks, safeguards, outstanding issues, and conditions for approval.

Step 8: Monitor after deployment. Reassess the vendor when the product, model, data flow, subcontractors, or intended use changes.

Security and privacy reviews should not stop at implementation.

This approach helps healthcare organizations evaluate both the business value of AI and the responsibilities that come with using it.

Does a HIPAA-Compliant AI Vendor Automatically Make Our Workflow Safe?

No. HIPAA compliance is not a blanket guarantee that every feature, configuration, or use of a product is appropriate for every healthcare organization.

A vendor may offer safeguards and agree to contractual obligations, but your organization still needs to understand its own environment, perform the risk analysis applicable to it, configure the product appropriately, train users, manage permissions, and monitor how the technology is used.

The same AI product can present different risks depending on what information it receives, who uses it, how it connects to other systems, and whether it can take actions without human approval.

Treat vendor review as one part of a broader privacy, security, and governance process.

What Should Healthcare Leaders Remember Before Sharing Patient Data With AI?

AI can help healthcare teams reduce administrative work, improve information handling, and make workflows more efficient. But adoption should not require organizations to accept uncertainty about patient privacy, clinical accuracy, or compliance.

Before choosing an AI vendor, ask what data the system needs, how it is used, how it is protected, who can access it, how it performs in real home health workflows, and what happens when something goes wrong.

Request evidence. Read the agreements.

Speak with customers. Test the workflow.

Define human oversight. Make sure your organization can retain control of its information.

The right AI vendor should be able to explain not only what its technology can do, but also how it protects the people whose data makes that work possible.

At Copper Digital, we believe healthcare AI should be evaluated on more than automation alone. It should fit the workflow, support human judgment, and be designed around the privacy and security expectations that healthcare demands.

Before you share patient data with any AI vendor, make sure you have the answers you need to make an informed decision.

Related reading

• Why Healthcare Teams Do Not Adopt AI.
• Can AI Read Home Health Referral Faxes?.
• AI Tools for Home Health Nurses.
• Home Health AI Solutions.