If AI Cannot Handle a Nurse Having a Bad Day, It Does Not Belong in Healthcare

I have been a nurse for 25 years: ICU, med-surg, pediatric psychiatry. I opened a brand new inpatient psych unit where we achieved zero suicide attempts in the first three months. I have trained hundreds of healthcare managers in Just Culture, led teams through COVID with retention rates most hospitals would envy, and spent years deploying patient safety technology across hospital systems as VP of Clinical Experience at Invisalert Solutions.

Then I left health tech to start my own consulting practice focused on one question nobody building AI for healthcare seems willing to sit with: what does this technology do to the people who have to use it? I recently sat down with Arvind Sarin, CEO of Copper Digital, on the Inside Home Health podcast, and he opened by saying something I have never heard a tech founder say: he wanted me to challenge everything they are building. So I did. Here is what came out of that conversation, because every company building AI for clinical environments needs to hear it.

Systems Fail, Not People

In Just Culture, we have a foundational principle: when an error happens, we investigate the system first, not the person. You look at the process and the environment, and ask whether it was human error, at-risk behavior, or reckless behavior, each of which has a different response. But you start with the system. Now apply that to AI.

Just Culture applied to AI: investigate the system's safeguards first, not the nurse who trusted it.

If a nurse follows an AI recommendation and something goes wrong, whose fault is it? In most organizations today, the nurse gets written up and the system gets a pass. That is the opposite of Just Culture. If your AI tool output the wrong medication dose, the wrong diagnosis suggestion, or the wrong treatment plan, the investigation should start with the AI system and its safeguards, not with the clinician who trusted it. And the other side: if a nurse uses her own clinical judgment and does not follow the AI, she should not be penalized for that either.

AI is advisory. The nurse's judgment is sacred. Any company that does not build that principle into their product does not understand healthcare.

AI Hallucinations Are Not a Hypothetical Problem in Healthcare

I did not even know what an AI hallucination was until about a year ago, when a company building technology to detect them explained it: you ask AI for directions to Boston College and it gives you the wrong address. With directions, that is annoying. With healthcare, that is life or death. We are talking about diagnoses, medication doses, treatment plans, and OASIS assessments that drive reimbursement and care planning. If an AI system outputs incorrect information and a clinician does not catch it because they trusted the system, the consequences are not just financial. They are clinical.

I will not work with or recommend any company that does not already have policies for what happens when their product outputs the wrong information, because otherwise it falls back on the healthcare worker. To his credit, Arvind described how Copper Digital handles this: they never auto-submit to the EMR, a human always reviews, and they run multiple QA rules checking for internal consistency, so if the OASIS says the patient cannot ambulate 15 feet but another field contradicts it, the system flags it. That kind of safeguard should be table stakes for every company in this space. It is not.

You Are Building for Two Very Different Nurses at the Same Time

When you deploy documentation technology to a home health agency, you are putting it in the hands of two very different populations. You have new-grad nurses who go straight into home health: tech-savvy, but without deep clinical experience, so they lean on AI for suggestions and need to trust those suggestions are accurate. And you have nurses with 30 or 40 years of experience who know the meds head to toe and are incredible in emergencies, but did not grow up on computers. When we transitioned to Epic on my med-surg unit, it was a real struggle for that generation, and the training modules were not great.

One-size onboarding fails both: the new grad who over-trusts AI and the veteran who rejects it out of frustration.

Every stage of experience has different barriers and anxieties around AI. If your onboarding does not account for that, you lose adoption from the people who need it most. Print things out, seriously; nurses are huge printer-outers. And have clinical people do the training, not engineers. When a nurse sees another nurse showing them the product, they trust it. When they see a tech person who has never been at the bedside, they tune out.

Do Not Let Your Users Feel Abandoned

I asked Arvind directly: do you have safeguards for when a nurse is out in a patient's home, alone, and something with the product is not working? Because that is the reality of home health. These clinicians are alone. No manager down the hall, no peer in the next room. If the tool is giving them trouble and they cannot reach anyone, they feel abandoned, and when a nurse feels abandoned by a system, they stop using it. The onus is on agency leadership to know their people and check in frequently, but the tech company has a responsibility too: make sure your users have a lifeline when they are stuck.

Trauma-Informed Technology Design Is Not a Buzzword

Nobody talks about what it means to deploy technology in environments where clinicians are already carrying years of workplace trauma. We talk about UX and workflow integration. We do not talk about the fact that the person using this product saw three to ten traumatic things on their last shift. I can recall several shifts from my 25 years in their entirety because they were so traumatic, and that is 25 years of them stacked on top of each other.

At Invisalert, I trained all company staff on trauma-informed language, because if you do a demo to a psychiatric hospital and refer to patients as difficult, or say your product will take away the bad behavior on the unit, the clinicians in that room immediately lose trust in you. My recommendation for every AI company entering healthcare: train your entire team, not just clinical staff, on trauma-informed approaches. Engineers, sales, product. Everyone who touches the product or the customer needs to understand the environment they are building for.

Deploying Technology in a Broken Culture Does Not Fix Anything

This was the last thing I said to Arvind, and it might be the most important. If a home health agency does not have psychological safety, if nurses do not feel they can make a mistake and come to their manager without getting yelled at, deploying AI will not help. It will make things worse. The nurses will not trust it, will not report when something goes wrong, and will not adopt it, and the same punitive culture that existed before will use the technology as another weapon.

This is not a product problem. It is a culture problem. Every tech company selling into healthcare needs to understand that their product will only work as well as the culture it is deployed into.

The One Thing I Would Change About How Every Healthcare AI Company Builds

If I could change one thing in the room where healthcare AI companies make product decisions, it would be this: make sure your product cannot be weaponized. Make sure it cannot be ripped off a wall and used to harm someone. Make sure it does not have blinking lights that trigger patients experiencing delirium. Use thermal monitoring instead of cameras where possible so you are not violating patient dignity. Be HIPAA compliant not in theory but in every interaction. I have had children on my psychiatric unit rip equipment off walls the strongest adult could not have removed. Safety is not a feature to add later. It is the foundation everything else is built on.

And put healthcare people on your team. Not as advisors you call once a quarter, but as builders, decision-makers, and people who can say no. Put an engineer, a tech person, and a nurse in a room and it is amazing what you come up with, but you have to put them in the room.

Eight non-negotiables before deploying AI in any clinical environment.

Why I Call Myself AI's Clinical Conscience

I did not start using that phrase to be clever. I use it because someone needs to ask the uncomfortable questions, not about features and integrations and time savings, but about what happens to the nurse who is already exhausted, already carrying trauma, already questioning whether she should stay in this profession. What happens when we hand her another system and tell her it is going to help?

It had better actually help. It had better not add to her burden. It had better not become another thing she gets blamed for when something goes wrong. That is the bar, and it is not as high as you think. You just have to care enough to meet it.

🎧 Listen to the full conversation between Susie Branagan and Arvind Sarin on the Inside Home Health podcast. To see how Copper Digital builds with human-in-the-loop safeguards, explore AI tools for nurses, the compliance details, pricing, or more resources.