The Real Cost of Unclear Accountability in Healthcare AI

Forsure answers the question most AI vendors avoid: Who is watching the system

The most honest way to talk about AI right now is to admit that a lot of people are right to be worried about it. The governance conversation is already happening, and it is not waiting for regulation to arrive, with concerns that are more specific than most AI marketing suggests. The sharpest of those concerns is about what is already happening inside the enterprise, on tools nobody approved, with data nobody is tracking.

The AI Already Inside Your Company

Before any of the larger risks arrive, there is one that is already here. Across mid-size and large enterprises, employees are using consumer AI tools on personal accounts to move faster. They paste in financial models, contract terms, member spreadsheets, and inside healthcare organizations, information that is functionally protected health information. None of this is malicious, and the behaviour is precisely what the tools were built to encourage, which is part of the problem because nobody thought to tell them where the data goes after they press enter.

This is tool sprawl, and it is the enterprise security problem hardly anyone is talking about. The organization has no inventory of which tools are in use, no log of what was submitted, no way to trace where a given piece of data traveled, and no answer to the only question that matters after an incident: what left the building, and when. Security and compliance teams are discovering the scale of this now, and the discovery is not going quietly.

There is a second problem underneath the first. These individual tools cannot reach the systems where the real work lives. They cannot pull from the claims platform, the eligibility file, or the pharmacy feed. So the data has to be carried to them by hand, copied out of governed systems and pasted into ungoverned ones. That interconnectability gap is what turns a productivity tool into an exposure. Every workaround is a person moving sensitive data across a boundary that was never designed to be crossed.

What enterprises actually need is a safe zone: an environment designed with clear boundaries, where AI operates within limits, data stays inside, connections to source systems are controlled and traceable, and every action leaves a footprint. Your intellectual property remains yours, and traceability is built in from the start. While the market debates future risks, this one is already inside the building.

Data centers, the physical machinery behind every AI product on the market, are getting bigger, thirstier, and closer to residential neighborhoods. Communities near new hyperscale sites report grid instability, falling property values, constant low-frequency hum, and water use so high that local aquifers show the strain. In several U.S. counties, homeowners have organized against permits for new facilities the way earlier generations organized against landfills.

The most powerful AI models require electricity at industrial scale. When a hyperscale cluster comes online, the surrounding grid absorbs it whether the surrounding grid is ready or not. A meaningful share of new capacity is being backed by natural gas because renewables cannot ramp fast enough, which means the AI boom is quietly reinflating fossil demand in states that had been trending the other way. Healthcare buyers who care about environmental, social, and governance (ESG) criteria will run into this footprint whether they engage with AI directly or not.

The cybersecurity environment is under growing strain. A subset of nation-state actors is already collecting encrypted health data with the explicit intention of decrypting it later, once quantum computers become powerful enough to break today’s encryption. The industry name for this is “harvest now, decrypt later,” and healthcare records are among the most attractive targets because they stay sensitive for a lifetime. A record stolen in 2026 is still worth stealing in 2036. NIST finalized the first post-quantum cryptography standards in 2024, so the tools now exist to defend against this; systems designed today can be aligned with those standards from the start, and the ones designed without them will eventually need to be rebuilt.

Frontier model risk is the concern professionals raise last, but it is the one that reveals the most about the state of the industry. The most capable AI models are powerful enough that mistakes are hard to undo, which is why the leading labs deliberately hold their most advanced tiers back from open public release. Anthropic did exactly this in 2026 when it kept its most capable model, Claude Mythos, out of general release over cybersecurity concerns, putting a safer version called Claude Fable 5 in front of the public while restricting the full model to vetted organizations for defensive use. That decision, a lab acknowledging that its most powerful product is too dangerous to release into general use, raises a question a benefits leader eventually has to answer: who is accountable when these systems touch real decisions. That is a governance question, not a technology one.

Healthcare Is Where Trust Has to Work First

We are also not going to pretend healthcare is neutral ground.

Healthcare is where every one of those risks is amplified. A data breach in retail is a stolen credit card. A data breach in healthcare is a stolen decade. A biased algorithm in advertising costs a company a lawsuit. A biased algorithm in a prior authorization tool costs someone chemotherapy.

Healthcare is one of the industries that must lead the way in building trust in AI, given what is at stake. The margin for error is smaller, and the people affected have no ability to shop around when something goes wrong. The industries that get this right will attract the AI they need on their own terms.

What a Trustworthy System Actually Does

If we move past the jargon, a trustworthy AI system in healthcare comes down to four clear things that any non-technical leader should be able to ask about directly and actually get a concrete answer to in return.

Each of these questions maps to something the system must be able to show on demand, not in theory or documentation, but in real operation, with evidence behind it.

Taken together, they define what “transparent AI controls” really means when it is more than a phrase used in marketing. And if even one of the four cannot be answered clearly and consistently, then the system is not yet ready for use in a healthcare environment.

The first question is whether the system can clearly explain its decisions in a way that holds up under real scrutiny. In healthcare, that means being able to take any outcome, whether it is a coverage decision, an authorization outcome, or a medication change, and trace it back through the actual inputs, reasoning, and governing rules that produced it, without gaps or post-hoc interpretation. The explanation should clearly show how the decision was actually made, rather than relying on generic wording or a simplified version written afterward. The standard is that this traceability is complete enough that a nurse can act on it, a member can understand it when it is communicated, and an auditor can verify it without ambiguity or missing logic.

The second is whether a person can always step in. Every serious workflow, at every point where the stakes rise, has a defined handoff to a human who can review, override, or take over. Autonomy and human authority both operate together, with the human role built in as an active and meaningful part of the system. The interface, tooling, and escalation logic are designed to support real intervention, so human involvement is practical and effective rather than a formality.

The third is whether the system keeps an honest record. Every meaningful decision leaves a trail that can be produced on request. If the record does not exist, the decision is not defensible. This is the part that dies quietly in most AI deployments, because logging costs money and no one notices what is missing until the day someone asks.

The fourth is whether the system is watched over time, not just at launch. AI behavior drifts as models update and data changes. A trustworthy system knows this about itself and expects the scrutiny, continuous monitoring for accuracy, drift, and fairness, rather than a one-time certification that gets forgotten by the second quarter.

AI vendors should be able to demonstrate all four and show how the system holds up when a member disputes a decision. If the answer is vague, the system is not ready.

The Security Posture Underneath

Data security  is the foundation of AI governance. HIPAA is the floor for protected health information, not the target, and meeting it is the minimum before an AI conversation can even begin.

Above HIPAA sits the NIST AI Risk Management Framework, which is becoming the de facto standard for governing AI systems in high-stakes environments. Forsure’s controls map directly to its three functions, and the trust center tags the relevant ones under Govern, Measure, and Manage. Govern is the cross-cutting function, the risk culture, policies, and accountability that hold everything else together, and it is what “governance built in, not bolted on” looks like made operational. Measure covers the ongoing assessment of systems against trustworthy-AI characteristics like validity, safety, security, accountability, explainability, privacy, and fairness, which is the layer that makes “it can explain itself” auditable rather than aspirational. Manage covers prioritizing and responding to risks over time through incident response and continuous monitoring, which is what makes “watched over time” operational instead of theoretical.

There is a piece of the framework that matters specifically for AI systems built today rather than five years ago. NIST extended the RMF to generative and agentic AI with the Generative AI Profile in July 2024, NIST AI 600-1. It explicitly covers the kind of AI this platform uses. Vendors that skip that profile skip the parts of governance most relevant to modern AI.

The quantum piece from earlier connects here as well. NIST post-quantum cryptography, also finalized in 2024, is the direct answer to the “harvest now, decrypt later” threat, and the standards now exist to be adopted rather than waited on.

Then there are the operational controls: SOC 2 Type II, role-based access, immutable audit logs. For any serious healthcare focus, alignment with HITRUST for protected health information handling, and TX-RAMP for public-sector deployments in Texas, are becoming baseline expectations rather than differentiators. Vendors that treat these as design targets from the start avoid the retrofit cost that catches everyone else two years in.

Zero data retention belongs in the same category, though it needs to be stated precisely. Prompt content and uploaded source data are not written to persistent storage. The decision record, access logs, and audit trail are retained, because a system that keeps nothing can prove nothing. What is retained is the evidence of what happened, not the sensitive material that passed through, which is the direct answer to the tool sprawl problem described earlier, where the exposure existed because data accumulated in places nobody was tracking.

Where AI Meets Reality

Not every AI is safe enough for a healthcare decision, and the leading labs say so themselves. Anthropic ties stronger safeguards to more capable models and holds the top tier back from open release. The safest system is the one built and supervised with the stakes in mind.

Policy is in flux around it. Washington is trying to set one national standard over the states. Colorado passed the most comprehensive AI law in the country, then walked it back to 2027 under legal pressure. Europe pushed its AI Act deadlines back. FDA and CMS keep layering healthcare-specific guidance on top. Every serious jurisdiction lands on the same four commitments. You do not need to be a compliance expert to read that pattern. When every regulator, on every continent, working independently, arrives at the same requirements, those requirements have stopped being a matter of opinion. The only open question is whether your vendor was built to meet them, or is planning to retrofit them later.

How Forsure Answers It

Forsure starts from a different point. Trust is the foundation of the operating system, not a layer on top. SureSystem™ is the intelligence engine. It connects claims, pharmacy, eligibility, and benefit-design data into one continuously learning environment, and runs agentic workflows across prior authorization, claims optimization, and pharmacy spend intelligence. Every decision leaves a record, workflows are able to explain what they did, and autonomous decisions include a clear path for human handoff when needed. Security is aligned to HIPAA, NIST AI RMF including the Generative AI Profile, and NIST post-quantum cryptography from day one, with HITRUST and TX-RAMP as design targets.

SureConsult™ is the human side. Consultants provide guidance on plan design, drug strategy, and pharmacy negotiations. Alone, technology produces decisions no one can defend, and consulting cannot keep pace with what AGI is asking of a benefits department. Together, they support the people accountable, not replace them.

None of this makes the dark side of AI go away. Data centers will still be built, some badly. But healthcare can insist on more, because the consequences of not insisting are too visible. Buyers here answer to employees, members, and patients. They know what happens when the wrong decision falls on the wrong person.

There is a fair question buried in all of this, and it deserves a direct answer. Who is actually watching the system? It is not us. A vendor telling you it is safe is a vendor telling you it is safe. What matters is that the watching is done by someone with no stake in the answer. HIPAA, SOC 2, NIST AI RMF, and eventually HITRUST are independent compliance systems, built and enforced by third parties, most of them in place long before this wave of AI arrived. Every certification is a verdict someone else reached about us, on their criteria, with their auditors. Ask any AI vendor who is watching them. Then ask them to show you who signed off, when, and against what standard.

‍