AI adoption is exploding, but governance is collapsing
Indian healthcare is undergoing a silent technological revolution. Doctors use ChatGPT to simplify complex cases. Hospital administrators draft SOPs using Copilot. Researchers depend on LLMs for literature reviews. Front desk staff type patient complaints into AI chat systems. Management uses AI-generated summaries to guide strategic decisions. Nurses ask AI tools to rewrite patient notes.
These tools are fast, cheap, powerful and increasingly embedded in clinical workflow. The convenience is undeniable. The danger is unprecedented.
The Digital Personal Data Protection Act, 2023 and the DPDP Rules 2025 have now made unregulated AI use a major legal violation. What healthcare considers innovation, the law may interpret as unlawful processing, cross-border data transfer, lack of purpose limitation, breach of patient confidentiality and failure to implement security safeguards.
Hospitals are not prepared for this shift. Doctors are not aware of these risks. IT teams cannot control these platforms. Management has no policies in place.
LLM adoption is now one of the biggest compliance threats in healthcare.
This article exposes the risks, explains the law and presents a full governance blueprint before enforcement begins.
The hidden reality: LLMs are already everywhere in healthcare
Most hospitals underestimate the scale of LLM usage across departments. Below is what is actually happening in clinics, diagnostic centres and research institutions.
Doctors
- Pasting patient symptoms into ChatGPT for differential diagnoses • Asking AI to rewrite medical summaries for patients • Using LLMs to interpret test results or draft clinical letters • Using AI to prescribe explanations in regional languages
Nurses
- Generating nursing notes with LLM help • Creating patient education materials • Seeking AI guidance on symptoms or medication queries
Researchers
- Conducting literature summaries • Cleaning datasets • Using AI-powered statistical interpretation • Generating hypotheses
Medical administration
- Drafting patient communication content • Creating SOPs or documentation • Categorising patient complaints
Hospital management
- Using AI to benchmark hospital metrics • Preparing board presentations • Reviewing operational workflows
Every one of these examples risks DPDP violations when identifiable or sensitive data is used.
Hospitals often do not know what their staff is inputting into these tools. That ignorance is now a legal liability.
Why LLM use is legally dangerous under the DPDP Act
DPDP is fundamentally built around strict processing boundaries: consent, purpose limitation, minimisation, storage location, retention, deletion and auditability.
LLMs violate most of these principles by design if used casually in healthcare.
- Unlawful processing of patient data
If a doctor copies a patient summary into ChatGPT for interpretation, this becomes unlawful digital processing without consent.
- Cross-border data transfer without compliance
Most LLMs store and process data on servers outside India. DPDP has explicit restrictions and conditions for foreign processing and storage. Using ChatGPT, Perplexity or Copilot for patient information creates automatic violations.
- No control over retention or deletion
LLMs can retain inputs: • in system logs • model caches • internal analytics • quality improvement datasets
DPDP requires deletion after purpose is fulfilled. LLM providers do not offer this guarantee.
- Absence of audit trails
Hospitals must prove: • what data was processed • for what purpose • by which system • when it was deleted
LLMs do not provide these audit-level logs.
- Vendor contracts are non-existent
Hospitals using external LLMs without processor agreements violate multiple DPDP rules: • purpose limitation • retention control • deletion rights • breach notification obligations • security safeguard assurances
- Sensitive data amplification
Medical data is sensitive under all global regulations. LLMs may also: • reveal patterns • infer new medical conditions • create biased outputs • hallucinate false medical details
If patients suffer harm from these AI outputs, hospitals face legal and ethical liability.
Global warnings: GDPR, HIPAA and EU AI Act precedents
India is not the first system to face LLM challenges in healthcare.
GDPR
- Several European regulators have warned hospitals to avoid uploading identifiable information into LLMs.
- GDPR requires explicit lawful basis, transparency and data minimisation, all of which are violated if clinicians use ChatGPT casually.
- Health providers have faced fines for allowing external vendors uncontrolled access to patient datasets.
HIPAA
- HIPAA prohibits sharing PHI with non-compliant platforms.
- Several US hospitals have issued internal bans on using ChatGPT or similar tools for patient data.
- LLM vendors are not business associates unless they sign BAAs, which most do not.
EU AI Act
- Healthcare AI is classified as “high-risk.”
- High-risk AI systems require strict controls, transparency, documentation and governance.
- LLMs integrated into clinical workflows must meet reliability and quality standards.
These precedents show where India’s regulatory trajectory will lead.
Fear Flash: How LLM misuse in healthcare can cause real-world disasters
Scenario 1: Doctor pastes a patient case into ChatGPT
The system records the entire query on foreign servers. DPDP violation: unlawful processing + cross-border transfer. Consequence: major penalty, investigation, loss of patient trust.
Scenario 2: Researcher uploads a dataset for cleaning
The dataset contains identifiable lab results. Once uploaded, the hospital cannot delete it from vendor logs. Consequence: violation of retention and purpose rules.
Scenario 3: Nurse generates instructions for a critical care patient
AI delivers incorrect medication timing. Patient harm follows. Consequence: medico-legal case + DPDP violation for unsafe automated advice.
Scenario 4: Admin staff uploads HR grievances to draft emails
This violates confidentiality for employees. DPDP applies to employee data as well.
Scenario 5: Management relies on AI-generated summaries
AI hallucinates and creates false benchmarking or performance metrics. Decisions made from incorrect data cause operational or reputational harm.
The danger is not theoretical. It is already happening everywhere in Indian healthcare.
DPDP Act obligations hospitals must apply immediately to LLM usage
- Consent architecture
LLM usage must be: • explicitly disclosed in patient consent forms • explained clearly • used only for defined purposes • withdrawn on request
- Data minimisation
Only strictly necessary, de-identified, anonymised information can be entered into an AI tool.
- Purpose limitation
LLMs cannot use patient data for training or improvement unless the patient explicitly consents.
- Security safeguards
Hospitals must ensure end-to-end encryption, controlled access, secure interfaces and contractual guarantees from vendors.
- Vendor governance
Hospitals must have processor agreements covering:
- data location • deletion • retention • audit rights • breach notification • limitations on training, reuse and analytics
- Data localisation considerations
Foreign storage must follow DPDP Rule conditions. Most LLM platforms do not qualify.
- Rights handling
If a patient requests deletion of data processed through an LLM, hospitals must have the technical ability to execute it.
Most do not.
Special risks for each healthcare stakeholder group
Doctors
- May unintentionally leak patient data
- May rely on hallucinated medical advice
- Could be held negligent if AI suggestions cause harm
Nurses
- Uploading photos or cases violates confidentiality
- Using AI-generated care instructions may mislead patients
Researchers
- LLM-generated citations may be fabricated
- Datasets uploaded to LLMs may breach ethics committee obligations
Administrative staff
- Uploading patient complaints or financial details creates DPDP exposure
Management
- Strategic decisions made using unreliable AI outputs create institutional risk
- Failure to control staff AI usage becomes organisational negligence
The LLM Governance Blueprint for DPDP-compliant healthcare
Hospitals must adopt the following governance model immediately.
- Create an organisation-wide AI usage policy
Define: • allowed tools • banned tools • approved use cases • high-risk activities • required approvals • documentation requirements
- Prohibit uploading identifiable patient data into public LLMs
Only anonymised or synthetic data allowed.
- Deploy private, secure hospital-grade LLMs
Use:
- on-prem models
- fine-tuned AI on isolated servers
- controlled enterprise-grade AI platforms
- Implement access controls and tiered permissions
Not all staff should have access to AI tools.
- Ensure human validation for all outputs
AI cannot be the final decision-maker in healthcare.
- Create audit logs of all AI interactions
Track who used the tool, when, why and for what data.
- Mandate staff training
All staff must understand:
- what cannot be entered into AI systems
- risks of hallucinations
- DPDP compliance requirements
- Form an AI Governance Committee
Include: • IT • clinicians • compliance • legal • quality • cybersecurity
Review AI use regularly.
Penalties and consequences of non-compliance under DPDP
Hospitals may face:
- Penalties up to 250 crores
- Search, inquiry and seizure of digital systems
- Suspension of all AI-related processing
- Loss of NABH accreditation
- Blacklisting from insurer and corporate panels
- Civil suits for privacy breach
- Criminal or medico-legal action for patient harm
- Media-driven reputational destruction
- Regulatory audits into IT infrastructure
Once a DPDP violation is published, trust is almost impossible to regain.
Take-home message
LLMs such as ChatGPT, Copilot and Perplexity bring extraordinary capabilities to healthcare. But without governance they become instruments of legal violation, patient harm and catastrophic reputational damage.
The question hospitals must ask is not whether to use AI. It is how to use AI without violating the DPDP Act and endangering patients.
Those who build strong governance will lead India’s AI-enabled healthcare future. Those who ignore DPDP will face penalties, disruptions, investigations and loss of trust.
AI is powerful. DPDP is clear. Healthcare must choose governance over convenience before enforcement begins.
Author: Sujeet Katiyar : | Data Privacy (DPDP Act, GDPR, HIPAA), GRC | Digital Health, AI, Telehealth, Rural Healthcare | CEO, Founder, Director, DPO | 27+ Years in Web, Mobile, Emerging Technologies