Encrypted AI Inference: Why Legacy AI Infrastructure Can't Protect Your Data

Every day, enterprises send their most sensitive data — patient records, financial models, legal briefs, defense specifications — into public AI models as plaintext. The data is stored, potentially used for training, and accessible to third-party employees.

Updated 2026-10-03: tier and cryptography descriptions in this post now match the published Data Handling Commitments; an earlier version overstated forward secrecy and scoped server-side memory protection too narrowly.

For CISOs, compliance officers, and decision-makers in healthcare, legal, and finance, this isn’t a theoretical risk. It’s a daily compliance violation.

The industry call this “enterprise AI.” We call it legacy AI — infrastructure built for a time when data privacy wasn’t a requirement. The alternative exists: encrypted AI inference, also known as privacy-preserving AI.

What CISOs Know But Can’t Say Out Loud

When your analysts feed portfolio data into Fireworks, your clinicians paste patient summaries into ChatGPT, and your lawyers draft contract language with Claude, something happens that no enterprise plan can prevent:

Your data is processed as plaintext on servers you don’t control.

Here’s what most enterprise AI looks like today:

  1. Employee types sensitive data into an AI interface
  2. Data travels over TLS to the provider’s servers
  3. Provider decrypts and processes the plaintext
  4. Provider stores logs, caches results, possibly uses data for training
  5. Result is returned — but the data lifecycle is now outside your control

What actually happens to your data:

  • Cached on provider servers indefinitely
  • Logged for troubleshooting and monitoring
  • Potentially used for model training (even on “enterprise” plans)
  • Accessible to provider employees with system access
  • Subject to the provider’s jurisdiction and legal obligations

“For consumer apps, this might be acceptable. For organizations bound by HIPAA, GDPR, SOC 2, or export controls? Unencrypted inference is a breach in progress.”

Why “Enterprise Plans” Are Still Legacy AI

Providers like Fireworks, Together, and Databricks offer enterprise tiers with promises of better data handling. But these solutions share a critical flaw: the data is still unencrypted when processed.

An “enterprise plan” typically means:

  • No data training (sometimes)
  • SLA guarantees
  • Dedicated support
  • Still processes your data as plaintext

This is like putting your bank statements in a locked envelope and handing it to a stranger who promises not to read them. The lock doesn’t matter — they see everything inside.

Encrypted AI inference means something different:

  • ✅ Data is encrypted on your infrastructure before anything is transmitted (AES-256-GCM, fresh key per request)
  • ✅ On the high and maximum tiers, plaintext is confined to an Intel SGX enclave and is never accessible to NOMYO, its personnel, or its host systems
  • ✅ Zero prompt/response retention by design for Encrypted Inference
  • ✅ Local Models only: inference runs on NOMYO-hosted models in the NOMYO hosting facility — payload content never leaves it
  • ✅ Verifiable evidence: every maximum-tier response carries a TPM 2.0 hardware quote you can check independently with the nomyo SDK

Privacy-preserving AI isn’t a feature toggle. It’s a fundamentally different architecture.

Real-World Scenarios: The Cost of Unencrypted Inference

These aren’t hypotheticals. They’re the scenarios CISOs at regulated organizations face every day.

Healthcare: HIPAA Violations in Real Time

A hospital uses a public AI assistant to help draft patient discharge summaries. The AI processes real patient names, diagnoses, and treatment histories. Under HIPAA, this is a breach — protected health information (PHI) was transmitted to a third party without a Business Associate Agreement, without encryption, without audit controls.

The cost: HIPAA fines range from $100 to $50,000 per violation, up to $1.5 million annually. But the real cost is patient trust — and once lost, it doesn’t come back.

Finance: Proprietary Data Leaked Through Model Training

A hedge fund analyst feeds portfolio data and market analysis into a public AI model for research assistance. The model’s training pipeline incorporates that data. Weeks later, similar patterns appear in the model’s outputs for other users. Information that should have been proprietary is now leaked.

The cost: Regulatory scrutiny, competitive disadvantage, potential insider trading implications. The SEC doesn’t care that your AI vendor “promised” not to train on your data.

A law firm uses AI to draft contract language and review discovery documents. Client confidences — trade secrets, settlement strategies, privileged communications — are processed by a third-party model. Attorney-client privilege requires that confidential information not be disclosed to third parties without consent.

The cost: Malpractice claims, loss of client trust, disqualification from cases. Once privilege is waived, there’s no undo.

Defense: Export Control and Classification Violations

Defense contractors working on classified programs use AI tools for analysis and documentation. Public AI models have no classification capabilities, no audit trails, and no guarantee that processed data won’t be stored or shared.

The cost: Federal contract termination, criminal liability, national security implications.

Encrypted AI Inference: The Category That Legacy AI Can’t Replicate

Encrypted AI inference — privacy-preserving AI where data remains encrypted throughout the entire processing pipeline — isn’t a new concept. The cryptography is well-understood. What’s new is the application to AI workloads at scale.

Legacy AI providers can’t add this to their existing infrastructure. It requires rebuilding from the ground up. NOMYO was built for this purpose.

The NOMYO Encryption Stack

AES-256-GCM for payload encryption:

  • Industry-standard symmetric encryption
  • Encrypts the actual prompt and response data
  • Authenticated encryption — tamper detection built in
  • Fast enough for real-time inference

RSA-OAEP 4096-bit for key exchange:

  • Asymmetric encryption establishes the secure channel
  • Key material is never transmitted in the clear: each request’s AES key travels only wrapped under the server’s RSA-OAEP 4096-bit public key
  • Server public key fingerprint verification prevents man-in-the-middle attacks

Per-request key rotation:

  • Every inference gets a unique AES-256 key
  • Keys are generated via cryptographically secure random number generation
  • Keys are zeroed immediately after use
  • Even if one key is compromised, only a single inference is affected

Memory Protection

Plaintext is shielded from disk exposure:

  • Client-side, the nomyo SDK’s secure memory handling attempts to protect key material and plaintext from swap (mlock/VirtualLock) and zeroes it after use — and reports per request whether the operating system actually granted the protection
  • Server-side, plaintext never touches disk on the high and maximum tiers; crypto material is zeroed after use and never swapped

Security Tiers

NOMYO offers three levels of protection, designed for the risk profiles that compliance teams understand:

  1. Standard: General business data, low sensitivity
  2. High: Sensitive business data, proprietary information
  3. Maximum: Regulated and special-category data — health data requires a signed addendum; NOMYO does not currently offer a HIPAA Business Associate Agreement

Each tier applies progressively stricter controls. Maximum adds password-protected keys, HTTPS enforcement, per-request TPM 2.0 attestation, and audit logging.

The Compliance Imperative: Why This Can’t Wait

Three regulatory forces are making encrypted AI inference a legal requirement, not a nice-to-have:

The EU AI Act

Now in force. Requires risk-based compliance for AI systems. High-risk AI applications (including those processing personal data) must implement appropriate technical measures. Encrypted AI inference is a demonstrable technical control that satisfies multiple requirements.

HIPAA Updates for the AI Era

The OCR has made clear that AI systems processing protected health information are subject to HIPAA’s Privacy and Security Rules. Without encrypted AI inference, healthcare organizations cannot demonstrate the required safeguards.

SOC 2 and Industry Standards

SOC 2 Type II audits increasingly scrutinize third-party data processing. If your AI provider processes plaintext data, it becomes part of your compliance perimeter. Encrypted AI inference removes that exposure.

Legacy AI vs. Encrypted AI Inference: The Structural Divide

The reason no major AI provider — Fireworks, Together, Databricks, or the public model providers — offers true encrypted AI inference is structural, not accidental:

Business model conflict: Public AI providers’ value proposition is data aggregation. Their models improve with more data. Encrypted AI inference makes that impossible — it’s fundamentally incompatible with their business model.

Technical debt: Adding E2EE to existing infrastructure designed for plaintext processing would require rebuilding the core architecture. It’s easier to sell “enterprise plans” than to rebuild.

Vendor lock-in: Plaintext processing creates dependency. Encrypted AI inference gives control back to the customer. No legacy provider wants to enable that.

NOMYO is different: We were built from the ground up for encrypted AI inference. Our architecture, our business model, and our mission are aligned. We don’t profit from your data — we profit from securing it.

This is the difference between legacy AI — infrastructure optimized for convenience — and privacy-preserving AI — infrastructure optimized for control.

The Bottom Line for CISOs and Compliance Teams

Encrypted AI inference is not a feature. It’s a baseline requirement for any organization that handles sensitive data.

If your data is sensitive enough that you can’t send it to a third party in plaintext, you need encrypted AI inference. Period.

The question is no longer “Should we encrypt our AI workloads?” The question is “Can we afford to keep sending plaintext to the cloud?”

Ready to move beyond legacy AI?

This is part of our blog series on privacy-first AI infrastructure. Read our companion piece: “The Privacy-First AI Stack: How to Set Up Encrypted Inference with NOMYO AI.”