Securing AI - Developing an Encryption Service for Agentic Workflows

Background

Al-powered product experiences are becoming increasingly sophisticated, and sensitive user data now flows through distributed systems rather than a single application or monolith. A single request may traverse multiple services, and persist for hours as it moves through a chain of tasks. While this architecture enables powerful product capabilities, it also expands attack surfaces and creates new security challenges. At Calendly, we set out to protect these sensitive customer data throughout their lifecycle—not simply in storage, but as they move through services and execution environments. We aimed to develop an Application-Level-Encryption (ALE) solution that would operate across a polyglot microservice ecosystem, support long-running asynchronous workflows, and introduce minimal user-facing latency.

Traditional encryption-at-rest secures data at the storage infrastructure layer. This control protects data confidentiality when the database is offline. However, if a user gains and/or has access to the database, content can be read in plaintext. ALE encrypts data prior to storage to provide an additional layer of protection. Calendly’s agentic dataflows contain sensitive customer content, and we wanted to use ALE to ensure data confidentiality. However, implementing ALE at scale introduces its own challenges; these include key management, key rotation, performance overhead, and service-to-service access patterns. Our solution needed to engineer a balance between security, performance and operational simplicity. 

Service Development Methodology 

The team evaluated several approaches for protecting sensitive workflows: direct KMS encryption, envelope encryption, and a derived-key model. Each option offered a different balance of centralized key control, latency, operational complexity, and developer experience. Agentic workflows can span multiple services and execution paths; we needed a pattern that was secure by default, easy for teams to adopt, and efficient enough to avoid adding noticeable latency to user-facing workflows.

Classic KMS Encryption 

Google Cloud Key Management Service (KMS) is Google’s managed service for creating, storing, and managing cryptographic keys. An ALE service that solely ran on KMS, with a single key for encrypting and decrypting payloads, would be the simplest to manage. The key would be stored in Google’s Cloud KMS, with controlled Google Cloud user access and audit logs. While simple, using a single key poses a security risk by creating a single point of failure. If the key is compromised, the entire dataset must be considered compromised.

Envelop-Style Encryption 

To address the limitations of a single, shared key we evaluated envelope encryption as an option as it uses two layers of keys: 

• A unique Data Encryption Key (DEK) encrypts the payload data

• A Key-Encryption-Key (KEK), or parent key, stored and managed by the KMS, encrypts the DEK.

In this operation, a distinct key would be used for data, and the key encryption key would encrypt the entire package; this solution addressed the KMS’s single point of failure and achieved security’s data confidentiality requirement. However, envelope-style encryption requires direct calls to the KMS’s KEK for encryption and decryption operations. A single call can average ~10 mms, and at the scale of asynchronous AI workflows, the time required for these repeated  calls would accumulate quickly, increasing both response times and costs to an unacceptable level.

Envelope-Style Encryption with an Epoch Key-Store

Our search for an approach that preserved security without sacrificing speed led us to explore a three-layer model that introduced  an intermediate ‘epoch key’ layer. The epoch key would serve as an intermediate KEK between the payload-specific DEKs and a KMS-managed root key. 

In this design, each payload would still be encrypted with a unique DEK. But, rather than encrypting every DEK directly with the root key, the DEK would be encrypted with an epoch key. The epoch keys would be encrypted and protected by the KMS root, while their metadata would be stored in an epoch key store. Epoch keys can be cached in memory, meaning encryption and decryption operations could occur on the application’s hot path without requiring a KMS call. The additional epoch key layer would significantly reduce the latency and cost associated with frequent KMS calls.

While this  solution met our security and performance requirements, it introduced the additional operational overhead of managing both a KMS and a dedicated epoch keystore. 

Derived Keys 

Due to our concerns with managing an epoch key-store, we evaluated a derived-key model. Similar to epoch keys, derived keys use a KMS to secure the master key, and use that master to create KEKs, accessible via a local cache. Derived keys eliminate the need for an additional datastore because they use the KEK cache to create subkeys via a NIST-approved Key Derivation Function. Generally, a subkey consists of the KEK, a salt of the KEK version number, and binding information such as the tenant, workflow, or data ID. Subkeys are deterministic; any agentic worker with access to the parent key cache and associated binding data can locally perform encryption and decryption. 

The derived key model ensured that every payload would be secured with a unique key, and eliminated the overhead associated with the storage and management of an envelope approach.

Building the ALE Service

Upon choosing to move forward with the derived-key architecture, we needed to operationalize it across the enterprise. The goal was a self-service derived-key solution with key rotation, audit logging, and standardized cryptographic controls. Calendly is a Kubernetes-based microservice platform, and we wanted to build the derived-key encryption into a self-service solution that could be adopted by any application team.

We chose to implement ALE derived-keys as a sidecar service that utilizes the current tech stack. The sidecar hosts the pod’s encryption and decryption functionality, while remaining logically separate from the application’s operations. This solution offers a few advantages; the sidecar is deployed through existing Helm-based deployment workflows, scales independently of application services, and can be adopted consistently across different programming languages and datastores. The skeletal dataflow is a solution where Google Cloud KMS serves as the root of trust, and Hashicorp Vault’s External Secrets Operator distributes parent keys to authorized workloads. The sidecar caches the parent key locally and uses a HMAC-based Key Derivation Function (HKDF) to derive unique, payload-specific encryption keys on demand. Payload data is encrypted with Google’s Tink AEAD library. 

The result is a simple ALE service that exposes a simple encrypt/decrypt API that implements Calendly’s cryptographic standards including key derivation, AEAD encryption, logging, and sensitive data-handling requirements. Application teams can adopt this service without addressing any cryptographic primitives, key management, or encryption policy decisions. At this time, this solution is limited to agentic Temporal workflows, but we envision it expanding to other workflows and encryption needs.

Summary

The distributed and non-deterministic nature of modern AI workflows make traditional encryption approaches more difficult to implement at scale. By combining key derivation, Google Cloud KMS, Hashicorp Vault, and Google’s Tink AEAD library within a sidecar architecture, Calendly developed a scalable, ALE service that balances security, performance, and operational simplicity.

The result is a service that enables teams to securely process sensitive data across dynamic operations, while maintaining centralized cryptographic controls and a consistent customer experience.