Securing Sensitive Data on Kafka with End-to-End Encryption

Many Kafka deployments expand quickly to include sensitive data, creating a serious challenge for managing business information in a secure and usable way.

These challenges increase when compliance and cloud are brought into the mix, managing privacy regulations, understanding data sovereignty, and enforcing security policies.

Using end-to-end encryption is critical for mitigating these business risks, as well as for understanding what data should be exposed to the consumer of that data. Data could be delivered in clear-text, de-identified, encrypted, or masked form depending on the sensitivity of the data and the actions that need to be taken on that data by the consumer.

Consider these factors first to help guide you to the right solutions and implementations:

• Know your corporate security policy
• Identify any industry or regulatory requirements that govern your data processing capabilities
• Consider the environment in which you plan to deploy your solution
• Understand your data boundaries

Basic traffic and the brokers storage encryption that is defaulted in the Cloud do not address these concerns.

What if you are required to encrypt data in-transit, at-rest, in logs and everywhere sensitive data might appear, whilst providing access to clear-text data for specific consumers?

In these situations, you need to resort to end-to-end data encryption from the producer to the consumer and beyond (the target application or user).

In order to achieve this level of data security, messages to and from the Byte-array using serializers and de-serializers need to be integrated with an encryption library so each message is encrypted and decrypted.

But what happens when you already have built-in producers and consumers deployed in production? Do you recode all your existing producers for new encryption to new sensitive messages? Where do you store your encryption keys? Managing Key rotation?

SecuPi for Kafka uses zero-code instrumentation agents on the producers and/or consumers so there is no need to change the code every time a new requirement is defined for sensitive messages across all producers and consumers in production.

SecuPi agents for Kafka are lightweight and their policies are driven from a central definition point meaning that a common level of data security can be provided across your data realm.

In addition, SecuPi encryption and data protection options include:

• Format preserving (FPE) Encryption – preserving form and length, based on the Data Format (for example. jsmith@SecuPi.com turns into ujckoi@xJekaP.com)
• Type-Safe Encryption – Transforming values into a different value in the same type
• AES and AEAD Encryption – where data is converted to binary ciphertext using a mathematical algorithm and encryption key
• Vaultless Tokenization – does not require a centralized storage vault to manage and store the tokens
• Blurring – adding Random Noise and Encrypt the value
• Bucketing/Rounding – grouping values into buckets
• Masking – transforming values into a non-readable form
• Replacing/Zeroing/Nullifying – Masking/hiding parts of the result with “X”, Zero or random values.
• Format Preserving Randomization – FPE with Random Key
• Obfuscation- where data is consistently randomized into a set of values