Securing Log Data: How to Mask Sensitive Data in Logs

Discover how to securely mask sensitive data in logs within your Spring Boot application. Learn effective techniques to mask JSON and toString output, ensuring the confidentiality of your data and preventing unauthorized access.

Introduction

Logging is an essential aspect of any application as it helps developers track system behavior and troubleshoot issues. However, sensitive information such as passwords, credit card numbers, or personal data may inadvertently get logged, posing a security risk. In this blog post, we will learn how we can mask sensitive data in logs. Let’s dive in!

Why Mask Sensitive Information in Logs?

Masking sensitive information in logs is crucial for several reasons:

1. Data Privacy: Logs often contain sensitive information such as passwords, credit card numbers, or personally identifiable information (PII). Masking this data prevents unauthorized access and helps maintain data privacy.
2. Compliance and Regulations: Many industries have strict regulations regarding the handling and storage of sensitive data. By masking sensitive information in logs, organizations can ensure compliance with regulations like GDPR, HIPAA, or PCI-DSS.
3. Risk Mitigation: Logs are a valuable source of information for identifying security incidents or troubleshooting issues. However, exposing sensitive data in logs increases the risk of data breaches or insider threats. Masking sensitive information minimizes these risks and helps protect against potential vulnerabilities.
4. Access Control: Log files may be accessible to various personnel, including developers, administrators, or third-party service providers. Masking sensitive data ensures that only authorized individuals can access the logs, reducing the chances of data misuse.
5. Client Confidence: In today’s data-driven world, customers and clients value the security and privacy of their information. Masking sensitive data in logs demonstrates a commitment to data protection, fostering trust and confidence among users.

By log masking, organizations can safeguard data, comply with regulations, mitigate risks, control access, and maintain trust with customers. It is an essential practice for ensuring data security and protecting sensitive information from unauthorized access.

How to Mask Sensitive Data in Logs?

Log masking in Spring Boot can be achieved using both the logback and log4j2 logging frameworks. Both logback and log4j2 provide flexible and extensible options to mask sensitive information in logs, ensuring the confidentiality and security of your data.

Understanding Example

Let’s consider an example where we have a simple POJO class called “Person” with attributes such as firstName, lastName, age, creditCardNumber, and address. Additionally, we have a controller endpoint in our Spring Boot application that returns a list of Person objects. The objective is to mask the sensitive attributes in the logs while still providing meaningful log information.

public class Person {

    private String firstName;
    private String lastName;
    private Integer age;
    private Integer creditCardNumber;
    private String address;

    // Constructor
    // Getters and Setters

    @Override
    public String toString() {
        return "Person{" +
                "firstName='" + firstName + '\'' +
                ", lastName='" + lastName + '\'' +
                ", age=" + age +
                ", creditCardNumber=" + creditCardNumber +
                ", address='" + address + '\'' +
                '}';
    }
}

@RestController
public class PersonController {

    private final Logger logger = LoggerFactory.getLogger(PersonController.class);

    @GetMapping("list/person")
    public ResponseEntity<List<Person>> getPersonList() throws JsonProcessingException {
        List<Person> personList = List.of(
                new Person("Foo", "Bar", 25, 123456789, "ABC XYZ Street"),
                new Person("Baz", "Qux", 30, 987654321, "MNO PQR Street")
        );

        logger.info("Person List {}", personList);

        String jsonOutput = new ObjectMapper().writeValueAsString(personList);

        logger.info("Person List - JSON output {}", jsonOutput);

        return new ResponseEntity<>(personList, HttpStatus.OK);
    }
}

In our example, the controller endpoint “getPersonList()” retrieves a list of Person objects and logs the information using different log statements. We use the logger provided by the logging framework (e.g., logback or log4j2) to log the list of Person objects in various formats, including plain text and JSON.

Person List Postman Output

Log Masking Using Logback

Here we will see two different approaches related to masking sensitive data in logs:

Approach 1: Using a Custom Converter Class

We will create a custom converter class called “LogMaskConverter” and configure logback.xml accordingly. The LogMaskConverter class will handle the transformation of sensitive data in the log messages, while logback.xml will define the conversion rule and pattern for applying the converter.

import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.pattern.CompositeConverter;

public class LogMaskConverter extends CompositeConverter<ILoggingEvent> {


    public String transform(ILoggingEvent event, String in) {

        in = in.replaceAll("(?<=firstName=')[^']+?(?=')|(?<=\"firstName\":\")[^\"]+?(?=\")", "****");
        in = in.replaceAll("(?<=lastName=')[^']+?(?=')|(?<=\"lastName\":\")[^\"]+?(?=\")", "****");
        in = in.replaceAll("(?<=age=)\\d+(?=(,|\\s|}))|(?<=\"age\":)\\d+(?=(,|\\s|}))", "****");
        in = in.replaceAll("(?<=creditCardNumber=)\\d+(?=(,|\\s|}))|(?<=\"creditCardNumber\":)\\d+(?=(,|\\s|}))", "****");
        return in;
    }
}

In the LogMaskConverter class, we implement the CompositeConverter<ILoggingEvent> interface provided by logback. This allows us to customize the transformation logic for log messages. In the transform() method, we apply regular expressions to replace sensitive information in the log message with asterisks (****).

Let’s break down the regular expressions used for the replacement:

1. firstName: "(?<=firstName=')[^']+?(?=')|(?<="firstName":")[^"]+?(?=")"
   This regex matches the value of firstName attribute in both single-quoted and double-quoted formats, ensuring that the value is replaced with asterisks.
2. lastName: "(?<=lastName=')[^']+?(?=')|(?<="lastName":")[^"]+?(?=")"
   Similar to the firstName regex, this one matches the value of lastName attribute and replaces it with asterisks.
3. age: "(?<=age=)\d+(?=(,|\s|}))|(?<="age":)\d+(?=(,|\s|}))"
   This regex captures the numeric value of the age attribute, whether it is surrounded by commas, spaces, or brackets. The value is then masked with asterisks.
4. creditCardNumber: "(?<=creditCardNumber=)\d+(?=(,|\s|}))|(?<="creditCardNumber":)\d+(?=(,|\s|}))"
   The regex targets the creditCardNumber attribute, capturing the numeric value and ensuring it is masked regardless of surrounding punctuation or whitespace.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <conversionRule conversionWord="mask" converterClass="com.example.helloworld.converter.LogMaskConverter" />
    <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d [%thread] %-5level %-50logger{40} - %mask(%msg) %n</pattern>
        </encoder>
    </appender>

    <appender name="RollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>MyApp.log</file>
        <encoder>
            <pattern>%d [%thread] %-5level %-50logger{40} - %mask(%msg)%n</pattern>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>MyApp-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <maxFileSize>1MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>10MB</totalSizeCap>
            <cleanHistoryOnStart>true</cleanHistoryOnStart>
        </rollingPolicy>
    </appender>

    <root level="INFO">
        <appender-ref ref="Console" />
        <appender-ref ref="RollingFile" />
    </root>
</configuration>

Moving on to the logback.xml configuration, we define the conversion rule by specifying the conversionWord as “mask” and associating it with our LogMaskConverter class. This allows us to use %mask as a placeholder in the log pattern.

In the ConsoleAppender and RollingFileAppender sections, we modify the pattern to include %mask(%msg). Here, %mask represents the conversion rule we defined earlier, and %msg refers to the log message itself. This configuration ensures that the LogMaskConverter is applied to the log message, masking the sensitive information using the defined regular expressions.

By following this approach and configuring logback.xml accordingly, sensitive data in log messages, such as firstName, lastName, age, and creditCardNumber, will be replaced with asterisks (****). This ensures that the log entries retain relevant information while protecting sensitive data from being exposed in logs.

Logback Mask Sensitive Data in Logs Using ConverterLogback Mask Sensitive Data in Logs Using Converter Logfile Ouput

Approach 2: Using Regex Replace in logback.xml

In this approach, we configure logback.xml to perform regex-based replacements directly in the log pattern using the %replace converter. We specify the regular expression pattern and the replacement value within the logback.xml configuration.

Let’s analyze the logback.xml configuration:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <appender name="Console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %replace(%msg){'(?<=(firstName=\'|\"firstName\":\")|(lastName=\'|\"lastName\":\")|(age=|\"age\":)|(creditCardNumber=|\"creditCardNumber\":)).+?(?=\'|\"|,)', '****'} %n</pattern>
        </encoder>
    </appender>

    <appender name="RollingFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>MyApp.log</file>

        <encoder>
            <pattern>%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level %logger{36} - %replace(%msg){'(?<=(firstName=\'|\"firstName\":\")|(lastName=\'|\"lastName\":\")|(age=|\"age\":)|(creditCardNumber=|\"creditCardNumber\":)).+?(?=\'|\"|,)', '****'} %n</pattern>
        </encoder>

        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>MyApp-%d{yyyy-MM-dd}.%i.log</fileNamePattern>
            <maxFileSize>1MB</maxFileSize>
            <maxHistory>30</maxHistory>
            <totalSizeCap>10MB</totalSizeCap>
            <cleanHistoryOnStart>true</cleanHistoryOnStart>
        </rollingPolicy>
    </appender>

    <root level="INFO">
        <appender-ref ref="Console" />
        <appender-ref ref="RollingFile" />
    </root>
</configuration>

In the pattern section of both ConsoleAppender and RollingFileAppender, we modify the log pattern to include the %replace converter. The %replace converter takes two parameters: the original log message (%msg) and the regex pattern for replacement.

The regex pattern used in this approach is: '(?<=(firstName='|"firstName":")|(lastName='|"lastName":")|(age=|"age":)|(creditCardNumber=|"creditCardNumber":)).+?(?='|"|,)' This regex pattern captures the sensitive data (firstName, lastName, age, creditCardNumber) in various formats (single-quoted, double-quoted) in the log message.

The %replace converter then replaces the captured sensitive data with asterisks (****), ensuring the sensitive information is masked in the log output.

By configuring logback.xml using this approach, the log messages containing sensitive information will have their specific parts masked using the regex pattern provided. This approach avoids the need for a custom converter class and allows us to perform the replacements directly within the logback.xml configuration.

Logback Mask Sensitive Data in Logs Using Replace ConverterLogback Mask Sensitive Data in Logs Using Replace Converter Logfile Output

Log Masking Using Log4j2

Here we will see two different approaches related to masking sensitive data in logs:

Approach 1: Using a Custom Converter Class

In this approach, we create a custom converter class, MaskingConverter, in log4j2. The converter class extends LogEventPatternConverter and overrides the format method to apply log masking logic. We also configure log4j2.xml to use the custom converter for log masking.

Let’s analyze the MaskingConverter class:

import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.layout.PatternLayout;
import org.apache.logging.log4j.core.pattern.ConverterKeys;
import org.apache.logging.log4j.core.pattern.LogEventPatternConverter;

import java.util.Arrays;

@Plugin(name = "MaskingConverter", category = "Converter")
@ConverterKeys({"mask"})
public class MaskingConverter extends LogEventPatternConverter {

    private final PatternLayout patternLayout;

    protected MaskingConverter(String[] options) {
        super("mask", "mask");
        this.patternLayout = createPatternLayout(options);
    }

    public static MaskingConverter newInstance(String[] options) {
        return new MaskingConverter(options);
    }

    private PatternLayout createPatternLayout(String[] options) {
        System.out.println("Options: " + Arrays.toString(options));
        if (options == null || options.length == 0) {
            throw new IllegalArgumentException("Options for MaskingConverter are missing.");
        }
        return PatternLayout.newBuilder().withPattern(options[0]).build();
    }

    @Override
    public void format(LogEvent event, StringBuilder toAppendTo) {
        String formattedMessage = patternLayout.toSerializable(event);
        String maskedMessage = maskSensitiveValues(formattedMessage);
        toAppendTo.setLength(0);
        toAppendTo.append(maskedMessage);
    }

    private String maskSensitiveValues(String message) {
        // Replace sensitive values with masked value
        message = message.replaceAll("(?<=firstName=')[^']+?(?=')|(?<=\"firstName\":\")[^\"]+?(?=\")", "****");
        message = message.replaceAll("(?<=lastName=')[^']+?(?=')|(?<=\"lastName\":\")[^\"]+?(?=\")", "****");
        message = message.replaceAll("(?<=age=)\\d+(?=(,|\\s|}))|(?<=\"age\":)\\d+(?=(,|\\s|}))", "****");
        message = message.replaceAll("(?<=creditCardNumber=)\\d+(?=(,|\\s|}))|(?<=\"creditCardNumber\":)\\d+(?=(,|\\s|}))", "****");
        return message;
    }
}

The MaskingConverter class is annotated with @Plugin to define it as a custom converter in log4j2. The name parameter specifies the name of the converter, and the category parameter specifies its category. The ConverterKeys annotation specifies the keys associated with the converter, allowing it to be referenced in log4j2.xml.

Within the class, we create a PatternLayout instance using the provided pattern from log4j2.xml. The format method applies the log masking logic by first formatting the log event using the patternLayout and then replacing sensitive values in the formatted message with asterisks (****).

Now, let’s analyze the log4j2.xml configuration:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="DEBUG">
    <Appenders>
        <Console name="LogToConsole" target="SYSTEM_OUT">
            <PatternLayout pattern="%mask{%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n}"/>
        </Console>
        <File name="LogToFile" fileName="logs/app.log">
            <PatternLayout pattern="%mask{%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n}"/>
        </File>
    </Appenders>
    <Loggers>
        <Logger name="com.example" level="info" additivity="false">
            <AppenderRef ref="LogToFile"/>
            <AppenderRef ref="LogToConsole"/>
        </Logger>
        <Logger name="org.springframework.boot" level="error" additivity="false">
            <AppenderRef ref="LogToConsole"/>
        </Logger>
        <Root level="error">
            <AppenderRef ref="LogToFile"/>
            <AppenderRef ref="LogToConsole"/>
        </Root>
    </Loggers>
</Configuration>

In the log4j2.xml configuration file, two appenders are defined: “LogToConsole” and “LogToFile”. The “LogToConsole” appender is associated with the console target and uses the %mask{} pattern within the PatternLayout to apply the custom masking converter to the log message. Similarly, the “LogToFile” appender writes log messages to a specified file and also applies the masking converter.

To summarize, in this approach, we create a custom converter class MaskingConverter that extends LogEventPatternConverter and overrides the format method to apply log masking logic. We configure log4j2.xml to use the custom converter by referencing it in the pattern layout with the %mask conversion. The log4j2.xml file defines the appender and logger configurations, including the pattern layout with the %mask conversion for log masking.

Log4j2 Mask Sensitive Data in Logs Using ConverterLog4j2 Mask Sensitive Data in Logs Using Converter Logfile Output

Approach 2: Using Regex Replace in log4j2.xml

In this approach, we configure log4j2.xml to perform regex-based replacements directly in the log pattern using the %replace converter. We specify the regular expression pattern and the replacement value within the log4j2.xml configuration.

Let’s analyze the log4j2.xml configuration:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="DEBUG">
    <Appenders>
        <Console name="LogToConsole" target="SYSTEM_OUT"
            <PatternLayout pattern='%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %replace{%msg}{(?<=(firstName='|"firstName":")|(lastName='|"lastName":")|(age=|"age":)|(creditCardNumber=|"creditCardNumber":)).+?(?='|"|,)}{****}%n'/>
        </Console>
        <File name="LogToFile" fileName="logs/app.log">
            <PatternLayout pattern='%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %replace{%msg}{(?<=(firstName='|"firstName":")|(lastName='|"lastName":")|(age=|"age":)|(creditCardNumber=|"creditCardNumber":)).+?(?='|"|,)}{****}%n'/>
        </File>
    </Appenders>
    <Loggers>
        <Logger name="com.example" level="info" additivity="false">
            <AppenderRef ref="LogToFile"/>
            <AppenderRef ref="LogToConsole"/>
        </Logger>
        <Logger name="org.springframework.boot" level="error" additivity="false">
            <AppenderRef ref="LogToConsole"/>
        </Logger>
        <Root level="error">
            <AppenderRef ref="LogToFile"/>
            <AppenderRef ref="LogToConsole"/>
        </Root>
    </Loggers>


</Configuration>

Here we are using the %replace pattern within the PatternLayout in the log4j2.xml configuration file to apply regex-based replacements for masking sensitive information in log messages.

Within the <Appenders> section, two appenders are defined: “LogToConsole” and “LogToFile”. The “LogToConsole” appender is associated with the console target, and the “LogToFile” appender writes log messages to a specified file.

Both appenders use the <PatternLayout> element to define the log message pattern. Instead of using the %mask{} pattern as in the previous approach, they use the %replace{%msg}{regex}{replacement} pattern. This pattern applies the %replace converter to the log message.

The regex specified within the %replace pattern is a regular expression that captures and replaces sensitive values. It uses lookbehind and lookahead assertions to identify specific patterns in the log message. The regex captures patterns such as firstName='...', "firstName":"...", lastName='...', "lastName":"...", age=..., "age":..., creditCardNumber=..., "creditCardNumber":.... The matched values are replaced with “****”.

With this configuration, log messages will be formatted using the specified pattern layout with the %replace converter. The regex specified within the %replace pattern will be applied to the log message, replacing sensitive information with asterisks before displaying it on the console or writing it to the log file.

Log4j2 Mask Sensitive Data in Logs Using Replace ConverterLog4j2 Mask Sensitive Data in Logs Using Replace Converter Logfile Output

FAQs

Things to Consider

While masking sensitive data in logs, there are a few things to keep in mind:

• Regular Expression Complexity: When using regular expressions to match sensitive information, consider the complexity and efficiency of the expressions. Complex regular expressions can impact application performance, so ensure that your patterns are optimized.
• Test Logging Configurations: Before deploying your application to a production environment, thoroughly test your logging configurations to ensure that sensitive information is properly masked. Verify that the masking logic is applied correctly and that the masked output is consistent across different log messages.
• Compliance and Privacy Regulations: Depending on your industry or region, there may be specific compliance and privacy regulations that govern the handling of sensitive information. Ensure that your logging practices comply with these regulations and that the masking techniques used are appropriate.
• Access Control and Encryption: Masking sensitive information in logs is just one aspect of securing data. Implement proper access control measures and encryption techniques to safeguard sensitive information throughout your application.
• Regularly Review and Update Masking Rules: As your application evolves and new sensitive information patterns emerge, regularly review and update your masking rules. Stay up to date with the latest security best practices and adjust your masking techniques accordingly.

Conclusion

Logging is an essential aspect of application development, but it’s crucial to protect sensitive information from being exposed in logs. In this blog post, we explored different approaches to mask sensitive information in Spring Boot logs. By implementing these techniques, you can enhance data security and comply with privacy regulations. Remember to consider the complexity of regular expressions, test your configurations, and stay up to date with security best practices. With the right approach, you can ensure that your logs remain a valuable tool for debugging while keeping sensitive information safe from unauthorized access.

Learn More