How to find passwords, tokens and personal data in your logs

In May 2018 Twitter asked every one of its users to change their password. A bug had written passwords to an internal log before they were hashed. The same year, GitHub told some users that a flaw in its password-reset flow had put their passwords in plain text in internal logs. In 2019 Facebook said hundreds of millions of passwords had been stored readably in internal systems, including logs.

These were companies with security teams, and they found the problem through audits. A team of five usually has neither, so the same problem simply sits in its logs, copied to wherever the logs are shipped and kept for as long as they are kept.

How it gets there

Nobody writes Log::info($password). Personal data and secrets get into logs by accident, through a short list of routes:

• Request logging. A middleware that logs the request body logs the login form too. Logging query strings logs password-reset tokens and magic-link codes.
• Headers. Logging headers “for debugging” logs Authorization: Bearer ... and session cookies.
• Exception messages. A database error that includes the failed SQL includes the values in it. A validation exception can include the submitted form.
• Third-party responses. A payment or identity API’s error response, logged whole, may echo back the card number or personal details you sent it.
• “Temporary” debug lines. logger.debug(user) added during an incident, and still there a year later. When teams trace a runaway logging bill back to its source, the culprits are usually the same: verbose HTTP tracing shipped to production and request payloads dumped in loops, which are exactly the lines that carry personal data.
• Normal lines that look harmless. “Password reset sent to jo@example.com” puts an email address in your logs on every reset.

The OWASP Logging Cheat Sheet lists what should usually not be recorded directly: session identifiers, access tokens, passwords, connection strings, encryption keys and secrets, payment card data, and sensitive personal data. It’s a good checklist to hold your logs up against. There’s even a weakness classification for the problem, CWE-532, “Insertion of Sensitive Information into Log File”.

Why it matters

Secrets in logs are secrets that have leaked. Whoever can read your logs can now use those tokens: everyone with access to the log tool, and anyone who gets into an account that has it.

Regulators take plain-text credentials seriously. In 2018 the Baden-Württemberg data protection authority fined the chat platform Knuddels €20,000 under the GDPR after a breach exposed passwords it had stored in plain text. That was a database, not a log file, but a password sitting readable in a log is the same failure.

Personal data in logs is personal data you’re responsible for. Under the GDPR, an email address or a user id in a log line is personal data like any other. The storage limitation principle in Article 5 says personal data should be kept “for no longer than is necessary” for its purpose. Security and audit are good purposes for keeping logs a long time, but that is much easier to stand behind when the logs hold ids rather than email addresses. If a customer asks what data you hold about them, your logs are part of the answer. Other regimes, from Australia’s Privacy Act to card-payment rules, point the same way.

Logs get copied. They are shipped to a log service, exported for an incident review, pasted into a ticket, and handed to a coding agent. Every copy carries whatever is in them.

Step 1: measure before you fix

The hard part isn’t fixing a line that logs an email address. It’s knowing which of your hundreds of log statements do it. So start by counting.

SaneLog checks every line as it arrives for formats that are clearly personal data or secrets: email addresses, payment card numbers that pass the card checksum, IBANs, international phone numbers, some national identifiers, private keys, bearer tokens, JSON web tokens, AWS access keys, API keys recognised by their vendor prefix, and the values after keys like password= or api_key=. Every new account starts in Detect only mode, which counts matches without changing anything.

That gives you a number per source and per type, for the month. In practice it tells you something like: api-1 logged 1,840 email addresses this month, worker-2 logged 12 bearer tokens. Show lines beside a count opens the actual lines, which leads you straight to the code that writes them.

A filter on redaction is present with an alert on it tells you the first time personal data or a secret appears from any source, including in code that was deployed this afternoon.

Step 2: fix it where it’s written

The only complete fix is to stop logging the data. With the lines in front of you, it’s usually a small change:

• Log an id instead of an email address. user_id=1812 is enough to look someone up and is far less sensitive.
• Never log request bodies or headers wholesale. Log the fields you need, by name.
• Strip query strings from logged URLs, or keep a list of parameters you allow.
• Catch database exceptions and log the error code, not the SQL with its values.
• Use your framework’s tools for this, such as marking parameters as sensitive so they’re hidden in stack traces.

Step 3: a safety net for what you miss

Code changes, and new log lines get written in a hurry during incidents. A safety net that replaces sensitive values before they are stored catches what slips through.

On SaneLog’s Business and Enterprise plans, Redact mode replaces each match before the line is stored, in place:

Payment failed for jo@example.com with card 4111 1111 1111 1111 from +61 412 345 678

is stored as

Payment failed for [email] with card [card] from [phone]

The original can’t be recovered. You can keep the last four digits of a card number or the domain of an email address if you need them for support. And you can still search for [card] to find every line that contained one.

Redaction works by recognising formats, so it has honest limits: it doesn’t detect names, street addresses or free-text identifiers, because in a log line those can’t be told apart from usernames and hostnames. It’s a backstop for the shapes that can be recognised reliably, not a substitute for step 2.

Step 4: keep logs for as long as practical, and keep them clean

Logs are evidence. When something goes wrong with security, the question is rarely about this afternoon. It is “when did this start?”, “what else did that account touch?”, “has this address been here before?”. Intrusions are often discovered weeks or months after they began, and the answer is only in your logs if they go back that far.

You will also be asked. Security reviews, customer vendor assessments and compliance frameworks routinely expect at least 12 months of logs. PCI DSS, for one, requires audit log history to be kept for at least a year, with the most recent three months immediately available. “We keep logs until logrotate deletes them” is a hard answer to give a customer’s security questionnaire.

So keep logs for as long as is practical, with a year as a sensible minimum. That is exactly why steps 1 to 3 matter. Clean logs, holding ids rather than email addresses and no secrets at all, can be kept for years without becoming a liability. Long retention is also easier to justify under privacy law when security and audit are the stated purpose and the logs carry no more personal data than they need.

In short

1. Count what’s in your logs before you try to fix it.
2. Fix it in the code that writes it: ids instead of emails, named fields instead of whole requests.
3. Redact on arrival as a safety net for what slips through.
4. Keep logs for as long as practical, a year at least. Clean logs make that safe.

Every SaneLog account starts in Detect only mode, so the count is there from your first log line, on every plan.