SaneLog
HomePricingDocsCoding agentsPostsLog inSign up

· rsyslog, syslog, tls, linux

rsyslog remote logging over TLS without losing logs

rsyslog forwarding fails quietly: lost logs in outages, short hostnames, 8 KB truncation, TLS errors. A remote syslog config that handles each one.

Sending a server’s logs somewhere central takes one line in rsyslog:

*.* @@logs.example.com:6514

It works the moment you restart rsyslog, and it keeps working until the first time something goes wrong. Then it fails quietly, in one of a handful of ways that are each well known on their own and rarely written down together. The same questions come up year after year: why remote logging works on one server and not another, how to encrypt it, why lines went missing while the central server was down. This post goes through the failures in order, and ends with a complete configuration you can adapt.

Everything here is about rsyslog, the default on Debian, Ubuntu, RHEL and most other distributions. The same ideas apply to syslog-ng.

Failure 1: lines disappear during an outage

The remote end restarts, or the network drops for a few minutes. With the one-line rule, rsyslog tries to send, fails, retries a few times, and then gives up on whatever it was holding. Those lines are gone.

The fix is an action queue for the forwarding rule, with a file name, which makes it disk-assisted:

queue.type="LinkedList"
queue.filename="fwd_central"
queue.maxDiskSpace="100m"
queue.saveOnShutdown="on"
action.resumeRetryCount="-1"

What each line does:

  • queue.type="LinkedList" gives the forwarding rule its own queue in memory, so it can fall behind without holding anything else up.
  • queue.filename makes that queue disk-assisted. Lines stay in memory while things are healthy and only spill to disk when the queue fills, so a short outage writes nothing to disk at all.
  • queue.maxDiskSpace caps how much disk an outage can use. 100 MB holds hours of a typical server’s logs.
  • queue.saveOnShutdown="on" writes whatever is still queued to disk when rsyslog stops, and sends it after it starts again. Without it, a restart during an outage loses the queue.
  • action.resumeRetryCount="-1" means retry forever. The default gives up.

Failure 2: the forward blocks, and local logging stops too

This one is worse. Without its own queue, a forwarding rule runs on rsyslog’s main queue. If the remote end stops accepting connections, the forward blocks, the main queue fills, and rsyslog stops writing local log files as well. Red Hat has a knowledge-base article titled, more or less, “rsyslog stops logging to all files after adding a TCP forwarding rule”. The action queue from failure 1 fixes this too: the forward can stall without taking anything else with it.

Failure 3: UDP can’t tell you it failed

A single @ (@logs.example.com:514) means UDP. UDP sends a packet and moves on. If the other end is down, rsyslog has no way of knowing, so nothing is queued or retried. Lines are simply lost. rsyslog’s own documentation says this is why reliable central logging needs TCP. Use @@ or, better, the full action() syntax with protocol="tcp".

Failure 4: anyone on the path can read it

Plain TCP syslog is unencrypted. Between two machines in the same private network that may be acceptable. From a cloud server to anywhere else, it crosses the public internet as readable text, along with whatever your applications put in their log lines. Use TLS.

rsyslog supports TLS through a “network stream driver”. On Debian and Ubuntu, install the GnuTLS one:

sudo apt install rsyslog-gnutls

And set it on the forwarding action:

StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="logs.example.com"
StreamDriver.CAFile="/etc/ssl/certs/ca-certificates.crt"

x509/name checks both that the certificate is valid and that it was issued to the name in StreamDriverPermittedPeers. Don’t switch to anon to get past an error: that encrypts the connection without checking who is on the other end. StreamDriver.CAFile on the action needs rsyslog 8.2108 or newer; older versions take the CA file as a global setting instead.

The errors people paste most often, and what they usually mean:

  • TLS handshake failed [gnutls error -43: Error in the certificate.]: the remote certificate didn’t validate against the CA file. Check that the CA file path exists and is the system bundle.
  • peer name not authorized - not permitted to talk to it: the certificate is valid but issued to a different name than StreamDriverPermittedPeers.
  • Validation failing only when the remote certificate comes from an intermediate CA: a known GnuTLS driver issue on some RHEL versions. The OpenSSL driver (StreamDriver="ossl", package rsyslog-openssl) is the usual workaround.

Failure 5: three servers called “web”

By default rsyslog sends the short hostname. web.prod.example.com and web.staging.example.com both arrive as web, and your central system files them together. Turn on fully qualified names, as a global setting near the top of the configuration:

global(preserveFQDN="on")

It also changes the hostname written in local log files, which is usually what you want anyway.

Failure 6: long messages are cut at 8 KB

rsyslog’s default maximum message size is 8 KB. Anything longer, typically a stack trace or a large JSON line, is truncated without warning. The first part survives, so it looks fine at a glance. Raise it at the very top of /etc/rsyslog.conf, before any module(load=...) line, because inputs read the value when they load:

global(maxMessageSize="64k")

Failure 7: a dead connection that looks alive

If a NAT gateway or firewall silently drops an idle TCP connection, rsyslog may keep writing into it for a long time before the operating system notices. TCP keepalives fix this:

KeepAlive="on"
KeepAlive.Probes="3"
KeepAlive.Interval="10"
KeepAlive.Time="60"

With these values a dead connection is noticed within about a minute and a half, and lines go to the queue instead of into a void.

The whole thing

Put the global settings at the top of /etc/rsyslog.conf:

global(maxMessageSize="64k")
global(preserveFQDN="on")

And the forwarding rule in /etc/rsyslog.d/60-forward.conf:

action(
  type="omfwd"
  target="logs.example.com" port="6514" protocol="tcp"

  StreamDriver="gtls" StreamDriverMode="1"
  StreamDriverAuthMode="x509/name"
  StreamDriverPermittedPeers="logs.example.com"
  StreamDriver.CAFile="/etc/ssl/certs/ca-certificates.crt"

  KeepAlive="on" KeepAlive.Probes="3" KeepAlive.Interval="10" KeepAlive.Time="60"

  queue.type="LinkedList" queue.filename="fwd_central"
  queue.maxDiskSpace="100m" queue.saveOnShutdown="on"
  action.resumeRetryCount="-1"
)

Check the syntax before restarting, then send a test line:

sudo rsyslogd -N1
sudo systemctl restart rsyslog
logger -t test "hello from $(hostname -f)"

To test the queue, block the remote port with a firewall rule, send a few lines with logger, wait a minute, remove the rule, and check the lines arrive. It’s worth doing once, because “it worked when I set it up” is precisely the state each of these failures starts from.

With SaneLog

SaneLog’s Sources page gives you this forwarding rule with your own endpoint filled in: TLS with name checking, full hostnames, the disk-assisted queue and keepalives. It also reconnects every 5,000 messages, so a change of address on our side reaches your servers without a restart. Add the maxMessageSize line yourself at the top of rsyslog.conf if you log long messages.

Each server then shows up named after its full hostname, waiting for your approval before its lines are counted, and lines that arrived while it waited are kept. syslog-ng users will find an equivalent configuration in the FAQ.

Once every server’s lines arrive in one place, you can also be told when one of them goes quiet: How to get alerted when a cron job stops running covers alerts on silence.

← all posts