How to get alerted when a cron job stops running, and other silent failures

It’s one of the most common stories in operations. A nightly backup cron job has been failing, or not running at all, for weeks. The server is fine, the uptime checks are green, nobody has raised anything. The team finds out on the day they need to restore.

Every check they had was green, and every one of them was measuring the wrong thing.

Most teams have error alerts. An exception gets thrown, a message lands in a log, (hopefully) someone looks. That covers the failures that announce themselves. The failures that hurt are the ones that don’t: the backup that never started, the queue worker that is still running but stopped taking jobs at 4am, the server that ran out of disk, stopped writing logs, and so stopped producing errors too. None of these throw anything. You find out when a customer does, or when you need the backup.

Alerts built on logs, not on exceptions, can catch all of them, because a log tool can notice that a line didn’t appear. Here are the four shapes those alerts take. The examples use SaneLog, but the ideas apply to any log tool worth paying for.

Exceptions are not the whole story

An error tracker receives an exception, groups it with others like it, and tells you when a group is new or busy. Keep one if you have one; it’s the right tool for that.

Logs are everything else your systems say: the line a backup prints when it finishes, a worker’s periodic “processed 120 jobs”, nginx’s status codes, a deploy script’s “release 2026-10-03.2 live”. Only from logs can you be told that something you expected did not happen, because nothing throws an exception when a job fails to start.

The usual fallback, having cron email you when a job fails, has the same blind spot. It only tells you about jobs that ran and complained. A job that never ran sends nothing, and nothing looks like success. The fix is to turn the signal around: the job reports success, and silence raises the alarm. This is sometimes called a dead man’s switch, or a heartbeat check.

Start with a filter

In SaneLog, every alert watches a saved filter: which sources it covers and what a line must match. It is the same thing you build when you narrow the live stream, so an alert is simply “tell me when this view changes in a way I care about”. Some examples:

Filter	Sources	Matches
app errors	all production servers	program is api and level is error or worse
backup finished	db-1	message contains backup complete
worker heartbeat	worker-1, worker-2	message contains processed
payment declines	all production servers	field decline_code is present

Changing the filter changes what its alerts watch. There’s no second query language to learn.

1. Something happened: “matching events occur”

The familiar one. It fires when the filter matches at least N lines within a time window.

• app errors, threshold 1, window 5 minutes: any production error at all.
• app errors, threshold 50, window 5 minutes: a burst, ignoring the trickle you’ve decided to live with.
• payment declines, threshold 20, window 10 minutes: no error anywhere, but twenty declines in ten minutes means your payment provider is struggling before anything throws.

That last one is the interesting one. Signals your customers feel first often live in ordinary info lines, which an error tracker never sees.

The same goes for security. A failed login attempt is mostly noise: anything exposed to the internet gets thousands of them. The line worth an alert is the successful login from somewhere you don’t expect, at an hour nobody on your team is awake. A threshold of 1 on a filter for successful sign-ins to an admin account is one of the most useful alerts a small team can have.

2. Something stopped: “matching events stop”

It fires when the filter matches nothing for the whole window plus a grace period. This is a heartbeat, and it’s the alert most teams don’t have.

• worker heartbeat, window 5 minutes, grace 5 minutes: workers that are up but no longer processing, or a machine that has gone away.
• Any filter that covers one server, window 15 minutes: that server went quiet. A server that stops logging entirely is either idle or in trouble, and you want to know which.

The grace period stops a job that runs a few minutes late from waking anyone.

3. Something didn’t happen on time: “an expected event is late”

It fires when nothing matches within a grace period after a scheduled time. This one is for anything with a timetable.

• backup finished, expected every day at 00:20, with 30 minutes’ grace: the backup must have logged its confirmation by 00:50.
• A weekly report, expected on Mondays at 06:00.
• Anything a cron expression can describe, like 0 */6 * * * for every six hours.

Each deadline comes from the schedule, never from the last time the job ran, so a job that slips five minutes later every day can’t quietly drag its own deadline along with it. Every missed run is its own incident. Schedules use the account’s timezone, including across daylight-saving changes, which is exactly where hand-written “did the backup run?” scripts break, twice a year.

It also catches a job that started and then hung. If the job logs “backup started” at 00:20 but never logs “backup complete”, the deadline for the completion line passes and the alert fires, even though the job is technically still running. Hung jobs are a common real cause of these incidents, and an alert that only listens for errors never sees them.

And it catches the scheduler itself failing. Schedulers skip runs more often than people expect: a cron daemon restarted at the wrong minute, a server rebooted overnight, a hosted CI service that runs a “every fifteen minutes” job with gaps of an hour or more under load. Every run that does happen is green. The failure is the runs that don’t.

4. Send me the lines: “every matching event”

Not an alarm, a delivery. Every line the filter matches goes to your channels, batched to at most one notification a minute. Use it to forward a particular stream somewhere: every critical line by email, or every line from a filter into a webhook that opens a ticket.

Where the alerts go

Email, Slack, Discord, Pushover, or any webhook. A channel is set up once and shared between alerts, so changing a webhook URL is one edit, and each channel has a Send test button. Webhook posts are signed, so the receiving end can check they came from you.

A notification carries the alert, its filter, how many lines matched, a few sample lines, and a link into the Log Explorer at that moment. An alert email also names its incident, so a coding agent connected to your account can be asked to look into it straight from your inbox.

One message per incident, not one per minute

Alerts are checked every minute, but a notification goes out when the condition starts to hold. That opens an incident. An alert that stays over its threshold for an hour sends one notification, not sixty. When the condition clears, the incident is closed, and you can choose to be told that too, with how long it lasted and the line that ended it. If you want reminders while an incident stays open, set a re-notify interval.

Snoozing silences notifications for an hour or a day without stopping the checks, so the record of what happened stays complete even while you sleep on purpose.

The habits that make this work

Alerts on silence only work if the good news is in your logs. Whatever your stack, three habits:

Log success, not just failure, and only after checking it. A job that only logs when something goes wrong can’t be watched for silence. But a success line that just means “the command exited 0” isn’t much better: a backup can exit 0 and write an empty file because a mount disappeared. Check the thing the job was supposed to produce, then log:

# a shell script run by cron
logger -t backup "backup started"
pg_dump "$DB" > "$OUT"
size=$(stat -c %s "$OUT")
if [ "$size" -gt 1000000 ]; then
  logger -t backup "backup complete size_bytes=$size"
else
  logger -t backup -p user.err "backup too small size_bytes=$size"
fi

The best check of all is a test restore into a scratch database. That turns “the job ran” into “we have a backup”, and it is the step almost everyone skips.

# a Python worker
log.info("processed batch", extra={"jobs": count, "duration_s": elapsed})

// a PHP job
Log::info('Backup complete', ['size_mb' => 412, 'duration_s' => 93]);

Log outcomes as data, not sentences. decline_code=card_declined gives you something to filter on. “Charge declined for order 88231” gives you something to search with a regular expression.

Send every server to one place. A heartbeat alert on worker-2 is useless if worker-2’s lines only exist on worker-2. Forwarding with rsyslog, over TLS with a disk queue so nothing is lost while the connection is down, takes one configuration file per server. SaneLog’s getting started guide has it ready to paste.

In short

Keep your error tracker for exceptions. Add four alerts from your logs: a burst threshold on errors, a heartbeat on your workers, a schedule on your nightly jobs, and a threshold on the business signal your customers feel first. The heartbeat and the schedule are the two nothing else will send you, and every SaneLog plan includes as many alerts as you want.