This Free Tool Watches My Server Logs So I Don’t Have To

I have a small home lab: three Linux boxes, a NAS, a router running OpenWrt (an open-source router operating system), and a Raspberry Pi doing Pi-hole duty (network-wide ad blocking). The combined log output is around 50 megabytes a day. I am not going to read that. I tried. The reading did not scale. I tried grep (a command-line text search tool) one-liners. The grep approach works for known issues (“show me the SSH (Secure Shell) failures in the last hour”) and fails for unknown ones (“is anything weird happening?”). The middle ground is a log watcher that summarizes patterns, surfaces anomalies, and stays out of my way the rest of the time. I have tried six of them in the last two years. The one I keep coming back to is called Vector, from the Datadog shop before they pivoted it. It is free, it is open source, it does the one job well, and the trade-offs are honest.

What the tool does

Vector is a log and metrics pipeline. You write a config file in TOML (a configuration file format that is easier to read than JSON, similar to INI) that defines sources (files, journald entries, syslog streams), transforms (parse, filter, sample, enrich), and sinks (where the parsed data goes: a file, a database, a webhook, a Slack message). The tool runs as a daemon (a long-running background process) and processes events in real time. The config is around 200 lines for my setup. The result is a steady stream of structured log events in a local SQLite database and a Slack channel that pings me only when something is off.

The killer feature for me is the sampling transform. Vector can keep 100% of error events and 1% of normal events. That sounds like dropping data, and it is. The result is that the local database does not explode in size, and the noise is gone. When I actually need to debug something, I switch the sample rate to 100% for that source. The tool does not pretend to keep everything. It admits that most logs are not interesting, and it gives me the controls to act on that admission.

Why I picked it over the alternatives

The alternatives I tried are real and worth mentioning. The first is the ELK stack (Elasticsearch, Logstash, Kibana: a popular logging stack that ingests, indexes, and visualizes logs). The ELK stack is the industry default. The stack is heavy. Running it on a Raspberry Pi is a joke. Running it on a small VPS (virtual private server) is a real cost. The second is Grafana Loki. Loki is the lighter cousin: a log aggregation system from the Grafana team that indexes by labels instead of full text. Loki is good. The query language takes a day to learn. The third is Promtail plus Loki, which is the “modern” combination. The combination is solid. The setup is non-trivial. The fourth is syslog-ng with a SQL sink. syslog-ng is venerable. The config is arcane. The fifth is a shell script with cron and grep. The script works. The script is the “before” state of every log pipeline project.

Vector wins for me on three axes. The first is the size. Vector compiles to a single binary. The binary is around 80 megabytes. It runs on my Raspberry Pi without breaking a sweat. The second is the config format. TOML is a breath of fresh air after YAML (a popular config format that is sensitive to indentation and prone to silent bugs). The third is the transform library. Vector has 30 or so built-in transforms. The most useful for me are the regex parser, the sample, the filter, and the route. The route transform sends events to different sinks based on a condition. That single transform is the one that made my Slack alerts stop being a firehose.

What the config looks like

The full config is long, but the shape is the kind of thing you can grasp in 5 minutes. The sources section lists where events come from:

• A file source for Nginx access logs (a popular web server)
• A journald source for systemd service logs (the standard logging system on modern Linux)
• A syslog source for the OpenWrt router
• A file source for the NAS auth log

The transforms section does the parsing and filtering. The Nginx parser uses a built-in grok pattern (a reusable regular expression template for extracting structured fields from unstructured log lines) to extract IP, status code, path, and user agent. The journald parser maps the standard journald fields to the Vector event model. The auth log parser uses a custom regex to extract username, source IP, and success/failure. The filter transform drops events that match known-good patterns. The sample transform keeps 1% of normal events and 100% of events that match “error” or “denied.”

Sinks are short. The structured events go to a local SQLite file. The high-priority events go to a Slack webhook. The “show me the last 24 hours” view is a SQLite query I run from a small shell script. The result is a log pipeline that costs me ten minutes a week to maintain, and that gives me exactly the information I need when something is off.

The honest trade-offs

Three trade-offs are worth knowing. The first is the learning curve. Vector’s transform model is powerful but takes a few days to internalize. The “aha” moment is when you realize transforms are stateless functions that produce events, and you can chain them. Before that moment, the config looks like a list of magic words. The second is the SQLite limit. SQLite is fine for a home lab. It is not fine for a real production system with gigabytes of logs. Vector scales by swapping the sink for ClickHouse, a Postgres database, or a cloud log service. The swap is one config change. The third is the upstream situation. Vector is now maintained by Datadog, and the project is less actively developed than it was in 2021 and 2022. The codebase is stable. The release cadence has slowed. For a home lab tool, the slowdown is fine. For a company betting on Vector as a core piece of infrastructure, the slowdown is a real concern.

A fourth trade-off that catches people out is the lack of a built-in alerting engine. Vector is a pipeline. The alerting is whatever you wire up on the sink side. For Slack, the alert is the message itself. For a more sophisticated system, you wire Vector to a Prometheus instance (a popular open-source monitoring system) and use Grafana for the alert rules. The lack of a built-in engine keeps Vector simple. It also means the “watch my logs” promise is only half-fulfilled by Vector alone.

When to use Vector and when to use something else

Vector is the right tool for a small-to-medium log pipeline that needs to be fast, cheap, and self-hosted. The list of cases where Vector is the right tool:

• A home lab or small business with a few servers and a few hundred megabytes of logs a day
• An edge deployment where the log pipeline runs on the device
• A developer who wants to pipe local logs into a custom sink (a database, a webhook, a custom Go service)
• A team that needs structured logs without standing up a full ELK stack

A short list of cases where Vector is the wrong tool:

• A large enterprise with gigabytes of logs a second (the SQLite approach will not scale, and the TOML config will get unwieldy)
• A team that needs a turnkey SaaS (software as a service) solution and does not want to host the pipeline
• A use case that requires full-text search across petabytes of historical logs (Elasticsearch or Loki with object storage is the better fit)

For me, Vector does the job, stays out of my way, and costs nothing. The trade-offs are the ones I can live with. The home lab is quieter, the alerts are useful, and the grep is back in the toolbox for ad-hoc queries instead of being the primary log interface.