Self-hosting

Log Dog is a single Go binary, logdog, with subcommands for serving the API, running migrations, and creating users and apps.

Quick start

Prerequisites: Go 1.26 (the module declares go 1.26.0; a Go 1.21+ toolchain with the default GOTOOLCHAIN=auto downloads it automatically) and Docker with Compose. Run these from the logdog_backend directory.

cp .env.example .env     # local config; defaults match the dev stack
make dev-up              # start NATS, ClickHouse, Postgres, Mailpit
make migrate             # apply Postgres + ClickHouse migrations
printf '%s' "$PASSWORD" | go run ./cmd/logdog create-user --email you@example.com --password-stdin --org "My Org"
go run ./cmd/logdog create-app --org-id 1 --name "My App"
make dev                 # runs `serve`; /healthz and /metrics on :8080

create-app shows the app's API key once (save it) and prints a curl snippet for sending a log.

Production

deploy/docker-compose.prod.yml in logdog_backend is the production deployment: one Compose project (logdog-prod) with Caddy as the router in front of two ingests, the API, the writer, the alerter, the web app, NATS, ClickHouse and Postgres. It is sized for a small box (8 GB of memory, 75 GB of disk). Both repos must be checked out side by side, since the web image is built from logdog_frontend.

There are two ways to face the internet:

Behind an existing nginx (the default). Caddy listens on plain HTTP on 127.0.0.1:8480, the only published port, and your nginx terminates TLS and proxies to it. deploy/nginx/ has the virtual host and its proxy lines.
Caddy terminating TLS, for a dedicated box, with the override deploy/docker-compose.tls.yml added to the compose command.

Four steps bring it up, run from the backend checkout:

deploy/bin/init-env.sh     # writes deploy/.env (mode 600) with generated passwords
$EDITOR deploy/.env        # fill in the public URL and the SMTP account

make image VERSION=$(git rev-parse --short HEAD)    # the backend image; put that tag in deploy/.env as LOGDOG_TAG
docker compose -f deploy/docker-compose.prod.yml build web    # the web image, tagged with LOGDOG_WEB_TAG
# or: docker build -t logdog-web:<tag> ../logdog_frontend

BOOTSTRAP_OWNER_EMAIL=you@example.com deploy/bin/bootstrap.sh

bootstrap.sh needs both images built first. It starts the data services, runs the migrations, creates the owner (it asks for the password, or reads BOOTSTRAP_OWNER_PASSWORD), initializes dogfooding for the owner's organization, creates a demo organization with a demo app (1-day retention), and then brings everything up and waits for health. It is safe to run again. deploy/.env never goes into git. The backend README's Production section lists every file in deploy/.

Sign-up is closed in production: LOGDOG_SIGNUP_ENABLED=false makes the API answer 403 signup_disabled and the web app show "Sign-ups are closed on this server." The API reads it at startup and the web page on every request, so changing it needs docker compose up -d api web and no rebuild.

make prod-smoke rehearses the whole stack on a spare machine and removes it afterwards. The runbook for deploying, rolling back, backing up and restoring is deploy/RUNBOOK.md in the backend repo.

Roles

One binary, four default roles (plus demo, below). logdog serve --roles=ingest,writer,api,alerter runs everything in one process for dev and small deployments; in production, run separate containers per role so ingest can scale out independently. Roles communicate only through NATS and the databases.

Retention

Each app keeps its logs for retention_days days, from 1 to 90 (new apps get 5). Set it on the Apps page, from the retention in the app's header, or with PATCH /v1/apps/{id} and {"retention_days": N}; create-app --retention-days N sets it when the app is created. A change applies to logs received from then on: logs already stored keep the retention they arrived with.

ClickHouse's logs table carries retention_days on every row and expires rows with TTL toDateTime(ts) + toIntervalDay(retention_days). ClickHouse removes expired rows during background merges, so they can outlive their retention by a few hours. Per-minute counts (the search histogram, each app's last_entry_at, alert counts) are kept 90 days for every app, so the histogram can show activity on days whose entries have already expired.

Dogfooding

Log Dog can log to itself. Run go run ./cmd/logdog dogfood init --org-id 1 --email you@example.com, paste the LOGDOG_SELF_LOG_KEY=… line it prints into .env and restart serve; it gives org 1 the app logdog, an API key self-log (shown that once), an email channel to the address and the rule Writer lag (component:writer event:lag, threshold 1 in 5 minutes, min level warn), which counts the writer lag lines the writer logs every 5 seconds while its pending count is above LOGDOG_WRITER_LAG_WARN. Run again, it creates nothing and prints no key; --new-key creates another self-log key and prints it. With the key set, the server ships every line it logs, gzipped, to that app through POST /v1/logs, from a buffer that drops the oldest lines when full so logging never waits on ingest, and leaves out Log Dog's own /v1/logs request lines, so it never logs about logging. When it ships to the process's own ingest (the default URL), whatever is logged after a shutdown signal is lost.

| Variable | Default | Meaning | | ------------------------ | ------------------------------- | --------------------------------------------------------------------------------------------------------- | | LOGDOG_SELF_LOG_KEY | none | An API key (ld_…); set, the server ships its own log to that key's app. | | LOGDOG_SELF_LOG_URL | http://127.0.0.1:8080/v1/logs | Where to post: an http or https URL without credentials or a query; by default this process's own ingest. | | LOGDOG_SELF_LOG_BUFFER | 10000 | Lines waiting to be shipped at most; past it the oldest are dropped. | | LOGDOG_ENV | dev | The env of the shipped entries. | | LOGDOG_WRITER_LAG_WARN | 2000 | Writer pending messages above which the writer logs writer lag. |

Public demo

The demo role feeds the demo app shown live on the landing page. It is not among the default roles (LOGDOG_ROLES=ingest,writer,api,alerter,demo): each second it posts a batch of LOGDOG_DEMO_RATE generated entries, a small shop's three services with no personal data, through the normal ingest path with the demo app's key, LOGDOG_DEMO_KEY. Give the demo app an org of its own with create-user and create-app. Given the same key, the api role serves GET /v1/demo/tail, a public WebSocket speaking /v1/tail's frames without session, cookie or parameter, streaming that one app at every level; it refuses to start with a key that is not active. Its cap of LOGDOG_DEMO_TAIL_CAP sockets is separate from /v1/tail's: past it, the preflight answers 429 too_many_tails and the socket closes with 1013. The landing page shows the panel only when GET /v1/demo/tail answers; without LOGDOG_DEMO_KEY it is hidden.

| Variable | Default | Meaning | | ------------------------ | ------------------------------- | -------------------------------------------------------------------------------------------- | | LOGDOG_DEMO_KEY | none | The demo app's API key; the demo role requires it, and with it api serves the demo tail. | | LOGDOG_DEMO_INGEST_URL | http://127.0.0.1:8080/v1/logs | Where the feeder posts; this process's own ingest by default. | | LOGDOG_DEMO_RATE | 10 | Entries per batch, one batch a second. | | LOGDOG_DEMO_TAIL_CAP | 200 | The most demo tail sockets open at once. |

Dev-stack ports

All bound to 127.0.0.1:

| Service | Ports | | ---------- | -------------------------------- | | NATS | 4222 (client), 8222 (monitoring) | | ClickHouse | 9000 (native), 8123 (HTTP) | | Postgres | 5432 | | Mailpit | 1025 (SMTP), 8025 (web UI) |

Limits

Ingest limits each API key to LOGDOG_INGEST_RATE_LIMIT entries per second (5,000), in bursts of up to LOGDOG_INGEST_RATE_BURST (10,000); past it, POST /v1/logs answers 429 rate_limited with Retry-After.

Give one key its own limit, with bursts of twice that: go run ./cmd/logdog set-key-limit <prefix|key-id> <entries-per-second|default>. Ingest picks it up at its next key reload, within 30 seconds.

Each ingest instance keeps its buckets in memory, so N instances allow N times the limit.

The server accepts at most LOGDOG_HTTP_MAX_CONNS (4,096) connections at once and cuts off a request whose headers and body take longer than LOGDOG_HTTP_READ_TIMEOUT (30 seconds); the backend README lists every limit and timeout.

Attribute storage

attrs is stored in a ClickHouse JSON(max_dynamic_paths = 256) column. Each storage part keeps up to 256 distinct attribute paths as columns of their own and stores the rest together, so entries with more paths are still stored and searchable in full, only slower to search on those paths. Raising the number is a change to the logs table, shipped as a ClickHouse migration.