lotel
look
Colourwayautodaynight

Still choosing a look for lotel. Your pick sticks on this browser.

Join the waitlist

Retention

lotel forgets on a schedule. Every table has a use-by policy: a maximum age and a maximum size. Old data is deleted by the storage layer, the [capped](https://github.com/hamin/duckdb-capped) DuckDB extension, not by a cleanup job you have to run.

The defaults

SignalTablesMax ageSize cap
Logsotlp_logs7 days2GB
Tracesotlp_traces3 days2GB
Metricsotlp_metrics_gauge, otlp_metrics_sum, otlp_metrics_histogram, otlp_metrics_exp_histogram30 days2GB each

The size cap applies to each table on its own, not to the store as a whole. With six tables, the store can hold up to six times the cap. 2GB means 2,000,000,000 bytes.

Rows age by their own timestamp: time_unix_nano for logs and metrics, start_time_unix_nano for spans. They do not age by when lotel received them.

How forgetting works

Each table is a set of Parquet files called segments. New rows go into one open segment. When it reaches about 128 MB, 1 million rows or a one hour span, it is sealed into a file and a new open segment starts.

Retention deletes whole sealed files:

The store applies retention when lotel starts, then again on a background thread every 60 seconds.

The result: data lives at least as long as the max age, and at most about one segment's span longer.

Change the policies

Set these environment variables before lotel starts:

VariableDefaultSets
LOTEL_TTL_LOGS7 daysMax age of the logs table
LOTEL_TTL_TRACES3 daysMax age of the traces table
LOTEL_TTL_METRICS30 daysMax age of all four metrics tables
LOTEL_MAX_BYTES2GBSize cap of every table

Ages take a number and a unit: s, m or min, h, d or day, w or week, with or without a space (12 hours, 12h, 2 weeks). Sizes take B, KB, MB, GB or TB (decimal) or KiB, MiB, GiB or TiB (binary).

LOTEL_TTL_LOGS='1 day' LOTEL_TTL_TRACES='12 hours' LOTEL_MAX_BYTES=500MB lotel

Policies are fixed when a table is created

lotel creates its tables on the first start in a data directory and writes the policies into them. On later starts it does not change tables that already exist. If you change LOTEL_TTL_* or LOTEL_MAX_BYTES for an existing data directory, the stored tables keep their original policy, and lotel prints a warning at start naming each value it ignored.

To apply new values, start lotel with a new data directory:

LOTEL_DATA_DIR=lotel-data-1d LOTEL_TTL_LOGS='1 day' lotel

The Storage page, the stickers in the top bar and the "kept" notes on the agent pages all show the policy the tables actually have.

The Storage page

/storage shows what is on disk and what forgets next.

The Storage page formats sizes in binary units, so a 2GB cap reads as 1.9 GB.

Other places retention shows up

Look at the files yourself

The store is plain Parquet files under <data dir>/store, one directory per table. You can read them with any Parquet tool. Treat the directory as sensitive: it holds everything your apps and agents sent.

Edit this page on GitHub