Local experiment viewer¶
The native development build includes a read-only Web workspace for existing JSONL experiments. Training jobs do not need to connect to the viewer, and the viewer does not modify their logs.
Open http://127.0.0.1:9110 locally, or http://<server-ip>:9110 remotely.
The service listens on 0.0.0.0 by default and prints its saved access key at
startup (a random key is generated on first use). Enter it in the viewer's Access key login form; no username
or browser authentication popup is needed. The login shell and static assets
are public, but experiment APIs require authentication. Successful login creates
a host-only HttpOnly, SameSite=Strict session cookie (Secure when logging in
through HTTPS). Keys are not stored in browser local storage or URLs.
Login is remembered for 30 days, including browser and server restarts.
Sign out revokes the session and clears its cookie. Scripted clients can still use HTTP Basic
with username et and the startup key. Previously added runs are restored after login.
--root sets the directory picker's initial location until a shared location is saved. Use --port to
choose another local port. Add up to six runs to the workspace and use their
checkboxes to compare them. Removing a run from the sidebar only removes it
from the workspace, not from disk. Each run has its own stream/rank selector;
sources are never silently merged.
et serve stores the key, active sessions, and shared workspace in
$XDG_STATE_HOME/expr_tracker/viewer-auth-<port>.json, or
~/.local/state/expr_tracker/viewer-auth-<port>.json when that variable is unset
or relative. The file is private (0600 on Unix), atomically updated, and locked
against concurrent use. Keep it outside repositories and do not share it.
--auth-file selects another private location. An invalid state file fails
explicitly rather than silently rotating the key. ET_VIEWER_INITIAL_KEY can
seed a new state file when migrating an existing key; it never replaces a saved
key. To rotate credentials, stop the server and replace its authentication state.
Click Add run in the sidebar to browse directories on the server, move up to a
parent directory, or enter a server path and press Go. Choose the directory
containing a single run's metric/span JSONL files and press Add run.
Only that directory's logs are loaded: subdirectories are never scanned for runs.
A parent directory without its own logs produces an explicit error. Add more
run directories to compare them; previously added runs remain in the list.
Display names come from tracker.name and tracker.project in the run's config,
not necessarily the directory basename. The viewer prefers config.json, then
config.trainer-r0.json, then the first stream config. This optional identity
read is size-limited and cached until the chosen config changes; missing or
invalid identity metadata falls back to directory names. Saved workspace entries
refresh their display names automatically.
The directory path is shown in the picker's editable path field.
This is not a browser upload picker: with SSH forwarding, paths
refer to the remote training host, not your laptop. Relative paths are resolved
against the startup --root; absolute paths can select any directory accessible
to the server process, including directories outside the startup root.
Added runs, their selected sources, colors, checkbox states, and the picker's last added directory are saved on the server, alongside its private authentication state. All browsers signed in to the same viewer share this workspace; visible pages check for changes every three seconds, even when Live is off, and also when focused. Reloading or restarting the server retains the saved workspace. Concurrent edits are revision-checked: a stale edit is rejected and the latest workspace is reloaded instead of silently overwriting another browser's changes. Loading or saving failures display an error and retry control; failed saves do not replace the last confirmed state. Removing a run only updates the shared list, not the experiment files. Unavailable saved directories remain listed with errors rather than being silently discarded.
Run directories are no longer read from or written to browser local storage.
Legacy browser-only lists are not automatically published to the shared server;
add those directories once to share them. Theme and per-chart preferences remain
local to each browser. No metric data or experiment config contents are persisted
in the shared workspace. Live and manual refresh only reload explicitly added
run directories. A ?root=... bookmark can override the picker's starting location
for that page until a run is added.
Experiment files are never uploaded, modified, or deleted. Browsing directories with more than 10,000 entries fails explicitly; enter a child directory path directly instead.
Curve comparison¶
The React + shadcn/ui workspace arranges independent rounded panels on a shared
canvas. The Runs panel contains the application icon and name, run search, and
the run list. The plus icon beside run search adds a directory. Its bottom row
groups the Live toggle button, Refresh, and Settings;
settings contains theme switching and sign out. On desktop, drag the panel's
right edge to resize its width without changing its height. The focused edge
also accepts Left/Right (Shift for larger changes), Home to reset, and Escape to cancel a drag.
The button beside Experiment tracker collapses the desktop panel into a narrow
rail. The application icon becomes the expand button, followed by Add and each
run's colored checkbox; hover or focus a checkbox to see its run name.
The rail shows all runs even when the expanded list is filtered, and keeps Live,
Refresh, and Settings available. Expand again to restore the previous width and
search without changing selection or reloading charts.
Desktop collapse and mobile management expansion are saved independently in this
browser and restored on reload. A warning appears if browser storage is unavailable.
Chart panels resize with the available space without reloading their data.
Source, details, and remove actions appear when a run is hovered or keyboard
focused, remain visible while its source menu is open, and are always available
on touch screens. Narrow screens use wider canvas margins and a compact Runs panel:
scroll the named run chips horizontally to select runs, or expand management for
source, details, and remove actions. Add, Live, Refresh, and Settings remain available.
Run and metric search fields are hidden on narrow screens, and / does not focus search.
Chart pin, settings, and expand actions likewise appear on hover or keyboard focus,
and stay visible on touch screens. The resize grip appears only when hovered,
keyboard focused, or dragged; search fields use a thin inset focus indicator.
Dropdown menus share the panels' compact radii, surface colors, and subtle borders.
The workspace uses single-line run rows. Metrics are grouped
by the first segment of their name:
train/loss and train/optimizer/lr appear under train; names without a
namespace appear under Other. Each section has independent Columns and
Rows dropdowns, both accepting 1–5 (default 4 columns and 2 rows).
Page capacity is columns times rows, so the default desktop
page shows up to eight charts in two rows. Left/right arrows navigate pages.
Narrow screens use one column and the same configured row count.
Search or a layout change resets pagination to the first matching
page; groups can also be collapsed. Only the current page is queried and rendered.
The sidebar contains
only runs; each row provides source selection, details, and removal controls.
Search and axis share one rounded sticky toolbar panel (horizontally
scrollable on narrow screens). Search filters charts immediately, matching full metric
names case-insensitively (multiple words must all match). Press / to focus
search and Escape to clear it. The header theme button switches between light
and dark themes, including chart axes and series colors, and saves the preference.
Each uPlot chart overlays the selected runs with consistent colors, with its legend centered above the plot. Values and the hovered step/time appear only in that chart's hover tooltip; cursors are not synchronized across charts. Each series shows its actual nearest sample, never an interpolated value. Ties choose the left sample. The matched sample is highlighted with a larger marker, and the tooltip heading snaps to an actual sampled step/time. Use Step, Wall time, or Relative time to change alignment. Plots show only the left and bottom axis borders, with short outward ticks and horizontal grid lines. Y-axis space fits its labels to avoid excessive left padding. The X axis includes the first and last plotted data values, suppressing nearby regular ticks; endpoint labels are stacked when needed to remain readable in narrow charts. These details also appear in PNG exports. Relative time is elapsed time since the selected source's earliest valid timestamp, independently for each run, displayed in seconds, minutes, hours, or days. Filtering steps or downsampling does not reset that origin; records without valid timestamps are skipped with a warning. Each chart's pin button adds a copy to a separate Pinned section above the namespace groups, keeping the chart in its original section. Unpinning only removes the pinned copy. Both copies share the metric's chart settings. Pinned charts support the same rows, paging, filtering, and focus controls. Hover a chart title to reveal its grab handle, then drag it before or after another chart in the same section. Hold within 48 pixels of either section edge for 650 ms to turn a page while dragging; release over a chart to place it. Escape cancels a drag. Focused titles also support arrow-key reordering: left/right move one position, up/down move one row. Pinned and original sections have independent ordering. Pins, ordering, and each section's column and row counts are remembered in the current browser, independently of the server-synchronized run directories. Each chart can expand into a focused full-width view. On narrow screens, the sidebar controls collapse so curves remain the primary content.
Per-chart settings¶
The settings button beside each metric title configures that chart independently. Press Apply settings to apply and save choices in browser local storage; they survive pagination and reloads without changing other metrics.
| Setting | Behavior |
|---|---|
| Smoothing · EMA | Weight from 0 to 0.99; 0 shows raw values. EMA starts at the first finite sample and ignores missing samples. |
| Y scale | Linear or logarithmic. Log views omit non-positive values after smoothing and report the omission. |
| Y minimum / maximum | Blank means automatic. Bounds must be finite, ordered, and positive for logarithmic axes. |
| Data points | Auto uses sample density; Show and Hide explicitly control markers. |
| Height | 160–640 CSS pixels per chart. |
| Export PNG | Downloads the currently displayed chart, including its metric title, run legend, theme, and applied transformations. |
Smoothing and log filtering run in the Worker using cached raw data; settings changes do not reread logs or alter recorded values. Hover tooltips show the displayed values and identify active EMA smoothing. Reset restores the default controls; apply them to save the reset.
Combining sources¶
Open an existing chart's settings and change Source mode from Selected source
to Overlay, Sum, Mean, Maximum, or Minimum. The chart always
shows the same metric; aggregation combines its values across source files within
each experiment, never different metrics or different experiments.
Choose sources with the checkboxes, or enter a wildcard under Source pattern:
rollout-* selects metrics.rollout-0.jsonl through metrics.rollout-7.jsonl;
trainer-* selects trainer sources. Full filenames such as
metrics.rollout-*.jsonl also work. * matches any text and ? matches one
character. A nonempty pattern replaces the manual selection and automatically
matches new available sources. Up to 128 inputs
are supported; an empty or excessive match is reported explicitly.
Overlay draws the selected sources as separate curves. Source colors distinguish sources, and different run line styles distinguish runs; legends and tooltips include both names. Reducers produce one curve per run, combining the selected sources in the same recorded step. Runs are never aggregated together. Only finite values present at that step participate: missing values are not interpolated, forward-filled, or replaced with zero, and an entirely missing step produces no sample. Means divide by the number of present values. Reduction happens on the server before downsampling, EMA smoothing, and log filtering. Time axes use the earliest selected-source timestamp at each step. Relative time starts at the earliest timestamp across the selected sources' entire histories, independently of metric presence, query window, and downsampling. Numeric overflow is reported and omitted.
The original metric remains the chart's title and position; its subtitle identifies the combination mode and input count. Other charts and source logs are unchanged. Choices are saved with that chart's browser-local settings, shared by its pinned copy, and included in PNG exports. Source aggregation does not change the run's selected source in the sidebar. Cached requests are reused when returning to the same operation and inputs. Legacy metric-aggregation settings are removed on load, while smoothing, axes, height, and other appearance settings are preserved.
Records and streaming¶
Each run's Details button opens config/summary, paginated records, and spans in a side panel without leaving the comparison workspace. The panel remains on the chosen tab and page during Live refresh. Records and spans can be downloaded as JSONL. Chart queries limit returned points and preserve per-bucket extrema. Charts skip NaN, infinity, and missing values, connecting the surrounding finite observations instead of inserting zeroes or breaks. Records still preserve the original missing/non-finite values. Record inspection preserves large integers through explicit exact-value tags instead of JavaScript rounding. The Records download contains the displayed API representation, including these tags; it is not a byte-for-byte download of the source JSONL. Record pages and each config/summary metadata group have a 16 MiB response budget. Requests exceeding it fail explicitly; narrow record requests rather than loading large tensor-like values into the browser. Constant and near-constant series receive a nonzero Y-axis range so small floating-point variations around a value such as 1 cannot collapse the plot. This changes only the display range, not the recorded values.
There is no workspace-wide metric limit. A Web Worker streams history over
/api/history/stream as NDJSON and prepares finite per-run x/y arrays
off the UI thread. Transferable Float64Array buffers move to uPlot's shared-axis,
per-series data mode without cloning thousands of boxed numbers on the UI thread.
Queries contain only metrics on the current section page (up to 20 with four
columns and five rows) and share two concurrent
streams. Each native chunk contains at most 256 points, with a bounded two-chunk
channel providing backpressure. Disconnecting cancels the scan.
Each query requests 10,000 points: ranges up to that size retain every source row (including 5,000+ points); larger ranges use extrema-preserving downsampling. Sections initialize when approached. Scrolling away pauses subscriptions, drawing, after a short grace period; initialized canvases stay mounted so scrolling back does not repeatedly destroy and recreate plots. Completed views with unchanged request and transform keys reuse their current data without resubscribing or transferring worker snapshots. Canvas drawing is skipped unless data, dimensions, theme, or chart settings change. Changed source metadata and manual refresh still invalidate offscreen views when revisited. Hover values update lightweight DOM labels rather than rerendering React charts. Shared worker streams avoid duplicate scans for neighboring charts, and completed sources are cached with targets of 128 MiB estimated storage and 64 sources. Active sources are pinned; they and rendered chart data can add memory beyond those cache targets. Worker updates are throttled to 100 ms and plot redraws are coalesced to one chart per animation frame, leaving room for input handling. Search and layout changes do not refetch run metadata; metadata polling omits full config/summary payloads and reuses the cached display identity.
New uncached streams wait briefly for search changes to settle, while chart filtering remains immediate. Cancelled requests do not produce chart errors. Transport failures and interrupted streams retry at most twice; data/protocol errors are not automatically retried. An explicit stream end frame completes loading without waiting for connection closure. Persistent failures display one message with Retry, which restarts failed sources without reloading the workspace or refetching successful sources.
Live is off by default. When enabled, it polls metadata every three seconds while the page is visible, fetching new
curves only when source metadata changes. Visible charts update in place; old
data remains visible during refresh. Each loading chart's border pulses instead
of displaying a spinner; reduced-motion mode uses a steady border. Manual Refresh
also invalidates curve queries. It refreshes selected runs, not unselected
sidebar entries, and sees only complete rows already written to disk. It cannot display
an open metric row or records still held in the training process's write buffer.
Original files and retained .parts directories are read together. Patches for
the same step are merged; span events remain separate even when they share a step.
The server lazily builds a temporary, rebuildable SQLite query index. JSONL remains the source of truth. Unchanged files reuse the index; append-only changes extend it, and detected truncation/replacement rebuilds it. The index bounds in-memory data and disappears when the viewer exits. Initial indexing of large runs can take time. Malformed records are skipped with a visible count; file/query failures are reported rather than shown as an empty successful result.
Up to 32 indexes are cached. Queries of different already indexed logs can run concurrently; each individual log's refresh and query remain serialized. Creating/rebuilding an index or evicting a cached index can still wait for active queries, and a queued cache update can delay new queries. Concurrent comparisons can use more memory even though the index count stays bounded. Limiting chart points limits response size, not the number of source rows scanned.
The viewer displays the last file modification time, not a guessed run outcome: absence of new rows does not establish that training succeeded, failed, or ended.
Remote training hosts¶
The default listener accepts remote connections. Authentication does not encrypt traffic: use a trusted network or put the server behind HTTPS. An HTTPS proxy must preserve Host, Origin, and cookies (and Authorization for scripted Basic-auth clients). Do not place keys in URLs. Alternatively, bind only to loopback and forward the port:
Open the same local URL in your browser and authenticate with the startup key.
The startup
--root is an initial directory, not a filesystem access sandbox. The viewer
can browse directories and read experiment logs anywhere the server account has
access; run it with an appropriately restricted account when needed. Log sources
and rotation segments must stay within each request's selected root. Source
selection, authentication, and same-origin checks apply to all native HTTP
endpoints, including directory browsing.
Build the development UI¶
Source development requires Rust, Python 3.10+, uv, and Node.js 22.12+:
npm --prefix web ci
npm --prefix web run build
uv sync --all-extras
uv run et serve --root ./tracker/jsonl
For frontend development, run the server on port 9110 and then
npm --prefix web run dev. Vite proxies /api to the native local server.
Production builds are included in native wheel/source distributions; installed
wheels do not require Rust or Node at runtime.