Hectiq Lab

Settings

Environment variables

VariableDescription
HECTIQLAB_API_KEYThe API key. Beats the credentials file.
HECTIQLAB_CREDENTIALSThe credentials file. Default is ~/.hectiq-lab/credentials.toml.
HECTIQLAB_PROFILEThe table of the credentials file that holds the key. The first one otherwise.
HECTIQLAB_PROJECTThe project, organisation/name, when no call names one.
HECTIQLAB_DISABLESet to 1 to record nothing. Every recording call does nothing.
HECTIQLAB_PROGRESSSet to false to draw no progress bar on uploads and downloads.
HECTIQLAB_CACHEWhere downloaded datasets and models are kept. Default is ~/.hectiqlab/cache. See Datasets.
HECTIQLAB_TIMEOUTSeconds to wait on one API request. Default is 30.
HECTIQLAB_API_URLThe API. Default is https://api.lab.hectiq.ai.
HECTIQLAB_WEB_URLThe web app, for the url of a run. Default is https://lab.hectiq.ai.
HECTIQLAB_PROMPTWhat asks on the terminal when it is created: 1 or all, 0 or none, or kinds among runs, datasets, models, artifacts, such as datasets,models. See Ask on the terminal.
HECTIQLAB_UPLOAD_LIMITThe most one upload may carry: one size, 20GB, or a size per kind, datasets:20GB,models:5GB,artifacts:1GB. No limit otherwise. See Size limits and disk room.
HECTIQLAB_DISK_MARGINThe room a download leaves free on its disk, 20GB. No margin otherwise.
HECTIQLAB_BUCKETA bucket of your own, gs://acme, where a dataset, a model or an artifact whose call names no location goes. Hectiq Lab's own bucket otherwise. See Datasets.
HECTIQLAB_RUN_*What hlab launch says about the first run.
HECTIQLAB_TUI_STATEWhere the dashboard keeps its preferences. Default is ~/.hectiqlab/tui.json.

Distributed training

When RANK or LOCAL_RANK is set to anything but 0, hectiqlab records nothing and asks nothing. Only the main process records, so the same script runs on every process unchanged. A create there gives a Future of None.

The project

A call records to the first project named by:

  1. The call's own project=.
  2. hl.set_project("organisation/name"), for the rest of the process.
  3. HECTIQLAB_PROJECT.
  4. The nearest .hectiqlab/config.toml above the working directory, which hlab init writes.

Progress bars

Uploads and downloads draw a progress bar on stderr when it is a terminal. Pass progress=False to a call to draw none, or progress=True to draw one anyway.

Ask on the terminal

hl.run, hl.datasets.create, hl.models.create and hlab launch can ask on the terminal when they create. Each question defaults to what the call says, so pressing Enter keeps it.

A call asks when, in this order:

  1. hectiqlab is switched on: never with HECTIQLAB_DISABLE, or on a process whose RANK or LOCAL_RANK is not 0.
  2. Its prompt=True says so. prompt=False never asks. With prompt=None, the default, HECTIQLAB_PROMPT decides.
  3. There is a terminal. Without one, in CI or behind a pipe, nothing is asked, and one log line says so.

HECTIQLAB_PROMPT takes 1 or all for every kind, 0 or none for none, or the kinds that ask, comma-separated: runs, datasets, models, artifacts. Case and spaces don't matter. A word that isn't a kind is ignored, with one warning naming it. hlab launch --prompt means every kind.

  • A run asks for its title, tags and category as it opens, then returns. A run that joins the one hlab launch opened asks nothing. As it closes, once its end has landed, a run that asked at its open prints a summary of what it did, with its link, and asks whether to keep it. No hides it; nothing is deleted, and the run's page shows it again.
  • A dataset or a model prints a plan of the version: its files, its largest folders, where they go, and the run behind it. It then asks whether to register it at all, its name, its version (empty for the next minor one) and, for a path with no location or with location="local" and no files, whether to upload its files: no keeps them on this machine. Last, an upload over its limit asks whether to go on, no by default. One you decline isn't registered, and its Future gives None.
  • A download of a dataset or a model that would leave less than HECTIQLAB_DISK_MARGIN free asks whether to go on.
  • An artifact over its limit asks whether to upload it, no by default. The answer holds for every artifact of the same name in the run, whatever its step.

Size limits and disk room

HECTIQLAB_UPLOAD_LIMIT caps what one call uploads: a dataset or a model counts all the files it uploads, an artifact its one file. The limit counts the same for Hectiq Lab's bucket and yours. It is checked before anything reaches Hectiq Lab. An upload over its limit is asked about when the prompt asks for its kind. Otherwise it isn't recorded: its Future gives None, one log line says why, and the run goes on. any_size=True on the call, or --any-size on the CLI, lets one through.

HECTIQLAB_UPLOAD_LIMIT=datasets:20GB,models:5GB,artifacts:1GB python train.py

A size is a number then a unit. KB, MB, GB, TB count in powers of 1000; KiB, MiB, GiB, TiB in powers of 1024. Case doesn't matter, and a bare number is bytes. 1GB,datasets:50GB gives datasets 50 GB and every other kind 1 GB. An item that doesn't read is ignored, with one warning naming it.

Before any byte of a download moves, the files it fetches are compared with the free space of the disk they land on. A download that doesn't fit fails, and nothing is downloaded. A dataset or a model that fits but leaves less than HECTIQLAB_DISK_MARGIN free asks when the prompt asks for its kind, and otherwise warns and goes on. An artifact never asks: it warns and goes on.

On this page