Settings
Environment variables
| Variable | Description |
|---|---|
HECTIQLAB_API_KEY | The API key. Beats the credentials file. |
HECTIQLAB_CREDENTIALS | The credentials file. Default is ~/.hectiq-lab/credentials.toml. |
HECTIQLAB_PROFILE | The table of the credentials file that holds the key. The first one otherwise. |
HECTIQLAB_PROJECT | The project, organisation/name, when no call names one. |
HECTIQLAB_DISABLE | Set to 1 to record nothing. Every recording call does nothing. |
HECTIQLAB_PROGRESS | Set to false to draw no progress bar on uploads and downloads. |
HECTIQLAB_CACHE | Where downloaded datasets and models are kept. Default is ~/.hectiqlab/cache. See Datasets. |
HECTIQLAB_TIMEOUT | Seconds to wait on one API request. Default is 30. |
HECTIQLAB_API_URL | The API. Default is https://api.lab.hectiq.ai. |
HECTIQLAB_WEB_URL | The web app, for the url of a run. Default is https://lab.hectiq.ai. |
HECTIQLAB_PROMPT | What asks on the terminal when it is created: 1 or all, 0 or none, or kinds among runs, datasets, models, artifacts, such as datasets,models. See Ask on the terminal. |
HECTIQLAB_UPLOAD_LIMIT | The most one upload may carry: one size, 20GB, or a size per kind, datasets:20GB,models:5GB,artifacts:1GB. No limit otherwise. See Size limits and disk room. |
HECTIQLAB_DISK_MARGIN | The room a download leaves free on its disk, 20GB. No margin otherwise. |
HECTIQLAB_BUCKET | A bucket of your own, gs://acme, where a dataset, a model or an artifact whose call names no location goes. Hectiq Lab's own bucket otherwise. See Datasets. |
HECTIQLAB_RUN_* | What hlab launch says about the first run. |
HECTIQLAB_TUI_STATE | Where the dashboard keeps its preferences. Default is ~/.hectiqlab/tui.json. |
Distributed training
When RANK or LOCAL_RANK is set to anything but 0, hectiqlab records nothing and asks nothing. Only the main process records, so the same script runs on every process unchanged. A create there gives a Future of None.
The project
A call records to the first project named by:
- The call's own
project=. hl.set_project("organisation/name"), for the rest of the process.HECTIQLAB_PROJECT.- The nearest
.hectiqlab/config.tomlabove the working directory, whichhlab initwrites.
Progress bars
Uploads and downloads draw a progress bar on stderr when it is a terminal. Pass progress=False to a call to draw none, or progress=True to draw one anyway.
Ask on the terminal
hl.run, hl.datasets.create, hl.models.create and hlab launch can ask on the terminal when they create. Each question defaults to what the call says, so pressing Enter keeps it.
A call asks when, in this order:
hectiqlabis switched on: never withHECTIQLAB_DISABLE, or on a process whoseRANKorLOCAL_RANKis not0.- Its
prompt=Truesays so.prompt=Falsenever asks. Withprompt=None, the default,HECTIQLAB_PROMPTdecides. - There is a terminal. Without one, in CI or behind a pipe, nothing is asked, and one log line says so.
HECTIQLAB_PROMPT takes 1 or all for every kind, 0 or none for none, or the kinds that ask, comma-separated: runs, datasets, models, artifacts. Case and spaces don't matter. A word that isn't a kind is ignored, with one warning naming it. hlab launch --prompt means every kind.
- A run asks for its title, tags and category as it opens, then returns. A run that joins the one
hlab launchopened asks nothing. As it closes, once its end has landed, a run that asked at its open prints a summary of what it did, with its link, and asks whether to keep it. No hides it; nothing is deleted, and the run's page shows it again. - A dataset or a model prints a plan of the version: its files, its largest folders, where they go, and the run behind it. It then asks whether to register it at all, its name, its version (empty for the next minor one) and, for a path with no
locationor withlocation="local"and nofiles, whether to upload its files: no keeps them on this machine. Last, an upload over its limit asks whether to go on, no by default. One you decline isn't registered, and itsFuturegivesNone. - A download of a dataset or a model that would leave less than
HECTIQLAB_DISK_MARGINfree asks whether to go on. - An artifact over its limit asks whether to upload it, no by default. The answer holds for every artifact of the same name in the run, whatever its step.
Size limits and disk room
HECTIQLAB_UPLOAD_LIMIT caps what one call uploads: a dataset or a model counts all the files it uploads, an artifact its one file. The limit counts the same for Hectiq Lab's bucket and yours. It is checked before anything reaches Hectiq Lab. An upload over its limit is asked about when the prompt asks for its kind. Otherwise it isn't recorded: its Future gives None, one log line says why, and the run goes on. any_size=True on the call, or --any-size on the CLI, lets one through.
HECTIQLAB_UPLOAD_LIMIT=datasets:20GB,models:5GB,artifacts:1GB python train.pyA size is a number then a unit. KB, MB, GB, TB count in powers of 1000; KiB, MiB, GiB, TiB in powers of 1024. Case doesn't matter, and a bare number is bytes. 1GB,datasets:50GB gives datasets 50 GB and every other kind 1 GB. An item that doesn't read is ignored, with one warning naming it.
Before any byte of a download moves, the files it fetches are compared with the free space of the disk they land on. A download that doesn't fit fails, and nothing is downloaded. A dataset or a model that fits but leaves less than HECTIQLAB_DISK_MARGIN free asks when the prompt asks for its kind, and otherwise warns and goes on. An artifact never asks: it warns and goes on.