Hectiq Lab

Runs

A run is one execution of your code: a training, an evaluation, a test. It belongs to a project, and it records how it ended and what produced it.

Open a run

import hectiqlab as hl

with hl.run("My first run", category="training", config={"lr": 1e-3}):
    ...
NameTypeDefaultDescription
titlestrWhat the run is.
descriptionstrNoneA longer description.
categorystrNoneOne of training, develop, evaluation, processing, test.
tagslist[str]NoneTags to attach.
configdictNoneThe hyperparameters.
groupGroup or strNoneThe group it joins, from hl.group, or the group's id. See Group runs.
projectstrNoneThe project. See Projects for how it's found otherwise.
promptboolNoneAsk on the terminal for its title, tags and category, and whether to keep it. HECTIQLAB_PROMPT otherwise. See Settings.

hl.run returns at once, or once answered when it asks: the run is created in the background.

Under hlab launch, the first run a script opens takes the title, description, category, tags and group the launch gives.

How a run ends

The blockThe run ends
Finishescompleted
Raisesfailed, with the traceback
Is interrupted with Ctrl-Cstopped
Exits with sys.exitcompleted with no status or 0, failed otherwise

A run opened without with ends the same way when the script exits. One exception: a script that calls sys.exit(1) ends such a run completed, because Python doesn't report that exit. Use with, or hlab launch, to record it.

The run waits for what it recorded before it ends, uploads included. A run that asked on the terminal as it opened then asks whether to keep it, under a summary of what it did. No hides it; nothing is deleted.

What a run records by itself

As it opens, the run records:

  • the git commit, branch, remote and diff;
  • every installed package and its version;
  • the machine: operating system, processor, GPU and host name;
  • the command line: the script, its arguments, the working directory and the Python used.

Read runs

A run is named by its rank, the number the web app shows, or by its id.

import hectiqlab as hl

run = hl.runs.get(42).unwrap()
runs = hl.runs.list(tags=["ENG-42"], statuses=["completed"]).unwrap()

hl.runs.list filters by search, tags, statuses, categories, authors and hidden, and takes a limit. It lists newest first, without diffs or environments; hl.runs.get reads everything.

Every run read carries url, its page in the web app. hlab run list --tag ENG-42 --json gives every run a ticket's work made, each with the address to cite.

Change a run

hl.runs.update(42, title="Baseline", note="Kept for the paper")
NameTypeDescription
titlestrThe new title.
descriptionstrThe new description.
categorystrThe new category.
statusstrOne of initiated, running, completed, failed, stopped.
notestrThe new note.
hiddenboolHide or show the run. --hide or --show on the CLI.
configdictThe new config. Python only.
groupGroup or strThe group to move it to, or the group's id. On the CLI, --group takes an id or a title, as hl.group does, and says when the title was new and the group made.

Without a run, hl.runs.update changes the open run.

Group runs

A group is the line of work a run belongs to: a sweep, an ablation, a series of attempts at one question. hl.group finds the group or makes it, and the run joins it:

sweep = hl.group("learning-rate sweep")
for lr in (1e-4, 3e-4, 1e-3):
    with hl.run(f"lr {lr}", group=sweep, config={"lr": lr}):
        ...
CallReturns
hl.group("learning-rate sweep")The group with exactly that title, made when none has it. When several have it, it refuses and lists each one's id, run count and latest run. Two scripts making one new title at the same moment can leave two such groups.
hl.group(id="0b6f…")The group with that id.
hl.group(run=42)The group run 42 belongs to.
hl.group("learning-rate sweep", new=True)A new group, whatever already has the title.

A group's id and a title are told apart by the id's shape, a UUID. Browse a project's groups, newest first:

groups = hl.runs.list_groups(search="sweep", tags=["ENG-42"]).unwrap()

hl.runs.list_groups filters by search, words the title contains, tags, the groups holding a run with any of them, and runs, the groups those runs belong to. An empty tags or runs list matches no group. It takes a limit, and order_by="last_run_at" puts the group with the latest run first; a group with no run counts from its creation. Each group carries num_runs and last_run_at, deleted runs aside.

Describe a group and write its note

A group has two texts of its own. Its description says what the experiment is about, in a line or two, and is written when the group is made. Its note is the summary over all its runs, in markdown, written when the line of work ends: what was tried, what held, what to do next. The note names runs by rank, linked, and never copies their numbers or notes.

sweep = hl.group("learning-rate sweep", description="Whether a warmup lets lr 1e-3 hold.")
...
hl.runs.update_group(sweep, note="A warmup lets lr 1e-3 hold ([run 43](https://lab.hectiq.ai/...)).")

hl.group writes the description only when it makes the group. A group it finds keeps its own, so a script run again never rewrites it. hl.runs.update_group changes the title, description or note given, takes the group or its id, and returns the group as it now stands. An empty text clears a field.

hlab run group create finds the group titled so, or makes it with its description, and says when it made one. hlab run group get and hlab run group update take the group's id or exact title, and never make a group: a title no group has is not found.

A group read with hl.group or hlab run group get carries its description, its note and url, its page in the web app. hl.runs.list_groups carries the description, not the note. A run read with hl.runs.get or hlab run get carries group, its group's id and title, or None when it is in no group.

Tag a run

hl.runs.tag("best", "baseline", run=42)
hl.runs.untag("baseline", run=42)

Without run=, the open run is tagged. Missing tags are created.

The open run

hl.run returns the run it opened:

NameDescription
run.idThe run's id. Waits for the run to be created.
run.rankThe run's number in its project. Waits too.
run.result()Waits for everything the run recorded, and raises the first failure. See Results and errors.

On this page