Runs
A run is one execution of your code: a training, an evaluation, a test. It belongs to a project, and it records how it ended and what produced it.
Open a run
import hectiqlab as hl
with hl.run("My first run", category="training", config={"lr": 1e-3}):
...| Name | Type | Default | Description |
|---|---|---|---|
title | str | What the run is. | |
description | str | None | A longer description. |
category | str | None | One of training, develop, evaluation, processing, test. |
tags | list[str] | None | Tags to attach. |
config | dict | None | The hyperparameters. |
group | Group or str | None | The group it joins, from hl.group, or the group's id. See Group runs. |
project | str | None | The project. See Projects for how it's found otherwise. |
prompt | bool | None | Ask on the terminal for its title, tags and category, and whether to keep it. HECTIQLAB_PROMPT otherwise. See Settings. |
hl.run returns at once, or once answered when it asks: the run is created in the background.
Under hlab launch, the first run a script opens takes the title, description, category, tags and group the launch gives.
How a run ends
| The block | The run ends |
|---|---|
| Finishes | completed |
| Raises | failed, with the traceback |
| Is interrupted with Ctrl-C | stopped |
Exits with sys.exit | completed with no status or 0, failed otherwise |
A run opened without with ends the same way when the script exits. One exception: a script that calls sys.exit(1) ends such a run completed, because Python doesn't report that exit. Use with, or hlab launch, to record it.
The run waits for what it recorded before it ends, uploads included. A run that asked on the terminal as it opened then asks whether to keep it, under a summary of what it did. No hides it; nothing is deleted.
What a run records by itself
As it opens, the run records:
- the git commit, branch, remote and diff;
- every installed package and its version;
- the machine: operating system, processor, GPU and host name;
- the command line: the script, its arguments, the working directory and the Python used.
Read runs
A run is named by its rank, the number the web app shows, or by its id.
import hectiqlab as hl
run = hl.runs.get(42).unwrap()
runs = hl.runs.list(tags=["ENG-42"], statuses=["completed"]).unwrap()hl.runs.list filters by search, tags, statuses, categories, authors and hidden, and takes a limit. It lists newest first, without diffs or environments; hl.runs.get reads everything.
Every run read carries url, its page in the web app. hlab run list --tag ENG-42 --json gives every run a ticket's work made, each with the address to cite.
Change a run
hl.runs.update(42, title="Baseline", note="Kept for the paper")| Name | Type | Description |
|---|---|---|
title | str | The new title. |
description | str | The new description. |
category | str | The new category. |
status | str | One of initiated, running, completed, failed, stopped. |
note | str | The new note. |
hidden | bool | Hide or show the run. --hide or --show on the CLI. |
config | dict | The new config. Python only. |
group | Group or str | The group to move it to, or the group's id. On the CLI, --group takes an id or a title, as hl.group does, and says when the title was new and the group made. |
Without a run, hl.runs.update changes the open run.
Group runs
A group is the line of work a run belongs to: a sweep, an ablation, a series of attempts at one question. hl.group finds the group or makes it, and the run joins it:
sweep = hl.group("learning-rate sweep")
for lr in (1e-4, 3e-4, 1e-3):
with hl.run(f"lr {lr}", group=sweep, config={"lr": lr}):
...| Call | Returns |
|---|---|
hl.group("learning-rate sweep") | The group with exactly that title, made when none has it. When several have it, it refuses and lists each one's id, run count and latest run. Two scripts making one new title at the same moment can leave two such groups. |
hl.group(id="0b6f…") | The group with that id. |
hl.group(run=42) | The group run 42 belongs to. |
hl.group("learning-rate sweep", new=True) | A new group, whatever already has the title. |
A group's id and a title are told apart by the id's shape, a UUID. Browse a project's groups, newest first:
groups = hl.runs.list_groups(search="sweep", tags=["ENG-42"]).unwrap()hl.runs.list_groups filters by search, words the title contains, tags, the groups holding a run with any of them, and runs, the groups those runs belong to. An empty tags or runs list matches no group. It takes a limit, and order_by="last_run_at" puts the group with the latest run first; a group with no run counts from its creation. Each group carries num_runs and last_run_at, deleted runs aside.
Describe a group and write its note
A group has two texts of its own. Its description says what the experiment is about, in a line or two, and is written when the group is made. Its note is the summary over all its runs, in markdown, written when the line of work ends: what was tried, what held, what to do next. The note names runs by rank, linked, and never copies their numbers or notes.
sweep = hl.group("learning-rate sweep", description="Whether a warmup lets lr 1e-3 hold.")
...
hl.runs.update_group(sweep, note="A warmup lets lr 1e-3 hold ([run 43](https://lab.hectiq.ai/...)).")hl.group writes the description only when it makes the group. A group it finds keeps its own, so a script run again never rewrites it. hl.runs.update_group changes the title, description or note given, takes the group or its id, and returns the group as it now stands. An empty text clears a field.
hlab run group create finds the group titled so, or makes it with its description, and says when it made one. hlab run group get and hlab run group update take the group's id or exact title, and never make a group: a title no group has is not found.
A group read with hl.group or hlab run group get carries its description, its note and url, its page in the web app. hl.runs.list_groups carries the description, not the note. A run read with hl.runs.get or hlab run get carries group, its group's id and title, or None when it is in no group.
Tag a run
hl.runs.tag("best", "baseline", run=42)
hl.runs.untag("baseline", run=42)Without run=, the open run is tagged. Missing tags are created.
The open run
hl.run returns the run it opened:
| Name | Description |
|---|---|
run.id | The run's id. Waits for the run to be created. |
run.rank | The run's number in its project. Waits too. |
run.result() | Waits for everything the run recorded, and raises the first failure. See Results and errors. |