Hectiq Lab

Models

A model is a versioned set of files a run produced for use elsewhere, such as weights. Models work exactly like datasets: hl.models has every function hl.datasets has, and hlab model every command of hlab dataset.

Create a model

import hectiqlab as hl

with hl.run("Train two-tower"):
    ...
    hl.models.create("checkpoints/", name="two-tower")

Created inside a run, the model is linked to it. The run waits for the upload before it ends.

location and files say where its folder lives and which files vouch for it, as for a dataset:

hl.models.create("checkpoints/", name="two-tower", location="gs://acme-ml/two-tower/v4")

Download a model

folder = hl.models.download("two-tower", "1.3").result()

Without a version, you get the latest. Without into, the files land in the cache, in <cache>/<project>/models/<name>/<version>/.

It fetches only what is missing or damaged, checks each file that vouches against the checksum Hectiq Lab keeps, and refuses a folder that holds another version. A model in a gs:// folder comes from that folder, with your Google login. A model kept on a machine, or data that is not files, cannot be downloaded: download fails and says where its data lives. Called while a run is open, it records that the run read this version.

Find a model on this machine

folder = hl.models.resolve("two-tower", "1.3").unwrap()

resolve gives the folder of an intact copy on this machine and never downloads. Without a version, it picks the highest major.minor with an intact copy on this machine and logs the pick at INFO; a higher version skipped because its copies are damaged is logged as a warning. When there is no intact copy, the error gives the hlab model download command to run, or, for a model kept on a machine or data that is not files, where its data lives. The CLI prints the path alone, so $(hlab model resolve two-tower) works in a shell.

A script that loads a model resolves it inside its run, after hlab model download was done once beforehand:

with hl.run("Evaluate two-tower"):
    folder = hl.models.resolve("two-tower", "1.3").unwrap()
    ...

Check a copy

checks = hl.models.verify("two-tower", "1.3").unwrap()

verify gives each copy on this machine and its state, and exits with 1 on the CLI when no copy is intact.

The marker and the cache

A model's folder carries the same .hectiqlab.json marker as a dataset's, and downloads land in the same cache. hlab cache list and hlab cache clear cover models too; see datasets.

Everything else

ToPythonCLI
Readhl.models.get(name, version)hlab model get
Listhl.models.list()hlab model list
List the fileshl.models.files(id)hlab model files
Changehl.models.update(id, ...)hlab model update
Taghl.models.tag(id, *tags)hlab model tag
Attach to a runhl.models.attach(id)
Deletehl.models.delete(id)hlab model delete

Each is described on the datasets page.

On this page