Hectiq Lab

Datasets

A dataset is a versioned set of files that runs read. A version is a folder, plus the files that vouch for it. The folder lives in Hectiq Lab, in your own Google Cloud Storage bucket, or on one machine.

Create a dataset

import hectiqlab as hl

hl.datasets.create("data/interactions/", name="interactions")
NameTypeDefaultDescription
pathstrA file, or a folder whose files keep their relative paths.
namestrIts name, shared by its versions.
versionstrNonemajor.minor. The next minor version otherwise.
descriptionstrNoneWhat it is.
metadatadictNoneAnything else to keep with it. A JSON object on the CLI.
runstr or intNoneThe run that produced it. The open run otherwise.
locationstrNoneWhere the folder lives: a "gs://bucket/folder", a "gs://bucket" for a new folder of it, "local", or another URI for data that is not files. HECTIQLAB_BUCKET, else Hectiq Lab's own bucket, otherwise. See where a version lives. --location on the CLI.
fileslist[str]NoneThe files that vouch for the version, relative to its folder. Every file otherwise. Only with a gs:// or "local" location. --file, once per file, on the CLI.
sourcestrNoneWhere the data came from. The path otherwise.
projectstrNoneThe project. The open run's otherwise.
progressboolNoneDraw a progress bar or not.
promptboolNoneAsk on the terminal, under a plan of the version. HECTIQLAB_PROMPT otherwise. See Settings.
any_sizeboolFalseUpload it whatever HECTIQLAB_UPLOAD_LIMIT says. --any-size on the CLI.

create returns a Future at once, and the files upload in the background. The open run waits for them before it ends. The Future gives None when nothing was recorded: the SDK is switched off, the prompt was told no, or the upload is over its limit. See Size limits and disk room.

Where a version lives

location is a URI. Its scheme says where the folder lives.

locationThe folder livesIts bytes move throughfiles
NoneIn Hectiq Lab's own bucketHectiq Lab's linksNo: every file vouches
None, with HECTIQLAB_BUCKET=gs://bucketIn a new folder of that bucket, as for "gs://bucket"Your own Google loginYes
"gs://bucket/folder"In your bucketYour own Google loginYes
"gs://bucket"In a new folder of your bucket, datasets/<id>, named after the versionYour own Google loginYes
"local"On this machine, at the path, recorded as file://host/pathNothing: they stay putYes
Any other URI, such as "bigquery://project.dataset"Wherever it says: data that is not filesNothingNo

Hectiq Lab keeps a checksum of each file that vouches for a version, so any machine can check its copy. A checksum names its method: md5:<hex>, or crc32c:<hex> for a composite object in a bucket, which has no MD5. By default every file vouches. With files, only the ones you name do, an index for example: the rest of the folder can grow without damaging the version.

hl.datasets.create("data/", name="clicks")
hl.datasets.create("data/", name="clicks", location="gs://acme-ml/clicks/v3")
hl.datasets.create(name="images", location="gs://acme-ml/images/", files=["images.parquet"])
hl.datasets.create("/data/images/", name="images", location="local", files=["images.parquet"])
hl.datasets.create(name="events", location="bigquery://acme.events")

A gs:// folder needs pip install "hectiqlab[gcs]" and a Google login on the machine, gcloud auth application-default login for example. Hectiq Lab never reads your bucket.

  • With a path, the files upload into the folder, which must be empty. Google computes each checksum.
  • With a bucket alone, gs://acme, the files upload into a new folder Hectiq Lab names after the version: gs://acme/datasets/<id>, or gs://acme/models/<id> for a model. It needs a path. HECTIQLAB_BUCKET=gs://acme sends there every dataset and model whose call names no location: a project that keeps its data in its own bucket says so once, in its environment. A location the call names still wins, "local" included. A HECTIQLAB_BUCKET that isn't a bucket alone is refused by the create that would use it, before anything moves.
  • Without a path, the files are already in the folder. Google says the checksum of each file that vouches; the data is never read.

A version whose files don't all land, in any bucket, is deleted, and the files it wrote into your bucket with it, so the folder is as it was. When Hectiq Lab can't be reached to delete it, the version and its files are kept, and a log line says so.

A bucket alone needs an API that knows it: Hectiq Lab's API ships it first. An older one refuses gs://acme before anything moves.

A "local" folder never moves. The files that vouch are hashed on this machine.

Any other URI is data that is not files, such as a BigQuery table. It takes no path and no files.

Read a dataset

A dataset is read by its name, and by its version or the latest one.

data = hl.datasets.get("interactions", "1.2").unwrap()
latest = hl.datasets.get("interactions").unwrap()

hl.datasets.get_by_id(id) reads one by its id.

Record that a run read a dataset

data = hl.datasets.get("interactions").unwrap()
hl.datasets.attach(data.id)

Without run=, the dataset is attached to the open run. hl.datasets.detach(data.id) removes it.

download and resolve attach it themselves when a run is open.

Download a dataset

folder = hl.datasets.download("interactions", "1.2").result()
NameTypeDefaultDescription
namestrThe dataset's name.
versionstrNonemajor.minor. The lab's latest otherwise.
intostrNoneA folder to download into. The cache otherwise.
projectstrNoneThe project. The open run's otherwise.
progressboolNoneDraw a progress bar or not.

download returns a Future with the folder's path. Without into, the files land in the cache, in <cache>/<project>/datasets/<name>/<version>/.

It fetches only what is missing or damaged. A second call on an intact copy fetches nothing, and a copy with a deleted or changed file gets just those files again. Each file that vouches is checked against the checksum Hectiq Lab keeps.

A version in a gs:// folder comes from that folder, with your Google login. Every file in the folder is fetched. A file that does not vouch is skipped when one of the same size is already there.

It refuses a folder that holds another version. A version kept on a machine, or data that is not files, cannot be downloaded: download fails and says where its data lives.

Before any byte moves, the files it fetches are compared with the free space of their disk. A download that doesn't fit fails, and nothing is downloaded. See Size limits and disk room.

Called while a run is open, it records that the run read this version.

hl.datasets.files(data.id) lists the files instead. A file in Hectiq Lab's bucket comes with an address valid for a day.

Find a dataset on this machine

folder = hl.datasets.resolve("interactions", "1.2").unwrap()

resolve gives the folder of an intact copy on this machine, found by name, wherever it was downloaded or created. It never downloads.

Without a version, it picks the highest major.minor with an intact copy on this machine, compared as integers, so 1.10 is above 1.9. A version like v1.3 is only found by naming it. The version picked is logged at INFO. A higher version skipped because its copies are damaged is logged as a warning, with their paths.

Called while a run is open, it records that the run read this version.

When there is no intact copy, the error says so and gives the hlab dataset download command to run. For a version kept on a machine, or data that is not files, it says where its data lives instead.

The CLI prints the path alone, so $(hlab dataset resolve interactions) works in a shell.

A training script resolves its data, and the download is done once beforehand, on a cluster's login node for example:

hlab dataset download interactions --version 1.2
with hl.run("Train two-tower"):
    folder = hl.datasets.resolve("interactions", "1.2").unwrap()
    ...

Check a copy

checks = hl.datasets.verify("interactions", "1.2").unwrap()

verify gives each copy on this machine and its state: intact, or which files are missing or changed. It reads only local files.

The CLI exits with 1 when no copy is intact.

The marker and the cache

A folder that holds a version carries a .hectiqlab.json in its root. It says which version the folder holds (kind, id, project, name and version) and the size, modification time and MD5 of each file that vouches. It is never uploaded. Deleting it is safe: the next download into that folder fetches everything again.

create writes it into a folder it registers, so resolve finds that folder at once. A single file gets no marker.

resolve checks only the files that vouch. A folder too large to hash can vouch with a few of its files, and files added later never damage the copy:

hl.datasets.create("/mnt/images/", name="images", location="local", files=["images.parquet"])

Downloads land in the cache, ~/.hectiqlab/cache unless HECTIQLAB_CACHE says otherwise. The index of every copy on this machine lives under it too, in .registry/.

hlab cache list
hlab cache clear interactions@1.2
hlab cache clear --all

hlab cache list shows every version on this machine: kind, project, name, version, size, copies and when it was last resolved. It takes --project.

hlab cache clear NAME[@VERSION] deletes copies inside the cache. It takes --kind dataset|model and --project. --all clears every copy and asks first; --yes skips the question. It deletes copies inside the cache only: a folder elsewhere, one you created or downloaded with --into, is kept and stays registered. A copy whose folder is already gone is forgotten.

List datasets

datasets = hl.datasets.list(search="interactions").unwrap()

Only the latest version of each dataset is listed. Pass keep_latest_version=False, or --all, for every version.

Change a dataset

hl.datasets.update(data.id, description="Clicks from May", metadata={"rows": 1200000})

metadata is merged into what the dataset already has. In Python, update also takes name, version and source.

Tag a dataset

hl.datasets.tag(data.id, "clean", "may")
hl.datasets.untag(data.id, "may")

Delete a dataset

hl.datasets.delete(data.id)

This deletes the version and its files. The CLI asks to confirm; --yes skips the question.

On this page