Datasets
A dataset is a versioned set of files that runs read. A version is a folder, plus the files that vouch for it. The folder lives in Hectiq Lab, in your own Google Cloud Storage bucket, or on one machine.
Create a dataset
import hectiqlab as hl
hl.datasets.create("data/interactions/", name="interactions")| Name | Type | Default | Description |
|---|---|---|---|
path | str | A file, or a folder whose files keep their relative paths. | |
name | str | Its name, shared by its versions. | |
version | str | None | major.minor. The next minor version otherwise. |
description | str | None | What it is. |
metadata | dict | None | Anything else to keep with it. A JSON object on the CLI. |
run | str or int | None | The run that produced it. The open run otherwise. |
location | str | None | Where the folder lives: a "gs://bucket/folder", a "gs://bucket" for a new folder of it, "local", or another URI for data that is not files. HECTIQLAB_BUCKET, else Hectiq Lab's own bucket, otherwise. See where a version lives. --location on the CLI. |
files | list[str] | None | The files that vouch for the version, relative to its folder. Every file otherwise. Only with a gs:// or "local" location. --file, once per file, on the CLI. |
source | str | None | Where the data came from. The path otherwise. |
project | str | None | The project. The open run's otherwise. |
progress | bool | None | Draw a progress bar or not. |
prompt | bool | None | Ask on the terminal, under a plan of the version. HECTIQLAB_PROMPT otherwise. See Settings. |
any_size | bool | False | Upload it whatever HECTIQLAB_UPLOAD_LIMIT says. --any-size on the CLI. |
create returns a Future at once, and the files upload in the background. The open run waits for them before it ends. The Future gives None when nothing was recorded: the SDK is switched off, the prompt was told no, or the upload is over its limit. See Size limits and disk room.
Where a version lives
location is a URI. Its scheme says where the folder lives.
location | The folder lives | Its bytes move through | files |
|---|---|---|---|
None | In Hectiq Lab's own bucket | Hectiq Lab's links | No: every file vouches |
None, with HECTIQLAB_BUCKET=gs://bucket | In a new folder of that bucket, as for "gs://bucket" | Your own Google login | Yes |
"gs://bucket/folder" | In your bucket | Your own Google login | Yes |
"gs://bucket" | In a new folder of your bucket, datasets/<id>, named after the version | Your own Google login | Yes |
"local" | On this machine, at the path, recorded as file://host/path | Nothing: they stay put | Yes |
Any other URI, such as "bigquery://project.dataset" | Wherever it says: data that is not files | Nothing | No |
Hectiq Lab keeps a checksum of each file that vouches for a version, so any machine can check its copy. A checksum names its method: md5:<hex>, or crc32c:<hex> for a composite object in a bucket, which has no MD5. By default every file vouches. With files, only the ones you name do, an index for example: the rest of the folder can grow without damaging the version.
hl.datasets.create("data/", name="clicks")
hl.datasets.create("data/", name="clicks", location="gs://acme-ml/clicks/v3")
hl.datasets.create(name="images", location="gs://acme-ml/images/", files=["images.parquet"])
hl.datasets.create("/data/images/", name="images", location="local", files=["images.parquet"])
hl.datasets.create(name="events", location="bigquery://acme.events")A gs:// folder needs pip install "hectiqlab[gcs]" and a Google login on the machine, gcloud auth application-default login for example. Hectiq Lab never reads your bucket.
- With a path, the files upload into the folder, which must be empty. Google computes each checksum.
- With a bucket alone,
gs://acme, the files upload into a new folder Hectiq Lab names after the version:gs://acme/datasets/<id>, orgs://acme/models/<id>for a model. It needs a path.HECTIQLAB_BUCKET=gs://acmesends there every dataset and model whose call names no location: a project that keeps its data in its own bucket says so once, in its environment. Alocationthe call names still wins,"local"included. AHECTIQLAB_BUCKETthat isn't a bucket alone is refused by thecreatethat would use it, before anything moves. - Without a path, the files are already in the folder. Google says the checksum of each file that vouches; the data is never read.
A version whose files don't all land, in any bucket, is deleted, and the files it wrote into your bucket with it, so the folder is as it was. When Hectiq Lab can't be reached to delete it, the version and its files are kept, and a log line says so.
A bucket alone needs an API that knows it: Hectiq Lab's API ships it first. An older one refuses gs://acme before anything moves.
A "local" folder never moves. The files that vouch are hashed on this machine.
Any other URI is data that is not files, such as a BigQuery table. It takes no path and no files.
Read a dataset
A dataset is read by its name, and by its version or the latest one.
data = hl.datasets.get("interactions", "1.2").unwrap()
latest = hl.datasets.get("interactions").unwrap()hl.datasets.get_by_id(id) reads one by its id.
Record that a run read a dataset
data = hl.datasets.get("interactions").unwrap()
hl.datasets.attach(data.id)Without run=, the dataset is attached to the open run. hl.datasets.detach(data.id) removes it.
download and resolve attach it themselves when a run is open.
Download a dataset
folder = hl.datasets.download("interactions", "1.2").result()| Name | Type | Default | Description |
|---|---|---|---|
name | str | The dataset's name. | |
version | str | None | major.minor. The lab's latest otherwise. |
into | str | None | A folder to download into. The cache otherwise. |
project | str | None | The project. The open run's otherwise. |
progress | bool | None | Draw a progress bar or not. |
download returns a Future with the folder's path. Without into, the files land in the cache, in <cache>/<project>/datasets/<name>/<version>/.
It fetches only what is missing or damaged. A second call on an intact copy fetches nothing, and a copy with a deleted or changed file gets just those files again. Each file that vouches is checked against the checksum Hectiq Lab keeps.
A version in a gs:// folder comes from that folder, with your Google login. Every file in the folder is fetched. A file that does not vouch is skipped when one of the same size is already there.
It refuses a folder that holds another version. A version kept on a machine, or data that is not files, cannot be downloaded: download fails and says where its data lives.
Before any byte moves, the files it fetches are compared with the free space of their disk. A download that doesn't fit fails, and nothing is downloaded. See Size limits and disk room.
Called while a run is open, it records that the run read this version.
hl.datasets.files(data.id) lists the files instead. A file in Hectiq Lab's bucket comes with an address valid for a day.
Find a dataset on this machine
folder = hl.datasets.resolve("interactions", "1.2").unwrap()resolve gives the folder of an intact copy on this machine, found by name, wherever it was downloaded or created. It never downloads.
Without a version, it picks the highest major.minor with an intact copy on this machine, compared as integers, so 1.10 is above 1.9. A version like v1.3 is only found by naming it. The version picked is logged at INFO. A higher version skipped because its copies are damaged is logged as a warning, with their paths.
Called while a run is open, it records that the run read this version.
When there is no intact copy, the error says so and gives the hlab dataset download command to run. For a version kept on a machine, or data that is not files, it says where its data lives instead.
The CLI prints the path alone, so $(hlab dataset resolve interactions) works in a shell.
A training script resolves its data, and the download is done once beforehand, on a cluster's login node for example:
hlab dataset download interactions --version 1.2with hl.run("Train two-tower"):
folder = hl.datasets.resolve("interactions", "1.2").unwrap()
...Check a copy
checks = hl.datasets.verify("interactions", "1.2").unwrap()verify gives each copy on this machine and its state: intact, or which files are missing or changed. It reads only local files.
The CLI exits with 1 when no copy is intact.
The marker and the cache
A folder that holds a version carries a .hectiqlab.json in its root. It says which version the folder holds (kind, id, project, name and version) and the size, modification time and MD5 of each file that vouches. It is never uploaded. Deleting it is safe: the next download into that folder fetches everything again.
create writes it into a folder it registers, so resolve finds that folder at once. A single file gets no marker.
resolve checks only the files that vouch. A folder too large to hash can vouch with a few of its files, and files added later never damage the copy:
hl.datasets.create("/mnt/images/", name="images", location="local", files=["images.parquet"])Downloads land in the cache, ~/.hectiqlab/cache unless HECTIQLAB_CACHE says otherwise. The index of every copy on this machine lives under it too, in .registry/.
hlab cache list
hlab cache clear interactions@1.2
hlab cache clear --allhlab cache list shows every version on this machine: kind, project, name, version, size, copies and when it was last resolved. It takes --project.
hlab cache clear NAME[@VERSION] deletes copies inside the cache. It takes --kind dataset|model and --project. --all clears every copy and asks first; --yes skips the question. It deletes copies inside the cache only: a folder elsewhere, one you created or downloaded with --into, is kept and stays registered. A copy whose folder is already gone is forgotten.
List datasets
datasets = hl.datasets.list(search="interactions").unwrap()Only the latest version of each dataset is listed. Pass keep_latest_version=False, or --all, for every version.
Change a dataset
hl.datasets.update(data.id, description="Clicks from May", metadata={"rows": 1200000})metadata is merged into what the dataset already has. In Python, update also takes name, version and source.
Tag a dataset
hl.datasets.tag(data.id, "clean", "may")
hl.datasets.untag(data.id, "may")Delete a dataset
hl.datasets.delete(data.id)This deletes the version and its files. The CLI asks to confirm; --yes skips the question.