Metrics

Metrics is a key feature to let you track the quality of your machine learning experiments. Metrics are tuples (key, step, value) and are always associated with a run.

Add metrics

Metrics must be created from within a run. First, create a run.

from pyhectiqlab import Run
run = Run(name="Push metrics", project="lab/demo")

Then, add metrics with run.add_metrics.

import numpy as np

for t in np.linspace(0, 10, 1000):
    run.add_metrics("sin(x)", value=np.sin(t), step=t)

Maximum rates and aggregation

Metrics push rate limit is fixed to 1000 metrics / 5 seconds. If you exceed this limit, an internal buffer will automatically and randomly downsample your metrics. The other option to handle high volume is to aggregate the metrics with one of the internal methods (mean, max, sum). Use Run.set_metrics_aggr to set an aggregation method.

run.set_metrics_aggr("mean")

The aggregation method will be applied on each batch before beeing sent to the lab. For example, if you push 2000 metrics under 5 seconds with run.set_metrics_aggr("mean"), only the average step and value will be sent. This feature is especially useful for experiments that produce high volume of metrics during a long period of time.

You can group metrics in a same graph using the delimeter :: such as

run.add_metrics("cos(wx)::w=0.2", step=x, val=np.cos(0.2*x))
run.add_metrics("cos(wx)::w=0.5", step=x, val=np.cos(0.5*x))

You can use run.set_metrics_group to automatically append a group to all metrics.

self.set_metrics_group("w=2")
self.add_metrics("sin(wx)", step=1, value=np.sin(2)) # Metrics will be "sin(wx)::w=2"

Tensorflow users

Tensorflow users can use the custom callback to automatically push the metrics.

Methods