Evaluator¶

bigdl.chronos.metric.forecast_metrics.symmetric_mean_absolute_percentage_error(preds: torch.Tensor, target: torch.Tensor) → torch.Tensor[source]¶

class bigdl.chronos.metric.forecast_metrics.Evaluator[source]¶

Bases: object

Evaluate metrics for y_true and y_pred.

static evaluate(metrics, y_true, y_pred, aggregate='mean')[source]¶

Evaluate a specific metrics for y_true and y_pred.

Parameters

metrics – String or list in [‘mae’, ‘mse’, ‘rmse’, ‘r2’, ‘mape’, ‘smape’] for built-in metrics. If callable function, it signature should be func(y_true, y_pred), where y_true and y_pred are numpy ndarray.
y_true – Array-like of shape = (n_samples, *). Ground truth (correct) target values.
y_pred – Array-like of shape = (n_samples, *). Estimated target values.
aggregate – aggregation method. Currently, “mean” and None are supported, ‘mean’ represents aggregating by mean, while None will return the element-wise result. The value defaults to ‘mean’.

Returns

Float or ndarray of floats. A floating point value, or an array of floating point values, one for each individual target.

get_latency(*args, num_running=100, **kwargs)[source]¶

Return the time cost in milliseconds of a specific function by running multiple times.

Parameters

func – The function to be tested for latency.
args – arguments for the tested function.
num_running – Int and the value is positive. Specify the running number of the function and the value defaults to 100.
kwargs – other arguments for the tested function.

Returns

Dictionary of str:float. Show the information of the time cost in milliseconds.

Example

>>> # to get the inferencing performance of a trained TCNForecaster
>>> x = next(iter(test_loader))[0]
>>> # run forecaster.predict(x.numpy()) for len(tsdata_test.df) times
>>> # to evaluate the time cost
>>> latency = Evaluator.get_latency(forecaster.predict, x.numpy(),                          num_running = len(tsdata_test.df))
>>> # an example output:
>>> # {"p50": 3.853, "p90": 3.881, "p95": 3.933, "p99": 4.107}