MultiTsFrame¶

hetida designer is often used for timeseries data analytics. Therefore it comes with an explicit data type for multivariate timeseries data: A MultiTSFrame represents multiple timeseries (of same dimension) data with non-necessarily common timestamps.

Long Format representation¶

Data is stored in records, one per metric per timestamp where this metric has value(s), i.e. in "long format".

Comparison to "wide" format¶

This is in contrast to "wide format" where each metric is its own column. A main advantage of the long-format is that it is more storage-efficient in case that multiple timeseries do not have the same timestamps.

MultiTsFrame details¶

A MultiTSFrame must have at least three columns:

• a "timestamp" column (datetime, no missing entries allowed)
• a "metric" column (string, no missing entries allowed)
• at least one value column: Per convention the third column is often named "value". Note that many base components operating on MultiTsFrames assume only the three columns "timestamp", "metric" and "value".

A MultiTSFrame can contain multi-dimensional timeseries data simply by having more than one value column. This implies that dimensions should agree for all metrics.

As an example consider a measurement by drones where location of measurement is relevant:

Or stock share trade events where price and number of trades shares is necessary:

Internal: Workflow & Components¶

Within workflows and components the MultiTSFrame object is a pandas.DataFrame following certain conventions:

• "metric" column with string and no missing data,
• "timestamp" column with timestamp information,
• additional value columns (at least one), often exactly one named "value".

Note that Pandas will handle a column build from values of differing types as dtype object and this may negatively impact efficiency / performance.

In contrast to pandas.Series the index of a MultiTSFrame should be considered irrelevant since timestamp information is in the "timestamp" column. When manipulating MultiTSFrame Pandas DataFrames you should ensure that the resulting index is duplicate-free. Ideally a generic integer index.

In the documentation of the workflow and components the convention is to write MultiTSFrame, e.g.:

- mutiple_timeseries (MultiTSFrame): This is an example for the documentation of an input/output with the type MultiTSFrame

External: Adapter System¶

Manual Input / Direct Provisioning¶

A simple json representation of a MultiTSFrame for the direct provisioning adapter is the following format:

{
    "value": [
        1,
        1.2,
        0.5
    ],
    "metric": [
        "a",
        "b",
        "c"
    ],
    "timestamp": [
        "2019-08-01T15:45:36.000Z",
        "2019-08-01T15:48:36.000Z",
        "2019-08-01T15:42:36.000Z"
    ]
}

print(df.to_json(orient="columns", date_format="iso", indent=2))

It is possible to define metadata for the MultiTSFrame. Conventions for the metadata keys can be found here.

For such cases, we recommend using the wrapped format, e.g.:

{
    "__hd_wrapped_data_object__": "DATAFRAME",
    "__metadata__": {
        "ref_data_frequency": {
            "a": "1h",
            "b": "1h"
        }
    },
    "__data__": {
        "value": {
            "0": 1,
            "1": 1.2,
            "2": 0.5,
            "3": 0.4,
        },
        "metric": {
            "0": "a",
            "1": "a",
            "2": "b",
            "3": "b"
        },
        "timestamp": {
            "0": "2019-08-01T01:00:00.000Z",
            "1": "2019-08-01T02:00:00.000Z",
            "2": "2019-08-01T01:00:00.000Z",
            "3": "2019-08-01T02:00:00.000Z"
        }
    }
}

Generic Rest Adapter¶

Sending a MultiTSFrames from a hetida designer workflow/component output to a generic Rest adapter sink of type MULTITSFRAME requires that the output pandas.DataFrame object passes several validations:

• column "timestamp" has no missing entries and a dtype of pandas.DatetimeTZDtype with timezone UTC
• column "metric" has no missing entries
• at least one additional column is defined.