Persisting ML / AI Models¶

Overview¶

After training a statistical / machine learning model, it is desirable to have a way to persist the model for future inference.

The recommended approach is to use a general custom adapter (e.g. the builtin blob storage adapter) to serialize / deserialize the model object as follows:

1. Run a training workflow which outputs the trained model to an ANY type output which is wired to a general custom adapter sink.
2. Employ a fitting inference workflow loading the trained model into an ANY input from the same general custom adapter.

General custom adapters are preferable here over generic rest adapters since read/write operations run directly from within the runtime Python code. This avoids having to transport potentially large amounts of binary data via http in json format.

hetida designer is equipped with several built-in such general custom adapters suitable for the task of persisiting arbitrary (binary) Python objects. The following use Python's built-in pickle module for serialization / deserialization:

• the local file adapter
• the blob storage adapter (for S3 compatible storage)

The component adapter allows to write components using arbitrary Python code that act as sources / sinks. This allows for easy custom / ad-hoc integration of additional storage solutions or special formats.

Apart from these, you can write your completely own general custom adapter tailored to your specific needs and persistence backends.

General warnings on deserialization :

• Deserializing executes code and therefore is a security risk - you should only load objects from trusted sources!
• If code or Python environments change, deserialization may fail at some point. See notes on deserialization pitfalls for details. You may prefer code-invariant serialization techniques like ONNX in some cases.

Model Management¶

hetida designer is not a model management/ experiment management framework. It simply treats models as data that is put out from workflows and read into workflow inputs.

You can of course establish the analytical steps of model management like for example automatic performance measuring / evaluation as part of hetida designer workflows.

But the management part, i.e. selecting the current/prod model based on evaluation results, fallback procedures on declining performance and so on has to be implemented around the systems which invoke hetida designer from the outside.

Custom Classes¶

You may import components in other components, which in particular allows to import classes from other components. This enables proper deserialization of trained models using custom classes: Just import the component containing the class definition in an inference component.

Persisting and Loading via General Custom Adapter¶

Built in general custom adapters¶

Mounted Files Adapter¶

The Mounted Files Adapter supports pickling objects from ANY typed outputs to a mounted volume/disk and loading them into ANY type inputs.

Blob Storage Adapter¶

The Blob Storage Adapter supports pickling objects from ANY typed outputs to a mounted Blob storage and loading them into ANY type inputs via the S3 standard.

Component Adapter¶

The Component Adapter is the most customizable solution apart from writing your own adapter. It allows to write custom Python code for accessing 3^rd party persistence solutions / storage. Typically you would use ANY as type for models.

Writing your own general custom adapter¶

If the builtin adapters are not suitable for your scenario, writing your own General Custom Adapter is the recommended way for persisting and loading models. It has the following advantages specific to model persistence:

• Keeps your workflow modular and reproducible in accordance with hetida designers separation of analytics and data ingestion/egestion (Adapter system, wirings etc.). We strongly recommend to follow this approach for production scenarios over for example storing/loading from inside workflow operators as described below!
• Faster and more efficient when handling large models as binary blobs compared to generic rest adapters.
• You may choose a format of your choice, such as ONNX which allows integration into other solutions, even non-Python-based ones.

To get started, besides the general custom adaper documentation you can checkout and read the mentioned builtin adapters code and documentation.

You typically will use the ANY type for arbitrary models / Python objects.

In case additional Python libraries such as onnx, tensorflow-onnx, onnx-tensorflow, or sklearn-onnx are needed, just follow the instructions on how to add Custom Python Dependencies to your runtime Docker container.

Persisting and Loading from within Workflows via Components¶

A simple option for development purposes is to use basic components for serializing objects to disk.

In order to persist objects from within workflows, the Store Object component is provided in the category Data Sinks. In addition to the object to be stored, this component expects a name string and a tag string as inputs, which in combination should uniquely identify the respective object. The name and the tag are used to determine the location of the serialized object file. The tag must not be "latest" because that is used internally and has a specific meaning.

In order to load a persisted object, the Load Object component is provided in the category Data Sources. This component takes a name string and a tag string as input and returns the object stored under that name and tag.

The directory where the serialized object files are stored within the container is defined by the MODEL_REPO_PATH environment variable, which by default is /mnt/obj_repo. To make this persistent a volume needs to be mounted there, like in the default docker-compose setup.