eml.callbacks

This package provides utility methods for Keras callbacks.

init_op_callback

To synchronize the model replica weights before training, you will need to create a callback with eml.callbacks.init_op_callback().

import engineml.keras as eml
callbacks = eml.callbacks.init_op_callback()
# Train model
model.fit(..., callbacks=callbacks)

preempted_callback

Training with spot or preemptible instances is significantly cheaper, but there is a small risk that your run could be preempted. With Keras, use eml.callbacks.preempted_callback('path/to/checkpoint') to automatically save a checkpoint before your run shuts down if preemption occurs. If you are using the prefer option for preemptible, then you can use preempted_callback to save your progress and resume from where you left off when your run is restarted.

import os
import engineml.keras as eml
callbacks = eml.callbacks.preempted_callback(os.path.join(eml.data.output_dir(), 'preempted.hdf5'))
# Train model
model.fit(..., callbacks=callbacks)