This package provides utility methods for Keras callbacks.
To synchronize the model replica weights before training, you will need to create a callback with
import engineml.keras as emlcallbacks = eml.callbacks.init_op_callback()# Train modelmodel.fit(..., callbacks=callbacks)
Training with spot or preemptible instances is significantly cheaper, but there is a small risk that your run could be
preempted. With Keras, use
eml.callbacks.preempted_callback('path/to/checkpoint') to automatically save a checkpoint
before your run shuts down if preemption occurs. If you are using the
prefer option for
then you can use
preempted_callback to save your progress and resume from where you left off when your run is restarted.
import osimport engineml.keras as emlcallbacks = eml.callbacks.preempted_callback(os.path.join(eml.data.output_dir(), 'preempted.hdf5'))# Train modelmodel.fit(..., callbacks=callbacks)