Launch your Run
Runs are launched when you run
engine run. Before launching any runs, ensure you have a
Engine ML uses
git to track source code, so please ensure that you are
engine run from a
git repository. If you have any unstaged changes,
engine run will automatically commit
those for you.
engine run will only track files that are staged and unstaged. Files that are not tracked by
git will not be
tracked by Engine ML.
View your Run's Progress
You can see your run's progress including logs, system metrics, and output files from the Engine ML Dashboard. Use
engine dashboard or visit https://app.engineml.com to view your recent runs. Data generated from runs is synced periodically.
Stop your Run
If your python code completes, exits with an error, or is interrupted with a
KeyboardInterrupt, Engine ML will
automatically mark the run as finished on the dashboard.
If you use a
KeyboardInterrupt to stop a run, or if you lose internet connection during a run, you should run
engine sync-cache to ensure that your files and logs are in sync with Engine ML.
Download Checkpoints and Events
When your experiment is complete, you can download your saved model and event files with
engine get-files. See the documentation if you want to download individual files.
$ engine get-files lateral-stick '*'Downloading outputs for run lateral-stick...Downloading outputs matching "*" for run lateral-stickSaved to file "lateral-stick-1544679573.tar.gz"
The system metrics dashboard is useful for finding bottlenecks in your code. Engine ML makes it easy to correlate
system metrics with code using
Metrics annotations are not visible by default. Make sure to check the box at the top of the page to turn them on.
Slow operations like image preprocessing
eml.annotate(title='Data', comment='Start Preprocessing', tags=('data'))data = preprocess_data(data)eml.annotate(title='Data', comment='Finished Preprocessing', tags=('data'))
The beginning of each epoch
for epoch in range(max_epochs):for data, label in data_iterator():train(data, label)eml.annotate(title='Epoch', comment=epoch)
The start of training and evaluation
eml.annotate(title='Starting training')for data, label in train_data.iterator():train(data, label)eml.annotate(title='Starting evaluation')for data, label in eval_data.iterator():evaluate(data, label)
Keeping track of Run IDs can be tricky. You can organize your runs with tags.
$ engine tag lateral-stick resnet50
To delete tags, use the
-d flag to
$ engine tag -d lateral-stick resnet50