File Patterns

Overview

Commands that interact with files accept a pattern argument. This enables filtering when listing and downloading run files.

PATTERN Language Rules

  • The * character matches zero or more characters of a name component without crossing directory boundaries.
  • The ** characters matches zero or more characters crossing directory boundaries.
  • The ? character matches exactly one character of a name component.
  • The backslash character \ is used to escape characters that would otherwise be interpreted as special characters.
  • The [ ] characters are a bracket expression that match a single character of a name component out of a set of characters. For example, [abc] matches a, b, or c.
info

Most shells will require any PATTERN with *, ?, or [ ] to be wrapped in single quotes

Examples

Use the ** glob PATTERN to recursively list all files in the outputs directory

$ engine list-files adaptive-strut '**'
DIR avod_cars_example/
4.5K avod_cars_example/avod_cars_example.config
DIR avod_cars_example/checkpoints/
DIR avod_cars_example/logs/
DIR avod_cars_example/logs/train/
DIR avod_cars_example/logs/train/2019-04-02 21:59:36.211486/
6.9M avod_cars_example/logs/train/2019-04-02 21:59:36.211486/events.out.tfevents.1554242377.ip-192-168-103-46.ec2.internal
418.0B checkpoint
581.0M checkpoint-0-00000000.data-00000-of-00001
14.3K checkpoint-0-00000000.index
3.6M checkpoint-0-00000000.meta
581.0M checkpoint-1000-00001000.data-00000-of-00001
14.3K checkpoint-1000-00001000.index
3.6M checkpoint-1000-00001000.meta
581.0M checkpoint-2000-00002000.data-00000-of-00001
14.3K checkpoint-2000-00002000.index
3.6M checkpoint-2000-00002000.meta
581.0M checkpoint-3000-00003000.data-00000-of-00001
14.3K checkpoint-3000-00003000.index
3.6M checkpoint-3000-00003000.meta
581.0M checkpoint-4000-00004000.data-00000-of-00001
14.3K checkpoint-4000-00004000.index
3.6M checkpoint-4000-00004000.meta

Use the * glob PATTERN to list all files at the root directory

$ engine list-files adaptive-strut '*'
DIR avod_cars_example/
418.0B checkpoint
581.0M checkpoint-0-00000000.data-00000-of-00001
14.3K checkpoint-0-00000000.index
3.6M checkpoint-0-00000000.meta
581.0M checkpoint-1000-00001000.data-00000-of-00001
14.3K checkpoint-1000-00001000.index
3.6M checkpoint-1000-00001000.meta
581.0M checkpoint-2000-00002000.data-00000-of-00001
14.3K checkpoint-2000-00002000.index
3.6M checkpoint-2000-00002000.meta
581.0M checkpoint-3000-00003000.data-00000-of-00001
14.3K checkpoint-3000-00003000.index
3.6M checkpoint-3000-00003000.meta
581.0M checkpoint-4000-00004000.data-00000-of-00001
14.3K checkpoint-4000-00004000.index
3.6M checkpoint-4000-00004000.meta

More complicated patterns can be constructed using the PATTERN language rules

$ engine list-files adaptive-strut 'avod*/*'
4.5K avod_cars_example/avod_cars_example.config
DIR avod_cars_example/checkpoints/
DIR avod_cars_example/logs/
$ engine list-files adaptive-strut 'avod*/**'
4.5K avod_cars_example/avod_cars_example.config
DIR avod_cars_example/checkpoints/
DIR avod_cars_example/logs/
DIR avod_cars_example/logs/train/
DIR avod_cars_example/logs/train/2019-04-02 21:59:36.211486/
6.9M avod_cars_example/logs/train/2019-04-02 21:59:36.211486/events.out.tfevents.1554242377.ip-192-168-103-46.ec2.internal
$ engine list-files adaptive-strut 'checkpoint-?000*.meta'
3.6M checkpoint-1000-00001000.meta
3.6M checkpoint-2000-00002000.meta
3.6M checkpoint-3000-00003000.meta
3.6M checkpoint-4000-00004000.meta
$ engine get-files adaptive-strut 'checkpoint-1*'
✔ Found 3 files (Uncompressed size: 584.6M) matching "checkpoint-1*" for run adaptive-strut
Downloading compressed archive: 45%|████▍ | 262M/585M [09:19<12:43, 444kB/s]
$ engine get-files adaptive-strut '*'
✔ Found 16 files (Uncompressed size: 2.9G) matching "*" for run adaptive-strut
Downloading compressed archive: 18%|█▊ | 521M/2.85G [02:51<11:28, 3.66MB/s]