Command-line interface

Strymon provides a command-line interface to build and run Timely dataflow programs on a running Strymon cluster.

The `strymon` utility

The strymon command-line utility is part of the Strymon run-time. It can be invoked either by using the binary generated by building the strymon_runtime crate, or by using the wrapper script provided in ./bin/strymon.

Strymon 0.2
Systems Group, ETH Zürich

USAGE:
    strymon [OPTIONS] <SUBCOMMAND>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -l, --log-level <RUST_LOG>    Set level and filters for logging

SUBCOMMANDS:
    help      Prints this message or the help of the given subcommand(s)
    status    Prints status information about a running Strymon instance
    submit    Submit a new Strymon application

All subcommands of the Strymon command-line utility can print additional debugging output by setting the log level to info or debug. The syntax for the value passed to the --log-level flag follows the same rules as the RUST_LOG environment variables of env_logger documented here.

`strymon status`

strymon status
Prints status information about a running Strymon instance

USAGE:
    strymon status [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -c, --coordinator <ADDR>    Address of the coordinator

The status subcommand connects to the specified coordinator (default: localhost:9189) and prints the status information extracted from the catalog.

`strymon submit`

strymon-submit
Submit a new Strymon application

USAGE:
    strymon submit [OPTIONS] <path|--binary-path <PATH>> [-- <args>...]

OPTIONS:
        --features <FEATURES>...
            Comma-separated list of features to activate

        --all-features                     Build all available features
        --no-default-features
            Do not build the `default` feature

        --bin <NAME>
            Build and submit only the specified binary

        --example <NAME>
            Build and submit only the specified example

        --debug
            Build in debug mode instead of release mode

    -e, --external-hostname <HOST>
            Externally reachable hostname of the spawned coordinator

    -c, --coordinator <ADDR>               Address of the coordinator
        --description <DESC>
            Human-readable description of the submitted job

        --workers <NUM>                    Number of workers per machine
        --placement-strategy <STRATEGY>
            Job placement strategy [values: pinned, random]

        --pinned-id <ID>...
            Comma-separated list of executor ids for the `pinned`
            placement strategy
        --pinned-host <HOST>...
            Comma-separated list of executor host names for the `pinned`
            placement strategy
        --num-executors <NUM>
            Number of executors for the `random` placement strategy

        --no-upload
            Let the executors read the binary from their local filesystem

        --binary-path <PATH>
            Submit a prebuilt binary (skips invoking `cargo build`)

    -h, --help                             Prints help information
    -V, --version                          Prints version information

ARGS:
    <path>       Path to the Cargo project directory
    <args>...    Arguments passed to the the spawned binary

Strymon submit is a rather large subcommand which performs following tasks:

Builds a Strymon job as a Rust Cargo project in <path> and extracts the binary crate artifact from the build.
Uploads and runs the binary on the Strymon cluster, using the placement configuration specified in the options.

Building the project

This subcommand invokes cargo build on the specified project path. For this reason, it mimics a subset of the arguments accepted by cargo build. Notably feature selection for conditional compilation with the --features, --all-features, --no-default-features options.

By default, jobs are built with optimizations. To submit debug builds, use the --debug flag.

The built project has only a one binary target, it will be chosen for submission. If there are multiple binary targets, please specify one using either the --bin or --example option.

Please refer to the Cargo documentation for a more detailed description of the option mentioned in this section.

Note: If you would like to skip the build step, specify the pre-built binary using the --binary-path option.

Job placement configuration

The built binary is submitted Strymon coordinator specified in the --coordinator otpion (default: localhost:9189).

By default, the submitted binary is uploaded using a randomly selected TCP port. For this reason, the external hostname of the local machine must be known. Either using the --external-hostname option, or by setting the TIMELY_SYSTEM_HOSTNAME environment variable. This functionality can be disabled by using the --no-upload option.

The --placement-strategy argument is used to specify to which machines the submitted binary is spawned on. By default, jobs will be placed on a single randomly selected executor. This is equivalent to --placement-strategy random --num-executors 1. To pin a job to a certain set of executors, use --placement-strategy pinned together with either --pinned-id 0,1,2 to select executors based on their executor id, or use --pinned-host host1,host2,host3 to specify them by hostname.

The number of worker threads per executors (default: 1) can set using the --workers option. The optional job name is given through the --description option.

`strymon terminate`

strymon-terminate
Send the termination signal to a running Strymon job

USAGE:
    strymon terminate [OPTIONS] <JOB_ID>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -c, --coordinator <ADDR>    Address of the coordinator

ARGS:
    <JOB_ID>    Numeric job identifier

Terminates all processes belonging to a running Strymon job. The job to terminate is identified by its numeric identifier which is assigned during strymon submit and can also be obtained through strymon status.

Cluster management scripts

We additionally provide shell scripts for starting and stopping the components of a Strymon cluster.

Usage: ./bin/start-strymon.sh [-c <HOSTNAME:PORT>] [-e <HOSTFILE>]
                              [-l <LOGDIR>]

    -c <HOSTNAME:PORT>   Address of the coordinator to be spawned
    -e <HOSTFILE>        Executor hosts file
    -l <LOGDIR>          Working directory for log and pid files

`./bin/start-strymon.sh`

Starts a coordinator instance on the host and port specified by the -c flag (default: localhost:9189) and then starts a series of executors, which can be specified in a host file (default: conf/executors).

Note that this script assumes that LOGDIR is shared among all machines participating in the cluster. You can inspect the output of the executors (and consequentially the jobs spawned by it) by reading the log files in logs/executor_$hostname.log.

`./bin/stop-strymon.sh`

Stops the instances spawned by start-strymon.sh. Make sure to pass the same coordinator and executor instances used for starting the cluster.

Note: Support for launching the coordinator or executors on remote machines with these scripts is currently still experimental. The script does currently not support starting multiple executors on the same machine.

The strymon utility

strymon status

strymon submit