Replace the --node-type flag with dedicated sub-commands on bacalhau serve

[frrist]

Background

At present there are different types of nodes, requester and compute. These types are best described as additive, where functionalities are combined, and commutative, where the order of combination does not change the outcome. Users select which type of node to run by passing a value to the --node-type flag. If the flag is not provided, its default value is requester.

• Requester
  • bacalhau serve --node-type=requester / bacalhau serve
• Compute
  • bacalhau serve --node-type=compute
• Hybrid
  • bacalhau serve --node-type=requester,compute / bacalhau serve --node-type=compute,requester

Problem

While Compute and Requester nodes share some configuration, they also have configuration unique to their respective types. This leads to the serve command have over 60 flags!! Some of the flags configure compute nodes, others requesters, while other apply to both. From the perspective of a user it's not easy to identify which flags go with which node types.

Flags:
      --advertised-address string                        Address to advertise to compute nodes to connect to.
      --allow-listed-local-paths strings                 Local paths that are allowed to be mounted into jobs
      --autocert string                                  Specifies a host name for which ACME is used to obtain a TLS Certificate.
                                                         Using this option results in the API serving over HTTPS
      --cluster-advertised-address string                Address to advertise to other orchestrators to connect to.
      --cluster-name string                              Name of the cluster to join.
      --cluster-peers strings                            Comma-separated list of other orchestrators to connect to to form a cluster.
      --cluster-port int                                 Port to listen for connections from other orchestrators to form a cluster.
      --compute-execution-store-path string              The path used for the compute execution store when using BoltDB
      --compute-execution-store-type storage-type        The type of store used by the compute node (BoltDB) (default BoltDB)
      --default-job-execution-timeout duration           default value for the execution timeout this compute node will assign to jobs with no timeout requirement defined. (default 10m0s)
      --default-publisher string                         A default publisher to apply to all jobs without a publisher
      --disable-engine strings                           Engine types to disable
      --disable-storage strings                          Storage types to disable
      --disabled-publisher strings                       Publisher types to disable
      --docker-manifest-cache-duration duration          The default time-to-live for each record in the manifest cache (default 1h0m0s)
      --docker-manifest-cache-frequency duration         The frequency that the checks for stale records is performed (default 1h0m0s)
      --docker-manifest-cache-size uint                  Specifies the number of items that can be held in the manifest cache (default 1000)
  -h, --help                                             help for serve
      --host string                                      The host to serve on. (default "0.0.0.0")
      --ignore-physical-resource-limits                  When set the compute node will ignore is physical resource limits
      --ipfs-connect string                              The ipfs host multiaddress to connect to, otherwise an in-process IPFS node will be created if not set.
      --job-execution-timeout-bypass-client-id strings   List of IDs of clients that are allowed to bypass the job execution timeout check
      --job-negotiation-timeout duration                 Timeout value to hold a bid for a job. (default 3m0s)
      --job-selection-accept-networked                   Accept jobs that require network access.
      --job-selection-data-locality local|anywhere       Only accept jobs that reference data we have locally ("local") or anywhere ("anywhere"). (default Anywhere)
      --job-selection-probe-exec string                  Use the result of a exec an external program to decide if we should take on the job.
      --job-selection-probe-http string                  Use the result of a HTTP POST to decide if we should take on the job.
      --job-selection-reject-stateless                   Reject jobs that don't specify any data.
      --labels stringToString                            Labels to be associated with the node that can be used for node selection and filtering. (e.g. --labels key1=value1,key2=value2) (default [])
      --limit-job-cpu string                             Job CPU core limit to run all jobs (e.g. 500m, 2, 8).
      --limit-job-gpu string                             Job GPU limit to run all jobs (e.g. 1, 2, or 8).
      --limit-job-memory string                          Job Memory limit to run all jobs  (e.g. 500Mb, 2Gb, 8Gb).
      --limit-total-cpu string                           Total CPU core limit to run all jobs (e.g. 500m, 2, 8).
      --limit-total-gpu string                           Total GPU limit to run all jobs (e.g. 1, 2, or 8).
      --limit-total-memory string                        Total Memory limit to run all jobs  (e.g. 500Mb, 2Gb, 8Gb).
      --local-publisher-address string                   The address for the local publisher's server to bind to (default "public")
      --local-publisher-directory string                 The directory where the local publisher will store content
      --local-publisher-port int                         The port for the local publisher's server to bind to (default: 6001) (default 6001)
      --max-job-execution-timeout duration               The maximum execution timeout this compute node supports. Jobs with higher timeout requirements will not be bid on. (default 2562047h47m16s)
      --min-job-execution-timeout duration               The minimum execution timeout this compute node supports. Jobs with lower timeout requirements will not be bid on. (default 500ms)
      --name string                                      The name of the node. If not set, the node name will be generated automatically based on the chosen name provider.
      --name-provider string                             The name provider to use to generate the node name, if no name is set. (default "puuid")
      --network-port int                                 Port to listen for connections from other nodes. Applies to orchestrator nodes. (default 4222)
      --network-store-dir string                         Directory that network can use for storage
      --node-info-store-ttl duration                     Sets the duration for which node information is retained in the node info store after which it
                                                         is automatically removed from the store. (default 10m0s)
      --node-type strings                                Whether the node is a compute, requester or both. (default [requester])
      --orchestrators strings                            Comma-separated list of orchestrators to connect to. Applies to compute nodes.
      --port int                                         The port to server on. (default 1234)
      --requester-job-store-path string                  The path used for the requester job store store when using BoltDB
      --requester-job-store-type storage-type            The type of job store used by the requester node (BoltDB) (default BoltDB)
      --requester-job-translation-enabled                Whether jobs should be translated at the requester node or not. Default: false
      --self-signed                                      Specifies whether to auto-generate a self-signed certificate for the requester node
      --tlscert string                                   Specifies a TLS certificate file to be used by the requester node
      --tlskey string                                    Specifies a TLS key file matching the certificate to be used by the requester node
      --web-ui                                           Whether to start the web UI alongside the bacalhau node.
      --web-ui-port int                                  The port number to listen on for web-ui connections. (default 8483)

Proposal

Introduce a bacalhau serve compute and bacalhau serve requester command with the following criteria

1. Flags that apply to both compute and requester nodes are persistent flags on the serve command. For example:
   • --labels
   • --port
   • --host
   • --name
   • --name-provider
   • --tls-*
   • etc.
2. Flags that only apply to compute nodes become flags on the serve compute command:
   • --allow-listed-local-paths
   • --default-publisher/engine/storages
   • --local-publisher-*
   • --orchestrators
   • --max-job-execution-timeout
   • --min-job-execution-timeout
   • etc.
3. Flags that only apply to requester nodes become flags on the serve requester command:
   • --advertised-address
   • --job-negotiation-timeout
   • --network-port
   • etc.
4. Deprecate (hide but don't remove) all but persistent flags on the serve command.
   • This will preserve backwards compatibility for our existing users, while guiding new users to the dedicated bacalhau serve compute and bacalhau serve requester commands.

The result should look similar to:

> bacalhau serve --help

Run a bacalhau server (see run subcommand)

Usage:
  bacalhau serve [command]

Available Commands:
  compute         Run a compute node
  requester       Run a requester node

Flags:
  -h, --help   help for docker

Global Flags:
      --host string         
      --port int
      --name string
      --name-provider string
      --tls-*
      --labels

> bacalhau serve compute
Runs compute node

Usage:
  bacalhau serve compute [flags]

Flags:
      --default-publisher string                         A default publisher to apply to all jobs without a publisher
      --disable-engine strings                           Engine types to disable
      --disable-storage strings                          Storage types to disable
      --disabled-publisher strings                       Publisher types to disable
      --docker-manifest-cache-duration duration          The default time-to-live for each record in the manifest cache (default 1h0m0s)
      --docker-manifest-cache-frequency duration         The frequency that the checks for stale records is performed (default 1h0m0s)
      --docker-manifest-cache-size uint                  Specifies the number of items that can be held in the manifest cache (default 1000)
      --ignore-physical-resource-limits                  When set the compute node will ignore is physical resource limits
      --ipfs-connect string                              The ipfs host multiaddress to connect to, otherwise an in-process IPFS node will be created if not set.
      --job-execution-timeout-bypass-client-id strings   List of IDs of clients that are allowed to bypass the job execution timeout check
      --job-selection-accept-networked                   Accept jobs that require network access.
      --job-selection-data-locality local|anywhere       Only accept jobs that reference data we have locally ("local") or anywhere ("anywhere"). (default Anywhere)
      --job-selection-reject-stateless                   Reject jobs that don't specify any data.
      --limit-job-cpu string                             Job CPU core limit to run all jobs (e.g. 500m, 2, 8).
      --limit-job-gpu string                             Job GPU limit to run all jobs (e.g. 1, 2, or 8).
      --limit-job-memory string                          Job Memory limit to run all jobs  (e.g. 500Mb, 2Gb, 8Gb).
      --limit-total-cpu string                           Total CPU core limit to run all jobs (e.g. 500m, 2, 8).
      --limit-total-gpu string                           Total GPU limit to run all jobs (e.g. 1, 2, or 8).
      --limit-total-memory string                        Total Memory limit to run all jobs  (e.g. 500Mb, 2Gb, 8Gb).
      --local-publisher-address string                   The address for the local publisher's server to bind to (default "public")
      --local-publisher-directory string                 The directory where the local publisher will store content
      --local-publisher-port int                         The port for the local publisher's server to bind to (default: 6001) (default 6001)
      --max-job-execution-timeout duration               The maximum execution timeout this compute node supports. Jobs with higher timeout requirements will not be bid on. (default 2562047h47m16s)
      --min-job-execution-timeout duration               The minimum execution timeout this compute node supports. Jobs with lower timeout requirements will not be bid on. (default 500ms)

> bacalhau serve requester
Runs requester node

Usage:
  bacalhau serve requester [flags]

Flags:
      --advertised-address string                        Address to advertise to compute nodes to connect to.
      --cluster-advertised-address string                Address to advertise to other orchestrators to connect to.
      --cluster-name string                              Name of the cluster to join.
      --cluster-peers strings                            Comma-separated list of other orchestrators to connect to to form a cluster.
      --cluster-port int                                 Port to listen for connections from other orchestrators to form a cluster.
      --job-negotiation-timeout duration                 Timeout value to hold a bid for a job. (default 3m0s)

[MichaelHoepler]

This is a good improvement for usability. How long would this take to implement? On another note: is there a nicer way to display this in the docs, to serve it even better to users? Optimizing the docs on CLI commands for usability could be an viable fix for now. What do you think @aronchick & @seanmtracey ?

[aronchick]

I like this a LOT. I think @MichaelHoepler's question is the key one, how long?

Questions:

• Are we preventing hybrid nodes? Do we think that's needed?
• How do we show "common" flags on the deeper nodes (since they will presumably want them).
• Do we want to hide (but not prevent) lots of the flags that we think are "optional/rare" in the list? We MOSTLY want people using config files, right?
• What about a command like "bacalhau serve config edit" that opens the config file with $EDITOR

Other than that, this is cool!

[wdbaruni]

I share the same questions as with Dave. Besides, most of the problems mentioned about having too many and overlapping flags are mitigated by the proposal to improve our configurations and move away fully from CLI flags and mainly use config files as documented in Rethinking Configurations, which already been reviewed and working on this quarter.

To add to Dave's questions:

• The proposal of having separate commands will prevent running hybrid nodes, which I have to say is a great feature we have. It simplifies trying out bacalhau and should be the first and main method for users to try running bacalhau locally. They shouldn't use devstack as it is more complicated and stateless by default, and we shouldn't ask them to run on two terminals with different configs or repos. Even in production it is not the worse idea to run your requester as a hybrid node if you only have a small network. In fact our demo network runs a hybrid node. I also use it on a daily basis in addition to devstack
• As we are moving more towards config files, how is this going to work? Our configurations are already split between parts that are common between both node types, with sections specific to each type.

Overall, I don't mind discussing a new proposal if there issues with the existing one, but you should mention the prior work, why it would't work, and why we need to explore other options