Workflow Fields Configuration Overview


Introduction

This page is designed to help illustrate each field's purpose in a given workflow and some tips on configuring them properly.

Master & Compute Nodes

The Master Node is where all of the software you'll be interacting with will be running. For example, this is where the RStudio / Desktop software is running from. The master node is always on and running while your workflow is Active.

When configuring your master node's vCPUs and RAM, in most cases 2 vCPUs should be computationally sufficient. For more detail around rightsizing workflows: https://kb.metworx.com/Users/Getting_Started/rightsizing-workflows/

Master Compute Nodes

The Compute Nodes are what do the heavy lifting of processing your jobs and simulations.

NOTE: When defining the configuration of compute nodes in your workflow, be aware that this configuration will be per compute node.

Therefore from the screenshot above, each compute node within your workflow will have 4vCPU / 16GB RAM.

Additionally, the cost you see (~$0.19/hr) will also be the cost per node.

The calculation of cost is essentially per core hour. Whether you have 1,000 cores for 1 hour or 10 cores for 100 hours, the biggest calculation is how long your model will take to run. Also, be aware that part of the cost calculation will be whatever the runtime is plus 20 minutes of idle time before the compute nodes shut down.

Cluster Size Configurations

There are 3 size configurations when creating a new workflow:

Initial Size

Maintain Initial Size

Maximum Size

Size Configurations

Initial size is the number of compute nodes that will be spun up when the cluster is initially created.

Maintain initial size is a selectable option to ensure that the cluster always maintains at least the definied initial size.

Maximum size is the max number of compute nodes you can spin up in a cluster.

Updating Size Configurations

You have the option to update your configuration settings after you've spun up a cluster within a given workflow.

To update an Active workflow, click the blue 'Update' link that has a wrench to the left of it as shown in the screenshot below.

Update Link

After clicking the Update link, you will be prompted with the Update Workflow screen.

Update Workflow Screen

NOTE - If you update the Master Node configuration, the machine will be rebooted, causing a brief outage and terminating running processes.

However, you can update the Compute Nodes and Maximum Size configurations without any impact to the running machine or your existing running jobs.

Desktop Konsole Tips

Submit a fake job to get a compute node to start up:

$ echo "Hello" | qsub

Konsole Dummy Job


Run 'qhost' to check number of queued jobs & node stat usage:

$ qhost

qhost konsole