Scope

A combination of cloud resources are available when launching a Metworx workflow. These include a head (or master) node and compute nodes. Collectively, the nodes are known as a high performance computation (HPC) cluster, and they remain running until you delete the workflow. There are different combinations of CPU and RAM for your head and compute nodes and they should be determined by your scientific or statistical analysis needs. Suggestions on how to pick the right size for your workflows are given below.

Head and Compute Nodes

The head node is always active and running while your workflow is active. The software you interact with, such as RStudio or the desktop software, run on the head node. In contrast, the compute nodes run as needed to process jobs and simulations (unless you select maintain initial size, see below).

The hourly cost associated with the head and compute node sizes (CPU and RAM) are shown in the dropdown menu for each CPU/RAM combination. In the screenshot above, the head node has a 2 vCPU /8 GB RAM combination, which is expected to cost approximately $0.12 per hour. Each compute node has a 4 vCPU /16 GB RAM combination, expected to cost ~$0.12 per hour, per node.

The total cost of the compute nodes is dependent on (1) the size of the compute nodes, (2) the number of nodes in use (the price is per node), and (3) the duration of use.

HPC Cluster Configurations

There are three size configurations when creating a new workflow:

1. Initial Size: the initial number of compute nodes that are spun up when the HPC cluster is created.
2. Maintain Initial Size: ensures that the HPC cluster always maintains (at least) the number of compute nodes defined above for initial size. Importantly, this prevents your workflow from auto-scaling the number of compute nodes below the initial size you selected and the compute nodes are considered active regardless of whether they're being used. This can have severe cost implications and you should exercise extreme caution when using this option.
3. Maximum Size: the maximum number of compute nodes that can spin up on a HPC cluster.

Updating the HPC Size Configurations

The HPC size can be updated at any time after you've spun up an HPC cluster within a given workflow.

Updates to the compute nodes and maximum size configurations do not impact the running machine or any existing jobs running. However, updating the head node configuration requires your workflow to reboot, this will cause a brief outage to your workflow and terminate any processes currently running.

To update an active workflow, navigate to the Metworx dashboard and click Update.

The Update Workflow screen appear, make the necessary changes and click Update Workflow.

General Guidance on HPC Size

CPUs

For less computationally intense activities on the head node, two vCPUs should be sufficient, e.g., data assembly, post-processing results, or submitting jobs to the grid.

For heavy processing, four vCPUs should be sufficient.

If using pkgr to install packages, and you need to install a large number of packages (e.g., at the start of the project or to perform code QC), we suggest using a 4- or 8-core workflow because pkgr can use multiple cores and this speeds up installation time.

RAM

In general, 16 GB of RAM should be sufficient, unless you're running large simulations or working with many datasets that are hundreds of megabytes or gigabytes.

Monitoring your workflow

There are a couple of options to monitor your workflows and confirm they are configured appropriately.

Starting with the Metworx 21.08 blueprints, you can use Grafana dashboards to view key metrics related to the head node and the grid. You can learn more about Grafana dashboards here.

Alternatively, you can check the RStudio admin dashboard, available here. This dashboard shows historical CPU and RAM usage to identify if you need additional CPU or RAM.

If you're using Metworx version 21.08 or newer, we recommend using the Grafana dashboards to monitor your workflows. However, if you are using Metworx version 20.12 or older, the RStudio admin dashboard is still a very useful tool.

Video tutorial