Attaching a Shared Volume to Metworx

Scope

The purpose of this document is to provide direction on how to set up an AWS environment in a way that enables Metworx workflows to mount a shared volume at workflow creation. Metworx workflows are set to look for specific secrets in AWS Secrets Manager, if those secrets are present it will connect and mount the shared volumes.

The Metworx workflow will look for these secrets using specific naming conventions. Due to this, any names of secrets must be exactly as they appear in this document.

This document outlines the steps to create and mount an AWS EFS Volume to Metworx, however other types of shared volumes can be used by populating the appropriate values into the secret.

Requirements

To attach a shared volume to Metworx, someone with Administrator Access in AWS must configure the following:
  • Create and modify AWS EFS Volumes
  • Allow Metworx workflow subnets network access to EFS Volume
  • Populate the Volume Secrets

Enable The Metworx Service IAM Role Access to the Secrets

The version of IAM objects provisioned after 22.06 should have the IAM permissions needed to read the Secret Manager configuration Secret

Create and Modify AWS EFS Volumes

In the AWS Account where Metworx workflows run, in the AWS Console navigate to EFS.

If you already have an EFS volume created that you would like to mount onto all Metworx workflows, click on the volume and note the dns name of the EFS volume and the names of the security groups that are being used for each availability zone.

To create a new EFS volume to be shared:

  • Click the "Create file system" button
  • Name the EFS volume
  • Select the VPC that contains your Metworx workflows
  • Configure any additional options you require such as availablility settings, backups, etc and create the volume.

    • Once the EFS Volume is created, click on the volume and select the network tag.

    Take note of the volume's DNS Name and the names of the security groups that are being used for each availability zone. These will need to be modified to allow traffic from Metworx.

    Allow Metworx Workflow Subnets Network Access to EFS Volume

    With the security groups' names noted (as mentioned above), navigate in the AWS Console the VPC.

    Click on Subnets and note the IPv4 CIDR of the Subnets in which workflows are created.

    Next on the left panel, select "Security Groups" for each security group that was noted previously for the EFS Volumes do the following steps:

    1. Select the security group that matches the group from the EFS Volume
    2. Click the "Edit Inbound Rules" button then click "Add Rule"
    3. In the "Type" field, select "NFS"
    4. In the "Source" field, select "Custom" and input the Subnet CIDR for the Metworx workflow subnet. Repeat this step, adding additional rules for each subnet if there is more than one.
    5. Click "Save Rules"

    Repeat these steps for each security group that was created for the EFS Volume.

    Populate the Volume Secrets

    To configure shared volumes to be mounted on a workflow, in the same AWS account and region as workflows, create AWS secret with an array of filesystem configuration values.

    The required parameters for each filesystem are:

    • device
    • mountpoint
    • fstype
    • options

    The following example will mount a single EFS filesystem on /data2 on all of the workflow cluster nodes.

    NOTE: You can specify multiple configurations in a single aws secret. 

    [
    {
        "device": "fs-0420cdf0c8a1a4e4b.efs.us-east-1.amazonaws.com:/",
        "mountpoint": "/data2",
        "fstype": "nfs4",
        "options": "nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport"
    }
]

    The name of the secret should be exactly as follows: metworx-config-efs-mount

    Setting Share Permissions

    Share permissions are set by the remote share for NFS volumes.

    If a new EFS volume has been created, it will need to be mounted and have the permissions set manually.

    Consider which users/groups will need access, mount the share on a workflow, and manually set the permissions prior to use in production environments.

    Final Step - Confirmation

    After the steps above have been completed, the mount should be mounted automatically at workflow creation with no further user interaction.

    To test, create a Metworx workflow and confirm that the shared directory is present at the configured mount point and is accessible with the correct users and permissions.