The Many-To-One Data Flow template is designed to help showcase SoftNAS FlexFiles ability to help you with more complex scenarios by leveraging templates. In this sample template, volumes from multiple sources are configured to be "Lifted and Shifted" simultaneously to a separate target node. An example of where this might be useful is if you have app data in multiple cloud locations and you wish to consolidate it. Or perhaps you have on-premise data from two branch offices and you wish to move it to a single cloud repository. 

Preliminary Steps

To implement your Many-To-One template, be sure that you have performed the following steps:

1. You have configured the Virtual Machines (Instances) you wish to migrate and the target instance you wish to migrate to. For more information on creating instances, see Creating Your Instance.
   
   
2. Make sure your VMs/instances are up to date. See Updating Software.
   
   
3. You have recorded the instance IP addresses, names and passwords for your instances. Having this information at hand is key to a stress-free deployment.
   
   
4. Security groups have been configured for each source instance. A single security group can be configured for multiple source instances on the same platform, provided the security group covers all required port/IP permutations necessary for the use case (including UltraFast ports if accelerated data transfer is desired for the migration.) See Configuring Ports and Security Groups for SoftNAS Cloud Platinum Beta components.
   
   
5. A security group must be configured for the target node. The target VM/instance's security group must include rules for each source node. Below is a sample target node inbound security group on Azure. See Configuring Ports and Security Groups for SoftNAS Cloud Platinum Beta components.
   
   
   
   
6. Configure FlexFile Settings on each node. See Configuring FlexFile Settings.
   
   

   
   

   Widget Connector
   url https://www.youtube.com/watch?v=pbkGYjr8tnI&feature=youtu.be

   Widget Connector
   url https://www.youtube.com/watch?v=cJX_rgNts4o

   
   
   

   Widget Connector
   url https://www.youtube.com/watch?v=IF-uwKSxAS4&feature=youtu.be

   
   
   

   Widget Connector
   url https://www.youtube.com/watch?v=Y9JneaT-zMM&feature=youtu.be

Configure Source Nodes using Lift and Shift

To prepare your source nodes for the Many-To-One workflow, each node must be pre-configured with the required credentials and source instance information to allow the flow to proceed. This is accomplished by performing approximately half of the Lift and Shift process. Select ‘Lift and Shift’ from the Storage Administration panel of the SoftNAS StorageCenter UI.

...

Configure Target Node FlexFile DataFlow Template

Adding the Template

Next, we will need to configure the Many-To-One DataFlow template on the target node.

1. Select ‘FlexFiles Architect’ from the Storage Administration panel of the SoftNAS StorageCenter UI.
   
2. From the SoftNAS FlexFiles options listed at the top of the panel, select ‘Template’ icon, second from the right., and drag it into the grid area.  This will open the ‘Add Template’ panel.
   
   
   
   
3. In the Add Template panel, select the desired template. In this case, it is the SoftNAS Many-To-One Target Template. Select ‘Add’ to apply the template to FlexFiles Architect.
   
   
   
   
4.  The template will appear in the FlexFiles Architect grid. 
   
   
   
   
   

Configuring Remote Node 1 Input Port

1. Click any open area within the grid to deselect the components of the template downloaded.
2. Select the ‘Remote Node 1 Input Port’ by clicking on it.
3. The ‘Operate’ panel should now show ‘Remote Node 1 Input Port’ as being selected.  Select the ‘key’ icon to configure Access Policies.
   
   
   
   
4. In Access Policies, select the dropdown by default labelled ‘view the component’, and change the default entry to ‘receive data via site-to-site’.
   
   
   
   
5. Select ‘Create’ a new policy.
   
   
   
   
6.  After selecting 'Create', the icons to the far right will be made available. Select the ‘Add user’ icon located to the left of the trashcan icon.
   
   
   
7. In the Add Users/Groups panel, type the Source Node 1 Public IP address in the box labeled ‘User identity’ (i.e. 13.228.114.1). If the preliminary configurations (security groups, FlexFiles Settings, and Lift and Shift steps) have been correctly configured, typing in the public IP address will call up the available nodes, allowing you to select the desired entry. Click 'Add'.
   
   
   
   
8.  Once the Remote Node 1 IP has been added to the Access Policy, close the panel by selecting the ‘X’ at the top right of the panel.
   
   
   

Configuring Remote Node 1 Output Port

1. Select ‘Remote Node 1 Output Port’ by clicking on it.
   
   
2. The ‘Operate’ panel should now show ‘Remote Node 1 Output Port’ as being selected.  Select the ‘key’ icon to configure Access Policies.
   
   
   
   
3. In Access Policies, select the drop-down by default labelled ‘view the component’, and change the default entry to ‘receive data via site-to-site’.
   
   
   
   
4. Select ‘Create’ a new policy
   
   
   
   
5. After selecting 'Create', the icons to the far right will be made available. Select the ‘Add user’ icon located to the left of the trashcan icon.
   
   
   
   
6. In the Add Users/Groups panel, type the Source Node 1 Public IP address in the box labeled ‘User identity’ (i.e. 13.228.114.1). If the preliminary configurations (security groups, FlexFiles Settings, and Lift and Shift steps) have been correctly configured, typing in the public IP address will call up the available nodes, allowing you to select the desired entry. Click 'Add'.
   
   
   
   
7. Once the Remote Node 1 IP has been added, close the panel by selecting the ‘X’ at the top right of the panel.
   

Configuring Remote Node 2 Input Port

1. Select the ‘Remote Node 2 Input Port’ by clicking it.
   
   
2. The ‘Operate’ panel should now show ‘Remote Node 2 Input Port’ as being selected.  Select the ‘key’ icon to configure Access Policies.
   
   
   
   
3. In Access Policies, select the drop-down by default labelled ‘view the component’, and change the default entry to ‘receive data via site-to-site’.
   
   
   
   
4. Select ‘Create’ a new policy.
   
   
   
   
5. After selecting 'Create', the icons to the far right will be made available. Select the ‘Add user’ icon located to the left of the trashcan icon.
   
   
   
   
6. In the Add Users/Groups panel, type the Remote Node 2 Public IP address in the box labeled ‘User identity’ (i.e. 13.228.101.117). If the preliminary configurations (security groups, FlexFiles Settings, and Lift and Shift steps) have been correctly configured, typing in the public IP address will call up the available nodes, allowing you to select the desired entry. Click 'Add'.
   
   
   
   
   

Once the Remote Node 2 IP has been added, close the panel by selecting the ‘X’ at the top right of the panel.

Image Modified

Configuring Remote Node 2 Output Port

1. Select the ‘Remote Node 2 Output Port’ by clicking it.
   
   
2. The ‘Operate’ panel should now show ‘Remote Node 2 Output Port’ as being selected.  Select the ‘key’ icon to configure Access Policies.
   
   
   
   
3. In Access Policies, select the drop-down by default labelled ‘view the component’, and change the default entry to ‘receive data via site-to-site’.
   
   
   
   
4. Select ‘Create’ a new policy.
   
   
   
   
5. After selecting 'Create', the icons to the far right will be made available. Select the ‘Add user’ icon located to the left of the trashcan icon.
   
   
   
   
6. In the Add Users/Groups panel, type the Remote Node 2 Public IP address in the box labeled ‘User identity’ (i.e. 13.228.101.117). If the preliminary configurations (security groups, FlexFiles Settings, and Lift and Shift steps) have been correctly configured, typing in the public IP address will call up the available nodes, allowing you to select the desired entry. Click 'Add'.
   
   
   
   
   
7. Once the Remote Node 2 IP has been added, close the panel by selecting the ‘X’ at the top right of the panel.
   
   
   

Configure Remote Factory Aggregation

Remote Node 1 Factory

1. Open the ‘Remote FactoryAggregation’ by double clicking it. It will split into two distinct boxes. 
   
   
   
   
2. Double-click on the ‘Remote Node 1 Factory‘ to open it.
   
   
   
   

3. Right-click the ‘RouteOnAttribute’  box, and select 'Configure'.  This will open the panel to configure the processor.
   
   
   
   

4. Select ‘Properties’ to add Source Node 1’s volumes.
   
   
   
   
   1. Double-click on the ‘TargetVolume1’ and replace “/remotenode1/volume1” with the mount point of the volume on Remote Node 1. In our case, this is simply "pool1/Volume1".
   2. If there are multiple volumes on Remote Node 1, repeat the above process by double-clicking TargetVolume2 and entering the next volume's mount point. Click the plus sign for each additional volume, and repeat as necessary.
      
      
      
      Note: Each volume that is to be migrated must be added, by clicking the plus icon as shown above. If there is only one volume to be moved on the given source node, delete the second target volume configured by default in the template (TargetVolume2) from Properties in Configure Processor. To determine the mount point, log in to the Remote Node in question using its IP address and login information, and open Volumes and LUNs. Look under the Mountpoint column. 
      
      
      
      
      
      
5. Next, right-click ‘SoftNAS ProcessFileListTarget Volume 1’ , and select 'Configure'.
   
   
6. In Configure Processor, select 'Properties'. Fill in the following values based on your configuration:
   
   
   1. Assign the Remote Node 1 Volume 1 name to the value for ‘Source Directory’ (i.e. /pool1/Volume1).  Select ‘OK’ to apply the change.
      
      
      
      
   2. For 'DestinationDirectory', provide a volume name to be used on the Target Node to receive the Source Directory content (i.e. /S1/pool1-Vol1). Select ‘OK’ to apply the change.
      
      
   3. Assign the Target Node Public IP to the value for ‘Node Name’ (i.e. 52.187.73.229). Select ‘OK’ to apply the change.
      
      
   4. Assign the Remote Node 1 Volume 1 name to the value for ‘Flow ID’ (i.e. Pool1-Volume1). Select ‘OK’ to apply the change.
      
      
   5. Review the values and click ‘Apply’ to complete the ProcessFileList configuration.
      
      

...

As this sample configuration only has the one volume on Remote Node 1 to transfer to the destination volume on the target node, we can now return to the Remote Factory Aggregation level of our FlexFiles Flow. To do this, simply double-click on 'Remote Factory Aggregation' at the bottom of the grid. 

Remote Node 2 Factory

Having configured Remote Note 1 on the target, it is now time to configure the second (Remote Node 2) node.

1. Open the ‘Remote Factory Aggregation’ by double-clicking it, then double-click on the ‘Remote Node 2 Factory’ to open it.
   
   
   
   
2. Right-click the ‘RouteOnAttribute’  box, and select 'Configure'.  This will open the panel to configure the processor.
   
   

3. 

   Select ‘Properties’ to add Source Node 2’s volumes.
   
   

   
   
   

4. Double-click on the ‘TargetVolume1’ and replace “/remotenode2/volume1” with the mount point of the volume on Remote Node 2. In our case, this is simply "pool1/Volume1".
   
   

5. If there are multiple volumes on Remote Node 2, repeat the above process by double-clicking TargetVolume2 and entering the next volume's mount point. Click the plus sign for each additional volume, and repeat as necessary. You will need to copy an existing routing strategy and modify the volume attribute, as adding the volume here does not add any routing strategy. Adding a routing strategy is covered in more detail in Modifying the Many-To-One Template.
   
   
   
   Note: Each volume that is to be migrated must be added, by clicking the plus icon as shown above. If there is only one volume to be moved on the given source node, delete the second target volume configured by default in the template (TargetVolume2) from Properties in Configure Processor. To determine the mount point, log in to the Remote Node in question using its IP address and login information, and open Volumes and LUNs. Look under the Mountpoint column. 
   
   

   
   
   

6. Next, right-click ‘SoftNAS ProcessFileListTarget Volume 1’ , and select 'Configure'.
   Note: If you see a warning sign on a processor or processor group, this means that there are required settings or configuration that need to be applied.
   
   
7. In Configure Processor, select 'Properties'. Fill in the following values based on your configuration:
   
   
   1. Assign the Remote Node 2 Volume 1 name to the value for ‘Source Directory’ (i.e. /pool1/Volume1).  Select ‘OK’ to apply the change.
      
      
      
      
   2. For 'DestinationDirectory', provide a volume name to be used on the Target Node to receive the Source Directory content (i.e. /S2/pool1-Vol1). Select ‘OK’ to apply the change.
      
      
   3. Assign the Target Node Public IP to the value for ‘Node Name’ (i.e. 52.187.73.229). Select ‘OK’ to apply the change.
      
      
   4. Assign the Remote Node 2 volume name to the value for ‘Flow ID’ (i.e. S2Pool1-Vol1). Select ‘OK’ to apply the change.
      
      
   5. Review the values and click ‘Apply’ to complete the ProcessFileList configuration.
      
      
   
   
   
   
8. Again, the Many-To-One template defaults to a configuration of two volumes per Remote (Source) Node. If you have only one volume for this particular node, simply delete the second flow for the template default 'TargetVolume2' volume, as covered earlier in this document, as well as in Modifying the Many-To-One Template. The Modifying the Many-To-One Template article can also help you if you are adding additional volumes to the workflow.

Configure Source (Remote) Node 1 FlexFile DataFlow Template

Adding the Template

Once the Target Node is configured to receive volumes, it is time to configure the Source Nodes to send them. 

1. To begin configuration, navigate via your browser to Source (Remote) Node 1, and select ‘FlexFiles Architect’ from the Storage Administration panel of the SoftNAS StorageCenter UI.
   
   
   
   
2. From the SoftNAS FlexFiles options listed at the top of the panel, select ‘Template’, the icon second from the right, and drag it into the grid area.  
   
   
   
   

3. This will open the ‘Add Template’ panel. Select the 'SoftNAS Many-to-One Source Template' and click ‘Add’ to apply the template to FlexFiles Architect.

   

   
   

Configure Remote Factory Aggregation 

Now that the template has been added, we can begin configuration.

1. Simply double-click Remote Factory Aggregation to begin.
   
   
   
   
2.  Once in Remote Factory Aggregation's configuration, select the ‘Remote Processor’ labeled https://0.0.0.0:9443/nifi,  and delete it by right-clicking and selecting 'Delete'. This is just a placeholder in the template.
   
   
   
   
   
3.  Replace the deleted Remote Process Group by selecting the ‘Remote Process Group’ icon (fourth from the right) from the control panel at the top of the page. Click and drag it onto the grid where the deleted one was located.
   
   
   
   
   

4. Add the URL for the target node (i.e. https://52.187.73.229:9443/nifi) and select ‘ADD’ to apply the change.  

   
   
   
   
   
   
   
5. Once the connection is established with the target, you will see the lock next to the URL and the Send and Received will each show a ‘1’. 
   
   
   
   
6. Create a connection from the newly created ‘Remote Process Group’ to the ‘RouteOnAttribute’ by positioning your pointer in the middle of the process group (the dark circle-arrow icon),  right-click, hold, and drag your pointer to the ‘RouteOnAttribute’ until the line turns green.
   
   

7. The Create Connection panel will open. Open the Settings tab, and configure the 'Selected Prioritizers' to read ‘PriorityAttributePrioritizer’ first, followed by ‘FirstInFirstOutPrioritizer’. Click ‘ADD’ to apply these values.

   
   
   
   
8. Create a connection from the ‘Remote Factory – Volume 1’ to the newly created ‘Remote Process Group’by positioning your pointer in the middle of the remote factory (the dark circle-arrow icon), right-click, hold, and drag your pointer to the ‘RouteOnAttribute’ until the line turns green.
   
   
   
   

9. The Create Connection panel will open. Open the Settings tab, and configure the 'Selected Prioritizers' to read ‘PriorityAttributePrioritizer’ first, followed by ‘FirstInFirstOutPrioritizer’. Click ‘ADD’ to apply these values.
   
   
   
   

10.  If you have more than one volume on the Source Node, you would repeat the above process once more, creating a connection between the ‘Remote Factory – Volume 2’ to the newly created Remote Process Group. Always be sure that the connector settings are configured with PriorityAttributePrioritizer first, and FirstInFirstOutPrioritizer second.  By default, the template provides for two volumes. 

    If your node only has the one volume, delete the 2nd (Remote Factory - Volume 2). To do this, first delete the connectors. For more information on how to delete volumes from your template, or instructions on how to add additional volumes, see Modifying the Many-To-One Template.
    
    

11. Next, we will configure 'RouteOnAttribute'. Simply right-click 'RouteOnAttribute' andselect 'Configure'.
    
    
    
    

12. Select ‘Properties’ to add Source Node 1’s volumes.

    1. Double-click on the ‘TargetVolume1’ and replace “/remotenode1/volume1” with the mount point of the volume on Remote Node 1. In our case, this is simply "pool1/Volume1".
       
       

    2. If there are multiple volumes on Remote Node 1, repeat the above process by double-clicking TargetVolume2 and entering the next volume's mount point. Click the plus sign for each additional volume, and repeat as necessary. Again, you will need to copy an existing routing strategy and modify the volume attribute, as adding the volume here does not add any routing strategy. This process is covered in more detail in Modifying the Many-To-One Template.
       
       
       
       Note: Each volume that is to be migrated must be added, by clicking the plus icon as shown above. If there is only one volume to be moved on the given source node, delete the second target volume configured by default in the template (TargetVolume2) from Properties in Configure Processor. To determine the mount point, log in to the Remote Node in question using its IP address and login information, and open Volumes and LUNs. Look under the Mountpoint column. 
       
       

13.  Next, configure 'Remote Factory-Volume 1' by double-clicking it.  It will expand to a larger configuration grouping.
    
    
    
    

14. Right-click ‘SoftNAS GetFileList Source Volume 1’ and select 'Configure'.
    
    
    
    

15. In 'Configure Processor', under 'Properties', assign the source volume 1 mount point (i.e. /pool1/volume1) to the value for the ‘Root Directory’ and select ‘APPLY’. 
    
    

    
    

    
    

16. If more than one volume was configured (and by default the template provides for two), then you would now return to Remote Factory Aggregation by clicking it at the bottom of the screen, in order to open and configure the second volume (by default labelled 'Remote Factory - Volume 2'). 

    

17.  In our case, the second volume does not exist. However, for any additional volumes, the process would be the same - Double-click 'Remote Factory- Volume 2' (or 3 or 4), then right-click ‘SoftNAS GetFileList Source Volume 2’ (or Volume 3, Volume 4, and so on) and select 'Configure'. In 'Configure Processor', under 'Properties', assign the source volume 2 mount point (i.e. /pool2/volume2) to the value for the ‘Root Directory’ and select ‘APPLY’.
    
    
    
    

Configure Source (Remote) Node 2 FlexFile DataFlow Template