FlexFiles Architect : Leveraging NiFi Site-to-Site

Owned by Former user (Deleted)
Last updated: May 04, 2020

Note: SoftNAS® FlexFiles Architect is a part of our legacy SoftNAS Platinum offering. It will not be visible in versions above 4.4.1 of our SoftNAS product, and is included here for legacy purposes.

Site To Site Benefits

When sending data from one instance of NiFi to another, there are many different protocols that can be used. The preferred protocol, though, is the NiFi Site-to-Site Protocol. Site-to-Site makes it easy to securely and efficiently transfer data to/from nodes in one NiFi instance or data producing application to nodes in another NiFi instance or other consuming application.

Using Site-to-Site provides the following benefits:

  • Easy to configure

    • After entering the URL of the remote NiFi instance, the available ports (endpoints) are automatically discovered and provided in a drop-down list

  • Secure

    • Site-to-Site optionally makes use of Certificates in order to encrypt data and provide authentication and authorization. Each port can be configured to allow only specific users, and only those users will be able to see that the port even exists. For information on configuring the Certificates, see the Security Configuration section of theSystem Administrator’s Guide.

  • Scalable

    • As nodes in the remote cluster change, those changes are automatically detected and data is scaled out across all nodes in the cluster.

  • Efficient

    • Site-to-Site allows batches of FlowFiles to be sent at once in order to avoid the overhead of establishing connections and making multiple round-trip requests between peers.

  • Reliable

    • Checksums are automatically produced by both the sender and receiver and compared after the data has been transmitted, in order to ensure that no corruption has occurred. If the checksums don’t match, the transaction will simply be canceled and tried again.

  • Automatically load balanced

    • As nodes come online or drop out of the remote cluster, or a node’s load becomes heavier or lighter, the amount of data that is directed to that node will automatically be adjusted.

  • FlowFiles maintain attributes

    • When a FlowFile is transferred over this protocol, all of the FlowFile’s attributes are automatically transferred with it. This can be very advantageous in many situations, as all of the context and enrichment that has been determined by one instance of NiFi travels with the data, making for easy routing of the data and allowing users to easily inspect the data.

  • Adaptable

    • As new technologies and ideas emerge, the protocol for handling Site-to-Site communications are able to change with them. When a connection is made to a remote NiFi instance, a handshake is performed in order to negotiate which protocol and which version of the protocol will be used. This allows new capabilities to be added while still maintaining backward compatibility with all older instances. Additionally, if a vulnerability or deficiency is ever discovered in a protocol, it allows a newer version of NiFi to forbid communication over the compromised versions of the protocol.

Site-to-Site is a protocol transferring data between two NiFi instances. Both end can be a standalone NiFi or a NiFi cluster. In this section, the NiFi instance initiates the communications is called Site-to-Site client NiFi instance and the other end as Site-to-Site server NiFi instance to clarify what configuration needed on each NiFi instances.

A NiFi instance can be both client and server for Site-to-Site protocol, however, it can only be a client or server within a specific Site-to-Site communication. For example, if there are three NiFi instances A, B and C. A pushes data to B, and B pulls data from C. A — push → B ← pull — C. Then B is not only a server in the communication between A and B, but also a client in B and C.

It is important to understand which NiFi instance will be the client or server in order to design your data flow, and configure each instance accordingly. Here is a summary of what components run on which side based on data flow direction:

  • Push: a client sends data to a Remote Process Group, the server receives it with an Input Port

  • Pull: a client receives data from a Remote Process Group, the server sends data through an Output Port

Site to Site Configuration Information

The following links provide details on Site To Site setup and configuration: