DataFlow Manager: A DataFlow Manager (DFM) is a FlexFiles user who has permissions to add, remove, and modify components of a FlexFiles dataflow.
FlowFile: The FlowFile represents a single piece of data in FlexFiles. A FlowFile is made up of two components: FlowFile Attributes and FlowFile Content. Content is the data that is represented by the FlowFile. Attributes are characteristics that provide information or context about the data; they are made up of key-value pairs. All FlowFiles have the following Standard Attributes:
uuid: A unique identifier for the FlowFile
filename: A human-readable filename that may be used when storing the data to disk or in an external service
path: A hierarchically structured value that can be used when storing data to disk or an external service so that the data is not stored in a single directory
Processor: The Processor is the FlexFiles component that is used to listen for incoming data; pull data from external sources; publish data to external sources; and route, transform, or extract information from FlowFiles.
Relationship: Each Processor has zero or more Relationships defined for it. These Relationships are named to indicate the result of processing a FlowFile. After a Processor has finished processing a FlowFile, it will route (or “transfer”) the FlowFile to one of the Relationships. A DFM is then able to connect each of these Relationships to other components in order to specify where the FlowFile should go next under each potential processing result.
Connection: A DFM creates an automated dataflow by dragging components from the Components part of the FlexFiles toolbar to the canvas and then connecting the components together via Connections. Each connection consists of one or more Relationships. For each Connection that is drawn, a DFM can determine which Relationships should be used for the Connection. This allows data to be routed in different ways based on its processing outcome. Each connection houses a FlowFile Queue. When a FlowFile is transferred to a particular Relationship, it is added to the queue belonging to the associated Connection.
Controller Service: Controller Services are extension points that, after being added and configured by a DFM in the User Interface, will start up when FlexFiles starts up and provide information for use by other components (such as processors or other controller services). A common Controller Service used by several components is the StandardSSLContextService. It provides the ability to configure keystore and/or truststore properties once and reuse that configuration throughout the application. The idea is that, rather than configure this information in every processor that might need it, the controller service provides it for any processor to use as needed.
Reporting Task: Reporting Tasks run in the background to provide statistical reports about what is happening in the FlexFiles instance. The DFM adds and configures Reporting Tasks in the User Interface as desired. Common reporting tasks include the ControllerStatusReportingTask, MonitorDiskUsage reporting task, MonitorMemory reporting task, and the StandardGangliaReporter.
Funnel: A funnel is a FlexFiles component that is used to combine the data from several Connections into a single Connection.
Process Group: When a dataflow becomes complex, it often is beneficial to reason about the dataflow at a higher, more abstract level. FlexFiles allows multiple components, such as Processors, to be grouped together into a Process Group. The FlexFiles User Interface then makes it easy for a DFM to connect together multiple Process Groups into a logical dataflow, as well as allowing the DFM to enter a Process Group in order to see and manipulate the components within the Process Group.
Port: Dataflows that are constructed using one or more Process Groups need a way to connect a Process Group to other dataflow components. This is achieved by using Ports. A DFM can add any number of Input Ports and Output Ports to a Process Group and name these ports appropriately.
Remote Process Group: Just as data is transferred into and out of a Process Group, it is sometimes necessary to transfer data from one instance of FlexFiles to another. While FlexFiles provides many different mechanisms for transferring data from one system to another, Remote Process Groups are often the easiest way to accomplish this if transferring data to another instance of FlexFiles.
Bulletin: The FlexFiles User Interface provides a significant amount of monitoring and feedback about the current status of the application. In addition to rolling statistics and the current status provided for each component, components are able to report Bulletins. Whenever a component reports a Bulletin, a bulletin icon is displayed on that component. System-level bulletins are displayed on the Status bar near the top of the page. Using the mouse to hover over that icon will provide a tool-tip that shows the time and severity (Debug, Info, Warning, Error) of the Bulletin, as well as the message of the Bulletin. Bulletins from all components can also be viewed and filtered in the Bulletin Board Page, available in the Global Menu.
Template: Often times, a dataflow is comprised of many sub-flows that could be reused. FlexFiles allows DFMs to select a part of the dataflow (or the entire dataflow) and create a Template. This Template is given a name and can then be dragged onto the canvas just like the other components. As a result, several components may be combined together to make a larger building block from which to create a dataflow. These templates can also be exported as XML and imported into another FlexFiles instance, allowing these building blocks to be shared.
flow.xml.gz: Everything the DFM puts onto the FlexFiles User Interface canvas is written, in real time, to one file called the flow.xml.gz. This file is located in the FlexFiles/conf directory by default. Any change made on the canvas is automatically saved to this file, without the user needing to click a "save" button. In addition, FlexFiles automatically creates a backup copy of this file in the archive directory when it is updated. You can use these archived files to rollback flow configuration. To do so, stop FlexFiles, replace flow.xml.gz with a desired backup copy, then restart FlexFiles. In a clustered environment, stop the entire FlexFiles cluster, replace the flow.xml.gz of one of nodes, and restart the node. Remove flow.xml.gz from other nodes. Once you confirmed the node starts up as a one-node cluster, start the other nodes. The replaced flow configuration will be synchronized across the cluster. The name and location of flow.xml.gz, and auto archive behavior are configurable. See the System Administrator’s Guide for further details.