102 – The Apache NiFi Expression Language
Use our Auto-Launching NiFi Image to Follow Along [Click Here]
All data that enters Apache NiFi is represented with an abstraction called a Flowfile. The attribute portion of a Flowfile is better known as the file’s meta data. Attributes give you information about the data that is passing through your system and/or held in your system. The NiFi expression language is used to interact with the attributes (metadata). It is a framework you need to learn if you wish to take advantage of NiFi.
Why Does The NiFi Expression Language Exist?
A Flowfile has an attribute called a filename, it has a path attribute, and it has an attribute called uuid. There are actually quite a few attributes that a Flowfile may posses, such as “Priority,” “MIME Type” and so forth. Placing attributes onto a Flowfile is pretty pointless if a user cannot put them to good use. The NiFi expression language gives you the ability to reference attributes. It allows you to manipulate their values, and allows you to compare them to various other values.
Learning The New Language
If you were to enter ${filename} then it would return the value of a filename attribute. The easiest and most basic attribute expression is the one used for a filename.
Each attribute starts with ${
Each attribute ends with }
Within the delimiters, you enter the expression’s text
As you learn, you will discover different manipulations for each value that is returned. For example, you could ask it to return a value that is all in upper-case. Let’s say we want the filename, but we want it all in upper case, we would write:
{filename:toUpper()}
As you may have guessed, “toUpper” is the function that tells NiFi to return the value (aka the file name) in an upper-case font.
A Function Has Five Elements
(1) There is the function call delimiter
(2) There is the name of the function
(3) Opened parenthesis
(4) Function arguments
There would usually be function arguments right here, but the toUpper function has no arguments
(5) Closed parenthesis
Use our Auto-Launching Nifi Image to Follow Along [Click Here]
A Few Examples of Attributes
Notice how some of these functions have arguments. For example, in the first example for reformatting a date, the argument is sat within the parenthesis. The argument in this case is “yyyy-MM-DD”
Reformat a date
${string_date:toDate(“yyyy-MM-DD”)}
Mathematical operations
${amount_owed:minus(5)}
Multi-variable greater than
${variable_one:gt(${variable_two})}
Adding a new directory to a path attribute
${path:append(‘/new_directory’)}
Check to see if the file has a specific name
${filename:contains(‘NiFi’)}
Boolean Logic
With NiFi, you are able to compare the value of an attribute against the value of another attribute. For example, the “equals” function is used very widely as it determines if another String value is equal to the subject. In other words, it gives you a direct comparison of both String values. You will receive a Boolean response.
For example, if you wish to find out if a filename is called “pickle.txt” you could check if the value of the filename attribute is equal to the value of the pickle attribute:
${pickle:equals( ${filename} )}.
That is just a quick example of how Boolean logic may be applied. Here is another expression your may use:
${filename:equals(‘pickle.txt’)}
JSONPath
If you reference JSONs with processors, then you must utilize the JSONPath expression language. Below is an example of a MyJSON attribute. It contains data we just made up. Pretend it was data inputted into somebody’s account when they signed up for an online subscription to something.
{
“firstName”: “Barry”,
“lastName”: “Dingle”,
“isAlive”: true,
“age”: 31,
“address”: {
“streetAddress”: “13 Norman Street”,
“city”: “South Park”,
“state”: “Denver”,
“postalCode”: “09011-5010”
},
“phoneNumbers”: [
{
“type”: “home”,
“number”: “666 555-4321”
},
{
“type”: “office”,
“number”: “666 555-8888”
}
],
“children”: [],
“spouse”: null
}
Using the attribute code above, here are a few JSONPath examples. Each is labeled as “Expression” or “Value.” If the expressions listed below were used in conjunction with the attribute shown above, then the values displayed are what would be returned.
Expression
${myJson:jsonPath(‘$.firstName’)}
Value
Barry
Expression
${myJson:jsonPath(‘$.address.postalCode’)}
Value
09011-5010
Expression
${myJson:jsonPath(‘$.phoneNumbers’)}
Value
[{“type”:”home”,”number”:”666 555-4321″}
{“type”:”office”,”number”:”666 555-8888″}]
XPath/XQuery
You may access data located in XMLs with processors, and you may do it with the XPath/Query language. You may use processors such as EvaluateXQuery and EvaluateXPath. Just like when you are using JSONPath, data may be searched or exactly specified. Here are a few examples:
» Search any and all levels of an XML and locate a field called Version
//Version
» Specify a value that shows the first name of an account holder
/account/user_name/first_name
» If numerous accounts exist on the XML, specify/give the value of the first account
/account[0]/user_name/first_name
» Filter for subversion, but only those that are greater than 5
//Version[subVersion>5]
Next: Learn to use Apache NiFi flowfile provenance
Learn more about Apache NiFi by downloading the free ebook Apache NiFi for Dummies: https://www.calculatedsystems.com/nifi-for-dummies
