Apache
/
nifi
mirror of https://github.com/apache/nifi.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

NIFI-6: Rebase from develop to include renaming of directory structure

This commit is contained in:
Mark Payne 2015-01-22 10:59:36 -05:00
parent b942daa65a cff8b0de40
commit 716e03b502
2205 changed files with 1241 additions and 13281 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

55
README.md
View File

@ -4,35 +4,15 @@ Apache NiFi is a dataflow system based on the concepts of flow-based programming
## Table of Contents
- [Features](#features)
- [Getting Started](#getting-started)
- [Getting Help](#getting-help)
- [Requirements](#requirements)
- [License](#license)
- [Disclaimer](#disclaimer)
## Features
Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Apache NiFi include:
- Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows
- Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing
- Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state
- Component-based extension model along well defined interfaces enabling rapid development and effective testing
## Getting Started
Execute <nifi install dir>/bin/nifi.sh start
## Getting Help
If you have questions, you can reach out to our mailing list: dev@nifi.incubator.apache.org
([archive](http://mail-archives.apache.org/mod_mbox/incubator-nifi-dev)).
We're also often available in IRC: #nifi on
[irc.freenode.net](http://webchat.freenode.net/?channels=#nifi).
## Requirements
* JDK 1.7 or higher
- Build the nifi-nar-maven-plugin. Change directory to 'nifi-nar-maven-plugin' and
follow the directions found there.
- Build nifi. Change directory to 'nifi' and follow the directions found there.
## License
@ -64,32 +44,3 @@ While incubation status is not necessarily a reflection of the completeness
or stability of the code, it does indicate that the project has yet to be
fully endorsed by the ASF.
## Export Control
This distribution includes cryptographic software. The country in which you
currently reside may have restrictions on the import, possession, use, and/or
re-export to another country, of encryption software. BEFORE using any
encryption software, please check your country's laws, regulations and
policies concerning the import, possession, or use, and re-export of encryption
software, to see if this is permitted. See <http://www.wassenaar.org/> for more
information.
The U.S. Government Department of Commerce, Bureau of Industry and Security
(BIS), has classified this software as Export Commodity Control Number (ECCN)
5D002.C.1, which includes information security software using or performing
cryptographic functions with asymmetric algorithms. The form and manner of this
Apache Software Foundation distribution makes it eligible for export under the
License Exception ENC Technology Software Unrestricted (TSU) exception (see the
BIS Export Administration Regulations, Section 740.13) for both object code and
source code.
The following provides more details on the included cryptographic software:
Apache NiFi uses BouncyCastle, Jasypt, JCraft Inc., and the built-in
java cryptography libraries for SSL, SSH, and the protection
of sensitive configuration parameters. See
http://bouncycastle.org/about.html
http://www.jasypt.org/faq.html
http://jcraft.com/c-info.html
http://www.oracle.com/us/products/export/export-regulations-345813.html
for more details on each of these libraries cryptography features.

75
maven-plugins/nar-maven-plugin/pom.xml
View File

@ -1,75 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>maven-plugins</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>nar-maven-plugin</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>maven-plugin</packaging>
<name>Apache NiFi NAR Plugin</name>
<description>Apache NiFi Nar Plugin. It is currently a part of the Apache Incubator.</description>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-plugin-plugin</artifactId>
<executions>
<execution>
<id>default-descriptor</id>
<goals>
<goal>descriptor</goal>
</goals>
<phase>process-classes</phase>
</execution>
<execution>
<id>help-descriptor</id>
<goals>
<goal>helpmojo</goal>
</goals>
<phase>process-classes</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-plugin-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<type>maven-plugin</type>
</dependency>
<dependency>
<!-- No code from maven-jar-plugin is actually used; it's included
just to simplify the dependencies list. -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
</dependency>
<dependency>
<groupId>org.apache.maven.plugin-tools</groupId>
<artifactId>maven-plugin-annotations</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
</project>

0
DISCLAIMER → nifi-nar-maven-plugin/DISCLAIMER
View File

202
nifi-nar-maven-plugin/LICENSE Normal file
View File

@ -0,0 +1,202 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

5
nifi-nar-maven-plugin/NOTICE Normal file
View File

@ -0,0 +1,5 @@
Apache NiFi
Copyright 2014-2015 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

57
nifi-nar-maven-plugin/README.md Normal file
View File

@ -0,0 +1,57 @@
# Apache NiFi NAR Maven Plugin
Apache NiFi NAR Maven Plugin helps to build NiFi Archive bundles to support the classloader isolation model of NiFi.
## Table of Contents
- [Getting Started](#getting-started)
- [Getting Help](#getting-help)
- [Requirements](#requirements)
- [License](#license)
- [Disclaimer](#disclaimer)
## Getting Started
To Build:
- Executed 'mvn clean install'
## Getting Help
If you have questions, you can reach out to our mailing list: dev@nifi.incubator.apache.org
([archive](http://mail-archives.apache.org/mod_mbox/incubator-nifi-dev)).
We're also often available in IRC: #nifi on
[irc.freenode.net](http://webchat.freenode.net/?channels=#nifi).
## Requirements
* JDK 1.7 or higher
## License
Except as otherwise noted this software is licensed under the
[Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0.html)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
## Disclaimer
Apache NiFi is an effort undergoing incubation at the Apache Software
Foundation (ASF), sponsored by the Apache Incubator PMC.
Incubation is required of all newly accepted projects until a further review
indicates that the infrastructure, communications, and decision making process
have stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the completeness
or stability of the code, it does indicate that the project has yet to be
fully endorsed by the ASF.

237
maven-plugins/pom.xml → nifi-nar-maven-plugin/pom.xml
View File

@ -20,14 +20,13 @@
<groupId>org.apache</groupId>
<artifactId>apache</artifactId>
<version>16</version>
<relativePath/>
<relativePath />
</parent>
<groupId>org.apache.nifi</groupId>
<artifactId>maven-plugins</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<name>Apache NiFi Maven Plugins</name>
<description>Apache NiFi Maven Plugins. It is currently a part of the Apache Incubator.</description>
<artifactId>nifi-nar-maven-plugin</artifactId>
<version>1.0.0-incubating-SNAPSHOT</version>
<packaging>maven-plugin</packaging>
<description>Apache NiFi Nar Plugin. It is currently a part of the Apache Incubator.</description>
<url>http://nifi.incubator.apache.org/maven-site/</url>
<organization>
<name>Apache NiFi (incubating) Project</name>
@ -55,16 +54,11 @@
<archive>http://mail-archives.apache.org/mod_mbox/incubator-nifi-commits</archive>
</mailingList>
</mailingLists>
<prerequisites>
<maven>${maven.min-version}</maven>
</prerequisites>
<modules>
<module>nar-maven-plugin</module>
</modules>
<scm>
<connection>scm:git:git://git.apache.org/incubator-nifi.git</connection>
<developerConnection>scm:git:https://git-wip-us.apache.org/repos/asf/incubator-nifi.git</developerConnection>
<url>https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git</url>
<tag>HEAD</tag>
</scm>
<issueManagement>
<system>JIRA</system>
@ -73,11 +67,13 @@
<properties>
<maven.compiler.source>1.7</maven.compiler.source>
<maven.compiler.target>1.7</maven.compiler.target>
<maven.version>3.0.5</maven.version>
<maven.min-version>3.0.5</maven.min-version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<sealJars>false</sealJars>
</properties>
<prerequisites>
<maven>${maven.min-version}</maven>
</prerequisites>
<build>
<pluginManagement>
<plugins>
@ -92,20 +88,6 @@
<showWarnings>true</showWarnings>
</configuration>
</plugin>
<plugin>
<artifactId>maven-war-plugin</artifactId>
<version>2.5</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-plugin-plugin</artifactId>
<version>3.3</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.9</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
@ -124,15 +106,9 @@
<artifactId>maven-assembly-plugin</artifactId>
<version>2.5.2</version>
<configuration>
<attach>false</attach>
<tarLongFileMode>gnu</tarLongFileMode>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>jaxb2-maven-plugin</artifactId>
<version>1.6</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
@ -166,22 +142,23 @@
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-release-plugin</artifactId>
<version>2.5.1</version>
<configuration>
<useReleaseProfile>true</useReleaseProfile>
<releaseProfiles>apache-release</releaseProfiles>
<autoVersionSubmodules>true</autoVersionSubmodules>
<goals>deploy</goals>
<tagNameFormat>@{project.artifactId}-@{project.version}</tagNameFormat>
<pushChanges>false</pushChanges>
<localCheckout>true</localCheckout>
</configuration>
<executions>
<execution>
<id>default</id>
<goals>
<goal>prepare</goal>
<goal>perform</goal>
</goals>
<configuration>
<pomFileName>platform/pom.xml</pomFileName>
<arguments>-P apache-release,check-licenses</arguments>
<autoVersionSubmodules>true</autoVersionSubmodules>
<releaseProfiles>apache-release</releaseProfiles>
<goals>deploy</goals>
<tagNameFormat>@{project.artifactId}-@{project.version}</tagNameFormat>
<pushChanges>false</pushChanges>
<localCheckout>true</localCheckout>
<pomFileName>nar-maven-plugin/pom.xml</pomFileName>
</configuration>
</execution>
</executions>
@ -189,138 +166,80 @@
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifestEntries>
<Sealed>${sealJars}</Sealed>
<Implementation-Build>${mvngit.commit.id}</Implementation-Build>
</manifestEntries>
</archive>
</configuration>
<version>2.5</version>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.sonatype.plugins</groupId>
<artifactId>nexus-staging-maven-plugin</artifactId>
<version>1.6.5</version>
<extensions>true</extensions>
<configuration>
<serverId>repository.apache.org</serverId>
<nexusUrl>https://repository.apache.org/</nexusUrl>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<artifactId>maven-plugin-plugin</artifactId>
<executions>
<execution>
<id>enforce-maven</id>
<id>default-descriptor</id>
<goals>
<goal>enforce</goal>
<goal>descriptor</goal>
</goals>
<configuration>
<rules>
<requireSameVersions>
<plugins>
<plugin>org.apache.maven.plugins:maven-surefire-plugin</plugin>
<plugin>org.apache.maven.plugins:maven-failsafe-plugin</plugin>
<plugin>org.apache.maven.plugins:maven-surefire-report-plugin</plugin>
</plugins>
</requireSameVersions>
<requireMavenVersion>
<version>${maven.version}</version>
</requireMavenVersion>
</rules>
</configuration>
<phase>process-classes</phase>
</execution>
<execution>
<id>help-descriptor</id>
<goals>
<goal>helpmojo</goal>
</goals>
<phase>process-classes</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.rat</groupId>
<artifactId>apache-rat-plugin</artifactId>
<configuration>
<excludes>
<exclude>nb-configuration.xml</exclude>
<exclude>nbactions.xml</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-plugin-api</artifactId>
<version>2.0.11</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.9</version>
<type>maven-plugin</type>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugin-tools</groupId>
<artifactId>maven-plugin-annotations</artifactId>
<version>3.3</version>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.maven</groupId>
<artifactId>maven-plugin-api</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<type>maven-plugin</type>
<version>2.9</version>
</dependency>
<dependency>
<!-- No code from maven-jar-plugin is actually used; it's included
just to simplify the dependencies list. -->
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.5</version>
</dependency>
<dependency>
<groupId>org.apache.maven.plugin-tools</groupId>
<artifactId>maven-plugin-annotations</artifactId>
<scope>provided</scope>
<version>3.3</version>
</dependency>
</dependencies>
<profiles>
<profile>
<id>apache-release</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<dependencies>
<dependency>
<groupId>org.apache.apache.resources</groupId>
<artifactId>apache-source-release-assembly-descriptor</artifactId>
<version>1.0.4</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>source-release-assembly</id>
<goals>
<goal>single</goal>
</goals>
<phase>validate</phase>
<configuration>
<runOnlyAtExecutionRoot>true</runOnlyAtExecutionRoot>
<finalName>nifi-${project.artifactId}-${project.version}</finalName>
<descriptorRefs>
<descriptorRef>source-release-zip-tar</descriptorRef>
</descriptorRefs>
<tarLongFileFormat>gnu</tarLongFileFormat>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<id>rename-source-release-assembly</id>
<goals>
<goal>exec</goal>
</goals>
<phase>validate</phase>
<configuration>
<executable>mv</executable>
<workingDirectory>${project.build.directory}</workingDirectory>
<commandlineArgs>-n nifi-${project.artifactId}-${project.version}-source-release.tar.gz nifi-${project.artifactId}-${project.version}-src.tar.gz</commandlineArgs>
<successCodes>
<successCode>0</successCode>
<successCode>1</successCode>
</successCodes>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
<profile>
<!-- Seal jars and skip tests when the
apache-release profile is activated. -->
<id>seal-jars</id>
<properties>
<sealJars>true</sealJars>
<skipTests>true</skipTests>
</properties>
</profile>
<profile>
<!-- Automatically check for licenses.
Activate with -P check-licenses -->

2
maven-plugins/nar-maven-plugin/src/main/java/nifi/NarMojo.java → nifi-nar-maven-plugin/src/main/java/org/apache/nifi/NarMojo.java
View File

@ -14,7 +14,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package nifi;
package org.apache.nifi;
import java.io.File;
import java.io.IOException;

2
maven-plugins/nar-maven-plugin/src/main/resources/META-INF/plexus/components.xml → nifi-nar-maven-plugin/src/main/resources/META-INF/plexus/components.xml
View File

@ -29,7 +29,7 @@
<process-test-resources>org.apache.maven.plugins:maven-resources-plugin:testResources</process-test-resources>
<test-compile>org.apache.maven.plugins:maven-compiler-plugin:testCompile</test-compile>
<test>org.apache.maven.plugins:maven-surefire-plugin:test</test>
<package>org.apache.nifi:nar-maven-plugin:nar</package>
<package>org.apache.nifi:nifi-nar-maven-plugin:nar</package>
<install>org.apache.maven.plugins:maven-install-plugin:install</install>
<deploy>org.apache.maven.plugins:maven-deploy-plugin:deploy</deploy>
</phases>

15
nifi/DISCLAIMER Normal file
View File

@ -0,0 +1,15 @@
Apache NiFi is an effort undergoing incubation at the Apache Software
Foundation (ASF), sponsored by the Apache Incubator PMC.
Incubation is required of all newly accepted projects until a further review
indicates that the infrastructure, communications, and decision making process
have stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the completeness
or stability of the code, it does indicate that the project has yet to be
fully endorsed by the ASF.
For more information about the incubation status of the Apache NiFi project
you can go to the following page:
http://nifi.incubator.apache.org/

0
LICENSE → nifi/LICENSE
View File

2
NOTICE → nifi/NOTICE
View File

@ -1,5 +1,5 @@
Apache NiFi
Copyright 2014 The Apache Software Foundation
Copyright 2014-2015 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).

101
nifi/README.md Normal file
View File

@ -0,0 +1,101 @@
# Apache NiFi
Apache NiFi is a dataflow system based on the concepts of flow-based programming. It is currently apart of the Apache Incubator.
## Table of Contents
- [Features](#features)
- [Getting Started](#getting-started)
- [Getting Help](#getting-help)
- [Requirements](#requirements)
- [License](#license)
- [Disclaimer](#disclaimer)
- [Export Control] (#export-control)
## Features
Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Some of the high-level capabilities and objectives of Apache NiFi include:
- Web-based user interface for seamless experience between design, control, feedback, and monitoring of data flows
- Highly configurable along several dimensions of quality of service such as loss tolerant versus guaranteed delivery, low latency versus high throughput, and priority based queuing
- Fine-grained data provenance for all data received, forked, joined, cloned, modified, sent, and ultimately dropped as data reaches its configured end-state
- Component-based extension model along well defined interfaces enabling rapid development and effective testing
## Getting Started
To build:
- Execute 'mvn clean install' or for parallel build execute 'mvn -T 2.0C clean install'
To start NiFi:
- Change directory to 'nifi-assembly'. In the target directory there should be a build of nifi.
- Unpack the build wherever you like or use the already unpacked build. '<install_location>/bin/nifi.sh start'
- Direct your browser to http://localhost:8080/nifi/
## Getting Help
If you have questions, you can reach out to our mailing list: dev@nifi.incubator.apache.org
([archive](http://mail-archives.apache.org/mod_mbox/incubator-nifi-dev)).
We're also often available in IRC: #nifi on
[irc.freenode.net](http://webchat.freenode.net/?channels=#nifi).
## Requirements
* JDK 1.7 or higher
## License
Except as otherwise noted this software is licensed under the
[Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0.html)
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
## Disclaimer
Apache NiFi is an effort undergoing incubation at the Apache Software
Foundation (ASF), sponsored by the Apache Incubator PMC.
Incubation is required of all newly accepted projects until a further review
indicates that the infrastructure, communications, and decision making process
have stabilized in a manner consistent with other successful ASF projects.
While incubation status is not necessarily a reflection of the completeness
or stability of the code, it does indicate that the project has yet to be
fully endorsed by the ASF.
## Export Control
This distribution includes cryptographic software. The country in which you
currently reside may have restrictions on the import, possession, use, and/or
re-export to another country, of encryption software. BEFORE using any
encryption software, please check your country's laws, regulations and
policies concerning the import, possession, or use, and re-export of encryption
software, to see if this is permitted. See <http://www.wassenaar.org/> for more
information.
The U.S. Government Department of Commerce, Bureau of Industry and Security
(BIS), has classified this software as Export Commodity Control Number (ECCN)
5D002.C.1, which includes information security software using or performing
cryptographic functions with asymmetric algorithms. The form and manner of this
Apache Software Foundation distribution makes it eligible for export under the
License Exception ENC Technology Software Unrestricted (TSU) exception (see the
BIS Export Administration Regulations, Section 740.13) for both object code and
source code.
The following provides more details on the included cryptographic software:
Apache NiFi uses BouncyCastle, Jasypt, JCraft Inc., and the built-in
java cryptography libraries for SSL, SSH, and the protection
of sensitive configuration parameters. See
http://bouncycastle.org/about.html
http://www.jasypt.org/faq.html
http://jcraft.com/c-info.html
http://www.oracle.com/us/products/export/export-regulations-345813.html
for more details on each of these libraries cryptography features.

81
nifi/nar-bundles/execute-script-bundle/execute-script-processors/pom.xml
View File

@ -1,81 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>execute-script-bundle</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>execute-script-processors</artifactId>
<description>NiFi Processors to Run Scripts</description>
<name>NiFi Script Execution Processors</name>
<dependencies>
<dependency>
<groupId>org.jruby</groupId>
<artifactId>jruby</artifactId>
<exclusions>
<exclusion>
<artifactId>jnr-netdb</artifactId>
<groupId>com.github.jnr</groupId>
</exclusion>
<exclusion>
<artifactId>jnr-posix</artifactId>
<groupId>com.github.jnr</groupId>
</exclusion>
<exclusion>
<artifactId>jffi</artifactId>
<groupId>com.github.jnr</groupId>
</exclusion>
<exclusion>
<artifactId>nailgun-server</artifactId>
<groupId>com.martiansoftware</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.python</groupId>
<artifactId>jython-standalone</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-core-flowfile-attributes</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-stream-utils</artifactId>
</dependency>
</dependencies>
</project>

566
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/processors/script/ExecuteScript.java
View File

@ -1,566 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.processors.script;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
import javax.script.ScriptException;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.components.ValidationContext;
import org.apache.nifi.components.ValidationResult;
import org.apache.nifi.components.Validator;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.io.BufferedInputStream;
import org.apache.nifi.io.BufferedOutputStream;
import org.apache.nifi.processor.AbstractProcessor;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.ProcessSession;
import org.apache.nifi.processor.ProcessorInitializationContext;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.processor.annotation.CapabilityDescription;
import org.apache.nifi.processor.annotation.EventDriven;
import org.apache.nifi.processor.annotation.Tags;
import org.apache.nifi.processor.exception.ProcessException;
import org.apache.nifi.processor.io.InputStreamCallback;
import org.apache.nifi.processor.io.StreamCallback;
import org.apache.nifi.processor.util.StandardValidators;
import org.apache.nifi.scripting.ConverterScript;
import org.apache.nifi.scripting.ReaderScript;
import org.apache.nifi.scripting.Script;
import org.apache.nifi.scripting.ScriptFactory;
import org.apache.nifi.scripting.WriterScript;
/**
* <!-- Processor Documentation ================================================== -->
* <h2>Description:</h2>
* <p>
* This processor provides the capability to execute scripts in various
* scripting languages, and passes into the scripts the input stream and output
* stream(s) representing an incoming flow file and any created flow files. The
* processor is designed to be thread safe, so multiple concurrent tasks may
* execute against a single script. The processor provides a framework which
* enables script writers to implement 3 different types of scripts:
* <ul>
* ReaderScript - which enables stream-based reading of a FlowFile's
* content</br> WriterScript - which enables stream-based reading and
* writing/modifying of a FlowFile's content</br> ConverterScript - which
* enables stream-based reading a FlowFile's content and stream-based writing to
* newly created FlowFiles</br>
* </ul>
* Presently, the processor supports 3 scripting languages: Ruby, Python, and
* JavaScript. The processor is built on the javax.script API which enables
* ScriptEngine discovery, thread management, and encapsulates much of the low
* level bridging-code that enables Java to Script language integration. Thus,
* it is designed to be easily extended to other scripting languages. </br> The
* attributes of a FlowFile and properties of the Processor are exposed to the
* script by either a variable in the base class or a getter method. A script
* may declare new Processor Properties and different Relationships via
* overriding the getPropertyDescriptors and getRelationships methods,
* respectively.
* </p>
* <p>
* <strong>Properties:</strong>
* </p>
* <p>
* In the list below, the names of required properties appear in bold. Any other
* properties (not in bold) are considered optional. If a property has a default
* value, it is indicated. If a property supports the use of the NiFi Expression
* Language (or simply, "expression language"), that is also indicated. Of
* particular note: This processor allows scripts to define additional Processor
* properties, which will not be initially visible. Once the processor's
* configuration is validated, script defined properties will become visible,
* and may affect the validity of the processor.
* </p>
* <ul>
* <li>
* <strong>Script File Name</strong>
* <ul>
* <li>Script location, can be relative or absolute path.</li>
* <li>Default value: no default</li>
* <li>Supports expression language: false</li>
* </ul>
* </li>
* <li>
* <strong>Script Check Interval</strong>
* <ul>
* <li>The time period between checking for updates to a script.</li>
* <li>Default value: 15 sec</li>
* <li>Supports expression language: false</li>
* </ul>
* </li>
* </ul>
*
* <p>
* <strong>Relationships:</strong>
* </p>
* <p>
* The initial 'out of the box' relationships are below. Of particular note is
* the ability of a script to change the set of relationships. However, any
* relationships defined by the script will not be visible until the processor's
* configuration has been validated. Once done, new relationships will become
* visible.
* </p>
* <ul>
* <li>
* success
* <ul>
* <li>Used when a file is successfully processed by a script.</li>
* </ul>
* </li>
* <li>
* failure
* <ul>
* <li>Used when an error occurs while processing a file with a script.</li>
* </ul>
* </li>
* </ul>
*
* <p>
* <strong>Example Scripts:</strong>
* </p>
* <ul>
* JavaScript example - the 'with' statement imports packages defined in the
* framework. Since the 'instance' variable is intended to be local scope (not
* global), it must be named 'instance' as it it not passed back to the
* processor upon script evaluation and must be fetched. If you make it global,
* you can name it whatever you'd like...but this is intended to be
* multi-threaded so do so at your own risk. Presently, there are issues with
* the JavaScript scripting engine that prevent sub-classing the base classes in
* the Processor's Java framework. So, what is actually happening is an instance
* of the ReaderScript is created with a provided callback object. When we are
* able to move to a more competent scripting engine, the code below will remain
* the same, but the 'instance' variable will actually be a sub-class of
* ReaderScript.
*
* <pre>
* with (Scripting) {
* var instance = new ReaderScript({
* route : function(input) {
* var str = IOUtils.toString(input);
* var expr = instance.getProperty("expr");
* filename = instance.attributes.get("filename");
* instance.setAttribute("filename", filename + ".modified");
* if (str.match(expr)) {
* return Script.FAIL_RELATIONSHIP;
* } else {
* return Script.SUCCESS_RELATIONSHIP;
* }
* }
* });
* }
* </pre>
*
* Ruby example - the 'OutputStreamHandler' is an interface which is called when
* creating flow files.
*
* <pre>
* java_import 'org.apache.nifi.scripting.OutputStreamHandler'
* class SimpleConverter < ConverterScript
* field_reader :FAIL_RELATIONSHIP, :SUCCESS_RELATIONSHIP, :logger, :attributes
*
* def convert(input)
* in_io = input.to_io
* createFlowFile("firstLine", FAIL_RELATIONSHIP, OutputStreamHandler.impl do |method, out|
* out_io = out.to_io
* out_io << in_io.readline.to_java_bytes
* out_io.close
* logger.debug("Wrote data to failure...this message logged with logger from super class")
* end)
*
* createFlowFile("otherLines", SUCCESS_RELATIONSHIP, OutputStreamHandler.impl do |method, out|
* out_io = out.to_io
* in_io.each_line { |line|
* out_io << line
* }
* out_io.close
* logger.debug("Wrote data to success...this message logged with logger from super class")
* end)
* in_io.close
* end
*
* end
*
* $logger.debug("Creating SimpleConverter...this message logged with logger from shared variables")
* SimpleConverter.new
* </pre>
*
* Python example - The difficulty with Python is that it does not return
* objects upon script evaluation, so the instance of the Script class must be
* fetched by name. Thus, you must define a variable called 'instance'.
*
* <pre>
* import re
*
* class RoutingReader(ReaderScript):
* A = Relationship.Builder().name("a").description("some good stuff").build()
* B = Relationship.Builder().name("b").description("some other stuff").build()
* C = Relationship.Builder().name("c").description("some bad stuff").build()
*
* def getRelationships(self):
* return [self.A,self.B,self.C]
*
* def getExceptionRoute(self):
* return self.C
*
* def route( self, input ):
* for line in FileUtil.wrap(input):
* if re.match("^bad", line, re.IGNORECASE):
* return self.B
* if re.match("^sed", line):
* raise RuntimeError("That's no good!")
*
* return self.A
*
* instance = RoutingReader()
* </pre>
*
* </ul>
* <p>
* <strong>Shared Variables</strong>
* </p>
* <ul>
* <li>logger : global scope</li>
* <li>properties : local/instance scope</li>
* </ul>
* <p>
* <strong>Script API:</strong>
* </p>
* <ul>
* <li>getAttribute(String) : String</li>
* <li>getAttributes() : Map(String,String)</li>
* <li>getExceptionRoute() : Relationship</li>
* <li>getFileName() : String</li>
* <li>getFlowFileEntryDate() : Calendar</li>
* <li>getFlowFileSize() : long</li>
* <li>getProperties() : Map(String, String)</li>
* <li>getProperty(String) : String</li>
* <li>getPropertyDescriptors() : List(PropertyDescriptor)</li>
* <li>getRelationships() : Collection(Relationship)</li>
* <li>getRoute() : Relationship</li>
* <li>setRoute(Relationship)</li>
* <li>setAttribute(String, String)</li>
* <li>validate() : Collection(String)</li>
* </ul>
* <p>
* <strong>ReaderScript API:</strong>
* </p>
* <ul>
* <li>route(InputStream) : Relationship</li>
* </ul>
* <p>
* <strong>WriterScript API:</strong>
* </p>
* <ul>
* <li>process(InputStream, OutputStream)</li>
* </ul>
* <p>
* <strong>ConverterScript API:</strong>
* </p>
* <ul>
* <li>convert(InputStream)</li>
* <li>createFlowFile(String, Relationship, OutputStreamHandler)</li>
* </ul>
* <p>
* <strong>OutputStreamHandler API:</strong>
* </p>
* <ul>
* <li>write(OutputStream)</li>
* </ul>
*/
@EventDriven
@Tags({"script", "ruby", "python", "javascript", "execute"})
@CapabilityDescription("Execute scripts in various scripting languages, and passes into the scripts the input stream and output stream(s) "
+ "representing an incoming flow file and any created flow files.")
public class ExecuteScript extends AbstractProcessor {
private final AtomicBoolean doCustomValidate = new AtomicBoolean(true);
private final AtomicReference<Set<Relationship>> relationships = new AtomicReference<>();
private final AtomicReference<List<PropertyDescriptor>> propertyDescriptors = new AtomicReference<>();
private volatile ScriptFactory scriptFactory;
private volatile Relationship exceptionRoute;
/**
* Script location, can be relative or absolute path -- passed as-is to
* {@link File#File(String) File constructor}
*/
public static final PropertyDescriptor SCRIPT_FILE_NAME = new PropertyDescriptor.Builder()
.name("Script File Name")
.description("Script location, can be relative or absolute path")
.required(true)
.addValidator(new Validator() {
@Override
public ValidationResult validate(String subject, String input, ValidationContext context) {
ValidationResult result = StandardValidators.FILE_EXISTS_VALIDATOR.validate(subject, input, context);
if (result.isValid()) {
int dotPos = input.lastIndexOf('.');
if (dotPos < 1) {
result = new ValidationResult.Builder()
.subject(subject)
.valid(false)
.explanation("Filename must have an extension")
.input(input)
.build();
}
}
return result;
}
})
.build();
static final PropertyDescriptor SCRIPT_CHECK_INTERVAL = new PropertyDescriptor.Builder()
.name("Script Check Interval")
.addValidator(StandardValidators.TIME_PERIOD_VALIDATOR)
.description("The time period between checking for updates to a script")
.required(true)
.defaultValue("15 sec")
.build();
@Override
protected void init(ProcessorInitializationContext context) {
Set<Relationship> empty = Collections.emptySet();
relationships.set(empty);
ArrayList<PropertyDescriptor> propDescs = new ArrayList<>();
propDescs.add(SCRIPT_FILE_NAME);
propDescs.add(SCRIPT_CHECK_INTERVAL);
propertyDescriptors.set(Collections.unmodifiableList(propDescs));
scriptFactory = new ScriptFactory(getLogger());
}
@Override
public List<PropertyDescriptor> getSupportedPropertyDescriptors() {
return propertyDescriptors.get();
}
@Override
protected PropertyDescriptor getSupportedDynamicPropertyDescriptor(String propertyDescriptorName) {
return new PropertyDescriptor.Builder()
.name(propertyDescriptorName)
.dynamic(true)
.addValidator(Validator.VALID)
.build();
}
@Override
public void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) {
doCustomValidate.set(true);
}
@Override
public Set<Relationship> getRelationships() {
return relationships.get();
}
/**
* Called by framework.
*
* Returns a list of reasons why this processor cannot be run.
* @return
*/
@Override
protected Collection<ValidationResult> customValidate(ValidationContext validationContext) {
if (doCustomValidate.getAndSet(false)) {
long interval = validationContext.getProperty(SCRIPT_CHECK_INTERVAL).asTimePeriod(TimeUnit.MILLISECONDS);
scriptFactory.setScriptCheckIntervalMS(interval);
List<ValidationResult> results = new ArrayList<>();
String file = validationContext.getProperty(SCRIPT_FILE_NAME).getValue();
try {
Script s = scriptFactory.getScript(file);
// set the relationships of the processor
relationships.set(new HashSet<>(s.getRelationships()));
// need to get script's prop. descs. and validate. May, or may not, have dynamic
// props already...depends if this is the first time the processor is being configured.
Map<PropertyDescriptor, String> properties = validationContext.getProperties();
// need to compare props, if any, against script-expected props that are required.
// script may be expecting required props that are not known, or some props may have invalid
// values.
// processor may be configured with dynamic props that the script will use...but does not declare which would
// be a bad thing
List<PropertyDescriptor> scriptPropDescs = s.getPropertyDescriptors();
getLogger().debug("Script is {}", new Object[]{s});
getLogger().debug("Script file name is {}", new Object[]{s.getFileName()});
getLogger().debug("Script Prop Descs are: {}", new Object[]{scriptPropDescs.toString()});
getLogger().debug("Thread is: {}", new Object[]{Thread.currentThread().toString()});
for (PropertyDescriptor propDesc : scriptPropDescs) {
// need to check for missing props
if (propDesc.isRequired() && !properties.containsKey(propDesc)) {
results.add(new ValidationResult.Builder()
.subject("Script Properties")
.valid(false)
.explanation("Missing Property " + propDesc.getName())
.build());
// need to validate current value against script provided validator
} else if (properties.containsKey(propDesc)) {
String value = properties.get(propDesc);
ValidationResult result = propDesc.validate(value, validationContext);
if (!result.isValid()) {
results.add(result);
}
} // else it is an optional prop according to the script and it is not specified by
// the configuration of the processor
}
// need to update the known prop desc's with what we just got from the script
List<PropertyDescriptor> pds = new ArrayList<>(propertyDescriptors.get());
pds.addAll(scriptPropDescs);
propertyDescriptors.set(Collections.unmodifiableList(pds));
if (results.isEmpty()) {
// so needed props are supplied and individually validated, now validate script
Collection<String> reasons;
reasons = s.validate();
if (null == reasons) {
getLogger().warn("Script had invalid return value for validate(), ignoring.");
} else {
for (String reason : reasons) {
ValidationResult result = new ValidationResult.Builder()
.subject("ScriptValidation")
.valid(false)
.explanation(reason)
.build();
results.add(result);
}
}
}
// get the exception route
exceptionRoute = s.getExceptionRoute();
return results;
} catch (ScriptException | IOException | NoSuchMethodException e) {
doCustomValidate.set(true);
results.add(new ValidationResult.Builder()
.subject("ScriptValidation")
.valid(false)
.explanation("Cannot create script due to " + e.getMessage())
.input(file)
.build());
getLogger().error("Cannot create script due to " + e, e);
return results;
}
}
return null;
}
@Override
public void onTrigger(ProcessContext context, ProcessSession session) throws ProcessException {
FlowFile flowFile = session.get();
if (flowFile == null) {
return; // fail-fast if there is no work to do
}
final String scriptFileName = context.getProperty(SCRIPT_FILE_NAME).getValue();
// doing this cloning because getProperties does not initialize props that have only their default values
// must do a getProperty for that value to be initialized
Map<String, String> props = new HashMap<>();
for (PropertyDescriptor propDesc : context.getProperties().keySet()) {
if (propDesc.isExpressionLanguageSupported()) {
props.put(propDesc.getName(), context.getProperty(propDesc).evaluateAttributeExpressions(flowFile).getValue());
} else {
props.put(propDesc.getName(), context.getProperty(propDesc).getValue());
}
}
Script script = null;
try {
final Script finalScript = scriptFactory.getScript(scriptFileName, props, flowFile);
script = finalScript;
if (finalScript instanceof ReaderScript) {
session.read(flowFile, new InputStreamCallback() {
@Override
public void process(InputStream in) throws IOException {
try {
((ReaderScript) finalScript).process(new BufferedInputStream(in));
} catch (NoSuchMethodException | ScriptException e) {
getLogger().error("Failed to execute ReaderScript", e);
throw new IOException(e);
}
}
});
} else if (finalScript instanceof WriterScript) {
flowFile = session.write(flowFile, new StreamCallback() {
@Override
public void process(InputStream in, OutputStream out) throws IOException {
try {
((WriterScript) finalScript).process(new BufferedInputStream(in), new BufferedOutputStream(out));
out.flush();
} catch (NoSuchMethodException | ScriptException e) {
getLogger().error("Failed to execute WriterScript", e);
throw new IOException(e);
}
}
});
} else if (finalScript instanceof ConverterScript) {
((ConverterScript) finalScript).process(session);
// Note that these scripts don't pass the incoming FF through,
// they always create new outputs
session.remove(flowFile);
return;
} else {
// only thing we can do is assume script has already run and done it's thing, so just transfer the incoming
// flowfile
getLogger().debug("Successfully executed script from {}", new Object[]{scriptFileName});
}
// update flow file attributes
flowFile = session.putAllAttributes(flowFile, finalScript.getAttributes());
Relationship route = finalScript.getRoute();
if (null == route) {
session.remove(flowFile);
getLogger().info("Removing flowfile {}", new Object[]{flowFile});
} else {
session.transfer(flowFile, route);
getLogger().info("Transferring flowfile {} to {}", new Object[]{flowFile, route});
}
} catch (ScriptException | IOException e) {
getLogger().error("Failed to create script from {} with flowFile {}. Rolling back session.",
new Object[]{scriptFileName, flowFile}, e);
throw new ProcessException(e);
} catch (Exception e) {
if (null != script) {
getLogger().error("Failed to execute script from {}. Transferring flow file {} to {}",
new Object[]{scriptFileName, flowFile, exceptionRoute}, e);
session.transfer(flowFile, exceptionRoute);
} else {
getLogger().error("Failed to execute script from {} with flowFile {}. Rolling back session",
new Object[]{scriptFileName, flowFile}, e);
throw new ProcessException(e);
}
}
}
}

131
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/ConverterScript.java
View File

@ -1,131 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.io.ByteArrayOutputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Map;
import javax.script.Invocable;
import javax.script.ScriptException;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.flowfile.attributes.CoreAttributes;
import org.apache.nifi.io.BufferedInputStream;
import org.apache.nifi.processor.ProcessSession;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.processor.io.InputStreamCallback;
import org.apache.nifi.processor.io.OutputStreamCallback;
/**
* <p>
* Script authors should extend this class if they want to perform complex
* conversions in a NiFi processor.
* </p>
*
* <p>
* Scripts must implement {@link #convert(FileInputStream)}. This method may
* create new FlowFiles and pass them to one or more routes. The input FlowFile
* will be removed from the repository after execution of this method completes.
* </p>
*
* <p>
* In general, the {@link #convert(FileInputStream)} will read from the supplied
* stream, then create one or more output sinks and route the result to the
* relationship of choice using
* {@link #routeStream(ByteArrayOutputStream, String, String)} or
* {@link #routeBytes(byte[], String, String)}.
*
* <p>
* Implement {@link #getProcessorRelationships()} to allow writing to
* relationships other than <code>success</code> and <code>failure</code>. The
* {@link #getRoute()} superclass method is *not* used by Converter Scripts.
* </p>
*
*/
public class ConverterScript extends Script {
private ProcessSession session; // used to create files
private Object convertCallback;
public ConverterScript() {
}
public ConverterScript(Object... callbacks) {
super(callbacks);
for (Object callback : callbacks) {
if (callback instanceof Map<?, ?>) {
convertCallback = convertCallback == null && ((Map<?, ?>) callback).containsKey("convert") ? callback : convertCallback;
}
}
}
// Subclasses should implement this to define basic logic
protected void convert(InputStream stream) throws NoSuchMethodException, ScriptException {
if (convertCallback != null) {
((Invocable) engine).invokeMethod(convertCallback, "convert", stream);
}
}
/**
* Owning processor uses this method to kick off handling of a single file
*
* @param aSession the owning processor's Repository (needed to make new
* files)
*/
public void process(ProcessSession aSession) {
this.session = aSession;
this.session.read(this.flowFile, new InputStreamCallback() {
@Override
public void process(InputStream in) throws IOException {
BufferedInputStream stream = new BufferedInputStream(in);
try {
convert(stream);
} catch (NoSuchMethodException | ScriptException e) {
logger.error("Failed to execute 'convert' function in script", e);
throw new IOException(e);
}
}
});
}
// this should go back to protected once we get Nashorn
public void createFlowFile(final String flowFileName, final Relationship relationship, final OutputStreamHandler handler) {
FlowFile result = session.create(this.flowFile);
result = session.putAttribute(result, CoreAttributes.FILENAME.key(), flowFileName);
try {
result = session.write(result, new OutputStreamCallback() {
@Override
public void process(OutputStream out) throws IOException {
handler.write(out);
}
});
this.logger.info("Transfer flow file {} to {}", new Object[]{result, relationship});
session.transfer(result, relationship);
} catch (Exception e) {
this.logger.error("Could not create new flow file from script", e);
session.remove(result);
}
}
}

46
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/JRubyScriptFactory.java
View File

@ -1,46 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.io.File;
import java.io.IOException;
import org.apache.commons.io.FileUtils;
public enum JRubyScriptFactory {
INSTANCE;
private static final String PRELOADS = "include Java\n"
+ "\n"
+ "java_import 'org.apache.nifi.components.PropertyDescriptor'\n"
+ "java_import 'org.apache.nifi.components.Validator'\n"
+ "java_import 'org.apache.nifi.processor.util.StandardValidators'\n"
+ "java_import 'org.apache.nifi.processor.Relationship'\n"
+ "java_import 'org.apache.nifi.logging.ProcessorLog'\n"
+ "java_import 'org.apache.nifi.scripting.ReaderScript'\n"
+ "java_import 'org.apache.nifi.scripting.WriterScript'\n"
+ "java_import 'org.apache.nifi.scripting.ConverterScript'\n"
+ "\n";
public String getScript(File scriptFile) throws IOException {
StringBuilder sb = new StringBuilder();
sb.append(PRELOADS)
.append(FileUtils.readFileToString(scriptFile, "UTF-8"));
return sb.toString();
}
}

56
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/JavaScriptScriptFactory.java
View File

@ -1,56 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.io.File;
import java.io.IOException;
import org.apache.commons.io.FileUtils;
import org.apache.commons.lang3.StringUtils;
public enum JavaScriptScriptFactory {
INSTANCE;
private static final String PRELOADS = "var Scripting = JavaImporter(\n"
+ " Packages.org.apache.nifi.components,\n"
+ " Packages.org.apache.nifi.processor.util,\n"
+ " Packages.org.apache.nifi.processor,\n"
+ " Packages.org.apache.nifi.logging,\n"
+ " Packages.org.apache.nifi.scripting,\n"
+ " Packages.org.apache.commons.io);\n"
+ "var readFile = function (file) {\n"
+ " var script = Packages.org.apache.commons.io.FileUtils.readFileToString("
+ " new java.io.File($PATH, file)"
+ " );\n"
+ " return \"\" + script;\n"
+ "}\n"
+ "var require = function (file){\n"
+ " var exports={}, module={};\n"
+ " module.__defineGetter__('id', function(){return file;});"
+ " eval(readFile(file));\n"
+ " return exports;\n"
+ "}\n";
public String getScript(File scriptFile) throws IOException {
StringBuilder sb = new StringBuilder();
final String parent = StringUtils.replace(scriptFile.getParent(), "\\", "/");
sb.append(PRELOADS).append("var $PATH = \"").append(parent).append("\"\n")
.append(FileUtils.readFileToString(scriptFile, "UTF-8"));
return sb.toString();
}
}

45
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/JythonScriptFactory.java
View File

@ -1,45 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.io.File;
import java.io.IOException;
import org.apache.commons.io.FileUtils;
public enum JythonScriptFactory {
INSTANCE;
private final static String PRELOADS = "from org.python.core.util import FileUtil\n"
+ "from org.apache.nifi.components import PropertyDescriptor\n"
+ "from org.apache.nifi.components import Validator\n"
+ "from org.apache.nifi.processor.util import StandardValidators\n"
+ "from org.apache.nifi.processor import Relationship\n"
+ "from org.apache.nifi.logging import ProcessorLog\n"
+ "from org.apache.nifi.scripting import ReaderScript\n"
+ "from org.apache.nifi.scripting import WriterScript\n"
+ "from org.apache.nifi.scripting import ConverterScript\n";
public String getScript(File scriptFile) throws IOException {
StringBuilder sb = new StringBuilder();
sb.append(PRELOADS)
.append(FileUtils.readFileToString(scriptFile, "UTF-8"));
return sb.toString();
}
}

24
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/OutputStreamHandler.java
View File

@ -1,24 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.io.OutputStream;
public interface OutputStreamHandler {
void write(OutputStream out);
}

79
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/ReaderScript.java
View File

@ -1,79 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.io.InputStream;
import java.util.Map;
import javax.script.Invocable;
import javax.script.ScriptException;
import org.apache.nifi.processor.Relationship;
/**
* <p>
* Script authors should extend this class if they want to follow the "reader"
* paradigm for NiFi processors.
* </p>
*
* <p>
* User scripts should implement {@link #route(InputStream)}. <code>route</code>
* uses a returned relationship name to determine where FlowFiles go. Scripts
* may also implement {@link #getProcessorRelationships()} to specify available
* relationship names.
* </p>
*
*/
public class ReaderScript extends Script {
private Object routeCallback;
public ReaderScript(Object... callbacks) {
super(callbacks);
for (Object callback : callbacks) {
if (callback instanceof Map<?, ?>) {
routeCallback = routeCallback == null && ((Map<?, ?>) callback).containsKey("route") ? callback : routeCallback;
}
}
}
public ReaderScript() {
}
// Simple helper
public void process(InputStream input) throws NoSuchMethodException, ScriptException {
lastRoute = route(input);
}
/**
* Subclasses should examine the provided inputstream, then determine which
* relationship the file will be sent down and return its name.
*
*
* @param in a Java InputStream containing the incoming FlowFile.
* @return a relationship name
* @throws ScriptException
* @throws NoSuchMethodException
*/
public Relationship route(InputStream in) throws NoSuchMethodException, ScriptException {
Relationship relationship = null;
Invocable invocable = (Invocable) this.engine;
relationship = (Relationship) invocable.invokeMethod(routeCallback, "route", in);
return relationship;
}
}

303
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/Script.java
View File

@ -1,303 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import javax.script.Invocable;
import javax.script.ScriptEngine;
import javax.script.ScriptException;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.logging.ProcessorLog;
import org.apache.nifi.processor.Relationship;
/**
* <p>
* Base class for all scripts. In this framework, only ScriptEngines that
* implement javax.script.Invocable are supported.
*
* </p>
*
*/
public class Script {
public static final Relationship SUCCESS_RELATIONSHIP = new Relationship.Builder()
.name("success")
.description("Destination of successfully created flow files")
.build();
public static final Relationship FAIL_RELATIONSHIP = new Relationship.Builder()
.name("failure")
.description("Destination of flow files when a error occurs in the script")
.build();
static final Set<Relationship> RELATIONSHIPS;
static {
Set<Relationship> rels = new HashSet<>();
rels.add(FAIL_RELATIONSHIP);
rels.add(SUCCESS_RELATIONSHIP);
RELATIONSHIPS = Collections.unmodifiableSet(rels);
}
FlowFile flowFile = null;
ScriptEngine engine = null;
protected Map<String, String> properties = new HashMap<>();
protected Relationship lastRoute = SUCCESS_RELATIONSHIP;
protected ProcessorLog logger;
protected String scriptFileName;
protected Map<String, String> attributes = new HashMap<>();
protected long flowFileSize = 0;
protected long flowFileEntryDate = System.currentTimeMillis();
// the following are needed due to an inadequate JavaScript ScriptEngine. It will not allow
// subclassing a Java Class, only implementing a Java Interface. So, the syntax of JavaScript
// scripts looks like subclassing, but actually is just constructing a Script instance and
// passing in functions as args to the constructor. When we move to Nashorn JavaScript ScriptEngine
// in Java 8, we can get rid of these and revert the subclasses of this class to abstract.
protected Object propDescCallback;
protected Object relationshipsCallback;
protected Object validateCallback;
protected Object exceptionRouteCallback;
/**
* Create a Script without any parameters
*/
public Script() {
}
public Script(Object... callbacks) {
for (Object callback : callbacks) {
if (callback instanceof Map<?, ?>) {
propDescCallback = propDescCallback == null && ((Map<?, ?>) callback).containsKey("getPropertyDescriptors") ? callback
: propDescCallback;
relationshipsCallback = relationshipsCallback == null && ((Map<?, ?>) callback).containsKey("getRelationships") ? callback
: relationshipsCallback;
validateCallback = validateCallback == null && ((Map<?, ?>) callback).containsKey("validate") ? callback : validateCallback;
exceptionRouteCallback = exceptionRouteCallback == null && ((Map<?, ?>) callback).containsKey("getExceptionRoute") ? callback
: exceptionRouteCallback;
}
}
}
/**
* Specify a set of properties with corresponding NiFi validators.
*
* Subclasses that do not override this method will still have access to all
* properties via the "properties" field
*
* @return a list of PropertyDescriptors
* @throws ScriptException
* @throws NoSuchMethodException
*/
@SuppressWarnings("unchecked")
public List<PropertyDescriptor> getPropertyDescriptors() throws NoSuchMethodException, ScriptException {
if (propDescCallback != null) {
return (List<PropertyDescriptor>) ((Invocable) engine).invokeMethod(propDescCallback, "getPropertyDescriptors", (Object) null);
}
return Collections.emptyList();
}
/**
* Specify a set of reasons why this processor should be invalid.
*
* Subclasses that do not override this method will depend only on
* individual property validators as specified in
* {@link #getPropertyDescriptors()}.
*
* @return a Collection of messages to display to the user, or an empty
* Collection if the processor configuration is OK.
* @throws ScriptException
* @throws NoSuchMethodException
*/
@SuppressWarnings("unchecked")
public Collection<String> validate() throws NoSuchMethodException, ScriptException {
if (validateCallback != null) {
return (Collection<String>) ((Invocable) engine).invokeMethod(validateCallback, "validate", (Object) null);
}
return Collections.emptyList();
}
void setFlowFile(FlowFile ff) {
flowFile = ff;
if (null != ff) {
// have to clone because ff.getAttributes is unmodifiable
this.attributes = new HashMap<>(ff.getAttributes());
this.flowFileSize = ff.getSize();
this.flowFileEntryDate = ff.getEntryDate();
}
}
void setProperties(Map<String, String> map) {
properties = new HashMap<>(map);
}
/**
* Required to access entire properties map -- Jython (at least) won't let
* you read the member variable without a getter
*
* @return entire parameter map
*/
// change back to protected when we get nashorn
public Map<String, String> getProperties() {
return properties;
}
/**
* Get the named parameter. Some scripting languages make a method call
* easier than accessing a member field, so this is a convenience method to
* look up values in the properties field.
*
* @param key a hash key
* @return the value pointed at by the key specified
*/
public String getProperty(String key) {
return properties.get(key);
}
/**
* Name the various relationships by which a file can leave this processor.
* Subclasses may override this method to change available relationships.
*
* @return a collection of relationship names
* @throws ScriptException
* @throws NoSuchMethodException
*/
@SuppressWarnings("unchecked")
public Collection<Relationship> getRelationships() throws NoSuchMethodException, ScriptException {
if (relationshipsCallback != null) {
return (Collection<Relationship>) ((Invocable) engine).invokeMethod(relationshipsCallback, "getRelationships", (Object) null);
}
return RELATIONSHIPS;
}
/**
* Determine what do with a file that has just been processed.
*
* After a script runs its "read" or "write" method, it should update the
* "lastRoute" field to specify the relationship to which the resulting file
* will be sent.
*
* @return a relationship name
*/
public Relationship getRoute() {
return lastRoute;
}
// Required because of a potential issue in Rhino -- protected methods are visible in
// subclasses but protected fields (like "lastRoute") are not
// change back to protected when we get nashorn
public void setRoute(Relationship route) {
lastRoute = route;
}
/**
* Determine where to send a file if an exception is thrown during
* processing.
*
* Subclasses may override this method to use a different relationship, or
* to determine the relationship dynamically. Returning null causes the file
* to be deleted instead.
*
* Defaults to "failure".
*
* @return the name of the relationship to use in event of an exception, or
* null to delete the file.
* @throws ScriptException
* @throws NoSuchMethodException
*/
public Relationship getExceptionRoute() throws NoSuchMethodException, ScriptException {
if (exceptionRouteCallback != null) {
return (Relationship) ((Invocable) engine).invokeMethod(exceptionRouteCallback, "getExceptionRoute", (Object) null);
}
return FAIL_RELATIONSHIP;
}
/*
* Some scripting languages make a method call easier than accessing a member field, so this is a convenience method to get
* the incoming flow file size.
*/
// Change back to protected when we get nashorn
public long getFlowFileSize() {
return flowFileSize;
}
/*
* Some scripting languages make a method call easier than accessing a member field, so this is a convenience method to get
* entry date of the flow file.
*/
// Change back to protected when we get nashorn
public long getFlowFileEntryDate() {
return flowFileEntryDate;
}
void setLogger(ProcessorLog logger) {
this.logger = logger;
}
/*
* Required so that scripts in some languages can read access the attribute. Jython (at least) won't let you read the member
* variable without a getter
*/
protected ProcessorLog getLogger() {
return this.logger;
}
void setFileName(String scriptFileName) {
this.scriptFileName = scriptFileName;
}
public String getFileName() {
return this.scriptFileName;
}
// this one's public because it's needed by ExecuteScript to update the flow file's attributes AFTER processing is done
public Map<String, String> getAttributes() {
return this.attributes;
}
/*
* Some scripting languages make a method call easier than accessing a member field, so this is a convenience method to look
* up values in the attributes field.
*/
// Change back to protected when we get nashorn
public String getAttribute(String key) {
return this.attributes.get(key);
}
/*
* Some scripting languages make a method call easier than accessing a member field, so this is a convenience method to set
* key/value pairs in the attributes field.
*/
// Change back to protected when we get nashorn
public void setAttribute(String key, String value) {
this.attributes.put(key, value);
}
void setEngine(ScriptEngine scriptEngine) {
this.engine = scriptEngine;
}
}

117
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/ScriptEngineFactory.java
View File

@ -1,117 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.io.File;
import java.util.concurrent.ConcurrentHashMap;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;
import javax.script.ScriptException;
import org.apache.commons.lang3.StringUtils;
import org.jruby.embed.PropertyName;
public class ScriptEngineFactory {
private static final String THREADING = "THREADING";
private static final String MULTITHREADED = "MULTITHREADED";
private static final String STATELESS = "STATELESS";
private static final String THREAD_ISOLATED = "THREAD-ISOLATED";
final static ScriptEngineManager scriptEngMgr;
static {
System.setProperty(PropertyName.LOCALCONTEXT_SCOPE.toString(), "singlethread");
System.setProperty(PropertyName.COMPILEMODE.toString(), "jit");
System.setProperty(PropertyName.COMPATVERSION.toString(), "JRuby1.9");
System.setProperty(PropertyName.LOCALVARIABLE_BEHAVIOR.toString(), "transient");
System.setProperty("compile.invokedynamic", "false");
System.setProperty(PropertyName.LAZINESS.toString(), "true");
scriptEngMgr = new ScriptEngineManager();
}
final ConcurrentHashMap<String, ScriptEngine> threadSafeEngines = new ConcurrentHashMap<>();
ScriptEngine getEngine(String extension) {
ScriptEngine engine = threadSafeEngines.get(extension);
if (null == engine) {
engine = scriptEngMgr.getEngineByExtension(extension);
if (null == engine) {
throw new IllegalArgumentException("No ScriptEngine exists for extension " + extension);
}
Object threading = engine.getFactory().getParameter(THREADING);
// the MULTITHREADED status means that the scripts need to be careful about sharing state
if (THREAD_ISOLATED.equals(threading) || STATELESS.equals(threading) || MULTITHREADED.equals(threading)) {
ScriptEngine cachedEngine = threadSafeEngines.putIfAbsent(extension, engine);
if (null != cachedEngine) {
engine = cachedEngine;
}
}
}
return engine;
}
ScriptEngine getNewEngine(File scriptFile, String extension) throws ScriptException {
ScriptEngine engine = scriptEngMgr.getEngineByExtension(extension);
if (null == engine) {
throw new IllegalArgumentException("No ScriptEngine exists for extension " + extension);
}
// Initialize some paths
StringBuilder sb = new StringBuilder();
switch (extension) {
case "rb":
String parent = scriptFile.getParent();
parent = StringUtils.replace(parent, "\\", "/");
sb.append("$:.unshift '")
.append(parent)
.append("'\n")
.append("$:.unshift File.join '")
.append(parent)
.append("', 'lib'\n");
engine.eval(sb.toString());
break;
case "py":
parent = scriptFile.getParent();
parent = StringUtils.replace(parent, "\\", "/");
String lib = parent + "/lib";
sb.append("import sys\n").append("sys.path.append('").append(parent)
.append("')\n").append("sys.path.append('")
.append(lib)
.append("')\n")
.append("__file__ = '")
.append(scriptFile.getAbsolutePath())
.append("'\n");
engine.eval(sb.toString());
break;
default:
break;
}
Object threading = engine.getFactory().getParameter(THREADING);
// the MULTITHREADED status means that the scripts need to be careful about sharing state
if (THREAD_ISOLATED.equals(threading) || STATELESS.equals(threading) || MULTITHREADED.equals(threading)) {
// replace prior instance if any
threadSafeEngines.put(extension, engine);
}
return engine;
}
boolean isThreadSafe(String scriptExtension) {
return threadSafeEngines.containsKey(scriptExtension);
}
}

269
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/ScriptFactory.java
View File

@ -1,269 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.security.DigestInputStream;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock.ReadLock;
import java.util.concurrent.locks.ReentrantReadWriteLock.WriteLock;
import javax.script.Bindings;
import javax.script.Compilable;
import javax.script.CompiledScript;
import javax.script.ScriptContext;
import javax.script.ScriptEngine;
import javax.script.ScriptException;
import javax.script.SimpleBindings;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.io.BufferedInputStream;
import org.apache.nifi.logging.ProcessorLog;
import org.apache.commons.io.FileUtils;
/**
* While this is a 'factory', it is not a singleton because we want a factory
* per processor. This factory has state, all of which belong to only one
* processor.
*
*/
public class ScriptFactory {
private final ScriptEngineFactory engineFactory = new ScriptEngineFactory();
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock();
private final ReadLock readLock = lock.readLock();
private final WriteLock writeLock = lock.writeLock();
private final ProcessorLog logger;
private volatile CompiledScript compiledScript;
private volatile String scriptText;
private volatile byte[] md5Hash;
private volatile long lastTimeChecked;
private volatile String scriptFileName;
private volatile long scriptCheckIntervalMS = 15000;
public ScriptFactory(ProcessorLog logger) {
this.logger = logger;
}
public void setScriptCheckIntervalMS(long msecs) {
this.scriptCheckIntervalMS = msecs;
}
/**
* @param aScriptFileName
* @param properties
* @param flowFile
* @return
* @throws IOException
* @throws ScriptException
*/
public Script getScript(final String aScriptFileName, final Map<String, String> properties, final FlowFile flowFile)
throws IOException, ScriptException {
final Script instance;
long now = System.currentTimeMillis();
readLock.lock();
try {
if (!aScriptFileName.equals(this.scriptFileName)) {
readLock.unlock();
writeLock.lock();
try {
if (!aScriptFileName.equals(this.scriptFileName)) {
// need to get brand new engine
compiledScript = null;
this.md5Hash = getMD5Hash(aScriptFileName);
this.lastTimeChecked = now;
this.scriptFileName = aScriptFileName;
updateEngine();
} // else another thread beat me to the change...so just get a script
} finally {
readLock.lock();
writeLock.unlock();
}
} else if (lastTimeChecked + scriptCheckIntervalMS < now) {
readLock.unlock();
writeLock.lock();
try {
if (lastTimeChecked + scriptCheckIntervalMS < now) {
byte[] md5 = getMD5Hash(this.scriptFileName);
if (!MessageDigest.isEqual(md5Hash, md5)) {
// need to get brand new engine
compiledScript = null;
updateEngine();
this.md5Hash = md5;
} // else no change to script, so just update time checked
this.lastTimeChecked = now;
} // else another thread beat me to the check...so just get a script
} finally {
readLock.lock();
writeLock.unlock();
}
}
try {
instance = getScriptInstance(properties);
instance.setFileName(this.scriptFileName);
instance.setProperties(properties);
instance.setLogger(logger);
instance.setFlowFile(flowFile);
} catch (ScriptException e) {
// need to reset state to enable re-initialization
this.lastTimeChecked = 0;
this.scriptFileName = null;
throw e;
}
} finally {
readLock.unlock();
}
return instance;
}
public Script getScript(String aScriptFileName) throws ScriptException, IOException {
Map<String, String> props = new HashMap<>();
return getScript(aScriptFileName, props, null);
}
private byte[] getMD5Hash(String aScriptFileName) throws FileNotFoundException, IOException {
byte[] messageDigest = null;
try (FileInputStream fis = new FileInputStream(aScriptFileName);
DigestInputStream dis = new DigestInputStream(new BufferedInputStream(fis), MessageDigest.getInstance("MD5"))) {
byte[] bytes = new byte[8192];
while (dis.read(bytes) != -1) {
// do nothing...just computing the md5 hash
}
messageDigest = dis.getMessageDigest().digest();
} catch (NoSuchAlgorithmException swallow) {
// MD5 is a legitimate format
}
return messageDigest;
}
private String getScriptText(File scriptFile, String extension) throws IOException {
final String script;
switch (extension) {
case "rb":
script = JRubyScriptFactory.INSTANCE.getScript(scriptFile);
break;
case "js":
script = JavaScriptScriptFactory.INSTANCE.getScript(scriptFile);
break;
case "py":
script = JythonScriptFactory.INSTANCE.getScript(scriptFile);
break;
default:
script = FileUtils.readFileToString(scriptFile);
}
return script;
}
private Script getScriptInstance(final Map<String, String> properties) throws ScriptException {
Map<String, Object> localThreadVariables = new HashMap<>();
final String extension = getExtension(scriptFileName);
String loggerVariableKey = getVariableName("GLOBAL", "logger", extension);
localThreadVariables.put(loggerVariableKey, logger);
String propertiesVariableKey = getVariableName("INSTANCE", "properties", extension);
localThreadVariables.put(propertiesVariableKey, properties);
localThreadVariables.put(ScriptEngine.FILENAME, scriptFileName);
final Bindings bindings = new SimpleBindings(localThreadVariables);
final ScriptEngine scriptEngine = engineFactory.getEngine(extension);
Script instance;
if (compiledScript == null) {
instance = (Script) scriptEngine.eval(scriptText, bindings);
if (instance == null) { // which it will be for python and also for local variables in javascript
instance = (Script) scriptEngine.eval("instance", bindings);
}
} else {
instance = (Script) compiledScript.eval(bindings);
if (instance == null) { // which it will be for python and also for local variables in javascript
instance = (Script) compiledScript.getEngine().eval("instance", bindings);
}
}
instance.setEngine(scriptEngine);
return instance;
}
/*
* Must have writeLock when calling this!!!!
*/
private void updateEngine() throws IOException, ScriptException {
final String extension = getExtension(scriptFileName);
// if engine is thread safe, it's being reused...if it's a JrubyEngine it
File scriptFile = new File(this.scriptFileName);
ScriptEngine scriptEngine = engineFactory.getNewEngine(scriptFile, extension);
scriptText = getScriptText(scriptFile, extension);
Map<String, Object> localThreadVariables = new HashMap<>();
String loggerVariableKey = getVariableName("GLOBAL", "logger", extension);
localThreadVariables.put(loggerVariableKey, logger);
String propertiesVariableKey = getVariableName("INSTANCE", "properties", extension);
localThreadVariables.put(propertiesVariableKey, new HashMap<String, String>());
localThreadVariables.put(ScriptEngine.FILENAME, scriptFileName);
if (scriptEngine instanceof Compilable) {
Bindings bindings = new SimpleBindings(localThreadVariables);
scriptEngine.setBindings(bindings, ScriptContext.ENGINE_SCOPE);
compiledScript = ((Compilable) scriptEngine).compile(scriptText);
}
logger.debug("Updating Engine!!");
}
private String getVariableName(String scope, String variableName, String extension) {
String result;
switch (extension) {
case "rb":
switch (scope) {
case "GLOBAL":
result = '$' + variableName;
break;
case "INSTANCE":
result = '@' + variableName;
break;
default:
result = variableName;
break;
}
break;
default:
result = variableName;
break;
}
return result;
}
private String getExtension(String aScriptFileName) {
int dotPos = aScriptFileName.lastIndexOf('.');
if (dotPos < 1) {
throw new IllegalArgumentException("Script file name must have an extension");
}
final String extension = aScriptFileName.substring(dotPos + 1);
return extension;
}
}

67
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/java/org/apache/nifi/scripting/WriterScript.java
View File

@ -1,67 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.scripting;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Map;
import javax.script.Invocable;
import javax.script.ScriptException;
/**
* <p>
* Script authors should extend this class if they want to follow the
* "processCallback" paradigm for NiFi processors.
* </p>
*
* <p>
* At a minimum, scripts must implement
* <code>process(FileInputStream, FileOutputStream)</code>.
* </p>
*
* <p>
* By default, all files processed will be sent to the relationship
* <em>success</em>, unless the scriptFileName raises an exception, in which
* case the file will be sent to <em>failure</em>. Implement
* {@link #getProcessorRelationships()} and/or {@link #getRoute()} to change
* this behavior.
* </p>
*
*/
public class WriterScript extends Script {
private Object processCallback;
public WriterScript() {
}
public WriterScript(Object... callbacks) {
super(callbacks);
for (Object callback : callbacks) {
if (callback instanceof Map<?, ?>) {
processCallback = processCallback == null && ((Map<?, ?>) callback).containsKey("process") ? callback : processCallback;
}
}
}
public void process(InputStream in, OutputStream out) throws NoSuchMethodException, ScriptException {
Invocable inv = (Invocable) engine;
inv.invokeMethod(processCallback, "process", in, out);
}
}

15
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor
View File

@ -1,15 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
org.apache.nifi.processors.script.ExecuteScript

264
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/main/resources/docs/org.apache.nifi.processors.script.ExecuteScript/index.html
View File

@ -1,264 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<head>
<meta charset="utf-8" />
<title>ExecuteScript</title>
<link rel="stylesheet" href="../../css/component-usage.css" type="text/css" />
</head>
<body>
<!-- Processor Documentation ================================================== -->
<h2>Description:</h2>
<p>
This processor provides the capability to execute scripts in various scripting languages, and passes into the scripts
the input stream and output stream(s) representing an incoming flow file and any created flow files. The processor is designed to be
thread safe, so multiple concurrent tasks may execute against a single script. The processor provides a framework which enables
script writers to implement 3 different types of scripts:
<ul>
ReaderScript - which enables stream-based reading of a FlowFile's content</br>
WriterScript - which enables stream-based reading and writing/modifying of a FlowFile's content</br>
ConverterScript - which enables stream-based reading a FlowFile's content and stream-based writing to newly created FlowFiles</br>
</ul>
Presently, the processor supports 3 scripting languages: Ruby, Python, and JavaScript. The processor is built on the
javax.script API which enables ScriptEngine discovery, thread management, and encapsulates much of the low level bridging-code that
enables Java to Script language integration. Thus, it is designed to be easily extended to other scripting languages. </br>
The attributes of a FlowFile and properties of the Processor are exposed to the script by either a variable in the base class or
a getter method. A script may declare new Processor Properties and different Relationships via overriding the getPropertyDescriptors
and getRelationships methods, respectively.
</p>
The processor provides some boilerplate script to aid in the creation of the three different types of scripts. For example,
the processor provides import statements for classes commonly used within a processor.
<pre>
'org.apache.nifi.components.PropertyDescriptor'
'org.apache.nifi.components.Validator'
'org.apache.nifi.processor.util.StandardValidators'
'org.apache.nifi.processor.Relationship'
'org.apache.nifi.logging.ProcessorLog'
'org.apache.nifi.scripting.ReaderScript'
'org.apache.nifi.scripting.WriterScript'
'org.apache.nifi.scripting.ConverterScript'
</pre>
The processor appends to the script's execution path the parent directory of the specified script file and a sub-directory
called 'lib', which may be useful for supporting scripts. </p>
<p>
<strong>Shared Variables</strong>
</p>
The following variables are provided as shared variables for the scripts:
<ul>
<li>logger
<ul>
<li> The processor's logger </li>
<li> Scope is GLOBAL, thus in Ruby the syntax is $logger</li>
</ul>
</li>
<li>properties
<ul>
<li> A Map of the processor's configuration properties; key and value are strings</li>
<li> Scope is INSTANCE, thus in Ruby the syntax is @properties</li>
</ul>
</li>
</ul>
<p>
<strong>Properties:</strong>
</p>
<p>
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered
optional. If a property has a default value, it is indicated. If a property supports the use of the NiFi Expression Language
(or simply, "expression language"), that is also indicated. Of particular note: This processor allows scripts to define additional
Processor properties, which will not be initially visible. Once the processor's configuration is validated, script defined properties
will become visible, and may affect the validity of the processor.
</p>
<ul>
<li>
<strong>Script File Name</strong>
<ul>
<li>Script location, can be relative or absolute path.</li>
<li>Default value: no default</li>
<li>Supports expression language: false</li>
</ul>
</li>
<li>
<strong>Script Check Interval</strong>
<ul>
<li>The time period between checking for updates to a script.</li>
<li>Default value: 15 sec</li>
<li>Supports expression language: false</li>
</ul>
</li>
</ul>
<p>
<strong>Relationships:</strong>
</p>
<p>
The initial 'out of the box' relationships are below. Of particular note is the ability of a script to change the set of
relationships. However, any relationships defined by the script will not be visible until the processor's configuration has been
validated. Once done, new relationships will become visible.
</p>
<ul>
<li>
success
<ul>
<li>Used when a file is successfully processed by a script.</li>
</ul>
</li>
<li>
failure
<ul>
<li>Used when an error occurs while processing a file with a script.</li>
</ul>
</li>
</ul>
<p>
<strong>Example Scripts:</strong>
</p>
<ul>
JavaScript example - the 'with' statement imports packages defined in the framework and limits the importing to the local scope,
rather than global. The 'Scripting' variable uses the JavaImporter class within JavaScript. Since the 'instance' variable is intended to
be local scope (not global), it must be named 'instance' as it it not passed back to the processor upon script evaluation and must be
fetched. If you make it global, you can name it whatever you'd like...but this is intended to be multi-threaded so do so at your own
risk.</p>
Presently, there are issues with the JavaScript scripting engine that prevent sub-classing the base classes in the Processor's Java
framework. So, what is actually happening is an instance of the ReaderScript is created with a provided callback object. When we are able
to move to a more competent scripting engine (supposedly in Java 8), the code below will remain the same, but the 'instance' variable
will actually be a sub-class of ReaderScript.
<pre>
with (Scripting) {
var instance = new ReaderScript({
route : function(input) {
var str = IOUtils.toString(input);
var expr = instance.getProperty("expr");
filename = instance.attributes.get("filename");
instance.setAttribute("filename", filename + ".modified");
if (str.match(expr)) {
return Script.FAIL_RELATIONSHIP;
} else {
return Script.SUCCESS_RELATIONSHIP;
}
}
});
}
</pre>
Ruby example - the 'OutputStreamHandler' is an interface which is called when creating flow files.
<pre>
java_import 'org.apache.nifi.scripting.OutputStreamHandler'
class SimpleConverter < ConverterScript
field_reader :FAIL_RELATIONSHIP, :SUCCESS_RELATIONSHIP, :logger, :attributes
def convert(input)
in_io = input.to_io
createFlowFile("firstLine", FAIL_RELATIONSHIP, OutputStreamHandler.impl do |method, out|
out_io = out.to_io
out_io << in_io.readline.to_java_bytes
out_io.close
logger.debug("Wrote data to failure...this message logged with logger from super class")
end)
createFlowFile("otherLines", SUCCESS_RELATIONSHIP, OutputStreamHandler.impl do |method, out|
out_io = out.to_io
in_io.each_line { |line|
out_io << line
}
out_io.close
logger.debug("Wrote data to success...this message logged with logger from super class")
end)
in_io.close
end
end
$logger.debug("Creating SimpleConverter...this message logged with logger from shared variables")
SimpleConverter.new
</pre>
Python example - The difficulty with Python is that it does not return objects upon script evaluation, so the instance of the Script
class must be fetched by name. Thus, you must define a variable called 'instance'.
<pre>
import re
class RoutingReader(ReaderScript):
A = Relationship.Builder().name("a").description("some good stuff").build()
B = Relationship.Builder().name("b").description("some other stuff").build()
C = Relationship.Builder().name("c").description("some bad stuff").build()
def getRelationships(self):
return [self.A,self.B,self.C]
def getExceptionRoute(self):
return self.C
def route( self, input ):
logger.info("Executing route")
for line in FileUtil.wrap(input):
if re.match("^bad", line, re.IGNORECASE):
return self.B
if re.match("^sed", line):
raise RuntimeError("That's no good!")
return self.A
logger.debug("Constructing instance")
instance = RoutingReader()
</pre>
</ul>
<p>
<strong>Script API:</strong>
</p>
<ul>
<li>getAttribute(String) : String</li>
<li>getAttributes() : Map(String,String)</li>
<li>getExceptionRoute() : Relationship</li>
<li>getFileName() : String</li>
<li>getFlowFileEntryDate() : Calendar</li>
<li>getFlowFileSize() : long</li>
<li>getProperties() : Map(String, String)</li>
<li>getProperty(String) : String</li>
<li>getPropertyDescriptors() : List(PropertyDescriptor)</li>
<li>getRelationships() : Collection(Relationship)</li>
<li>getRoute() : Relationship</li>
<li>setRoute(Relationship)</li>
<li>setAttribute(String, String)</li>
<li>validate() : Collection(String)</li>
</ul>
<p>
<strong>ReaderScript API:</strong>
</p>
<ul>
<li>route(InputStream) : Relationship</li>
</ul>
<p>
<strong>WriterScript API:</strong>
</p>
<ul>
<li>process(InputStream, OutputStream)</li>
</ul>
<p>
<strong>ConverterScript API:</strong>
</p>
<ul>
<li>convert(InputStream)</li>
<li>createFlowFile(String, Relationship, OutputStreamHandler)</li>
</ul>
<p>
<strong>OutputStreamHandler API:</strong>
</p>
<ul>
<li>write(OutputStream)</li>
</ul>
</body>
</html>

939
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/java/org/apache/nifi/processors/script/TestExecuteScript.java
View File

@ -1,939 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.processors.script;
import org.apache.nifi.processors.script.ExecuteScript;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import java.io.File;
import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.util.MockFlowFile;
import org.apache.nifi.util.TestRunner;
import org.apache.nifi.util.TestRunners;
import org.apache.commons.io.FileUtils;
import org.junit.Before;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
* @author unattributed
*
*/
public class TestExecuteScript {
static Logger LOG;
static {
System.setProperty("org.slf4j.simpleLogger.defaultLogLevel", "info");
System.setProperty("org.slf4j.simpleLogger.showDateTime", "true");
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.script.ExecuteScript", "trace");
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.script.TestExecuteScript", "debug");
System.setProperty("org.slf4j.simpleLogger.log.nifi.processors.AbstractProcessor", "debug");
LOG = LoggerFactory.getLogger(TestExecuteScript.class);
}
private TestRunner controller;
private final String multiline = "Lorem ipsum dolor sit amet,\n"
+ "consectetur adipisicing elit,\n"
+ "sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.\n"
+ "Ut enim ad minim veniam,\n"
+ "quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.\n"
+ "Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.\n"
+ "Excepteur sint occaecat cupidatat non proident,\n"
+ "sunt in culpa qui officia deserunt mollit anim id est laborum.";
/**
* Create a mock SingleProcessorController using our processor and pass data
* to it via byte array. Returns the Sink that provides access to any files
* that pass out of the processor
*/
@Before
public void setupEach() throws IOException {
controller = TestRunners.newTestRunner(ExecuteScript.class);
controller.setValidateExpressionUsage(false);
// copy all scripts to target directory and run from there. some python
// scripts create .class files that end up in src/test/resources.
FileUtils.copyDirectory(new File("src/test/resources"), new File("target/test-scripts"));
}
// Fail if the specified relationship does not contain exactly one file
// with the expected value
private void assertRelationshipContents(String expected, String relationship) {
controller.assertTransferCount(relationship, 1);
MockFlowFile ff = controller.getFlowFilesForRelationship(relationship).get(0);
ff.assertContentEquals(expected);
}
// Fail if the specified relationship does not contain specified number of files
// with the expected value
private void assertRelationshipContents(String expected, String relationship, int count) {
controller.assertTransferCount(relationship, count);
MockFlowFile ff = controller.getFlowFilesForRelationship(relationship).get(count - 1);
ff.assertContentEquals(expected);
}
// ////////////////////////////////////
// General tests
@Test(expected = IllegalArgumentException.class)
public void failOnBadName() {
LOG.info("Supplying bad script file names");
// None of these should result in actually setting the property, because they're non-existent / bad files
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "not/really.rb");
controller.assertNotValid();
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "fakey/fake.js");
controller.assertNotValid();
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "pom.xml");
controller.assertNotValid();
}
// ////////////////////////////////////
// Ruby script tests
@Test
public void testSimpleReadR() {
LOG.info("Ruby script: fail file based on reading contents");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readTest.rb");
controller.setThreadCount(2);
controller.run(2);
assertRelationshipContents(multiline, "failure");
assertRelationshipContents("This stuff is fine", "success");
controller.getFlowFilesForRelationship("success").get(0).assertAttributeEquals("filename", "NewFileNameFromReadTest");
}
@Test
public void testParamReadR() {
LOG.info("Ruby script: Failing file based on reading contents");
Map<String, String> attrs1 = new HashMap<>();
attrs1.put("filename", "StuffIsFine.txt");
Map<String, String> attrs2 = new HashMap<>();
attrs2.put("filename", "multiline.txt");
controller.enqueue("This stuff is fine".getBytes(), attrs1);
controller.enqueue(multiline.getBytes(), attrs2);
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readWithParams.rb");
controller.setProperty("expr", "rehenderit");
controller.run(2);
assertRelationshipContents(multiline, "failure");
assertRelationshipContents("This stuff is fine", "success");
}
@Test
public void testWriteLastLineR() {
LOG.info("Running Ruby script to output last line of file");
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/writeTest.rb");
controller.run();
List<MockFlowFile> files = controller.getFlowFilesForRelationship("success");
assertEquals("Process did not generate an output file", 1, files.size());
byte[] blob = files.get(0).toByteArray();
String[] lines = new String(blob).split("\n");
assertEquals("File had more than one line", 1, lines.length);
assertEquals("sunt in culpa qui officia deserunt mollit anim id est laborum.", lines[0]);
}
@Test
public void testWriteOptionalParametersR() {
LOG.info("Ruby script that uses optional parameters");
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/paramTest.rb");
controller.setProperty("repeat", "3");
controller.run();
List<MockFlowFile> files = controller.getFlowFilesForRelationship("success");
assertEquals("Process did not generate an output file", 1, files.size());
byte[] blob = files.get(0).toByteArray();
String[] lines = new String(blob).split("\n");
assertEquals("File did not have 3 lines", 3, lines.length);
assertEquals("sunt in culpa qui officia deserunt mollit anim id est laborum.", lines[0]);
}
@Test
public void testSetupOptionalValidationR() {
LOG.info("Ruby script creating validators for optional properties");
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/optionalValidators.rb");
controller.assertNotValid();
controller.setProperty("int", "abc");
controller.assertNotValid();
controller.setProperty("url", "not@valid");
controller.assertNotValid();
controller.setProperty("nonEmpty", "");
controller.assertNotValid();
controller.setProperty("int", "123");
controller.setProperty("url", "http://localhost");
controller.setProperty("nonEmpty", "abc123");
controller.assertValid();
}
@Test
public void testTwoScriptsSameThreadSameClassName() {
LOG.info("Test 2 different scripts with the same ruby class name");
Map<String, String> attrs1 = new HashMap<>();
attrs1.put("filename", "StuffIsFine.txt");
Map<String, String> attrs2 = new HashMap<>();
attrs2.put("filename", "multiline.txt");
controller.enqueue("This stuff is fine".getBytes(), attrs1);
controller.enqueue(multiline.getBytes(), attrs2);
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readWithParams.rb");
controller.setProperty("expr", "rehenderit");
controller.run(2);
assertRelationshipContents(multiline, "failure");
assertRelationshipContents("This stuff is fine", "success");
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/optionalValidators.rb");
controller.assertNotValid();
controller.setProperty("int", "abc");
controller.assertNotValid();
controller.setProperty("url", "not@valid");
controller.assertNotValid();
controller.setProperty("nonEmpty", "");
controller.assertNotValid();
controller.setProperty("int", "123");
controller.setProperty("url", "http://localhost");
controller.setProperty("nonEmpty", "abc123");
controller.assertValid();
}
@Test
public void testUpdateScriptR() throws Exception {
LOG.info("Test one script with updated class");
File testFile = File.createTempFile("script", ".rb");
File original = new File("target/test-scripts/readWithParams.rb");
FileUtils.copyFile(original, testFile);
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, testFile.getPath());
controller.assertValid();
original = new File("target/test-scripts/optionalValidators.rb");
FileUtils.copyFile(original, testFile);
controller.setProperty(ExecuteScript.SCRIPT_CHECK_INTERVAL, "5 secs");
Thread.sleep(6000);
controller.assertNotValid();
controller.setProperty("int", "abc");
controller.assertNotValid();
controller.setProperty("url", "not@valid");
controller.assertNotValid();
controller.setProperty("nonEmpty", "");
controller.assertNotValid();
controller.setProperty("int", "123");
controller.setProperty("url", "http://localhost");
controller.setProperty("nonEmpty", "abc123");
controller.assertValid();
FileUtils.deleteQuietly(testFile);
}
@Test
public void testMultiThreadExecR() {
LOG.info("Ruby script 20 threads: Failing file based on reading contents");
Map<String, String> attrs1 = new HashMap<>();
attrs1.put("filename", "StuffIsFine.txt");
Map<String, String> attrs2 = new HashMap<>();
attrs2.put("filename", "multiline.txt");
controller.setThreadCount(20);
for (int i = 0; i < 10; i++) {
controller.enqueue("This stuff is fine".getBytes(), attrs1);
controller.enqueue(multiline.getBytes(), attrs2);
}
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readWithParams.rb");
controller.setProperty("expr", "rehenderit");
controller.run(20);
controller.assertTransferCount("failure", 10);
controller.assertTransferCount("success", 10);
for (int i = 0; i < 10; i++) {
MockFlowFile ff = controller.getFlowFilesForRelationship("failure").get(i);
ff.assertContentEquals(multiline);
assertTrue(ff.getAttribute("filename").endsWith("modified"));
ff = controller.getFlowFilesForRelationship("success").get(i);
ff.assertContentEquals("This stuff is fine");
assertTrue(ff.getAttribute("filename").endsWith("modified"));
}
}
@Test
public void testManualValidationR() {
LOG.info("Ruby script defining manual validator");
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/alwaysFail.rb");
controller.assertNotValid();
}
@Test
public void testGetRelationshipsR() {
LOG.info("Ruby script: getRelationships");
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/routeTest.rb");
// at this point, the script has not been instantiated so the processor simply returns an empty set
Set<Relationship> rels = controller.getProcessor().getRelationships();
assertEquals(0, rels.size());
// this will instantiate the script
controller.assertValid();
// this will call the script
rels = controller.getProcessor().getRelationships();
assertEquals(3, rels.size());
}
@Test
public void testGetExceptionRouteR() {
LOG.info("Ruby script defining route taken in event of exception");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue("Bad things go to 'b'.".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/routeTest.rb");
controller.run(3);
assertRelationshipContents("This stuff is fine", "a");
assertRelationshipContents("Bad things go to 'b'.", "b");
assertRelationshipContents(multiline, "c");
}
@Test
public void testSimpleConverterR() {
LOG.info("Running Ruby converter script");
for (int i = 0; i < 20; i++) {
controller.enqueue(multiline.getBytes());
}
controller.setThreadCount(20);
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/simpleConverter.rb");
controller.run(20);
List<MockFlowFile> successFiles = controller.getFlowFilesForRelationship("success");
List<MockFlowFile> failFiles = controller.getFlowFilesForRelationship("failure");
assertEquals("Process did not generate 20 SUCCESS files", 20, successFiles.size());
assertEquals("Process did not generate 20 FAILURE files", 20, failFiles.size());
MockFlowFile sFile = successFiles.get(19);
MockFlowFile fFile = failFiles.get(19);
byte[] blob = fFile.toByteArray();
String[] lines = new String(blob).split("\n");
assertEquals("File had more than one line", 1, lines.length);
assertEquals("Lorem ipsum dolor sit amet,", lines[0]);
blob = sFile.toByteArray();
lines = new String(blob).split("\n");
assertEquals("SUCCESS had wrong number of lines", 7, lines.length);
assertEquals("consectetur adipisicing elit,", lines[0]);
}
@Test
public void testLoadLocalR() {
LOG.info("Ruby: load another script file");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/loadLocal.rb");
controller.run(2);
assertRelationshipContents(multiline, "failure");
assertRelationshipContents("This stuff is fine", "success");
}
@Test
public void testFlowFileR() {
LOG.info("Ruby: get FlowFile properties");
controller.enqueue(multiline.getBytes());
HashMap<String, String> meta = new HashMap<String, String>();
meta.put("evict", "yup");
controller.enqueue("This would be plenty long but it's also evicted.".getBytes(), meta);
controller.enqueue("This is too short".getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/ffTest.rb");
controller.run(3);
assertRelationshipContents(multiline, "success");
assertRelationshipContents("This is too short", "failure");
assertRelationshipContents("This would be plenty long but it's also evicted.", "evict");
}
// //////////////////////////////////// // JS tests
@Test
public void testSimpleReadJS() {
LOG.info("Javascript: fail file based on reading contents");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readTest.js");
controller.run(2);
assertRelationshipContents(multiline, "failure");
assertRelationshipContents("This stuff is fine", "success");
}
@Test
public void testParamReadJS() {
LOG.info("Javascript: read contents and fail based on parameter");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readWithParams.js");
controller.setProperty("expr", "sed do");
controller.run(2);
assertRelationshipContents(multiline, "failure");
assertRelationshipContents("This stuff is fine", "success");
}
@Test
public void testWriteLastLineJS() {
LOG.info("Running Javascript to output last line of file");
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/writeTest.js");
controller.run();
List<MockFlowFile> sunkFiles = controller.getFlowFilesForRelationship("success");
assertEquals("Process did not generate an output file", 1, sunkFiles.size());
MockFlowFile sunkFile = sunkFiles.iterator().next();
byte[] blob = sunkFile.toByteArray();
String[] lines = new String(blob).split("\n");
assertEquals("File had more than one line", 1, lines.length);
assertEquals("sunt in culpa qui officia deserunt mollit anim id est laborum.", lines[0]);
}
@Test
public void testWriteOptionalParametersJS() {
LOG.info("Javascript processCallback that uses optional parameters");
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/paramTest.js");
controller.setProperty("repeat", "3");
controller.run();
List<MockFlowFile> sunkFiles = controller.getFlowFilesForRelationship("success");
assertEquals("Process did not generate an output file", 1, sunkFiles.size());
MockFlowFile sunkFile = sunkFiles.iterator().next();
byte[] blob = sunkFile.toByteArray();
String[] lines = new String(blob).split("\n");
assertEquals("File did not have 3 lines", 3, lines.length);
assertEquals("sunt in culpa qui officia deserunt mollit anim id est laborum.", lines[0]);
}
@Test
public void testSetupOptionalValidationJS() {
LOG.info("Javascript creating validators for optional properties");
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/optionalValidators.js");
controller.setProperty("int", "abc");
controller.setProperty("url", "not@valid");
controller.setProperty("nonEmpty", "");
assertEquals(2, controller.getProcessor().getPropertyDescriptors().size());
controller.assertNotValid(); // due to invalid values above
assertEquals(5, controller.getProcessor().getPropertyDescriptors().size());
controller.setProperty("int", "123");
controller.setProperty("url", "http://localhost");
controller.setProperty("nonEmpty", "abc123");
assertEquals(5, controller.getProcessor().getPropertyDescriptors().size());
controller.assertValid();
}
@Test
public void testManualValidationJS() {
LOG.info("Javascript defining manual validator");
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/alwaysFail.js");
controller.assertNotValid();
}
@Test
public void testGetExceptionRouteJS() {
LOG.info("Javascript defining route taken in event of exception");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue("Bad things go to 'b'.".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/routeTest.js");
controller.run(3);
assertRelationshipContents("This stuff is fine", "a");
assertRelationshipContents("Bad things go to 'b'.", "b");
assertRelationshipContents(multiline, "c");
}
@Test
public void testSimpleConverterJS() {
LOG.info("Running Javascript converter script");
for (int i = 0; i < 20; i++) {
controller.enqueue(multiline.getBytes());
}
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/simpleConverter.js");
controller.run(20);
List<MockFlowFile> successFiles = controller.getFlowFilesForRelationship("success");
List<MockFlowFile> failFiles = controller.getFlowFilesForRelationship("failure");
assertEquals("Process did not generate 20 SUCCESS files", 20, successFiles.size());
assertEquals("Process did not generate 20 FAILURE file", 20, failFiles.size());
MockFlowFile sFile = successFiles.get(19);
MockFlowFile fFile = failFiles.get(0);
byte[] blob = sFile.toByteArray();
String[] lines = new String(blob).split("\n");
assertEquals("SUCCESS had wrong number of lines", 7, lines.length);
assertTrue(lines[0].startsWith("consectetur adipisicing elit,"));
blob = fFile.toByteArray();
lines = new String(blob).split("\n");
assertEquals("File had more than one line", 1, lines.length);
assertTrue(lines[0].startsWith("Lorem ipsum dolor sit amet,"));
}
@Test
public void testLoadLocalJS() {
LOG.info("Javascript: load another script file");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/loadLocal.js");
controller.run(2);
assertRelationshipContents(multiline, "failure");
assertRelationshipContents("This stuff is fine", "success");
}
@Test
public void testXMLJS() {
LOG.info("Javascript: native XML parser");
controller.enqueue("<a><b foo='bar'>Bad</b><b good='true'>Good</b><b good='false'>Bad</b></a>".getBytes());
controller.enqueue("<a><b>Hello</b><b>world</b></a>".getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/parseXml.js");
controller.run(2);
assertRelationshipContents("Good", "success");
assertRelationshipContents("<a><b>Hello</b><b>world</b></a>", "failure");
}
@Test
public void testFlowFileJS() {
LOG.info("JavaScript: get FlowFile properties");
controller.enqueue("This is too short".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/ffTest.js");
controller.run(2);
assertRelationshipContents(multiline, "success");
assertRelationshipContents("This is too short", "failure");
}
@Test
public void testMultiThreadExecJS() {
LOG.info("JavaScript script 20 threads: Failing file based on reading contents");
Map<String, String> attrs1 = new HashMap<>();
attrs1.put("filename", "StuffIsFine.txt");
Map<String, String> attrs2 = new HashMap<>();
attrs2.put("filename", "multiline.txt");
controller.setThreadCount(20);
for (int i = 0; i < 10; i++) {
controller.enqueue("This stuff is fine".getBytes(), attrs1);
controller.enqueue(multiline.getBytes(), attrs2);
}
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readWithParams.js");
controller.setProperty("expr", "rehenderit");
controller.run(20);
controller.assertTransferCount("failure", 10);
controller.assertTransferCount("success", 10);
for (int i = 0; i < 10; i++) {
MockFlowFile ff = controller.getFlowFilesForRelationship("failure").get(i);
ff.assertContentEquals(multiline);
assertTrue(ff.getAttribute("filename").endsWith("modified"));
ff = controller.getFlowFilesForRelationship("success").get(i);
ff.assertContentEquals("This stuff is fine");
assertTrue(ff.getAttribute("filename").endsWith("modified"));
}
}
@Test
public void testUpdateScriptJS() throws Exception {
LOG.info("Test one script with updated class");
File testFile = File.createTempFile("script", ".js");
File original = new File("target/test-scripts/readWithParams.js");
FileUtils.copyFile(original, testFile);
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, testFile.getPath());
controller.assertValid();
original = new File("target/test-scripts/optionalValidators.js");
FileUtils.copyFile(original, testFile);
controller.setProperty(ExecuteScript.SCRIPT_CHECK_INTERVAL, "5 secs");
Thread.sleep(6000);
controller.assertNotValid();
controller.setProperty("int", "abc");
controller.assertNotValid();
controller.setProperty("url", "not@valid");
controller.assertNotValid();
controller.setProperty("nonEmpty", "");
controller.assertNotValid();
controller.setProperty("int", "123");
controller.setProperty("url", "http://localhost");
controller.setProperty("nonEmpty", "abc123");
controller.assertValid();
FileUtils.deleteQuietly(testFile);
}
// ////////////////////////////////// // Python script tests
@Test
public void testSimpleReadP() {
LOG.info("Python script: fail file based on reading contents");
for (int i = 0; i < 20; i++) {
Map<String, String> attr1 = new HashMap<>();
attr1.put("filename", "FineStuff");
attr1.put("counter", Integer.toString(i));
Map<String, String> attr2 = new HashMap<>();
attr2.put("filename", "MultiLine");
attr2.put("counter", Integer.toString(i));
controller.enqueue("This stuff is fine".getBytes(), attr1);
controller.enqueue(multiline.getBytes(), attr2);
}
controller.setThreadCount(40);
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readTest.py");
controller.run(40);
assertRelationshipContents(multiline, "failure", 20);
assertRelationshipContents("This stuff is fine", "success", 20);
List<MockFlowFile> fails = controller.getFlowFilesForRelationship("failure");
List<MockFlowFile> successes = controller.getFlowFilesForRelationship("success");
for (int i = 0; i < 20; i++) {
assertTrue(fails.get(i).getAttribute("filename").matches("^.*\\d+$"));
assertTrue(successes.get(i).getAttribute("filename").matches("^.*\\d+$"));
}
}
@Test
public void testParamReadP() {
LOG.info("Python script: read contents and fail based on parameter");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readWithParams.py");
controller.setProperty("expr", "sed do");
controller.run(2);
assertRelationshipContents(multiline, "failure");
assertRelationshipContents("This stuff is fine", "success");
}
@Test
public void testWriteLastLineP() {
LOG.info("Running Python script to output last line of file");
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/writeTest.py");
controller.run();
List<MockFlowFile> sunkFiles = controller.getFlowFilesForRelationship("success");
assertEquals("Process did not generate an output file", 1, sunkFiles.size());
MockFlowFile sunkFile = sunkFiles.iterator().next();
byte[] blob = sunkFile.toByteArray();
String[] lines = new String(blob).split("\n");
assertEquals("File had more than one line", 1, lines.length);
assertEquals("sunt in culpa qui officia deserunt mollit anim id est laborum.", lines[0]);
}
@Test
public void testWriteOptionalParametersP() {
LOG.info("Python script processCallback that uses optional parameters");
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/paramTest.py");
controller.setProperty("repeat", "3");
controller.run();
List<MockFlowFile> sunkFiles = controller.getFlowFilesForRelationship("success");
assertEquals("Process did not generate an output file", 1, sunkFiles.size());
MockFlowFile sunkFile = sunkFiles.iterator().next();
byte[] blob = sunkFile.toByteArray();
String[] lines = new String(blob).split("\n");
assertEquals("File did not have 3 lines", 3, lines.length);
assertTrue(lines[2].startsWith("sunt in culpa qui officia deserunt mollit anim id est laborum."));
}
@Test
public void testManualValidationP() {
LOG.info("Python defining manual validator");
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/alwaysFail.py");
controller.assertNotValid();
}
@Test
public void testSetupOptionalValidationP() {
LOG.info("Python script creating validators for optional properties");
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/optionalValidators.py");
controller.setProperty("int", "abc");
controller.setProperty("url", "not@valid");
controller.setProperty("nonEmpty", "");
assertEquals(2, controller.getProcessor().getPropertyDescriptors().size());
controller.assertNotValid();
controller.setProperty("int", "123");
controller.setProperty("url", "http://localhost");
controller.setProperty("nonEmpty", "abc123");
assertEquals(5, controller.getProcessor().getPropertyDescriptors().size());
controller.assertValid();
}
@Test
public void testGetExceptionRouteP() {
LOG.info("Python script defining route taken in event of exception");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue("Bad things go to 'b'.".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/routeTest.py");
// Don't put the error in the logs
// TestableAppender ta = new TestableAppender();
// ta.attach(Logger.getLogger(ExecuteScript.class));
controller.run(3);
// ta.detach();
assertRelationshipContents("This stuff is fine", "a");
assertRelationshipContents("Bad things go to 'b'.", "b");
assertRelationshipContents(multiline, "c");
// ta.assertFound("threw exception");
}
@Test
public void testLoadLocalP() throws Exception {
final Thread t = new Thread(new Runnable() {
@Override
public void run() {
try {
testGetExceptionRouteP();
setupEach();
} catch (Exception e) {
}
}
});
t.start();
t.join();
LOG.info("Python: load another script file");
controller.enqueue("This stuff is fine".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/loadLocal.py");
controller.run(2);
assertRelationshipContents(multiline, "failure");
assertRelationshipContents("This stuff is fine", "success");
}
@Test
public void testSimpleConverterP() {
LOG.info("Running Python converter script");
for (int i = 0; i < 20; i++) {
controller.enqueue(multiline.getBytes());
}
controller.setThreadCount(20);
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/simpleConverter.py");
controller.run(20);
List<MockFlowFile> successFiles = controller.getFlowFilesForRelationship("success");
List<MockFlowFile> failFiles = controller.getFlowFilesForRelationship("failure");
assertEquals("Process did not generate 20 SUCCESS files", 20, successFiles.size());
assertEquals("Process did not generate 20 FAILURE files", 20, failFiles.size());
MockFlowFile sFile = successFiles.iterator().next();
MockFlowFile fFile = failFiles.iterator().next();
byte[] blob = sFile.toByteArray();
String[] lines = new String(blob).split("\n");
assertEquals("SUCCESS had wrong number of lines", 7, lines.length);
assertTrue(lines[0].startsWith("consectetur adipisicing elit,"));
blob = fFile.toByteArray();
lines = new String(blob).split("\n");
assertEquals("File had more than one line", 1, lines.length);
assertTrue(lines[0].startsWith("Lorem ipsum dolor sit amet,"));
}
@Test
public void testFlowFileP() {
LOG.info("Python: get FlowFile properties");
controller.enqueue("This is too short".getBytes());
controller.enqueue(multiline.getBytes());
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/ffTest.py");
controller.run(2);
assertRelationshipContents(multiline, "success");
assertRelationshipContents("This is too short", "failure");
}
@Test
public void testMultiThreadExecP() {
LOG.info("Pthon script 20 threads: Failing file based on reading contents");
Map<String, String> attrs1 = new HashMap<>();
attrs1.put("filename", "StuffIsFine.txt");
Map<String, String> attrs2 = new HashMap<>();
attrs2.put("filename", "multiline.txt");
for (int i = 0; i < 10; i++) {
controller.enqueue("This stuff is fine".getBytes(), attrs1);
controller.enqueue(multiline.getBytes(), attrs2);
}
controller.setThreadCount(20);
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, "target/test-scripts/readWithParams.py");
controller.setProperty("expr", "sed do");
controller.run(20);
controller.assertTransferCount("failure", 10);
controller.assertTransferCount("success", 10);
for (int i = 0; i < 10; i++) {
MockFlowFile ff = controller.getFlowFilesForRelationship("failure").get(i);
ff.assertContentEquals(multiline);
assertTrue(ff.getAttribute("filename").endsWith("modified"));
ff = controller.getFlowFilesForRelationship("success").get(i);
ff.assertContentEquals("This stuff is fine");
assertTrue(ff.getAttribute("filename").endsWith("modified"));
}
}
@Test
public void testUpdateScriptP() throws Exception {
LOG.info("Test one script with updated class");
File testFile = File.createTempFile("script", ".py");
File original = new File("target/test-scripts/readTest.py");
FileUtils.copyFile(original, testFile);
controller.setProperty(ExecuteScript.SCRIPT_FILE_NAME, testFile.getPath());
controller.assertValid();
original = new File("target/test-scripts/readWithParams.py");
FileUtils.copyFile(original, testFile);
controller.setProperty(ExecuteScript.SCRIPT_CHECK_INTERVAL, "5 secs");
Thread.sleep(6000);
controller.assertNotValid(); // need to set 'expr'
controller.setProperty("int", "abc");
controller.assertNotValid();
controller.setProperty("url", "not@valid");
controller.assertNotValid();
controller.setProperty("nonEmpty", "");
controller.assertNotValid();
controller.setProperty("expr", "sed do");
controller.assertValid();
assertEquals(6, controller.getProcessContext().getProperties().size());
FileUtils.deleteQuietly(testFile);
}
}

24
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/alwaysFail.js
View File

@ -1,24 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var instance = new ReaderScript({
validate: function () {
return ["This will never work."];
}
});
}

19
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/alwaysFail.py
View File

@ -1,19 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# type(name, superclass_tuple, member_dict) is a shorthand for defining an
# anonymous class. Note the trailing parens (), because scriptBuilder must
# return an *instance* of the class.
instance = type("FailingReader", (ReaderScript, object),\
{"validate": lambda self : ["I am broken"]})()

21
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/alwaysFail.rb
View File

@ -1,21 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class NeverSatisfied < ReaderScript
def validate
return ["This is supposed to fail"]
end
end
NeverSatisfied.new

28
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/ffTest.js
View File

@ -1,28 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var instance = new ReaderScript({
route: function (input) {
if (instance.getFlowFileSize() < 20) {
return Script.FAIL_RELATIONSHIP;
} else {
return Script.SUCCESS_RELATIONSHIP;
}
}
});
}

22
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/ffTest.py
View File

@ -1,22 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class SimpleJythonReader(ReaderScript):
def route(self, input):
if self.getFlowFileSize() < 20 : return self.FAIL_RELATIONSHIP
return self.SUCCESS_RELATIONSHIP
instance = SimpleJythonReader()

30
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/ffTest.rb
View File

@ -1,30 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class SimpleJRubyReader < ReaderScript
field_reader :FAIL_RELATIONSHIP, :SUCCESS_RELATIONSHIP
def getRelationships
@@evict = Relationship::Builder.new.name("evict").description("some evicted stuff").build()
[FAIL_RELATIONSHIP, SUCCESS_RELATIONSHIP, @@evict]
end
def route( input )
return FAIL_RELATIONSHIP if getFlowFileSize < 20
return @@evict if !getAttribute("evict").nil?
return SUCCESS_RELATIONSHIP
end
end
SimpleJRubyReader.new

18
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/lib/Sub.py
View File

@ -1,18 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class TestFilter:
def notAllowed(self):
return "^sed"

22
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/lib/sub.js
View File

@ -1,22 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
function notAllowed() { // Works for eval(readFile(...))
return /sed do/i;
}
exports.notAllowed = notAllowed; // Works for require(...)

17
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/lib/sub.rb
View File

@ -1,17 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
def notAllowed
return /^sed/i
end

30
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/loadLocal.js
View File

@ -1,30 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
var sub = require("lib/sub.js");
with (Scripting) {
var instance = new ReaderScript({
route: function (input) {
var str = IOUtils.toString(input);
if (str.match(sub.notAllowed())) {
return Script.FAIL_RELATIONSHIP;
} else {
return Script.SUCCESS_RELATIONSHIP;
}
}
});
}

26
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/loadLocal.py
View File

@ -1,26 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import re
from Sub import TestFilter
class SimpleJythonReader(ReaderScript):
def route(self, input):
tf = TestFilter()
for line in FileUtil.wrap(input):
if re.match(tf.notAllowed(),line): return self.FAIL_RELATIONSHIP
return self.SUCCESS_RELATIONSHIP
instance = SimpleJythonReader()

29
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/loadLocal.rb
View File

@ -1,29 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
require 'sub'
class SimpleJRubyReader < ReaderScript
field_reader :FAIL_RELATIONSHIP, :SUCCESS_RELATIONSHIP
def route( input )
input.to_io.each_line do |line|
return FAIL_RELATIONSHIP if line.match notAllowed
end
return SUCCESS_RELATIONSHIP
end
end
SimpleJRubyReader.new

54
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/log4j.xml
View File

@ -1,54 +0,0 @@
<?xml version="1.0" encoding="UTF-8" ?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.out"/>
<param name="Threshold" value="DEBUG"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{MM-dd-yy HH:mm:ss.SSS} %-5p %c{2} %x - %m%n"/>
</layout>
</appender>
<logger name="org.nifi.model.processor.FlowFileProcessor" additivity="false">
<level value="INFO"/>
<appender-ref ref="CONSOLE"/>
</logger>
<logger name="junit.TestableAppender" additivity="false">
<level value="TRACE"/>
<appender-ref ref="CONSOLE"/>
</logger>
<logger name="org.nifi.model.processor.impl.ScriptRunnerProcessor" additivity="false">
<level value="WARN"/>
<appender-ref ref="CONSOLE"/>
</logger>
<logger name="org.nifi.model.processor.impl.ScriptRunnerProcessorTest" additivity="false">
<level value="INFO"/>
<appender-ref ref="CONSOLE"/>
</logger>
<root>
<level value="INFO"/>
<appender-ref ref="CONSOLE" />
</root>
</log4j:configuration>

28
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/optionalValidators.js
View File

@ -1,28 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var instance = new WriterScript({
getPropertyDescriptors: function () {
i = new PropertyDescriptor.Builder().name("int").description("an int").required(true).addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR).build();
u = new PropertyDescriptor.Builder().name("url").description("a url").required(true).addValidator(StandardValidators.URL_VALIDATOR).build();
s = new PropertyDescriptor.Builder().name("nonEmpty").description("a non empty property").required(true).addValidator(StandardValidators.NON_EMPTY_VALIDATOR).build();
return [i, u, s];
}
});
}

22
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/optionalValidators.py
View File

@ -1,22 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class SimpleJythonReader(ReaderScript):
def getPropertyDescriptors( self ):
nev = PropertyDescriptor.Builder().name("nonEmpty").required(1).addValidator(StandardValidators.NON_EMPTY_VALIDATOR).build()
iv = PropertyDescriptor.Builder().name("int").required(1).addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR).build()
uv = PropertyDescriptor.Builder().name("url").required(1).addValidator(StandardValidators.URL_VALIDATOR).build()
return [nev, iv, uv]
instance = SimpleJythonReader()

39
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/optionalValidators.rb
View File

@ -1,39 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class SimpleJRubyReader < ReaderScript
field_reader :FAIL_RELATIONSHIP, :SUCCESS_RELATIONSHIP, :logger
def getPropertyDescriptors
logger.debug("Defining descriptors");
i = StandardValidators::INTEGER_VALIDATOR
u = StandardValidators::URL_VALIDATOR
s = StandardValidators::NON_EMPTY_VALIDATOR
intPropDesc = PropertyDescriptor::Builder.new().name("int").required(true).addValidator(i).build()
urlPropDesc = PropertyDescriptor::Builder.new().name("url").required(true).addValidator(u).build()
nonEmptyPropDesc = PropertyDescriptor::Builder.new().name("nonEmpty").addValidator(s).build()
return [intPropDesc, urlPropDesc, nonEmptyPropDesc]
end
def route( input )
logger.debug("Routing input");
input.to_io.each_line do |line|
return FAIL_RELATIONSHIP if line.match /^sed/i
end
return SUCCESS_RELATIONSHIP
end
end
$logger.debug("Creating SimpleJRubyReader with props" + @properties.to_s)
SimpleJRubyReader.new

28
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/paramTest.js
View File

@ -1,28 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var instance = new WriterScript({
process: function (input, output) {
var str = IOUtils.toString(input);
var last = str.split("\n").pop() + "\n";
for (var i = 0; i < instance.getProperty("repeat"); i++) {
IOUtils.write(last, output);
}
output.flush();
}
});
}

26
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/paramTest.py
View File

@ -1,26 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class ParamUsingWriter(WriterScript):
def process ( self, input, output ):
last = FileUtil.wrap(input).readlines()[-1] + '\n'
writer = FileUtil.wrap(output)
times = int(self.getProperty("repeat"))
lines = [last] * times
writer.writelines(lines)
writer.close()
instance = ParamUsingWriter()

31
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/paramTest.rb
View File

@ -1,31 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class ParamUsingWriter < WriterScript
def process ( input, output )
reader = input.to_io
writer = output.to_io
last = reader.readlines.last
getProperty("repeat").to_i.times do
writer << last + "\n"
end
writer.close
reader.close
end
end
ParamUsingWriter.new

36
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/parseXml.js
View File

@ -1,36 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var instance = new WriterScript({
process: function (istream, ostream) {
var str = IOUtils.toString(istream);
var obj = new XML(str);
print(obj)
var good = obj.b.(@good == "true");
if (good.length() == 0) {
instance.setRoute(Script.FAIL_RELATIONSHIP);
IOUtils.write(str, ostream);
} else {
instance.setRoute(Script.SUCCESS_RELATIONSHIP);
for each (var goodStr in good) {
IOUtils.write(goodStr, ostream);
}
}
ostream.flush();
}
});
}

30
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/readTest.js
View File

@ -1,30 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var instance = new ReaderScript({
route: function (input) {
str = IOUtils.toString(input);
if (str.match(/sed do/i)) {
return Script.FAIL_RELATIONSHIP;
} else {
return Script.SUCCESS_RELATIONSHIP;
}
}
});
logger.debug("Got a logger and properties" + properties);
}

32
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/readTest.py
View File

@ -1,32 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os, re, subprocess
class SimpleJythonReader(ReaderScript):
def route(self, input):
logger.info("In route")
returnid = os.system("c:\\cygwin\\bin\\echo GOOD")
fname = self.getAttribute("filename")
counter = self.getAttribute("counter")
fname = fname + '.' + counter
self.setAttribute("filename", fname)
for line in FileUtil.wrap(input):
if re.match("^sed",line): return self.FAIL_RELATIONSHIP
return self.SUCCESS_RELATIONSHIP
instance = SimpleJythonReader()

30
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/readTest.rb
View File

@ -1,30 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class SimpleJRubyReader < ReaderScript
field_reader :FAIL_RELATIONSHIP, :SUCCESS_RELATIONSHIP, :logger, :attributes
def route( input )
logger.info("Route Input")
input.to_io.each_line do |line|
return FAIL_RELATIONSHIP if line.match /^sed/i
end
attributes.put("filename", "NewFileNameFromReadTest")
return SUCCESS_RELATIONSHIP
end
end
$logger.info("Logger is made available in shared variables...however, the SimpleJRubyReader.logger is not set till after this script returns")
SimpleJRubyReader.new

32
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/readWithParams.js
View File

@ -1,32 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var instance = new ReaderScript({
route: function (input) {
var str = IOUtils.toString(input);
var expr = instance.getProperty("expr");
filename = instance.attributes.get("filename");
instance.setAttribute("filename", filename + ".modified");
if (str.match(expr)) {
return Script.FAIL_RELATIONSHIP;
} else {
return Script.SUCCESS_RELATIONSHIP;
}
}
});
}

32
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/readWithParams.py
View File

@ -1,32 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import re
class SimpleJythonReader(ReaderScript):
def getPropertyDescriptors( self ):
nev = StandardValidators.NON_EMPTY_VALIDATOR
return [PropertyDescriptor.Builder().name("expr").required(1).addValidator(nev).build()]
def route( self, input ):
expr = self.getProperty("expr")
filename = self.getAttribute("filename")
self.setAttribute("filename", filename + ".modified")
for line in FileUtil.wrap(input):
if re.match(expr, line): return self.FAIL_RELATIONSHIP
return self.SUCCESS_RELATIONSHIP
instance = SimpleJythonReader()

33
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/readWithParams.rb
View File

@ -1,33 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class SimpleJRubyReader < ReaderScript
field_reader :FAIL_RELATIONSHIP, :SUCCESS_RELATIONSHIP, :properties, :attributes
def route( input )
expr = properties.get "expr"
raise "Must specify the 'expr' property!" if expr.nil?
filename = attributes.get "filename"
setAttribute("filename", filename + ".modified")
input.to_io.each_line do |line|
return FAIL_RELATIONSHIP if line.match expr
end
return SUCCESS_RELATIONSHIP
end
end
$logger.debug("Can access logger and properties via shared instance variables...props = " + @properties.to_s)
SimpleJRubyReader.new

41
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/routeTest.js
View File

@ -1,41 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var a = new Relationship.Builder().name("a").description("some good stuff").build()
var b = new Relationship.Builder().name("b").description("some other stuff").build()
var c = new Relationship.Builder().name("c").description("some bad stuff").build()
var instance = new ReaderScript({
getExceptionRoute: function () {
return c;
},
getRelationships: function () {
return [a, b, c];
},
route: function (input) {
var str = IOUtils.toString(input);
var lines = str.split("\n");
for (var line in lines) {
if (lines[line].match(/^bad/i)) {
return b;
} else if (lines[line].match(/^sed/i)) {
throw "That's no good!";
}
}
return a;
}
});
}

37
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/routeTest.py
View File

@ -1,37 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import re
class RoutingReader(ReaderScript):
A = Relationship.Builder().name("a").description("some good stuff").build()
B = Relationship.Builder().name("b").description("some other stuff").build()
C = Relationship.Builder().name("c").description("some bad stuff").build()
def getRelationships(self):
return [self.A,self.B,self.C]
def getExceptionRoute(self):
return self.C
def route( self, input ):
for line in FileUtil.wrap(input):
if re.match("^bad", line, re.IGNORECASE):
return self.B
if re.match("^sed", line):
raise RuntimeError("That's no good!")
return self.A
instance = RoutingReader()

39
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/routeTest.rb
View File

@ -1,39 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class SimpleJRubyReader < ReaderScript
@@a = Relationship::Builder.new().name("a").description("some good stuff").build()
@@b = Relationship::Builder.new().name("b").description("some bad stuff").build()
@@c = Relationship::Builder.new().name("c").description("some other stuff").build()
def getRelationships
return [@@a, @@b, @@c]
end
def getExceptionRoute
@@c
end
def route( input )
input.to_io.each_line do |line|
return @@b if line.match /^bad/i
raise "That's no good!" if line.match /^sed/i
end
@@a
end
end
SimpleJRubyReader.new

45
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/simpleConverter.js
View File

@ -1,45 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var instance = new ConverterScript({
convert: function (input) {
var buffReader = new java.io.BufferedReader(new java.io.InputStreamReader(input));
instance.createFlowFile("firstLine", Script.FAIL_RELATIONSHIP, function (output) {
var out = new java.io.BufferedWriter(new java.io.OutputStreamWriter(output));
var firstLine = buffReader.readLine();
out.write(firstLine, 0, firstLine.length());
out.flush();
out.close();
});
instance.createFlowFile("otherLines", Script.SUCCESS_RELATIONSHIP, function (output) {
var out = new java.io.BufferedWriter(new java.io.OutputStreamWriter(output));
var line = buffReader.readLine();
while (line != null) {
out.write(line, 0, line.length());
out.newLine();
line = buffReader.readLine();
}
out.flush();
out.close();
});
}
});
logger.debug("Processor props" + properties)
}

60
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/simpleConverter.py
View File

@ -1,60 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from org.python.core.io import TextIOWrapper,BufferedReader,BufferedWriter,StreamIO
from org.apache.nifi.scripting import OutputStreamHandler
class WriteFirstLine(OutputStreamHandler):
def __init__(self, wrappedIn):
self.wrappedIn = wrappedIn
def write(self, output):
streamOut = StreamIO(output, False)
bufWrtr = BufferedWriter(streamOut, 8192)
wrappedOut = TextIOWrapper(bufWrtr)
wrappedOut.write(self.wrappedIn.readline(8192))
wrappedOut.flush()
wrappedOut.close()
class WriteOtherLines(OutputStreamHandler):
def __init__(self, wrappedIn):
self.wrappedIn = wrappedIn
def write(self, output):
streamOut = StreamIO(output, False)
bufWrtr = BufferedWriter(streamOut, 8192)
wrappedOut = TextIOWrapper(bufWrtr)
line = self.wrappedIn.readline(8192)
while line != '':
wrappedOut.write(line)
line = self.wrappedIn.readline(8192)
wrappedOut.flush()
wrappedOut.close()
class SimpleConverter(ConverterScript):
def convert(self, input):
streamIn = StreamIO(input, False)
bufRdr = BufferedReader(streamIn, 8192)
wrappedIn = TextIOWrapper(bufRdr)
writeFirstLine = WriteFirstLine(wrappedIn)
self.createFlowFile("firstLine", self.FAIL_RELATIONSHIP, writeFirstLine)
writeOtherLines = WriteOtherLines(wrappedIn)
self.createFlowFile("otherLines", self.SUCCESS_RELATIONSHIP, writeOtherLines)
instance = SimpleConverter()

42
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/simpleConverter.rb
View File

@ -1,42 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
java_import 'org.apache.nifi.scripting.OutputStreamHandler'
class SimpleConverter < ConverterScript
field_reader :FAIL_RELATIONSHIP, :SUCCESS_RELATIONSHIP, :logger, :attributes
def convert(input)
in_io = input.to_io
createFlowFile("firstLine", FAIL_RELATIONSHIP, OutputStreamHandler.impl do |method, out|
out_io = out.to_io
out_io << in_io.readline.to_java_bytes
out_io.close
logger.debug("Wrote data to failure...this message logged with logger from super class")
end)
createFlowFile("otherLines", SUCCESS_RELATIONSHIP, OutputStreamHandler.impl do |method, out|
out_io = out.to_io
in_io.each_line { |line|
out_io << line
}
out_io.close
logger.debug("Wrote data to success...this message logged with logger from super class")
end)
in_io.close
end
end
$logger.debug("Creating SimpleConverter...this message logged with logger from shared variables")
SimpleConverter.new

26
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/writeTest.js
View File

@ -1,26 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
with (Scripting) {
var instance = new WriterScript({
process: function (input, output) {
var str = IOUtils.toString(input);
IOUtils.write(str.split("\n").pop(), output);
output.flush();
}
});
}

22
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/writeTest.py
View File

@ -1,22 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class SimpleWriter(WriterScript):
def process( self, input, output ):
last = FileUtil.wrap(input).readlines()[-1]
writer = FileUtil.wrap(output)
writer.write(last)
writer.close()
instance = SimpleWriter()

32
nifi/nar-bundles/execute-script-bundle/execute-script-processors/src/test/resources/writeTest.rb
View File

@ -1,32 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class SimpleJRubyRunner < WriterScript
def process( input, output )
in_io = input.to_io
out_io = output.to_io
last = nil
in_io.each_line do |line|
last = line
end
out_io << last unless last.nil?
in_io.close
out_io.close
end
end
SimpleJRubyRunner.new

36
nifi/nar-bundles/execute-script-bundle/nar/pom.xml
View File

@ -1,36 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>execute-script-bundle</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>execute-script-nar</artifactId>
<name>Execute Script NAR</name>
<packaging>nar</packaging>
<description>NiFi Script Running NAR</description>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>execute-script-processors</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</project>

81
nifi/nar-bundles/execute-script-bundle/pom.xml
View File

@ -1,81 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>nar-container-common</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>execute-script-bundle</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>Execute Script Bundle</name>
<packaging>pom</packaging>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-stream-utils</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-core-flowfile-attributes</artifactId>
<version>0.0.1-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<version>0.0.1-SNAPSHOT</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.jruby</groupId>
<artifactId>jruby</artifactId>
<version>1.7.16.1</version>
</dependency>
<dependency>
<groupId>org.python</groupId>
<artifactId>jython-standalone</artifactId>
<version>2.7-b3</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
</dependencies>
</dependencyManagement>
<modules>
<module>execute-script-processors</module>
<module>nar</module>
</modules>
</project>

76
nifi/nar-bundles/kafka-bundle/kafka-processors/pom.xml
View File

@ -1,76 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>kafka-bundle</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>kafka-processors</artifactId>
<packaging>jar</packaging>
<name>kafka-processors</name>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.8.2</artifactId>
<version>0.8.1</version>
<exclusions>
<!-- Transitive dependencies excluded because they are located
in a legacy Maven repository, which Maven 3 doesn't support. -->
<exclusion>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>

35
nifi/nar-bundles/kafka-bundle/pom.xml
View File

@ -1,35 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>nar-bundle-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>kafka-bundle</artifactId>
<packaging>pom</packaging>
<name>kafka-bundle</name>
<modules>
<module>kafka-processors</module>
<module>kafka-nar</module>
</modules>
</project>

39
nifi/nar-bundles/monitor-threshold-bundle/nar/pom.xml
View File

@ -1,39 +0,0 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>monitor-threshold-bundle</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>monitor-threshold-nar</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>MonitorThreshold-NAR</name>
<packaging>nar</packaging>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>monitor-threshold-processor</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>monitor-threshold-ui</artifactId>
<type>war</type>
</dependency>
</dependencies>
</project>

45
nifi/nar-bundles/monitor-threshold-bundle/processor/pom.xml
View File

@ -1,45 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>monitor-threshold-bundle</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>monitor-threshold-processor</artifactId>
<name>MonitorThreshold-Processor</name>
<description>Processor that counts files with particular attribute values and notifies when a count exceeds a user specified threshold</description>
<dependencies>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-mock</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
</project>

904
nifi/nar-bundles/monitor-threshold-bundle/processor/src/main/java/org/apache/nifi/processors/monitor/MonitorThreshold.java
View File

@ -1,904 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.processors.monitor;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Collection;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TimeZone;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.atomic.AtomicReference;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantReadWriteLock;
import java.util.regex.Pattern;
import org.apache.nifi.annotation.behavior.TriggerWhenEmpty;
import org.apache.nifi.annotation.documentation.CapabilityDescription;
import org.apache.nifi.annotation.documentation.Tags;
import org.apache.nifi.annotation.lifecycle.OnStopped;
import org.apache.nifi.components.PropertyDescriptor;
import org.apache.nifi.components.ValidationContext;
import org.apache.nifi.components.ValidationResult;
import org.apache.nifi.util.file.FileUtils;
import org.apache.nifi.flowfile.FlowFile;
import org.apache.nifi.logging.ProcessorLog;
import org.apache.nifi.processor.AbstractProcessor;
import org.apache.nifi.processor.ProcessContext;
import org.apache.nifi.processor.ProcessSession;
import org.apache.nifi.processor.ProcessorInitializationContext;
import org.apache.nifi.processor.Relationship;
import org.apache.nifi.processor.exception.ProcessException;
import org.apache.nifi.processor.util.StandardValidators;
import org.apache.nifi.util.Tuple;
/**
* This processor has an advanced user interface that is used to specify
* attributes that should be monitored, as well as thresholds for individual
* values that are expected to appear in the monitored attributes.
*/
/* Because this processor maintains counts (in memory) that must be persisted
* to disk according to a prescribed schedule (See OPT_COUNT_RESET_TIME/OPT_COUNT_RESET_MINUTES
* properties), this processor must be timer driven. This processor cannot be event driven.
* Using @TriggerWhenEmpty ensures that the processor will be timer driven, and that its
* onTrigger method will be called even when no Flowfiles are in an incoming queue.
*/
@TriggerWhenEmpty
@Tags({"monitor", "threshold", "throttle", "limit", "counts", "size", "bytes", "files", "attribute", "values", "notify"})
@CapabilityDescription("Examines values found in selected FlowFile attributes and "
+ "maintains counts of the number of times the values have been encountered."
+ "The counts are then used to check against user defined thresholds.")
public class MonitorThreshold extends AbstractProcessor {
// default property values
public static final String DEFAULT_THRESHOLDS_KEY = "Default";
public static final String DEFAULT_COUNTS_PERSISTENCE_FILE_PREFIX = "conf/MonitorThreshold";
public static final int DEFAULT_SAVE_COUNTS_FREQ_SECS = 30;
public static final String DEFAULT_DELIMITER_TO_USE_FOR_COUNTS_ATTRIBUTES = ".";
public static final String DEFAULT_ADD_ATRIBUTES_WHEN_THRESHOLD_EXCEEDED = "true";
public static final String ALWAYS = "Always";
public static final String ONLY_WHEN_THRESHOLD_EXCEEDED = "Only When Threshold Exceeded";
public static final String NEVER = "Never";
public static final String DEFAULT_ADD_ATRIBUTES = ONLY_WHEN_THRESHOLD_EXCEEDED;
// constants/variables, used for clearing/persisting counts
public static final long MILLIS_IN_MINUTE = 60000L;
public static final long MILLIS_IN_HOUR = MILLIS_IN_MINUTE * 60;
public static final String TIME_REGEX = "^((?:[0-1]?[0-9])|(?:[2][0-3])):([0-5]?[0-9]):([0-5]?[0-9])";
public static final Pattern TIME_PATTERN = Pattern.compile(TIME_REGEX);
private Long timeCountsLastCleared = System.currentTimeMillis();
private Long timeCountsLastSaved = null;
// other constants
public static final String NUM_APPLICABLE_THRESHOLDS_KEY = "numApplicableThresholds";
public static final String NUM_THRESHOLDS_EXCEEDED_KEY = "numThresholdsExceeded";
public static final String FILE_COUNT_PREFIX = "fileCount";
public static final String FILE_THRESHOLD_PREFIX = "fileThreshold";
public static final String BYTE_COUNT_PREFIX = "byteCount";
public static final String BYTE_THRESHOLD_PREFIX = "byteThreshold";
public static final String AGGREGATE_COUNTS_WHEN_NO_THRESHOLD_PROVIDED_PROPERTY = "Aggregate Counts When No Threshold Provided";
public static final String COUNT_RESET_TIME_PROPERTY = "Count Reset Time";
public static final String MINUTES_TO_WAIT_BEFORE_RESETTING_COUNTS_PROPERTY = "Minutes to Wait Before Resetting Counts";
public static final String ADD_ATTRIBUTES_PROPERTY = "Add Attributes";
public static final String MAX_ATTRIBUTE_PAIRS_TO_ADD_PROPERTY = "Maximum Attribute Pairs to Add When Multiple Thresholds Exceeded";
public static final String ATTRIBUTE_TO_USE_FOR_COUNTING_BYTES_PROPERTY = "Attribute to use for Counting Bytes";
public static final String DELIMITER_TO_USE_FOR_COUNTS_ATTRIBUTES_PROPERTY = "Delimiter to use for Counts Attributes";
public static final String STANDARD_FLOWFILE_FILESIZE_PROPERTY = "fileSize";
public static final String FREQUENCY_TO_SAVE_COUNTS_PROPERTY = "Frequency to Save Counts (seconds)";
public static final String PREFIX_FOR_COUNTS_PERSISTENCE_FILE_PROPERTY = "Prefix for Counts Persistence File";
// locks for concurrency protection
private final ReentrantReadWriteLock rwLock = new ReentrantReadWriteLock();
private final Lock readLock = rwLock.readLock();
private final Lock writeLock = rwLock.writeLock();
// file and byte counts, etc...
private ConcurrentHashMap<String, ConcurrentHashMap<String, CompareRecord>> counts = null; // Guarded by rwLock
private Map<String, Map<String, CompareRecord>> thresholds; // Guarded by rwLock
// optional properties
private List<PropertyDescriptor> properties;
public static final PropertyDescriptor OPT_AGGREGATE_COUNTS_WHEN_NO_THRESHOLD_PROVIDED = new PropertyDescriptor.Builder()
.name(AGGREGATE_COUNTS_WHEN_NO_THRESHOLD_PROVIDED_PROPERTY)
.description("When a value (in a monitored attribute) is encountered, but no thresholds have been specified "
+ "for that value, how should counts be maintained? "
+ "If this property is true (default), two behaviors are adopted. "
+ "First, counts for ALL non-threshold values are aggregated into a single count. "
+ "Second, when a non-threshold value is encountered, the aggregate count will be compared to the default threshold."
+ "If this property is false, the behaviors change, as follows. "
+ "Separate, individual counts are maintained for each non-threshold, value that is encountered. "
+ "When a non-threshold value is encountered, it's unique count will be compared with the default threshold. "
+ "NOTE: Counts are never maintained for non-monitored attributes. ")
.required(false)
.addValidator(StandardValidators.BOOLEAN_VALIDATOR)
.defaultValue("true")
.allowableValues("true", "false")
.expressionLanguageSupported(false)
.build();
public static final PropertyDescriptor OPT_COUNT_RESET_TIME = new PropertyDescriptor.Builder()
.name(COUNT_RESET_TIME_PROPERTY)
.description("The time (24-hour clock, hh:mm:ss format, GMT Timezone) when the byte and file counts will be reset to zero. "
+ "Note: To prevent counts from increasing forever, you must specify either a '" + COUNT_RESET_TIME_PROPERTY + "' or '" + MINUTES_TO_WAIT_BEFORE_RESETTING_COUNTS_PROPERTY + "'")
.required(false)
.addValidator(StandardValidators.createRegexMatchingValidator(TIME_PATTERN))
.expressionLanguageSupported(false)
.build();
public static final PropertyDescriptor OPT_COUNT_RESET_MINUTES = new PropertyDescriptor.Builder()
.name(MINUTES_TO_WAIT_BEFORE_RESETTING_COUNTS_PROPERTY)
.description("Minutes to delay count reset, beginning at '" + COUNT_RESET_TIME_PROPERTY + "', if provided. "
+ "If '" + COUNT_RESET_TIME_PROPERTY + "' is not provided, then the minutes to delay are counted from the last reset. "
+ "(Last reset is initially the time when the processor is created (added to a flow).)")
.required(false)
.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
.expressionLanguageSupported(false)
.build();
public static final PropertyDescriptor OPT_ADD_ATTRIBUTES = new PropertyDescriptor.Builder()
.name(ADD_ATTRIBUTES_PROPERTY)
.description("Setting this property to'" + ONLY_WHEN_THRESHOLD_EXCEEDED + "' will cause two additional attributes to be added to"
+ "FlowFiles for every threshold that is exceeded. For example, if a file count threshold is exceeded, "
+ " a fileCount.attributeName.value and a fileThreshold.attributeName.value will be added to the FlowFile. "
+ "Setting this property to'" + ALWAYS + "' will cause attributes to be added, no matter whether a threshold is exceeded."
+ "Note 1: Attributes are only added if a monitored attribute is present on the FlowFile, no matter if '" + "' " + ADD_ATTRIBUTES_PROPERTY + "' is set to '" + ALWAYS + "'. "
+ "Note 2: This processor makes a log entry for every threshold that is exceeded, no matter the value of this property. ")
.required(false)
.defaultValue(DEFAULT_ADD_ATRIBUTES)
.allowableValues(ALWAYS, ONLY_WHEN_THRESHOLD_EXCEEDED, NEVER)
.expressionLanguageSupported(false)
.build();
public static final PropertyDescriptor OPT_MAX_ATTRIBUTE_PAIRS_TO_ADD_WHEN_MULTIPLE_THRESHOLD_EXCEEDED = new PropertyDescriptor.Builder()
.name(MAX_ATTRIBUTE_PAIRS_TO_ADD_PROPERTY)
.description("Controls/limits the number of FlowFile attributes that are added when "
+ "multiple thresholds are exceeded. Recall that when a threshold is exceeded, "
+ "a pair of two attributes are added to the FlowFile, one is the current count and "
+ "the other is the threshold. If 100 thresholds are exceeded, then 200 "
+ "attributes will be added to the FlowFile. Setting this property to zero means add "
+ "all count/threshold pairs that were exceeded. Any setting greater than zero indicates "
+ "how many count/threshold pairs to add when multiple thresholds are exceeded. "
+ "Only non-negative settings are supported. The default is 0. "
+ "In effect only when '" + ADD_ATTRIBUTES_PROPERTY + "' is '" + ONLY_WHEN_THRESHOLD_EXCEEDED + "'."
+ "Setting " + ADD_ATTRIBUTES_PROPERTY + " to" + " '" + ALWAYS + "' or '" + NEVER
+ "' causes '" + MAX_ATTRIBUTE_PAIRS_TO_ADD_PROPERTY + "' to be ignored.")
.required(false)
.addValidator(StandardValidators.NON_NEGATIVE_INTEGER_VALIDATOR)
.defaultValue("0")
.expressionLanguageSupported(false)
.build();
public static final PropertyDescriptor OPT_DELIMITER_TO_USE_FOR_COUNTS_ATTRIBUTES = new PropertyDescriptor.Builder()
.name(DELIMITER_TO_USE_FOR_COUNTS_ATTRIBUTES_PROPERTY)
.description("The delimiter to use for naming threshold exceeded attributes,"
+ "e.g., fileCount.attributeName.value. Defaults to \'.\'")
.required(false)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.defaultValue(DEFAULT_DELIMITER_TO_USE_FOR_COUNTS_ATTRIBUTES)
.expressionLanguageSupported(false)
.build();
public static final PropertyDescriptor OPT_ATTRIBUTE_TO_USE_FOR_COUNTING_BYTES = new PropertyDescriptor.Builder()
.name(ATTRIBUTE_TO_USE_FOR_COUNTING_BYTES_PROPERTY)
.description("Setting this property allows a FlowFile attribute to be used for counting bytes, "
+ "in place of the actual fileSize. Note that the attribute named by this property "
+ "must contain only numeric, non-negative integer values for each FlowFile. "
+ "Non-numeric or negative values that are encountered will be ignored, causing the actual fileSize "
+ "to be used instead.")
.required(false)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.defaultValue(STANDARD_FLOWFILE_FILESIZE_PROPERTY)
.expressionLanguageSupported(false)
.build();
public static final PropertyDescriptor OPT_FREQUENCY_TO_SAVE_COUNTS_SECS = new PropertyDescriptor.Builder()
.name(FREQUENCY_TO_SAVE_COUNTS_PROPERTY)
.description("How often the counts should be written to disk. The default value is 30 seconds.")
.required(false)
.addValidator(StandardValidators.POSITIVE_INTEGER_VALIDATOR)
.defaultValue("30")
.expressionLanguageSupported(false)
.build();
public static final PropertyDescriptor OPT_COUNTS_PERSISTENCE_FILE_PREFIX = new PropertyDescriptor.Builder()
.name(PREFIX_FOR_COUNTS_PERSISTENCE_FILE_PROPERTY)
.description("The prefix for the file that is saved (to disk) to maintain counts across NIFI restarts. "
+ "By default, the value is " + DEFAULT_COUNTS_PERSISTENCE_FILE_PREFIX + ". The actual name of the counts file will "
+ "be this value plus \"-XXXX.counts\" where XXXX is the processor ID.")
.required(false)
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.defaultValue(DEFAULT_COUNTS_PERSISTENCE_FILE_PREFIX)
.expressionLanguageSupported(false)
.build();
// relationships
private final AtomicReference<Set<Relationship>> relationships = new AtomicReference<>();
public static final String SUCCESS = "success";
public static final Relationship RELATIONSHIP_SUCCESS = new Relationship.Builder()
.name(SUCCESS)
.description("All FlowFiles follow this relationship, unless there is a problem with the FlowFile.")
.build();
public static final String FAILURE = "failure";
public static final Relationship RELATIONSHIP_FAILURE = new Relationship.Builder()
.name(FAILURE)
.description("FlowFiles follow this relationship path if there is a problem with the FlowFile, or if there is a unrecoverable configuration error.")
.build();
@Override
public Collection<ValidationResult> customValidate(ValidationContext validationContext) {
final Collection<ValidationResult> problems = new ArrayList<>();
String countResetTime = validationContext.getProperty(OPT_COUNT_RESET_TIME).getValue();
String minutesToWaitBeforeResettingCounts = validationContext.getProperty(OPT_COUNT_RESET_MINUTES).getValue();
if (countResetTime == null && minutesToWaitBeforeResettingCounts == null) {
ValidationResult rolloverTimeOrMinutesRequired = new ValidationResult.Builder()
.subject("Invalid processor configuration.")
.explanation("To prevent counts from increasing forever, you must specify either a '" + COUNT_RESET_TIME_PROPERTY + "' or '" + MINUTES_TO_WAIT_BEFORE_RESETTING_COUNTS_PROPERTY + "'.")
.input(COUNT_RESET_TIME_PROPERTY + ": " + countResetTime + " "
+ MINUTES_TO_WAIT_BEFORE_RESETTING_COUNTS_PROPERTY + ": " + minutesToWaitBeforeResettingCounts)
.valid(false)
.build();
problems.add(rolloverTimeOrMinutesRequired);
}
int maxPairsToAdd = Integer.parseInt(validationContext.getProperty(OPT_MAX_ATTRIBUTE_PAIRS_TO_ADD_WHEN_MULTIPLE_THRESHOLD_EXCEEDED).getValue());
if (maxPairsToAdd < 0) {
ValidationResult maxPairsToAddCannotBeNegative = new ValidationResult.Builder()
.subject("Invalid processor configuration.")
.explanation(MAX_ATTRIBUTE_PAIRS_TO_ADD_PROPERTY + " cannot be negative.")
.input(MAX_ATTRIBUTE_PAIRS_TO_ADD_PROPERTY + ": " + maxPairsToAdd)
.valid(false)
.build();
problems.add(maxPairsToAddCannotBeNegative);
}
return problems;
}
@Override
protected void init(final ProcessorInitializationContext context) {
final List<PropertyDescriptor> properties = new ArrayList<>();
properties.add(OPT_AGGREGATE_COUNTS_WHEN_NO_THRESHOLD_PROVIDED);
properties.add(OPT_COUNT_RESET_TIME);
properties.add(OPT_COUNT_RESET_MINUTES);
properties.add(OPT_ADD_ATTRIBUTES);
properties.add(OPT_MAX_ATTRIBUTE_PAIRS_TO_ADD_WHEN_MULTIPLE_THRESHOLD_EXCEEDED);
properties.add(OPT_DELIMITER_TO_USE_FOR_COUNTS_ATTRIBUTES);
properties.add(OPT_ATTRIBUTE_TO_USE_FOR_COUNTING_BYTES);
properties.add(OPT_FREQUENCY_TO_SAVE_COUNTS_SECS);
properties.add(OPT_COUNTS_PERSISTENCE_FILE_PREFIX);
this.properties = Collections.unmodifiableList(properties);
final Set<Relationship> relationships = new HashSet<>();
relationships.add(RELATIONSHIP_SUCCESS);
relationships.add(RELATIONSHIP_FAILURE);
this.relationships.set(Collections.unmodifiableSet(relationships));
}
@Override
protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
return properties;
}
@Override
public Set<Relationship> getRelationships() {
return relationships.get();
}
@OnStopped
public void processorTasksStopped() {
writeLock.lock();
try {
// Since the user can change thresholds when the processor is stopped,
// force the user thresholds to be reloaded on the next call to onTrigger()
this.thresholds = null;
} finally {
writeLock.unlock();
}
}
@Override
public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
final ProcessorLog logger = getLogger();
//ProvenanceReporter provenanceReporter = session.getProvenanceReporter();
getCounts(context, session, logger);
if (shouldClearCounts(context)) {
clearCounts(context, session, logger);
}
if (shouldSaveCounts(context)) {
saveCounts(getCountsFile(context), logger);
}
FlowFile flowFile = null;
try {
final Map<String, Map<String, CompareRecord>> thresholds = getThresholds(context, logger);
if (thresholds == null) {
logger.error(this + " unable to read thresholds.");
return;
}
flowFile = session.get();
if (flowFile == null) {
return;
}
// prep to loop through all thresholds for this FlowFile
final long fileSize = getFileSize(context, logger, flowFile);
final String delimiter = context.getProperty(OPT_DELIMITER_TO_USE_FOR_COUNTS_ATTRIBUTES).getValue();
boolean aggregateCountsWhenNoThresholdProvided = Boolean.parseBoolean(context.getProperty(OPT_AGGREGATE_COUNTS_WHEN_NO_THRESHOLD_PROVIDED).getValue());
String addAttributes = context.getProperty(OPT_ADD_ATTRIBUTES).getValue();
int maxPairsToAdd = Integer.parseInt(context.getProperty(OPT_MAX_ATTRIBUTE_PAIRS_TO_ADD_WHEN_MULTIPLE_THRESHOLD_EXCEEDED).getValue());
int countOfPairsAdded = 0;
final List<String> reasonsThresholdsExceeded = new ArrayList<>();
long numApplicableThresholds = 0, numThresholdsExceeded = 0;
// loop through all thresholds for this FlowFile
for (final String attributeName : thresholds.keySet()) { //user configured thresholds
String attributeValue = flowFile.getAttribute(attributeName);
if (attributeValue == null) { // attribute does not exist on this FlowFile
continue; // check for any remaining thresholds that might apply to attributes on this FlowFile
}
CompareRecord threshold = thresholds.get(attributeName).get(attributeValue); //has (user configured) thresholds only, not actual counts
CompareRecord countsForAttrValue = null;
if (threshold == null) { //user did not specify a threshold for the attributeValue found in the FlowFile
threshold = thresholds.get(attributeName).get(DEFAULT_THRESHOLDS_KEY); //use default thresholds
if (aggregateCountsWhenNoThresholdProvided) {
// ensure there is a default count record
countsForAttrValue = getCountsForAttributeValue(getCountsForAttribute(attributeName), DEFAULT_THRESHOLDS_KEY, attributeName, threshold); // if no default count record available, one will be created
} else {
countsForAttrValue = getCountsForAttributeValue(getCountsForAttribute(attributeName), attributeValue, attributeName, threshold); // if no count record available, one will be created
}
} else { // there is a user specified threshold
countsForAttrValue = getCountsForAttributeValue(getCountsForAttribute(attributeName), attributeValue, attributeName, threshold); // if no count record available, one will be created
}
// update internal (processor) counts
countsForAttrValue.addToFlowFileCounts(fileSize);
String attributeNameAndValue = delimiter + countsForAttrValue.getAttributeName() + delimiter + attributeValue;
if (addAttributes.equals(ALWAYS)) {
flowFile = session.putAttribute(flowFile, FILE_COUNT_PREFIX + attributeNameAndValue, Integer.toString(countsForAttrValue.getFileCount()));
flowFile = session.putAttribute(flowFile, FILE_THRESHOLD_PREFIX + attributeNameAndValue, Integer.toString(countsForAttrValue.getFileThreshold()));
countOfPairsAdded++;
flowFile = session.putAttribute(flowFile, BYTE_COUNT_PREFIX + attributeNameAndValue, Long.toString(countsForAttrValue.getByteCount()));
flowFile = session.putAttribute(flowFile, BYTE_THRESHOLD_PREFIX + attributeNameAndValue, Long.toString(countsForAttrValue.getByteThreshold()));
countOfPairsAdded++;
}
// check thresholds
if (countsForAttrValue.fileThresholdExceeded()) {
numThresholdsExceeded++;
reasonsThresholdsExceeded.add(countsForAttrValue.getReasonFileLimitExceeded());
if (addAttributes.equals(ONLY_WHEN_THRESHOLD_EXCEEDED) && (maxPairsToAdd >= 0) && ((maxPairsToAdd == 0) || (countOfPairsAdded < maxPairsToAdd))) {
//String attributeNameAndValue = delimiter + countsForAttrValue.getAttributeName() + delimiter + attributeValue;
flowFile = session.putAttribute(flowFile, FILE_COUNT_PREFIX + attributeNameAndValue, Integer.toString(countsForAttrValue.getFileCount()));
flowFile = session.putAttribute(flowFile, FILE_THRESHOLD_PREFIX + attributeNameAndValue, Integer.toString(countsForAttrValue.getFileThreshold()));
countOfPairsAdded++;
}
}
if (countsForAttrValue.byteThresholdExceeded()) {
numThresholdsExceeded++;
reasonsThresholdsExceeded.add(countsForAttrValue.getReasonByteLimitExceeded());
if (addAttributes.equals(ONLY_WHEN_THRESHOLD_EXCEEDED) && (maxPairsToAdd >= 0) && ((maxPairsToAdd == 0) || (countOfPairsAdded < maxPairsToAdd))) {
//String attributeNameAndValue = delimiter + countsForAttrValue.getAttributeName() + delimiter + attributeValue;
flowFile = session.putAttribute(flowFile, BYTE_COUNT_PREFIX + attributeNameAndValue, Long.toString(countsForAttrValue.getByteCount()));
flowFile = session.putAttribute(flowFile, BYTE_THRESHOLD_PREFIX + attributeNameAndValue, Long.toString(countsForAttrValue.getByteThreshold()));
countOfPairsAdded++;
}
}
numApplicableThresholds = numApplicableThresholds + 2; // the UI requires that both a file and byte threshold be supplied
} // end loop through all attributes with thresholds
if (numThresholdsExceeded > 0) {
logLimitsWereExceeded(logger, flowFile, reasonsThresholdsExceeded);
}
flowFile = session.putAttribute(flowFile, NUM_APPLICABLE_THRESHOLDS_KEY, Long.toString(numApplicableThresholds));
flowFile = session.putAttribute(flowFile, NUM_THRESHOLDS_EXCEEDED_KEY, Long.toString(numThresholdsExceeded));
session.transfer(flowFile, RELATIONSHIP_SUCCESS);
} catch (final Throwable t) {
if (flowFile != null) {
logger.warn("Failed to process " + flowFile + ". Sending to failure relationship.", t);
session.transfer(flowFile, RELATIONSHIP_FAILURE);
}
}
} //end onTrigger(...)
private void logLimitsWereExceeded(ProcessorLog logger, final FlowFile flowFile, final List<String> reasonsThresholdExceeded) {
final StringBuilder sb = new StringBuilder();
sb.append("Threshold(s) Exceeded for: ").append(flowFile);
int numberOfReasons = reasonsThresholdExceeded.size();
for (int i = 0; i < numberOfReasons; i++) {
sb.append(reasonsThresholdExceeded.get(i));
}
logger.info("\n" + sb.toString());
}
private long getFileSize(final ProcessContext context, final ProcessorLog logger, FlowFile flowFile) {
long fileSize = flowFile.getSize();
String attributeToUseForCountingBytes = context.getProperty(OPT_ATTRIBUTE_TO_USE_FOR_COUNTING_BYTES).getValue();
if (attributeToUseForCountingBytes != null) {
if (attributeToUseForCountingBytes.equals(STANDARD_FLOWFILE_FILESIZE_PROPERTY)) {
return fileSize;
}
final String valueToUseForCountingBytes = flowFile.getAttribute(attributeToUseForCountingBytes);
if (valueToUseForCountingBytes != null) {
try {
final int intVal = Integer.parseInt(valueToUseForCountingBytes);
if (intVal >= 0) {
fileSize = intVal;
} else {
logger.warn(this + " Value to use for counting bytes must be numeric and non-negative. Found: " + valueToUseForCountingBytes);
}
} catch (final NumberFormatException e) {
logger.warn(this + " Value to use for counting bytes must be numeric and non-negative. Found: " + valueToUseForCountingBytes);
}
} else {
logger.warn(this + " Value to use for counting bytes cannot be null. It must be numeric and non-negative. Found: " + valueToUseForCountingBytes);
}
} else {
logger.warn(this + ATTRIBUTE_TO_USE_FOR_COUNTING_BYTES_PROPERTY + " property cannot be empty.");
}
return fileSize;
}
private ConcurrentHashMap<String, CompareRecord> getCountsForAttribute(final String attributeName) {
ConcurrentHashMap<String, CompareRecord> countsForThisAttribute = null;
readLock.lock();
try {
countsForThisAttribute = this.counts.get(attributeName);
} finally {
readLock.unlock();
}
if (countsForThisAttribute == null) { // if no count record is available, create one
countsForThisAttribute = new ConcurrentHashMap<>();
ConcurrentHashMap<String, CompareRecord> putResult = null;
writeLock.lock();
try {
putResult = this.counts.putIfAbsent(attributeName, countsForThisAttribute);
} finally {
writeLock.unlock();
}
if (putResult != null) {
countsForThisAttribute = putResult;
}
}
return countsForThisAttribute;
}
private CompareRecord getCountsForAttributeValue(ConcurrentHashMap<String, CompareRecord> countsForThisAttribute,
final String attributeValue, final String attributeName, final CompareRecord threshold) {
CompareRecord countsForThisAttributeValue = null;
readLock.lock();
try {
countsForThisAttributeValue = countsForThisAttribute.get(attributeValue);
} finally {
readLock.unlock();
}
if (countsForThisAttributeValue == null) { // if no count record is available, create one
countsForThisAttributeValue = new CompareRecord(threshold, attributeName, attributeValue);
CompareRecord returnedState = null;
writeLock.lock();
try {
returnedState = countsForThisAttribute.putIfAbsent(attributeValue, countsForThisAttributeValue);
} finally {
writeLock.unlock();
}
if (returnedState != null) {
countsForThisAttributeValue = returnedState;
}
}
return countsForThisAttributeValue;
}
protected boolean shouldSaveCounts(ProcessContext context) {
writeLock.lock();
try {
if (this.timeCountsLastSaved == null) {
this.timeCountsLastSaved = System.currentTimeMillis();
}
final String saveSeconds = context.getProperty(OPT_FREQUENCY_TO_SAVE_COUNTS_SECS).getValue();
final int seconds = saveSeconds == null ? DEFAULT_SAVE_COUNTS_FREQ_SECS : Integer.parseInt(saveSeconds);
final long timeToSave = seconds * 1000 + this.timeCountsLastSaved;
return timeToSave <= System.currentTimeMillis();
} finally {
writeLock.unlock();
}
}
protected void saveCounts(final File file, ProcessorLog logger) {
readLock.lock();
try {
if (this.counts == null) {
return;
}
try {
final FileOutputStream fos = new FileOutputStream(file);
final ObjectOutputStream oos = new ObjectOutputStream(fos);
try {
oos.writeObject(this.counts);
Long time = this.timeCountsLastCleared;
if (time == null) {
time = 0L;
}
oos.writeObject(time);
} finally {
FileUtils.closeQuietly(oos);
}
logger.debug(this + " saved current counts to file " + file.getAbsolutePath());
} catch (final IOException e) {
logger.error("Unable to save counts to file " + file.getAbsolutePath() + " due to " + e);
}
} finally {
readLock.unlock();
}
writeLock.lock();
try {
this.timeCountsLastSaved = System.currentTimeMillis();
} finally {
writeLock.unlock();
}
}
protected synchronized ConcurrentHashMap<String, ConcurrentHashMap<String, CompareRecord>> getCounts(final ProcessContext context, final ProcessSession session, final ProcessorLog logger) {
if (this.counts != null) {
return this.counts;
}
final File file = getCountsFile(context);
if (file.exists()) {
try {
final Tuple<Long, ConcurrentHashMap<String, ConcurrentHashMap<String, CompareRecord>>> tuple = readCountsFile(file);
writeLock.lock();
try {
timeCountsLastCleared = tuple.getKey();
this.counts = tuple.getValue();
} finally {
writeLock.unlock();
}
if (shouldClearCounts(context)) {
logger.warn(this + " restored saved counts but the counts have already expired. Clearing counts now.");
clearCounts(context, session, logger);
} else {
logger.info(this + " restored saved counts");
}
return this.counts;
} catch (final IOException e) {
logger.error(this + " unable to read counts from file " + file.getAbsolutePath() + " due to " + e);
this.counts = new ConcurrentHashMap<>();
}
} else {
this.counts = new ConcurrentHashMap<>();
}
return this.counts;
}
private File getCountsFile(ProcessContext context) {
final String countsFilePrefixVal = context.getProperty(OPT_COUNTS_PERSISTENCE_FILE_PREFIX).getValue();
final String countsFilePrefix = countsFilePrefixVal == null ? DEFAULT_COUNTS_PERSISTENCE_FILE_PREFIX : countsFilePrefixVal;
final String countsFilename = countsFilePrefix + "-" + this.getIdentifier() + ".counts";
final File file = new File(countsFilename);
return file;
}
@SuppressWarnings("unchecked")
protected Tuple<Long, ConcurrentHashMap<String, ConcurrentHashMap<String, CompareRecord>>> readCountsFile(final File file)
throws IOException {
final FileInputStream fis = new FileInputStream(file);
final ObjectInputStream ois = new ObjectInputStream(fis);
try {
final Object mapObject = ois.readObject();
final Long rolloverTime = (Long) ois.readObject();
return new Tuple<>(rolloverTime, (ConcurrentHashMap<String, ConcurrentHashMap<String, CompareRecord>>) mapObject);
} catch (final ClassNotFoundException e) {
throw new IOException(e);
} finally {
FileUtils.closeQuietly(ois);
}
}
private Map<String, Map<String, CompareRecord>> getThresholds(ProcessContext context, ProcessorLog logger) {
Map<String, Map<String, CompareRecord>> thresholds;
readLock.lock();
try {
if (this.thresholds == null || this.thresholds.isEmpty()) {
thresholds = null;
} else {
thresholds = new HashMap<>(this.thresholds);
}
} finally {
readLock.unlock();
}
if (thresholds == null) {
writeLock.lock();
try {
if (this.thresholds == null) {
final ThresholdsParser parser = new ThresholdsParser(logger);
this.thresholds = parser.readThresholds(context.getAnnotationData()); //read thresholds from flow.xml file
}
thresholds = Collections.unmodifiableMap(this.thresholds);
} catch (final Exception e) {
logger.error("Unable to read Thresholds.", e);
return null;
} finally {
writeLock.unlock();
}
}
return thresholds;
}
protected boolean shouldClearCounts(ProcessContext context) {
readLock.lock();
try {
// If we are rolling over based on time, rather than a certain number of minutes,
// then we don't want to allow the millisToClear to be greater than 24 hours.
final boolean usingTime = (context.getProperty(OPT_COUNT_RESET_TIME).getValue() != null);
long millisToClear = calculateCountResetMillis(context);
if (usingTime) {
millisToClear = Math.min(MILLIS_IN_HOUR * 24, millisToClear);
}
long timeToClear = this.timeCountsLastCleared + millisToClear;
return timeToClear < System.currentTimeMillis();
} finally {
readLock.unlock();
}
}
protected long calculateCountResetMillis(ProcessContext context) {
final String rolloverTimeVal = context.getProperty(OPT_COUNT_RESET_TIME).getValue();
final String rolloverMinVal = context.getProperty(OPT_COUNT_RESET_MINUTES).getValue();
if (rolloverTimeVal == null) {
final int minutes = Integer.parseInt(rolloverMinVal);
final long millis = minutes * MILLIS_IN_MINUTE;
return millis;
} else {
final Calendar calendar = Calendar.getInstance(TimeZone.getTimeZone("GMT"));
final String[] rolloverSplit = rolloverTimeVal.split(":");
final int rolloverHour = Integer.parseInt(rolloverSplit[0]);
final int rolloverMinutes = Integer.parseInt(rolloverSplit[1]);
final int rolloverSeconds = Integer.parseInt(rolloverSplit[2]);
calendar.set(Calendar.HOUR_OF_DAY, rolloverHour);
calendar.set(Calendar.MINUTE, rolloverMinutes);
calendar.set(Calendar.SECOND, rolloverSeconds);
calendar.set(Calendar.MILLISECOND, 0);
// Make sure we have the right day.
final Calendar lastRolloverCal = Calendar.getInstance(TimeZone.getTimeZone("GMT"));
readLock.lock();
try {
lastRolloverCal.setTimeInMillis(this.timeCountsLastCleared);
} finally {
readLock.unlock();
}
while (calendar.before(lastRolloverCal)) {
calendar.add(Calendar.DAY_OF_YEAR, 1);
}
return calendar.getTimeInMillis() - lastRolloverCal.getTimeInMillis();
}
}
protected void clearCounts(final ProcessContext context, final ProcessSession session, final ProcessorLog logger) {
writeLock.lock();
try {
if (shouldClearCounts(context)) {
logger.info("\n" + this + " throttling period has elapsed; resetting counts.");
logCounts(context, logger);
counts.clear();
this.timeCountsLastCleared = System.currentTimeMillis();
}
} finally {
writeLock.unlock();
}
}
protected synchronized void logCounts(ProcessContext context, ProcessorLog logger) {
final StringBuilder sb = new StringBuilder();
sb.append("\n>>>>>>>> MonitorThreshold Report: <<<<<<<<");
for (String count : this.counts.keySet()) {
sb.append("\nAttribute with Threshold: ").append(count);
ConcurrentHashMap<String, CompareRecord> countsForThisAttribute = getCountsForAttribute(count);
for (String value : countsForThisAttribute.keySet()) {
sb.append("\nAttribute: ").append(count).append("\tValue: '").append(value).append("'");
CompareRecord countsForThisAttributeValue = countsForThisAttribute.get(value);
sb.append("\n\tFiles Threshold: ").append(countsForThisAttributeValue.getFileThreshold())
.append("\tBytes Threshold: ").append(countsForThisAttributeValue.getByteThreshold())
.append("\n\tNum files seen: ").append(countsForThisAttributeValue.getFileCount())
.append("\tNum bytes seen: ").append(countsForThisAttributeValue.getByteCount());
}
}
sb.append("\n>>>>>> End MonitorThreshold Report. <<<<<<");
String logReport = sb.toString();
logger.info(logReport);
}
// CompareRecord does double duty.
// A single CompareRecord instance is used to contain thresholds (but kept in the count variables).
// Other CompareRecord instances contain actual counts, each with a reference back to their 'governing' threshold instance.
public static class CompareRecord implements Serializable {
private static final long serialVersionUID = -2759458735761055366L;
private int fileCount;
private long byteCount;
private CompareRecord threshold = null; // reference to the instance containing the thresholds
private final String attributeName;
private final String attributeValue;
private final ReentrantReadWriteLock rwLock = new ReentrantReadWriteLock();
private final Lock compareRecordReadLock = rwLock.readLock();
private final Lock compareRecordWriteLock = rwLock.writeLock();
// Used for creating a 'threshold' record.
public CompareRecord(final String attributeName, final String attributeValue, final int fileThreshold, final long byteThreshold) {
this.attributeName = attributeName;
this.attributeValue = attributeValue;
this.fileCount = fileThreshold; // for threshold records, save thresholds in count fields
this.byteCount = byteThreshold; // for threshold records, save thresholds in count fields
}
// Used for creating a 'count' record (that is compared against thresholds to determine whether or not a FlowFile exceeds a threshold).
public CompareRecord(final CompareRecord threshold, final String attributeName, final String attributeValue) {
this(attributeName, attributeValue, 0, 0); // begin with 0 for the counts
this.threshold = threshold; // "connect" this count record to its corresponding threshold record
}
public void addToFlowFileCounts(final long fileSize) {
if (threshold == null) {
throw new IllegalStateException("Attempted to add to counts when no threshold exists!");
}
compareRecordWriteLock.lock();
try {
byteCount += fileSize;
fileCount++;
} finally {
compareRecordWriteLock.unlock();
}
}
public String getReasonFileLimitExceeded() {
long fileCnt = getFileCount();
long fileThreshold = getFileThreshold();
if (getFileThreshold() < fileCnt) {
return "\n\tAttribute: '" + attributeName + "' Value '" + attributeValue + "'"
+ "\tFile threshold: " + fileThreshold
+ "\tExpected Count: " + fileCnt + " (counting this file)"
+ ((fileThreshold < fileCnt) ? "\t[threshold exceeded]" : "\t[below thresholds]");
}
return null;
}
public String getReasonByteLimitExceeded() {
long byteCnt = getByteCount();
long byteThreshold = getByteThreshold();
if (byteThreshold < byteCnt) {
return "\n\tAttribute: '" + attributeName + "' Value '" + attributeValue + "'"
+ "\tByte threshold: " + byteThreshold
+ "\tExpected Count: " + byteCnt + " (counting this file)"
+ ((byteThreshold < byteCnt) ? "\t[threshold exceeded]" : "\t[below thresholds]");
}
return null;
}
public boolean fileThresholdExceeded() {
return getFileThreshold() < getFileCount();
}
public boolean byteThresholdExceeded() {
return getByteThreshold() < getByteCount();
}
public long getByteCount() {
compareRecordReadLock.lock();
try {
return byteCount;
} finally {
compareRecordReadLock.unlock();
}
}
public int getFileCount() {
compareRecordReadLock.lock();
try {
return fileCount;
} finally {
compareRecordReadLock.unlock();
}
}
public long getByteThreshold() {
compareRecordReadLock.lock();
try {
return threshold.getByteCount();
} finally {
compareRecordReadLock.unlock();
}
}
public int getFileThreshold() {
compareRecordReadLock.lock();
try {
return threshold.getFileCount();
} finally {
compareRecordReadLock.unlock();
}
}
public String getAttributeName() {
return attributeName;
}
public String getAttributeValue() {
return this.attributeValue;
}
} //end CompareRecord inner class
} // end MonitorThreshold class

134
nifi/nar-bundles/monitor-threshold-bundle/processor/src/main/java/org/apache/nifi/processors/monitor/ThresholdsParser.java
View File

@ -1,134 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.processors.monitor;
/*
* NOTE: rule is synonymous with threshold
*/
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import org.apache.nifi.logging.ProcessorLog;
import org.apache.nifi.processors.monitor.MonitorThreshold.CompareRecord;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.NamedNodeMap;
import org.w3c.dom.Node;
import org.w3c.dom.NodeList;
import org.xml.sax.InputSource;
import org.xml.sax.SAXException;
public class ThresholdsParser {
private final ProcessorLog logger;
public ThresholdsParser(ProcessorLog logger) {
this.logger = logger;
}
private CompareRecord parseThresholds(final String attributeName, final Node valueNode) {
final NamedNodeMap attributes = valueNode.getAttributes();
final long byteLimit = Long.parseLong(attributes.getNamedItem("size").getTextContent());
final int filesLimit = Integer.parseInt(attributes.getNamedItem("count").getTextContent());
String value = null;
if (valueNode.getNodeName().equals("noMatchRule")) {
value = MonitorThreshold.DEFAULT_THRESHOLDS_KEY;
} else {
value = attributes.getNamedItem("id").getTextContent();
}
if (value == null) {
throw new IllegalArgumentException("Thresholds for " + attributeName + ", size=" + byteLimit + ", count=" + filesLimit + " has no ID");
}
final CompareRecord thresholdsState = new CompareRecord(attributeName, value, filesLimit, byteLimit);
return thresholdsState;
}
public Map<String, Map<String, CompareRecord>> readThresholds(String values)
throws ParserConfigurationException, SAXException, IOException {
final Map<String, Map<String, CompareRecord>> thresholdsMap = new HashMap<>();
final DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance();
final DocumentBuilder db = factory.newDocumentBuilder();
final Document doc = db.parse(new InputSource(new ByteArrayInputStream(values.getBytes())));
final Element configurationElement = doc.getDocumentElement();
final NodeList secondLevelElementList = configurationElement.getChildNodes();
if (secondLevelElementList.getLength() == 0) {
logger.warn("Trying to read thresholds, but no thresholds were parsed.");
}
// Loop through the the list of FlowFile attributes (i.e., <flowFileAttribute> elements) and their list of values with limits
for (int i = 0; i < secondLevelElementList.getLength(); i++) {
final Node element = secondLevelElementList.item(i);
final String localName = element.getNodeName();
if (localName != null && localName.equalsIgnoreCase("flowFileAttribute")) {
if (element.getNodeType() != Node.ELEMENT_NODE) {
logger.debug("Encountered non-ELEMENT_NODE while parsing thresholds: " + element.getNodeName());
continue;
}
final String attributeName = element.getAttributes().getNamedItem("attributeName").getNodeValue();
final NodeList children = element.getChildNodes();
CompareRecord noMatchRule = null;
List<CompareRecord> thresholds = new ArrayList<>();
for (int j = 0; j < children.getLength(); j++) {
final Node valueNode = children.item(j);
if (valueNode == null || valueNode.getNodeType() != Node.ELEMENT_NODE) {
continue;
}
final String nodeName = valueNode.getNodeName();
if (nodeName.equalsIgnoreCase("noMatchRule")) {
noMatchRule = parseThresholds(attributeName, valueNode);
} else if (nodeName.equalsIgnoreCase("rule")) {
thresholds.add(parseThresholds(attributeName, valueNode));
} else {
throw new SAXException("Invalid Threshold Configuration: child of 'attributeName' element was not 'noMatchRule', or 'threshold'");
}
}
if (noMatchRule == null) {
throw new SAXException("Invalid Threshold Configuration: no 'noMatchRule' element found for 'attributeName' element");
}
Map<String, CompareRecord> comparisonRecords = thresholdsMap.get(attributeName);
if (comparisonRecords == null) {
comparisonRecords = new HashMap<>();
thresholdsMap.put(attributeName, comparisonRecords);
}
comparisonRecords.put(noMatchRule.getAttributeValue(), noMatchRule);
for (final CompareRecord rs : thresholds) {
comparisonRecords.put(rs.getAttributeValue(), rs);
}
}
} // end loop through all listed attributes
return thresholdsMap;
}
}

15
nifi/nar-bundles/monitor-threshold-bundle/processor/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor
View File

@ -1,15 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
org.apache.nifi.processors.monitor.MonitorThreshold

485
nifi/nar-bundles/monitor-threshold-bundle/processor/src/main/resources/docs/org.apache.nifi.processors.monitor.MonitorThreshold/index.html
View File

@ -1,485 +0,0 @@
<!DOCTYPE html>
<html lang="en">
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<head>
<meta charset="utf-8" />
<title>MonitorThreshold</title>
<link rel="stylesheet" href="../css/component-usage.css" type="text/css" />
</head>
<body>
<h2>Description:</h2>
<p>This processor examines values found in selected FlowFile attributes and
maintains counts of the number of times the values have been encountered.
The counts are then used to check against user defined thresholds.
Both the counts and the thresholds can be exposed as FlowFile attributes, for
later examination and use by downstream processors.
</p>
<strong>Thresholds for Each Identified Attribute Value</strong>
<p>One or more FlowFile attributes can be identified for monitoring purposes, using
this processor’s advanced user interface (use the Advanced button when configuring the processor).
</p>
<p>In the following example, two FlowFile attributes have been identified: category and priority.
In other words, for each FlowFile that is presented to this processor both the category and priority
attributes (if they exist) will be examined to see if thresholds have been exceeded. For each attribute
that is identified, different thresholds (and counts) are used for every unique attribute value.
Finally, for each attribute value, two types of thresholds are supported; size (bytes) and number of files.
</p>
<p><strong>Example Thresholds:</strong></p>
<table border="1">
<tr><th>Attribute Names</th><th>Attribute Values</th><th>Size Threshold (bytes)</th><th>File Threshold<th></tr>
<tr><td>category</td><td>Default</td><td>10000</td><td>100</td></tr>
<tr><td></td><td>red</td><td>1000</td><td>10</td></tr>
<tr><td></td><td>green</td><td>1000</td><td>5</td></tr>
<tr><th></th><th>Attribute Values</th><th>Size Threshold (bytes)</th><th>File Threshold<th></tr>
<tr><td>priority</td><td>Default</td><td>100000</td><td>200</td></tr>
<tr><td></td><td>blue</td><td>1000</td><td>50</td></tr>
<tr><td></td><td>yellow</td><td>1000</td><td>100</td></tr>
</table>
<br><strong>When Thresholds are Exceeded</strong>
<p>
To allow downstream flows (processors) to react, this processor adds two new attributes
(<strong>numApplicableThresholds</strong> and <strong>numThresholdsExceeded</strong>) to
every FlowFile that passes through the processor, no matter whether a threshold is exceeded.
</p>
<p>
In addition, if a monitored attribute value is present on a FlowFile, and
<strong>'Add Attributes'</strong> is set to 'Only When Threshold Exceeded',
this processor will add an additional pair of attributes to each FlowFile that exceeds
the threshold; one attribute will contain the threshold and the other attribute will contain the count
(that exceeded the threshold), e.g.,
<ul>
<li>If a byte count threshold is exceeded, both
<ul>
<li><strong>byteCount<i>.attributeName.value</i></strong> and </li>
<li><strong>byteThreshold<i>.attributeName.value</i></strong> </li>
</ul>
attributes will be added to the FlowFile
</li>
<li>If a file count threshold is exceeded, both
<ul>
<li><strong>fileCount<i>.attributeName.value</i></strong> and </li>
<li><strong>fileThreshold<i>.attributeName.value</i></strong> </li>
</ul>
attributes will be added to the FlowFile,
where <strong><i>attributeName</i></strong> is replaced with the
actual attribute name and <strong><i>value</i></strong> is replaced with the actual attribute value.
</li>
</ul>
</p>
<p>For example, assuming the aforementioned thresholds (from the preceding table) are in effect, if
<ul>
<li>a FlowFile is encountered with a category value of red,</li>
<li>and it is the 11th such file encountered,</li>
</ul>
then the file count threshold (=10) would be exceeded.
This would cause the following new attributes to be added to the FlowFile:
<p>
<table border="1">
<tr><th>Attribute Name</th><th>Value</th></tr>
<tr><td><strong>fileCount.category.red</strong></td><td>11</td></tr>
<tr><td><strong>fileThreshold.category.red</strong></td><td>10</td></tr>
</table>
</p>
</p>
<p>
Note that when a threshold is exceeded, it is always noted in the
log, no matter the value of <strong>'Add Attributes'</strong>.
</p>
<p>
Finally, if a monitored attribute value is present on the FlowFile, and
<strong>'Add Attributes'</strong> is set to 'Always', <strong>four</strong> additional
attributes will be added to FlowFiles, <i>no matter whether a threshold has been exceeded</i>.
(This can be a convenient mechanism for counting values found in FlowFile attributes.)
<br>The four additional attributes are: <strong>fileCount<i>.attributeName.value</i></strong>,
a <strong>fileThreshold<i>.attributeName.value</i></strong>, a <strong>byteCount<i>.attributeName.value</i></strong>,
and a <strong>byteThreshold<i>.attributeName.value</i></strong> where <strong><i>attributeName</i></strong> is replaced with the
actual attribute name and <strong><i>value</i></strong> is replaced with the actual attribute value.
</p>
<p>
Note that a FlowFile must have a monitored attribute before corresponding count/threshold attributes will be added.
In other words, if a particular FlowFile does not have a monitored attribute, the associated
count/threshold attributes will not be added under any circumstances (no matter the value of 'Add Attributes').
</p>
<br><strong>When Multiple Thresholds are Exceeded</strong>
<p>
If more than one threshold is exceeded, a pair of attributes are added <i>for each</i> threshold
that is exceeded (subject to the values of the
<strong>'Add Attributes'</strong> and
<strong>'Maximum Attribute Pairs to Add When Multiple Thresholds Exceeded' </strong>properties).
</p>
<br><strong>Default Thresholds Required</strong>
<p>
Since the range of possible values (for a particular FlowFile attribute) can be large/unknown
in advance, it is conceivable that thresholds may not be supplied for all possible values.
To account for this possibility, a <strong><i>default threshold must be specified for every attribute that is
monitored</i></strong>. Default thresholds are used when a "monitored" attribute has a value for
which no thresholds were supplied.
</p>
<p>
For example, in the following table, assume that thresholds have been specified for FlowFiles
with a category value of red or green, but no thresholds were specified for category values of cyan or magenta.
FlowFiles that are encountered with a category value of cyan or magenta will be checked using the default
thresholds.
</p>
<table border="1">
<tr><th>Attribute Names</th><th>Attribute Values</th><th>Size Threshold (bytes)</th><th>File Threshold<th></tr>
<tr><td>category</td><td>Default</td><td>10000</td><td>100</td></tr>
<tr><td></td><td>red</td><td>1000</td><td>10</td></tr>
<tr><td></td><td>green</td><td>1000</td><td>5</td></tr>
<tr><td></td><td><strike>cyan value not supplied</strike></td><td>(use Default)</td><td>(use Default)</td></tr>
<tr><td></td><td><strike>magenta value not supplied</strike></td><td>(use Default)</td><td>(use Default)</td></tr>
<tr><th></th><th>Attribute Values</th><th>Size Threshold (bytes)</th><th>File Threshold<th></tr>
<tr><td>priority</td><td>Default</td><td>100000</td><td>200</td></tr>
<tr><td></td><td>blue</td><td>1000</td><td>50</td></tr>
<tr><td></td><td>yellow</td><td>1000</td><td>100</td></tr>
</table>
<br><strong>Counting Without Thresholds</strong>
<p>
When a value (in a monitored attribute) is encountered, but no thresholds have been specified
for that value, the <strong>'Aggregate Counts When No Threshold Provided'</strong> determines how counts are maintained.
<ul>
<li>If false, then :
<ul>
<li>Separate, individual counts are maintained for each non-threshold, value that is encountered.
As shown in the following table, if a FlowFile has a category value of cyan, its counts would be added to the cyan counts.
If a FlowFile has a category value of magenta, its counts would be added to the magenta counts
- even though no thresholds were provided for category values cyan or magenta.
<table border="1">
<tr><th>Attribute Names</th><th>Attribute Values</th><th>Size Threshold (bytes)</th><th><i>byte count</i></th><th>File Threshold</th><th><i>file count</i></th></tr>
<tr><td>category</td><td>Default</td><td>10000</td><td><strike>n/a</strike></td><td>100</td><td><strike>n/a</strike></td></tr>
<tr><td></td><td>red</td><td>1000</td><td></td><td>10</td><td></td></tr>
<tr><td></td><td>green</td><td>1000</td><td></td><td>5</td><td></td></tr>
<tr><td></td><td><strike>cyan value not supplied</strike></td><td>(use Default threshold)</td><td></td><td>(use Default threshold)</td><td></td></tr>
<tr><td></td><td><strike>magenta value not supplied</strike></td><td>(use Default threshold)</td><td></td><td>(use Default threshold)</td><td></td></tr>
<tr><th></th><th>Attribute Values</th><th>Size Threshold (bytes)</th><th><i>byte count</i></th><th>File Threshold</th><th><i>file count</i></th></tr>
<tr><td>priority</td><td>Default</td><td>100000</td><td><strike>n/a</strike></td><td>200</td><td><strike>n/a</strike></td></tr>
<tr><td></td><td>blue</td><td>1000</td><td></td><td>50</td><td></td></tr>
<tr><td></td><td>yellow</td><td>1000</td><td></td><td>100</td><td></td></tr>
</table>
NOTE: Counts (shown in the above table) can <strong>not</strong> be viewed in the Advanced user interface. Only thresholds can be viewed in the Advanced user interface.
</li>
<li>When a non-threshold value is encountered, it's unique count will be compared with the default threshold.
So, if a FlowFile has a category value of cyan, the cyan count will be checked against the default thresholds (since no
thresholds were provided for a cyan category value). If a FlowFile has a category value of magenta, magenta
counts will be checked against the (same) default thresholds (since no thresholds were provided for a magenta category).
</li>
</ul>
</li>
<li>If true (default), then:
<ul>
<li>Counts for ALL non-threshold values are aggregated into a single default count. As shown in the following table,
all FlowFiles with a category value of cyan or magenta will have their counts added to a single, default count.</li>
<table border="1">
<tr><th>Attribute Names</th><th>Attribute Values</th><th>Size Threshold (bytes)</th><th><i>byte count</i></th><th>File Threshold</th><th><i>file count</i></th></tr>
<tr><td>category</td><td>Default</td><td>10000</td><td></td><td>100</td><td></td></tr>
<tr><td></td><td>red</td><td>1000</td><td></td><td>10</td><td></td></tr>
<tr><td></td><td>green</td><td>1000</td><td></td><td>5</td><td></td></tr>
<tr><td></td><td><strike>cyan value not supplied</strike></td><td>(use Default threshold)</td><td>(use Default count)</td><td>(use Default threshold)</td><td>(use Default count)</td></tr>
<tr><td></td><td><strike>magenta value not supplied</strike></td><td>(use Default threshold)</td><td>(use Default count)</td><td>(use Default threshold)</td><td>(use Default count)</td></tr>
<tr><th></th><th>Attribute Values</th><th>Size Threshold (bytes)</th><th><i>byte count</i></th><th>File Threshold</th><th><i>file count</i></th></tr>
<tr><td>priority</td><td>Default</td><td>100000</td><td></td><td>200</td><td></td></tr>
<tr><td></td><td>blue</td><td>1000</td><td></td><td>50</td><td></td></tr>
<tr><td></td><td>yellow</td><td>1000</td><td></td><td>100</td><td></td></tr>
</table>
NOTE: Counts (shown in the above table) can <strong>not</strong> be viewed in the Advanced user interface. Only thresholds can be viewed in the Advanced user interface.
<li>When a non-threshold value is encountered, the default count (containing counts for both cyan and magenta
category values) will be compared to the default threshold.</li>
</ul>
</li>
</ul>
</p>
<p>NOTE 1: Counts are never maintained for attributes that have not been
identified for monitoring, i.e., do not have a default threshold. Once an attribute has
been identified for monitoring (has a default threshold), counts are always maintained
for thresholds that have been specified. The <strong>'Aggregate Counts When No Threshold Provided'</strong>
property only applies in situations where an attribute has been identified for monitoring, and a value is
encountered with no threshold.
</p>
<p>
NOTE 2: If separate counts are maintained for each unique value for each attribute with thresholds
(<strong>'Aggregate Counts When No Threshold Provided'</strong> is set to <strong>true</strong>),
consideration should be given to the choice of attributes to be monitored. An attribute that is
known to have a range of 1 million values, could (ultimately) result in this processor maintaining
2 million counts in memory.
If the range of values for an attribute are expected to be larger than available memory resources
(for maintaining counts), set the <strong>'Aggregate Counts When No Threshold Provided'</strong> property
to <strong>false</strong>. Adopting this approach will require that specific thresholds be provided for any
attribute values that should be "counted" separately.
</p>
<p><strong>Separate Instance Equals Separate Counts</strong></p>
<p>
Each MonitorThreshold instance maintains its own, separate counts. This is likely the expected
behavior when there are multiple MonitorThreshold instances on a flow. However, this has significant
implications when only a single MonitorThreshold instance appears on a flow, but the flow is running
on a NiFi <strong><i>cluster</i></strong>. In this case, each node (in the cluster) will have its
own instance of MonitorThreshold, each with its own,independent set of counts (unless MonitorThreshold is running isolated).
Assuming an even distribution of FlowFiles across all of the nodes in the
cluster will result in counts that increase slower, and therefore exceed
thresholds later, than might otherwise be expected. Consequently, thresholds may need to be set lower
in clustered environments, in order to achieve desired results.
</p>
<p><strong>Resetting Counts</strong></p>
<p>
To avoid counts that increase forever (that might eventually force all thresholds to be exceeded),
counts are reset on a periodic basis. Resets can be scheduled for a particular <strong>'Count Reset Time'</strong>
every day, or after a specific number of <strong>'Minutes to Wait Before Resetting Counts'</strong> have elapsed. At
least one value must be provided. If <strong>“Count Reset Timeâ€<C3A2></strong> is not provided, then the minutes to delay
are counted from the last reset.
</p>
<p>
When counts are reset, the current thresholds, total number of bytes and the number of files seen
for each Attribute and Value pair is printed in the logs, and the counts are then reset to zero.
</p>
<p><strong>Changing Thresholds</strong></p>
<p>
If changes are made to other parameters, such as the size or file thresholds,
after the processor has already been running, then the changes will not go into effect until the
Minutes to Wait… point has been reached. To circumvent this problem, consider temporarily changing the
<strong>'Count Reset Time'</strong> (and stop/restart the processor) to force the changes to be implemented sooner; then
revert back to the desired <strong>'Count Reset Time'</strong>.
</p>
<p>
<p><strong>Persisting Counts</strong></p>
<p>
To avoid unplanned count resets (due to system restarts, etc…), counts are persisted to disk on
a user defined periodic basis according to the <strong>'Frequency to Persist Counts (seconds)'</strong> property.
</p>
<p><strong>Modifies Attributes:</strong>
<table border="1">
<tr><th>Attribute Name</th><th>Description</th></tr>
<tr><td>numApplicableThresholds</td><td>The number of thresholds that were applicable for the FlowFile. Added to every FlowFile.</td></tr>
<tr><td>numThresholdsExceeded</td><td>The number of thresholds that the FlowFile exceeded. Added to every FlowFile.</td></tr>
<tr><td>fileThreshold<i>.attributeName.value</i></td><td>Contains the file threshold for an attribute name and value.
<br>A separate fileThreshold.attributeName.value attribute is added for each file threshold that is exceeded.
<br><i>attributeName</i> will be the name of the attribute who's file threshold was exceeded.
<br><i>value</i> will be the value of the attribute encountered on the FlowFile.
<br><br>Added <u>only</u> if <i>attributeName</i> is a monitored attribute <u>and</u> <i>attributeName</i>
is present on the FlowFile <u>and</u> one of the following is true:
<ul><li><strong>'Add Attributes'</strong> is set to 'Always' <u>or</u></li>
<li><strong>'Add Attributes'</strong> is set to 'Only When Threshold Exceeded' <u>and</u> a file threshold is exceeded.</li></ul>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(See also: <strong>'Maximum Attribute Pairs to Add When Multiple Thresholds Exceeded'</strong>).
<br><br>Not added when <strong>'Add Attributes'</strong> is set to 'Never'
</td></tr>
<tr><td>fileCount<i>.attributeName.value</i></td><td>Contains the file count for an attribute name and value who's threshold was exceeded.
<br>A separate fileCount.attributeName.value attribute is added for each file threshold that is exceeded.
<br><i>attributeName</i> will be the name of the attribute who's file threshold was exceeded.
<br><i>value</i> will be the value of the attribute encountered on the FlowFile.
<br><br>Added <u>only</u> if <i>attributeName</i> is a monitored attribute <u>and</u> <i>attributeName</i>
is present on the FlowFile <u>and</u> one of the following is true:
<ul><li><strong>'Add Attributes'</strong> is set to 'Always' <u>or</u></li>
<li><strong>'Add Attributes'</strong> is set to 'Only When Threshold Exceeded' <u>and</u> a file threshold is exceeded.</li></ul>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(See also: <strong>'Maximum Attribute Pairs to Add When Multiple Thresholds Exceeded'</strong>).
<br><br>Not added when <strong>'Add Attributes'</strong> is set to 'Never'
</td></tr>
<tr><td>byteThreshold<i>.attributeName.value</i></td><td>Contains the byte threshold for an attribute name and value.
<br>A separate byteThreshold.attributeName.value attribute is added for each byte threshold that is exceeded.
<br><i>attributeName</i> will be the name of the attribute who's byte threshold was exceeded.
<br><i>value</i> will be the value of the attribute encountered on the FlowFile.
<br><br>Added <u>only</u> if <i>attributeName</i> is a monitored attribute <u>and</u> <i>attributeName</i>
is present on the FlowFile <u>and</u> one of the following is true:
<ul><li><strong>'Add Attributes'</strong> is set to 'Always' <u>or</u></li>
<li><strong>'Add Attributes'</strong> is set to 'Only When Threshold Exceeded' <u>and</u> a file threshold is exceeded.</li></ul>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(See also: <strong>'Maximum Attribute Pairs to Add When Multiple Thresholds Exceeded'</strong>).
<br><br>Not added when <strong>'Add Attributes'</strong> is set to 'Never'
</td></tr>
<tr><td>byteCount<i>.attributeName.value</i></td><td>Contains the byte count for an attribute name and value who's threshold was exceeded.
<br>A separate byteCount.attributeName.value attribute is added for each byte threshold is exceeded.
<br><i>attributeName</i> will be the name of the attribute who's byte threshold that was exceeded.
<br><i>value</i> will be the value of the attribute encountered on the FlowFile.
<br><br>Added <u>only</u> if <i>attributeName</i> is a monitored attribute <u>and</u> <i>attributeName</i>
is present on the FlowFile <u>and</u> one of the following is true:
<ul><li><strong>'Add Attributes'</strong> is set to 'Always' <u>or</u></li>
<li><strong>'Add Attributes'</strong> is set to 'Only When Threshold Exceeded' <u>and</u> a file threshold is exceeded.</li></ul>
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;(See also: <strong>'Maximum Attribute Pairs to Add When Multiple Thresholds Exceeded'</strong>).
<br><br>Not added when <strong>'Add Attributes'</strong> is set to 'Never'
</td></tr>
</table>
<p><strong>Properties:</strong>
<br>Required properties appear in bold. Non-bold properties are optional.
If a property has a default, it is indicated.
If a property supports the NiFi Expression Language, it is indicated.
</p>
<p><ul>
<li>Aggregate Counts When No Threshold Provided</li>
<ul>
<li>When a value (in a monitored attribute) is encountered, but no thresholds have been specified
for that value, this property determines how counts are maintained.
<ul>
<li>If true (default):
<ul>
<li>Counts for ALL non-threshold values are aggregated into a single count.</li>
<li>When a non-threshold value is encountered, the aggregate count will be compared to the default threshold.</li>
</ul>
</li>
<li>If false:
<ul>
<li>Separate, individual counts are maintained for each non-threshold, value that is encountered.</li>
<li>When a non-threshold value is encountered, it's unique count will be compared with the default threshold.</li>
</ul>
</li>
</ul>
</p>
<p>NOTE 1: Counts are never maintained for attributes that have not been
identified for monitoring, i.e., do not have a default threshold. Once an attribute has
been identified for monitoring (has a default threshold), counts are always maintained
for thresholds that have been specified. This property only applies in situations
where an attribute has been identified for monitoring, and a value is encountered for which there is no user supplied threshold.
</p>
<p>NOTE 2: Changing this property from false to true will not remove unique, "non-threshold" counts
that are being maintained. It will simply suspend accumulating unique, "non-threshold" counts
and begin accumulating an aggregate/"default" count for all "non-threshold" values.
Setting the property back to false, will resume accumulating unique, "non-threshold" counts.
Since repeatedly changing this property, may give the mistaken appearance that counts are not
accurately maintained, it is advised that this property be set to one value and left unchanged
during a counting "period". See <strong>Count Reset Time</strong> and
<strong>Minutes to Wait Before Resetting Counts </strong> for information on starting a new counting period.
</p>
</li>
<li>Default: true</li>
<li>Supports expression language: false.</li>
</ul>
<li>Count Reset Time</li>
<ul>
<li>The time (24-hour clock, hh:mm:ss format, GMT Timezone) when the byte and file counts
will be reset to zero. All of hours, minutes, and seconds are required, i.e., hh:mm is insufficient.</li>
<li>Default: none – either <strong>Count Reset Time</strong> or <strong>Minutes to Wait Before Resetting Counts </strong>must be provided.</li>
<li>Supports expression language: false.</li>
</ul>
<li>Minutes to Wait Before Resetting Counts</li>
<ul>
<li>Minutes to delay count reset, beginning at Count Reset Time, if provided. If Count
Reset Time is not provided, then the minutes to delay are counted from the last reset.
(Last reset is initially set to the time when the processor is created (added to a flow).)
<br> NOTE: If changes are made to other
parameters, such as the size or file thresholds after the processor has already been running,
then the changes will not go into effect until this Minutes to Wait… point has been reached.
To circumvent this problem, consider temporarily changing the Count Reset Time (and
stop/restart the processor) to force the changes to be implemented sooner; then revert back
to the desired settings.
</li>
<li>Default: none – either <strong>Count Reset Time</strong> or <strong>Minutes to Wait
Before Resetting Counts </strong>must be provided.</li>
<li>Supports expression language: false.</li>
</ul>
<li>Add Attributes</li>
<ul>
<li>Setting this property to 'Only When Threshold Exceeded' (the default) will cause <strong>two</strong> additional attributes to be added to
FlowFiles for every threshold that is exceeded, <i>but only when the threshold is exceeded</i>.
<br>For example, if a file count threshold
is exceeded, a <strong>fileCount.attributeName.value</strong> and a <strong>fileThreshold.attributeName.value</strong>
will be added to the FlowFile.
<li>Setting this property to 'Always' will cause <strong>four</strong> additional attributes to be added to
FlowFiles <i>no matter whether the threshold has been exceeded</i>.
<br>The four additional attributes include a <strong>fileCount.attributeName.value</strong>,
a <strong>fileThreshold.attributeName.value</strong>, a <strong>byteCount.attributeName.value</strong>,
and a <strong>byteThreshold.attributeName.value</strong>.
<li>Setting this property to 'Never' means do not add additional attributes to FlowFiles.
<li>Note 1: Attributes are only added if a monitored attribute is present on the FlowFile, no matter
the value of this property.
<li>Note 2: This processor makes a log entry for every threshold
that is exceeded, no matter the value of this property.</li>
<li>Defaults to: 'Only When Threshold Exceeded'.</li>
<li>Supports expression language: false.</li>
</ul>
<li>Maximum Attribute Pairs to Add When Multiple Thresholds Exceeded</li>
<ul>
<li>Controls/limits the number of FlowFile attributes that are added when
multiple thresholds are exceeded.
<br>Recall that when a threshold is exceeded,
a pair of two attributes are added to the FlowFile, one is the current count and
the other is the threshold. If 100 thresholds are exceeded, then 200
attributes will be added to the FlowFile. Setting this property to zero means add
all count/threshold pairs that were exceeded. Any setting greater than zero indicates
how many count/threshold pairs to add when multiple thresholds are exceeded.
Only non-negative settings are supported. The default is 0.
<br>NOTE: This property is in effect only when <strong>'Add Attributes'</strong> is 'Only When Threshold Exceeded'.
In other words, setting <strong>'Add Attributes'</strong> to 'Always' or 'Never'
causes this property to be ignored.
</li>
<li>Defaults to: 0</li>
<li>Supports expression language: false.</li>
</ul>
<li>Delimiter to Use for Counts Attributes</li>
<ul>
</li><li>The delimiter to use when naming counts that are published as FlowFile
attributes, e.g., fileCount.attributeName.value.
</li><li>Defaults to: “.â€<C3A2>
</li><li>Supports expression language: false.
</ul>
<li>Attribute to use for Counting Bytes</li>
<ul>
<li>Setting this property allows a FlowFile attribute to be used for counting bytes,
in place of the actual fileSize. Note that the attribute named by this property must
contain only numeric, non-negative integer values for each FlowFile. Non-numeric or
negative values that are encountered will be ignored, causing the actual fileSize to
be used instead.</li>
<li>Defaults to: fileSize.</li>
<li>Supports expression language: false.</li>
</ul>
<li>Frequency to Save Counts (seconds)</li>
<ul>
<li>How often all of the counts should be written to disk.</li>
<li>Defaults to: 30 seconds.</li>
<li>Supports expression language: false.</li>
</ul>
<li>Prefix for Counts Persistence File </li>
<ul>
<li>The prefix for the file that is persisted to maintain counts across NIFI restarts.
The actual name of the state file will be this value plus "-XXXX.state" where XXXX is
the processor ID.</li>
<li>Defaults to: conf/MonitorThreshold</li>
<li>Supports expression language: false.</li>
</ul>
</ul></p>
<p>
<strong>Relationships:</strong>
</p>
<ul>
<li>success - all FlowFiles follow this relationship, unless there is a problem with the FlowFile.</li>
<li>failure - FlowFiles follow this relationship path if there is a problem processing the FlowFile, or if there is a unrecoverable configuration error.</li>
</ul>
</p>
</body>
</html>

1560
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/java/org/apache/nifi/processors/monitor/TestMonitorThreshold.java
View File

File diff suppressed because it is too large Load Diff

23
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/testFile
View File

@ -1,23 +0,0 @@
This is the test file used in MonitorThreshold unit tests.
It's content is irrelevant (hence the gibberish below),
since the primary focus is on its attributes;
which are setup in the unit tests.
However, its size is relevant, as it affects the unit tests.
The size of this file should be 1,668 bytes.
There is nothing special about the number 1,668.
It just happened to be the size of the test file that was used for most of the unit tests for the precursor processor,
and it was easier to create a file containing 1,668 bytes than to modify all of the unit tests.
Besides, what difference does it make how big the test file is, as long as it is a known quantity.
1 2 3 4 5 6 7 8
012345678901234567890123456789012345678901234567890123456789012345678901234567890
dfadfsfgasdfgdgbhadg2346475789087908907u3aDFAdfu5675868978aswgt45yeryh56767768rt
dfsfg567878649468947957989078907890689068fjfhjshhjjtre57yjuyju469r4y5h65yuy65476
DFSgfgfsd3578684686fghsd579057890fgadghadghsfyhjsffnsfgn65756757846956h6yh6h6sgh
DFSgfgfsd3578684686fghsd579057890fgadghadghsfyhjsffnsfgn65756757846956h6yh6h6sgh
dfsfg567878649468947957989078907890689068fjfhjshhjjtre57yjuyju469r4y5h65yuy65476
dfsfg567878649468947957989078907890689068fjfhjshhjjtre57yjuyju469r4y5h65yuy65476
dfsfg567878649468947957989078907890689068fjfhjshhjjtre57yjuyju469r4y5h65yuy65476
dfsfg567878649468947957989078907890689068fjfhjshhjjtre57yjuyju469r4y5h65yuy65476
dfsfg567878649468947957989078907890689068fjfhjshhjjtre57yjuyju469r4y5h65yuy65476
wx
01234567890123456789012345678901234567890123456789012345678901234567890123456789

21
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_allow_0.xml
View File

@ -1,21 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="32000" count="6">Default</noMatchRule>
<rule id="2345" size="1000000000" count="0"/>
</flowFileAttribute>
</configuration>

21
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_allow_1.xml
View File

@ -1,21 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="32000" count="6">Default</noMatchRule>
<rule id="2345" size="1000000000" count="1"/>
</flowFileAttribute>
</configuration>

21
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_allow_10_KB.xml
View File

@ -1,21 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="32000" count="6">Default</noMatchRule>
<rule id="2345" size="10000" count="1000000000"/>
</flowFileAttribute>
</configuration>

26
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_allow_10_KB_and_2_KB.xml
View File

@ -1,26 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="32000" count="6">Default</noMatchRule>
<rule id="2345" size="10000" count="1000000000"/>
</flowFileAttribute>
<flowFileAttribute attributeName="priority" id="2">
<noMatchRule size="100000000" count="10000000">Default</noMatchRule>
<rule id="0" size="4000" count="1000000000"/>
<rule id="1" size="2000" count="1000000000"/>
</flowFileAttribute>
</configuration>

21
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_allow_1669_bytes.xml
View File

@ -1,21 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="320000" count="6">Default</noMatchRule>
<rule id="2345" size="1669" count="2"/>
</flowFileAttribute>
</configuration>

21
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_allow_2.xml
View File

@ -1,21 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="320000" count="6">Default</noMatchRule>
<rule id="2345" size="1000000000" count="2"/>
</flowFileAttribute>
</configuration>

21
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_allow_2_default.xml
View File

@ -1,21 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="320000" count="2">Default</noMatchRule>
<rule id="1234" size="1000000000" count="9"/>
</flowFileAttribute>
</configuration>

21
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_allow_3336_bytes_default.xml
View File

@ -1,21 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="3336" count="1000">Default</noMatchRule>
<rule id="1234" size="1000000000" count="9"/>
</flowFileAttribute>
</configuration>

21
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_with_empty_attribute_value.xml
View File

@ -1,21 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="32000" count="6">Default</noMatchRule>
<rule id="" size="1668" count="1"/>
</flowFileAttribute>
</configuration>

22
nifi/nar-bundles/monitor-threshold-bundle/processor/src/test/resources/threshold_settings_with_spaces_for_attribute_values.xml
View File

@ -1,22 +0,0 @@
<?xml version="1.0"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<configuration>
<flowFileAttribute attributeName="category" id="1">
<noMatchRule size="1668" count="1">Default</noMatchRule>
<rule id=" " size="1668" count="1"/>
<rule id=" " size="1668" count="1"/>
</flowFileAttribute>
</configuration>

134
nifi/nar-bundles/monitor-threshold-bundle/ui/pom.xml
View File

@ -1,134 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.nifi</groupId>
<artifactId>monitor-threshold-bundle</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
<artifactId>monitor-threshold-ui</artifactId>
<packaging>war</packaging>
<name>MonitorThreshold-UI</name>
<build>
<resources>
<resource>
<directory>src/main/resources</directory>
</resource>
<resource>
<directory>src/main/xsd</directory>
</resource>
</resources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-war-plugin</artifactId>
<configuration>
<failOnMissingWebXml>false</failOnMissingWebXml>
</configuration>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>jaxb2-maven-plugin</artifactId>
<executions>
<execution>
<id>xjc</id>
<goals>
<goal>xjc</goal>
</goals>
<configuration>
<packageName>org.apache.nifi.settings.generated</packageName>
</configuration>
</execution>
</executions>
<configuration>
<generateDirectory>${project.build.directory}/generated-sources/jaxb</generateDirectory>
</configuration>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>javax.servlet</groupId>
<artifactId>javax.servlet-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.servlet.jsp</groupId>
<artifactId>javax.servlet.jsp-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-client</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-server</artifactId>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-core</artifactId>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-json</artifactId>
</dependency>
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-servlet</artifactId>
</dependency>
<dependency>
<groupId>javax.servlet.jsp.jstl</groupId>
<artifactId>javax.servlet.jsp.jstl-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>javax.el</groupId>
<artifactId>javax.el-api</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>custom-ui-utilities</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.nifi</groupId>
<artifactId>nifi-processor-utils</artifactId>
<scope>provided</scope>
</dependency>
</dependencies>
</project>

36
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/java/org/apache/nifi/monitor/thresholds/ui/AttributeComparator.java
View File

@ -1,36 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.monitor.thresholds.ui;
import java.util.Comparator;
import org.apache.nifi.settings.generated.Attribute;
public class AttributeComparator implements Comparator<Attribute> {
private final String column;
public AttributeComparator(String col) {
column = col;
}
@Override
public int compare(Attribute attr1, Attribute attr2) {
return attr1.getAttributeName().compareTo(attr2.getAttributeName());
}
}

242
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/java/org/apache/nifi/monitor/thresholds/ui/AttributeResource.java
View File

@ -1,242 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.monitor.thresholds.ui;
/*
* NOTE: rule is synonymous with threshold
*/
import java.util.UUID;
import javax.net.ssl.SSLEngineResult.Status;
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.Consumes;
import javax.ws.rs.DefaultValue;
import javax.ws.rs.FormParam;
import javax.ws.rs.DELETE;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.settings.generated.ObjectFactory;
import org.apache.nifi.settings.generated.Rule;
import org.apache.nifi.settings.generated.Attribute;
/**
* MonitorThreshold REST Web Service (for handling changes to attributes made by
* the user in the advanced ui)
*
*/
@Path("/settings/processor/{procid}")
public class AttributeResource extends ThresholdSettingsResource {
@Context
private HttpServletRequest request;
@Context
private ServletContext servletContext;
public AttributeResource() {
super();
//logger.debug("Invoking AttributeResource.java AttributeResource() constructor.");
}
/**
* Returns a list of all attributes
*
* @param processorid
* @param sortcolumn
* @param sord
* @param attributefilter
* @param rowNum
* @param pageCommand
* @return
*/
@GET
@Path("/attributes")
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Produces(MediaType.APPLICATION_XML)
public Response getAllAttributes(@PathParam(PROCID) String processorid,
@QueryParam("sidx") @DefaultValue(ATTRIBUTENAME) String sortcolumn,
@QueryParam("sord") @DefaultValue("asc") String sord,
@QueryParam("attributefilter") @DefaultValue("") String attributefilter,
@QueryParam("rownum") @DefaultValue("-1") String rowNum,
@QueryParam("pagecom") @DefaultValue("") String pageCommand) {
//logger.debug("Invoking AttributeResource.java @GET @PATH(\"/attributes\"): getAllAttributes(...)");
Integer rownum = Integer.parseInt(rowNum);
return generateOkResponse(getConfigFile(request, servletContext).getConfigAttributes(sortcolumn, getIsAsc(sord), attributefilter, rownum, pageCommand)).build();
}
@GET
@Path("/attributeInfo")
@Produces(MediaType.WILDCARD)
public Response getAllAttributes(@PathParam(PROCID) String processorid) {
//logger.debug("Invoking AttributeResource.java @GET @PATH(\"/attributeInfo\"): getAllAttributes(...)");
return generateOkResponse(getConfigFile(request, servletContext).getListStats()).build();
}
/**
* Inserts/adds a new attribute.
*
* @param processorid
* @param attributeName
* @param size
* @param count
* @return
*/
@PUT
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Produces(MediaType.APPLICATION_XML)
@Path("/attribute")
public Response createAttribute(@PathParam(PROCID) String processorid,
@FormParam(ATTRIBUTENAME) @DefaultValue(StringUtils.EMPTY) String attributeName,
@FormParam(SIZE) @DefaultValue(StringUtils.EMPTY) String size,
@FormParam(COUNT) @DefaultValue(StringUtils.EMPTY) String count) {
//logger.debug("Invoking AttributeResource.java @PUT @PATH(\"/attribute\"): createAttribute(" + attributeName + ", " + size + ", " + count + ")");
try {
if (attributeName.isEmpty()) {
validation_error_list.add(INVALID_ATTRIBUTE_NAME);
}
if (!validateStringAsLong(size)) {
validation_error_list.add(INVALID_SIZE);
}
if (!validateStringAsInt(count)) {
validation_error_list.add(INVALID_COUNT);
}
ThresholdsConfigFile thresholds = getConfigFile(request, servletContext);
Attribute tg = thresholds.findAttributebyAttributeName(attributeName);
if (tg != null) {
validation_error_list.add(DUPLICATE_ATTRIBUTE_NAME + " " + attributeName);
}
if (!validation_error_list.isEmpty()) {
return Response.status(400).entity(setValidationErrorMessage(validation_error_list)).build();
}
ObjectFactory of = new ObjectFactory();
//create new attribute
Attribute attr = of.createAttribute();
attr.setAttributeName(attributeName);
attr.setId(UUID.randomUUID().toString());
//create default rule and add to attribute rules
Rule rule = of.createRule();
rule.setValue("Default");
rule.setCount(getBigIntValueOf(count));
rule.setSize(getBigIntValueOf(size));
attr.setNoMatchRule(rule);
thresholds.getFlowFileAttributes().add(attr);
//Save values to config file
thresholds.save();
} catch (Exception ex) {
logger.error(ex.getMessage());
return Response.status(400).entity(GENERAL_ERROR + ex.getMessage()).build();
}
return generateOkResponse(Status.OK.toString()).build();
}
/**
* Updates an attribute.
*
* @param processorid
* @param uuid
* @param attributeName
* @return
*/
@POST
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Produces(MediaType.APPLICATION_XML)
@Path("/attribute")
public Response updateAttribute(@PathParam(PROCID) String processorid,
@FormParam(ATTRIBUTE_UUID) String uuid,
@FormParam(ATTRIBUTENAME) @DefaultValue(StringUtils.EMPTY) String attributeName) {
//logger.debug("Invoking AttributeResource.java @POST @PATH(\"/attribute\"): updateAttribute(" + attributeName + ")");
try {
if (attributeName.isEmpty()) {
validation_error_list.add(INVALID_ATTRIBUTE_NAME);
}
ThresholdsConfigFile config = getConfigFile(request, servletContext);
Attribute tg = config.findAttributebyAttributeName(attributeName);
if (tg != null) {
if (tg.getId().compareTo(uuid) != 0) {
validation_error_list.add(DUPLICATE_ATTRIBUTE_NAME + " " + attributeName);
}
}
if (!validation_error_list.isEmpty()) {
return Response.status(400).entity(setValidationErrorMessage(validation_error_list)).build();
}
Attribute attr = config.findAttribute(uuid);
attr.setAttributeName(attributeName);
//Save values to config file
config.save();
} catch (Exception ex) {
logger.error(ex.getMessage());
return Response.status(400).entity(GENERAL_ERROR + ex.getMessage()).build();
}
return generateOkResponse(Status.OK.toString()).build();
}
/**
* Deletes an attribute.
*
* @param processorid
* @param uuid
* @return
*/
@DELETE
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Produces(MediaType.APPLICATION_XML)
@Path("/attribute")
public Response deleteAttribute(@PathParam(PROCID) String processorid,
@FormParam(ID) String uuid) {
//logger.debug("Invoking AttributeResource.java @DELETE @PATH(\"/attribute\"): deleteAttribute(...)");
try {
ThresholdsConfigFile config = getConfigFile(request, servletContext);
Attribute attr = config.findAttribute(uuid);
if (attr != null) {
config.getFlowFileAttributes().remove(attr);
//Save values to config file
config.save();
}
} catch (Exception ex) {
logger.error(ex.getMessage());
return Response.status(400).entity(GENERAL_ERROR + ex.getMessage()).build();
}
return generateOkResponse(Status.OK.toString()).build();
}
}

46
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/java/org/apache/nifi/monitor/thresholds/ui/RuleComparator.java
View File

@ -1,46 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.monitor.thresholds.ui;
/*
* NOTE: rule is synonymous with threshold
*/
import java.util.Comparator;
import org.apache.nifi.settings.generated.Thresholds;
public class RuleComparator implements Comparator<Thresholds> {
private final String column;
public RuleComparator(String col) {
column = col;
}
@Override
public int compare(Thresholds rule1, Thresholds rule2) {
if (column.compareTo("attributevalue") == 0) {
return rule1.getId().compareTo(rule2.getId());
}
if (column.compareTo("size") == 0) {
return rule1.getSize().compareTo(rule2.getSize());
} else {
return rule1.getCount().compareTo(rule2.getCount());
}
}
}

227
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/java/org/apache/nifi/monitor/thresholds/ui/ThresholdResource.java
View File

@ -1,227 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.monitor.thresholds.ui;
/*
* NOTE: rule is synonymous with threshold
*/
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.Consumes;
import javax.ws.rs.DELETE;
import javax.ws.rs.DefaultValue;
import javax.ws.rs.FormParam;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.PUT;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.Status;
import org.apache.commons.lang3.StringUtils;
import org.apache.nifi.settings.generated.Thresholds;
import org.apache.nifi.settings.generated.Attribute;
import org.apache.nifi.settings.generated.ObjectFactory;
import org.apache.nifi.settings.generated.Rule;
/**
* MonitorThreshold REST Web Service (for handling changes to thresholds made by
* the user in the Advanced UI)
*
*/
@Path("/settings/processor/{procid}/attribute/{attributeuuid}")
public class ThresholdResource extends ThresholdSettingsResource {
@Context
private HttpServletRequest request;
@Context
private ServletContext servletContext;
public ThresholdResource() {
super();
}
@GET
@Path("/rules")
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Produces(MediaType.APPLICATION_XML)
public Response getSubSettings(
@PathParam(PROCID) String processorid,
@PathParam(ATTRIBUTE_UUID) String attributeid,
@QueryParam("sidx") @DefaultValue(ID) String sortcolumn,
@QueryParam("sord") @DefaultValue("asc") String sord,
@QueryParam("attributevaluefilter") @DefaultValue("") String attributevaluefilter,
@QueryParam("sizefilter") @DefaultValue("") String sizefilter,
@QueryParam("filecountfilter") @DefaultValue("") String filecountfilter) {
//logger.debug("Invoking ThresholdResource.java @GET @PATH(\"/rules\"): getSubSettings(...)");
String result = getConfigFile(request, servletContext).getRules(attributeid, sortcolumn, getIsAsc(sord), attributevaluefilter, sizefilter, filecountfilter);
return generateOkResponse(result).build();
}
/**
* Inserts/adds a new threshold.
*
* @param processorid
* @param attributeid
* @param value
* @param size
* @param count
* @return
*/
@PUT
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Produces(MediaType.APPLICATION_XML)
@Path("/rule")
public Response ruleInsert(
@PathParam(PROCID) String processorid,
@PathParam(ATTRIBUTE_UUID) String attributeid,
@FormParam(RULEVALUE) @DefaultValue(StringUtils.EMPTY) String value,
@FormParam(SIZE) @DefaultValue(StringUtils.EMPTY) String size,
@FormParam(COUNT) @DefaultValue(StringUtils.EMPTY) String count) {
//logger.debug("Invoking ThresholdResource.java @PUT @PATH(\"/rule\"): ruleInsert(...). RULEVALUE is: " + value);
try {
ThresholdsConfigFile config = getConfigFile(request, servletContext);
if (value.isEmpty()) {
validation_error_list.add(INVALID_RULE_ID);
}
if (!validateStringAsLong(size)) {
validation_error_list.add(INVALID_SIZE);
}
if (!validateStringAsInt(count)) {
validation_error_list.add(INVALID_COUNT);
}
if (config.containsRule(attributeid, value)) {
validation_error_list.add(String.format(DUPLICATE_VALUE + "%s", value));
}
if (!validation_error_list.isEmpty()) {
return Response.status(400).entity(setValidationErrorMessage(validation_error_list)).build();
}
ObjectFactory of = new ObjectFactory();
//create new attribute
Thresholds rule = of.createThresholds();
rule.setId(value);
rule.setSize(getBigIntValueOf(size));
rule.setCount(getBigIntValueOf(count));
Attribute attr = config.findAttribute(attributeid);
attr.getRule().add(rule);
config.save();
} catch (Exception ex) {
logger.error(ex.getMessage());
return Response.status(400).entity(GENERAL_ERROR + ex.getMessage()).build();
}
return generateOkResponse(Status.OK.toString()).build();
}
/**
* Updates a threshold value, size or count.
*
* @param processorid
* @param uuid
* @param rulevalue
* @param size
* @param count
* @return
*/
@POST
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Produces(MediaType.APPLICATION_XML)
@Path("/rule/{" + RULEVALUE + "}")
public Response ruleUpdate(@PathParam(PROCID) String processorid,
@PathParam(ATTRIBUTE_UUID) String uuid,
// @PathParam(RULEUUID) String ruleuuid,
@PathParam(RULEVALUE) String rulevalue,
// @FormParam(RULEVALUE) @DefaultValue(StringUtils.EMPTY) String rulevalue,
@FormParam(SIZE) @DefaultValue(StringUtils.EMPTY) String size,
@FormParam(COUNT) @DefaultValue(StringUtils.EMPTY) String count) {
//logger.debug("Invoking ThresholdResource.java @POST @PATH(\"/rule/{\"+RULEVALUE+\"}\"): ruleUpdate(...). RULEVALUE is: " + rulevalue);
try {
// if(rulevalue.isEmpty()&& ruleuuid.compareTo("-1")!=0)
// validation_error_list.add(INVALIDRULEID);
if (!validateStringAsLong(size)) {
validation_error_list.add(INVALID_SIZE);
}
if (!validateStringAsInt(count)) {
validation_error_list.add(INVALID_COUNT);
}
if (!validation_error_list.isEmpty()) {
return Response.status(400).entity(setValidationErrorMessage(validation_error_list)).build();
}
ThresholdsConfigFile config = getConfigFile(request, servletContext);
Attribute attr = config.findAttribute(uuid);
// if(ruleuuid.compareTo("-1")==0){
if (rulevalue.compareToIgnoreCase("default") == 0) {
Rule rule = attr.getNoMatchRule();
rule.setCount(getBigIntValueOf(count));
rule.setSize(getBigIntValueOf(size));
} else {
Thresholds rule = config.findRule(attr, rulevalue);//ruleuuid);
// rule.setValue(rulevalue);
rule.setCount(getBigIntValueOf(count));
rule.setSize(getBigIntValueOf(size));
}
config.save();
} catch (Exception ex) {
logger.error(ex.getMessage());
return Response.status(400).entity(GENERAL_ERROR + ex.getMessage()).build();
}
return generateOkResponse(Status.OK.toString()).build();
}
/**
* Deletes a threshold.
*
* @param processorid
* @param uuid
* @param rulevalue
* @return
*/
@DELETE
@Consumes(MediaType.APPLICATION_FORM_URLENCODED)
@Produces(MediaType.APPLICATION_XML)
@Path("/rule/{" + ID + "}")
public Response ruleDelete(@PathParam(PROCID) String processorid,
@PathParam(ATTRIBUTE_UUID) String uuid,
@PathParam(ID) String rulevalue) {
//logger.debug("Invoking ThresholdResource.java @DELETE @PATH(\"/rule/{\"+ID+\"}\"): ruleDelete(...). RULEVALUE is: " + rulevalue);
try {
ThresholdsConfigFile config = getConfigFile(request, servletContext);
Attribute attr = config.findAttribute(uuid);
Thresholds rule = config.findRule(attr, rulevalue);//ruleuuid);
attr.getRule().remove(rule);
config.save();
} catch (Exception ex) {
logger.error(ex.getMessage());
return Response.status(400).entity(GENERAL_ERROR + ex.getMessage()).build();
}
return generateOkResponse(Status.OK.toString()).build();
}
}

191
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/java/org/apache/nifi/monitor/thresholds/ui/ThresholdSettingsResource.java
View File

@ -1,191 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.monitor.thresholds.ui;
/*
* NOTE: rule is synonymous with threshold
*/
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import javax.servlet.ServletContext;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.core.CacheControl;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.Response;
import javax.ws.rs.core.Response.ResponseBuilder;
import javax.ws.rs.core.UriInfo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.nifi.web.NiFiWebContext;
/**
* MonitorThreshold REST Web Service (superclass)
*/
public class ThresholdSettingsResource {
@Context
protected UriInfo context;
protected static final Logger logger = LoggerFactory.getLogger(ThresholdSettingsResource.class);
protected List<String> validation_error_list = new ArrayList<>();
protected static final String DUPLICATE_ATTRIBUTE_NAME = "Please enter a unique Attribute Name. An attribute with this name already exists: ";
protected static final String INVALID_ATTRIBUTE_NAME = "Attribute Name must contain a value.";
protected static final String INVALID_RULE_ID = "Please enter a value.";
protected static final String INVALID_SIZE = "Please enter an integer Size value in bytes. Punctuation (including decimals/commas) and other forms of text are not allowed. Max allowed size is 9223372036854775807.";
protected static final String INVALID_COUNT = "Please enter an integer File Count. Punctuation (including decimals/commas) and other forms of text are not allowed. Max allowed size is 2147483647.";
protected static final String GENERAL_ERROR = "A general error has occurred. Detailed Error Message: ";
protected static final String DUPLICATE_VALUE = "Please enter a unique value. A threshold with this value already exists: ";
protected static final String PROCID = "procid";
protected static final String ID = "id";
protected static final String ATTRIBUTE_UUID = "attributeuuid";//uuid of the attribute selected
protected static final String ATTRIBUTENAME = "attributename";//name of the attribute selected
protected static final String RULEUUID = "ruuid";
protected static final String RULEVALUE = "attributevalue";
protected static final String SIZE = "size";//rule/threshold size
protected static final String COUNT = "count";//rule/threshold count
/**
* Creates a new instance of ThresholdSettingsResource
*/
public ThresholdSettingsResource() {
}
/**
*
* @param response
* @return
*/
protected ResponseBuilder noCache(ResponseBuilder response) {
CacheControl cacheControl = new CacheControl();
cacheControl.setPrivate(true);
cacheControl.setNoCache(true);
cacheControl.setNoStore(true);
return response.cacheControl(cacheControl);
}
/**
* Returns a standard web response with a status 200.
*
* @param value
* @return
*/
protected ResponseBuilder generateOkResponse(String value) {
ResponseBuilder response = Response.ok(value);
return noCache(response);
}
/**
* Validates a number. If the string value cannot be parsed as a long then
* false is returned.
*
* @param value
* @return
*/
protected boolean validateStringAsLong(String value) {
try {
Long.parseLong(value);
} catch (NumberFormatException ex) {
logger.error(ex.getMessage());
return false;
}
return true;
}
/**
* Validates a number. If the string value cannot be parsed as an int then
* false is returned.
*
* @param value
* @return
*/
protected boolean validateStringAsInt(String value) {
try {
Integer.parseInt(value);
} catch (NumberFormatException ex) {
logger.error(ex.getMessage());
return false;
}
return true;
}
/**
* Validates a number. If the string value cannot be parsed as a float then
* false is returned.
*
* @param value
* @return
*/
protected boolean validateStringAsFloat(String value) {
try {
Float.parseFloat(value);
} catch (NumberFormatException ex) {
logger.error(ex.getMessage());
return false;
}
return true;
}
protected ThresholdsConfigFile getConfigFile(HttpServletRequest request, ServletContext servletContext) {
//logger.debug("Running ThresholdSettingsResource: getConfigFile(...).");
NiFiWebContext nifiContext = (NiFiWebContext) servletContext.getAttribute("nifi-web-context");
ThresholdsConfigFile file = new ThresholdsConfigFile(nifiContext, request);
return file;
}
protected Boolean getIsAsc(String sord) {
Boolean isAsc = true;
if (sord.compareTo("desc") == 0) {
isAsc = false;
}
return isAsc;
}
protected BigInteger getBigIntValueOf(String value) {
Long _value = Long.valueOf(value);
return BigInteger.valueOf(_value);
}
protected String setValidationErrorMessage(List<String> messages) {
String result = new String();
for (String message : messages) {
result += "<div>";
result += message;
result += "</div>";
}
return result;
}
@SuppressWarnings("unused")
private Map<String, String> getQueryStringValues(String querystring) {
Map<String, String> queryvalues = new HashMap<String, String>();
String[] splitvals = querystring.split("&");
queryvalues.put("processorid", splitvals[0]);
queryvalues.put(splitvals[1].split("=")[0], splitvals[1].split("=")[1]);
queryvalues.put(splitvals[2].split("=")[0], splitvals[2].split("=")[1]);
return queryvalues;
}
}

296
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/java/org/apache/nifi/monitor/thresholds/ui/ThresholdsConfigFile.java
View File

@ -1,296 +0,0 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.nifi.monitor.thresholds.ui;
/*
* NOTE: rule is synonymous with threshold
*/
import java.io.StringReader;
import java.io.StringWriter;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.UUID;
import javax.servlet.http.HttpServletRequest;
import javax.xml.XMLConstants;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBElement;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;
import javax.xml.bind.Unmarshaller;
import javax.xml.transform.stream.StreamSource;
import javax.xml.validation.Schema;
import javax.xml.validation.SchemaFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.nifi.settings.generated.Thresholds;
import org.apache.nifi.settings.generated.Attribute;
import org.apache.nifi.settings.generated.Configuration;
import org.apache.nifi.settings.generated.ObjectFactory;
import org.apache.nifi.web.HttpServletRequestContextConfig;
import org.apache.nifi.web.NiFiWebContext;
public class ThresholdsConfigFile {
protected final Logger logger = LoggerFactory.getLogger(ThresholdsConfigFile.class);
private final String THRESHOLD_SETTINGS_XSD = "/threshold_settings.xsd";
private final String JAXB_GENERATED_PATH = "org.apache.nifi.settings.generated";
private final JAXBContext JAXB_CONTEXT = initializeJaxbContext();
private Configuration configuration;
private ObjectFactory objectFactory;
private final List<Attribute> filtered_attribute_list = new ArrayList<>();
private final NiFiWebContext nifiWebCtx;
private final HttpServletRequestContextConfig contextConfig;
private final HttpServletRequest request;
/**
* Load the JAXBContext.
*/
private JAXBContext initializeJaxbContext() {
try {
return JAXBContext.newInstance(JAXB_GENERATED_PATH);
} catch (JAXBException e) {
logger.error("Unable to create JAXBContext.");
throw new RuntimeException("Unable to create JAXBContext.");
}
}
public ThresholdsConfigFile(final NiFiWebContext ctx, final HttpServletRequest request) {
//logger.debug("Running ThresholdsConfigFile(...) constructor.");
nifiWebCtx = ctx;
this.request = request;
contextConfig = new HttpServletRequestContextConfig(request);
getState();
}
private void getState() {
objectFactory = new ObjectFactory();
String state = nifiWebCtx.getProcessor(contextConfig).getAnnotationData();
if (state != null && !state.isEmpty()) {
try {
//logger.debug("Running ThresholdsConfigFile.getState(). Getting (and unmarshalling) Annotation Data.");
// find the schema
SchemaFactory schemaFactory = SchemaFactory.newInstance(XMLConstants.W3C_XML_SCHEMA_NS_URI);
Schema schema = schemaFactory.newSchema(ThresholdsConfigFile.class.getResource(THRESHOLD_SETTINGS_XSD));
// attempt to unmarshal
Unmarshaller unmarshaller = JAXB_CONTEXT.createUnmarshaller();
unmarshaller.setSchema(schema);
JAXBElement<Configuration> element = unmarshaller.unmarshal(new StreamSource(new StringReader(state)), Configuration.class);
configuration = element.getValue();
} catch (Exception ex) {
logger.error(ex.getMessage());
}
} else {
try {
configuration = objectFactory.createConfiguration();
save();
} catch (Exception ex) {
logger.error(ex.getMessage());
}
}
}
public List<Attribute> getFlowFileAttributes() {
return configuration.getFlowFileAttribute();
}
/**
* Helper method to determine if an attribute is already in the attributes
* ArrayList.
*
* @param id
* @return
*/
public boolean containsAttribute(String id) {
boolean result = false;
for (Attribute attr : configuration.getFlowFileAttribute()) {
if (attr.getId().compareTo(id) == 0) {
result = true;
break;
}
}
return result;
}
public boolean containsRule(String id, String value) {
boolean result = false;
Attribute attr = findAttribute(id);
if (attr != null) {
Thresholds rule = findRule(attr, value);
if (rule != null) {
result = true;
}
}
return result;
}
/**
* Iterates through attributes list and returns the attribute that contains
* the id passed into the method. Returns null if attribute is not found in
* attribute list.
* @param id
* @return
*/
public Attribute findAttribute(String id) {
for (Attribute attr : configuration.getFlowFileAttribute()) {
if (attr.getId().compareTo(id) == 0) {
return attr;
}
}
return null;
}
public Attribute findAttributebyAttributeName(String attributeName) {
for (Attribute attr : configuration.getFlowFileAttribute()) {
if (attr.getAttributeName().compareTo(attributeName) == 0) {
return attr;
}
}
return null;
}
public Thresholds findRule(Attribute attr, String value) {
for (Thresholds rule : attr.getRule()) {
if (rule.getId().compareTo(value) == 0) {
return rule;
}
}
return null;
}
/**
* returns xml without attributes, used for presentation.
*
* @param col
* @param isAsc
* @param attributeFilter
* @param rowNum
* @param pageCom
* @return
*/
public String getConfigAttributes(String col, Boolean isAsc, String attributeFilter, Integer rowNum, String pageCom) {
return getFormattedAttributes(configuration.getFlowFileAttribute(), col, isAsc, attributeFilter, "", "", "");//,rowNum,pageCom);
}
/**
* UI Specific. Returns xml only related to the attributeid passed into the
* method.
* @param attrid
* @param col
* @param isAsc
* @param attributevalueFilter
* @param sizeFilter
* @param filecountfilter
* @return
*/
public String getRules(String attrid, String col, Boolean isAsc, String attributevalueFilter, String sizeFilter, String filecountfilter) {
List<Attribute> attributeList = new ArrayList<>();
Attribute attr = findAttribute(attrid);
if (attr != null) {
attributeList.add(attr);
}
Collections.sort(attributeList.get(0).getRule(), new RuleComparator(col));
if (!isAsc) {
Collections.reverse(attributeList.get(0).getRule());
}
return getFormattedAttributes(attributeList, col, isAsc, "", attributevalueFilter, sizeFilter, filecountfilter);//,-1,"");
}
public String getListStats()//Integer rowNum)
{
return String.format("Results %s", filtered_attribute_list.size());
}
private String getFormattedAttributes(List<Attribute> list, String col, Boolean isAsc, String attributeFilter, String attributevalueFilter,
String sizeFilter, String filecountfilter)//, Integer rowNum, String pageCom)
{
String result = "<?xml version=\"1.0\" ?><configuration><attributes>";
getFilteredList(list, attributeFilter);
for (Attribute attribute : getSortedPagedList(col, isAsc)) {
result += "<attribute>";
result += String.format("<uuid>%s</uuid>", attribute.getId());
result += String.format("<attributename>%s</attributename>", attribute.getAttributeName());
result += "<rules>";
result += "<rule>";
result += String.format("<ruuid>%s</ruuid>", UUID.randomUUID());
result += String.format("<attributevalue>Default</attributevalue>");
result += String.format("<size>%s</size>", attribute.getNoMatchRule().getSize());
result += String.format("<count>%s</count>", attribute.getNoMatchRule().getCount());
result += "</rule>";
for (Thresholds rule : attribute.getRule()) {
if (rule.getId().toLowerCase().contains(attributevalueFilter.toLowerCase())
&& rule.getSize().toString().contains(sizeFilter)
&& rule.getCount().toString().contains(filecountfilter)) {
result += "<rule>";
result += String.format("<ruuid>%s</ruuid>", UUID.randomUUID());
result += String.format("<attributevalue>%s</attributevalue>", rule.getId());
result += String.format("<size>%s</size>", rule.getSize());
result += String.format("<count>%s</count>", rule.getCount());
result += "</rule>";
}
}
result += "</rules></attribute>";
}
result += "</attributes></configuration>";
return result;
}
private void sortList(List<Attribute> list, String col, boolean isAsc) {
Collections.sort(list, new AttributeComparator(col));
if (!isAsc) {
Collections.reverse(list);
}
}
private void getFilteredList(List<Attribute> list, String attributeFilter) {
filtered_attribute_list.clear();
for (Attribute attribute : list) {
if (attribute.getAttributeName().toLowerCase().contains(attributeFilter.toLowerCase())) {
filtered_attribute_list.add(attribute);
}
}
}
private List<Attribute> getSortedPagedList(String col, boolean isAsc) {
sortList(filtered_attribute_list, col, isAsc);//sub, col, isAsc);
return filtered_attribute_list;// sub;
}
public void save() throws Exception {
final StringWriter strWriter = new StringWriter();
Marshaller marshaller = JAXB_CONTEXT.createMarshaller();
marshaller.setProperty(Marshaller.JAXB_FORMATTED_OUTPUT, true);
marshaller.setProperty(Marshaller.JAXB_FRAGMENT, true);
marshaller.marshal(configuration, strWriter);
// save thresholds (in the framework/flow.xml) for MonitorThreshold processor to read and use.
nifiWebCtx.setProcessorAnnotationData(contextConfig, strWriter.toString());
}
}

15
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/webapp/META-INF/nifi-processor
View File

@ -1,15 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
org.apache.nifi.processors.monitor.MonitorThreshold

43
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/webapp/WEB-INF/jsp/attribute-add-dialog.jsp
View File

@ -1,43 +0,0 @@
<%--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--%>
<div id="attribute-add-dialog">
<div class="dialog-content" >
<div id="attribute-error-message" class="error-message" ></div>
<div class="setting">
<div class="setting-name">Attribute Name</div>
<div class="setting-field">
<input id="new-attribute-name" name="new-property-name" style="width: 200px;" type="text"/>
</div>
</div>
<div id="setting-title">Enter default thresholds (required):</div>
<div class="setting">
<div class="setting-name">Default Size Threshold (bytes)</div>
<div class="setting-field">
<input id="size-value" name="new-property-value" style="width: 200px;" type="text"/>
</div>
</div>
<div class="setting">
<div class="setting-name">Default File Threshold</div>
<div class="setting-field">
<input id="file-count-value" name="new-property-value" style="width: 200px;" type="text"/>
</div>
</div>
</div>
</div>

21
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/webapp/WEB-INF/jsp/attribute-confirm-dialog.jsp
View File

@ -1,21 +0,0 @@
<%--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--%>
<div id="attribute-confirm-dialog">
<div class="dialog-content">
<div id="attribute-dialog-content"></div>
</div>
</div>

27
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/webapp/WEB-INF/jsp/attribute-edit-dialog.jsp
View File

@ -1,27 +0,0 @@
<%--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--%>
<div id="attribute-edit-dialog">
<div class="dialog-content" >
<div id="attribute-error-message" class="error-message" ></div>
<div class="setting">
<div class="setting-name">Attribute Name</div>
<div class="setting-field">
<input id="edit-attribute-name" name="new-property-name" style="width: 200px;" type="text"/>
</div>
</div>
</div>
</div>

26
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/webapp/WEB-INF/jsp/attribute-filter-dialog.jsp
View File

@ -1,26 +0,0 @@
<%--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--%>
<div id="attribute-filter-dialog">
<div class="dialog-content" >
<div class="setting">
<div class="setting-name">Filter Attribute Name</div>
<div class="setting-field">
<input id="filter-attribute" name="new-property-value" style="width: 200px;" type="text"/>
</div>
</div>
</div>
</div>

21
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/webapp/WEB-INF/jsp/error-dialog.jsp
View File

@ -1,21 +0,0 @@
<%--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--%>
<div id="error-dialog">
<div class="dialog-content">
<div id="error-dialog-content" ></div>
</div>
</div>

101
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/webapp/WEB-INF/jsp/index.jsp
View File

@ -1,101 +0,0 @@
<%--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--%>
<%@page contentType="text/html" pageEncoding="UTF-8"%>
<!DOCTYPE html>
<html>
<head>
<title>Threshold Settings</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=EmulateIE7"/>
<link rel="shortcut icon" href="images/nifi16.ico"/>
<link rel="stylesheet" href="../nifi/js/jquery/ui-smoothness/jquery-ui-1.10.4.min.css" type="text/css" />
<link rel="stylesheet" href="../nifi/js/jquery/modal/jquery.modal.css" type="text/css" />
<link rel="stylesheet" href="js/jquery/jqgrid/css/ui.jqgrid.css" type="text/css" />
<!-- Following is missing. Not sure where it might be. Will need to search further. -->
<link rel="stylesheet" href="../nifi/css/nf-canvas-all.css" type="text/css" />
<link rel="stylesheet" href="css/threshold_styles.css" type="text/css" />
<script type="text/javascript" src="../nifi/js/jquery/jquery-2.1.1.min.js"></script>
<script type="text/javascript" src="../nifi/js/jquery/ui-smoothness/jquery-ui-1.10.4.min.js"></script>
<script type="text/javascript" src="../nifi/js/jquery/jquery.center.js"></script>
<script type="text/javascript" src="../nifi/js/jquery/modal/jquery.modal.js"></script>
<script type="text/javascript" src="js/jquery/jqgrid/js/i18n/grid.locale-en.js"></script>
<script type="text/javascript" src="js/jquery/jqgrid/js/jquery.jqGrid.min.js"></script>
<script type="text/javascript" src="js/nf-common.js" ></script>
</head>
<body>
<jsp:include page="/WEB-INF/jsp/error-dialog.jsp"/>
<jsp:include page="/WEB-INF/jsp/attribute-confirm-dialog.jsp"/>
<jsp:include page="/WEB-INF/jsp/attribute-add-dialog.jsp"/>
<jsp:include page="/WEB-INF/jsp/threshold-add-dialog.jsp"/>
<jsp:include page="/WEB-INF/jsp/threshold-confirm-dialog.jsp"/>
<jsp:include page="/WEB-INF/jsp/attribute-edit-dialog.jsp"/>
<jsp:include page="/WEB-INF/jsp/threshold-edit-dialog.jsp"/>
<jsp:include page="/WEB-INF/jsp/attribute-filter-dialog.jsp"/>
<jsp:include page="/WEB-INF/jsp/threshold-filter-dialog.jsp"/>
<div id="threshold-header-text">Threshold Settings </div>
<div class="list-header" >
<div id="control-box">
<div id="save-cancel">
<div id="attribute-list-header" class="list-header-text">Attributes</div>
<img alt="Separator" src="images/separator.gif">
<input id="attribute-add-main" class="control-style" type="image" src="images/addWorksheetRow.png" title = "Add New Attribute" />
<div id="attribute-filter-values" class="clear-filter"></div>
</div>
</div>
<div id="filter-selection" style="left:20px;">
<table width="80%">
<tr>
<td><div id="attribute-names-header" class="abbreviated-header" >Names</div></td>
</tr>
</table>
</div>
</div>
<div id="list1" class="list_container">
<table id="list"></table>
</div>
<div id="rowdetails"></div>
<div id="status-bar" style="position: relative; bottom: 0px; padding: 2px; height: 15px; width:100%; display: inline-block; background-color: silver; color: black;" ></div>
<div id="faded-background"></div>
<div id="glass-pane"></div>
<div id="attribute-context-menu" class="context-menu">
<div id="attribute-add" class="context-control" ><img src="images/addWorksheetRow.png"/> <span class="context-label">Add Attribute</span></div>
<div id="attribute-edit" class="context-control" ><img style="width:16px; height:16px;" src="../../nifi/images/iconEdit.png"/> <span class="context-label">Edit Attribute</span></div>
<div id="attribute-delete" class="context-control" ><img src="images/removeWorksheetRow.png"/> <span class="context-label">Delete Attribute</span></div>
<div id="attribute-filter" class="context-control" ><img src="images/filter.gif" /> <span class="context-label">Filter Attributes</span></div>
</div>
<div id="threshold-context-menu" class="context-menu">
<div id="threshold-add" class="context-control" ><img src="images/addWorksheetRow.png"/> <span class="context-label">Add Threshold</span></div>
<div id="threshold-edit" class="context-control" ><img style="width:16px; height:16px;" src="../../nifi/images/iconEdit.png"/> <span class="context-label">Edit Threshold</span></div>
<div id="threshold-delete" class="context-control" ><img src="images/removeWorksheetRow.png"/> <span class="context-label">Delete Threshold</span></div>
<div id="threshold-filter" class="context-control" ><img src="images/filter.gif" /> <span class="context-label">Filter Thresholds</span></div>
</div>
</body>
</html>

45
nifi/nar-bundles/monitor-threshold-bundle/ui/src/main/webapp/WEB-INF/jsp/threshold-add-dialog.jsp
View File

@ -1,45 +0,0 @@
<%--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
--%>
<div id="threshold-add-dialog">
<div class="dialog-content" >
<div id="threshold-add-error-message" class="error-message" ></div>
<input type="hidden" id="attribute-row-id"></input>
<div class="setting">
<div class="setting-name">Attribute Value</div>
<div class="setting-field">
<input id="add-attribute-value" name="new-property-value" style="width: 200px;" type="text"/>
</div>
</div>
<div class="setting">
<div class="setting-name">Size Threshold (bytes)</div>
<div class="setting-field">
<input id="add-size-value" name="new-property-value" style="width: 200px;" type="text"/>
</div>
</div>
<div class="setting">
<div class="setting-name">File Threshold</div>
<div class="setting-field">
<input id="add-file-count-value" name="new-property-value" style="width: 200px;" type="text"/>
</div>
</div>
</div>
</div>

Some files were not shown because too many files have changed in this diff Show More