Commit Graph

1939 Commits

Author SHA1 Message Date
Peter Gyori 91dd59dbdf NIFI-7312: Enable search in variable registry of root process group
This closes #4303.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-02 13:25:00 -04:00
Mark Payne e31c323aa7 NIFI-7447: When returning an object from a Controller Service, if that object is defined as an interface, proxy that interface. This way, any method call into the object will also change the classloader to the appropriate classloader. 2020-06-02 12:13:51 -04:00
KovalevIV 7034d7e44c NIFI-7471 fix bug with property validation 2020-05-20 10:49:49 -04:00
Andy LoPresto 441781cec5
NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in shared constant).
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.

This closes #4263.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-19 12:56:59 -07:00
mtien 3fec4d8c27
NIFI-7331 Fixed grammatical errors in log output.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4283.
2020-05-19 10:38:45 +02:00
Matthieu Cauffiez 179675f0b4 NIFI-7380 - fix for controller service validation in NiFi Stateless
This closes #4264.

Signed-off-by: Matthieu Cauffiez <matthieu.cauffiez@bell.ca>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-18 16:11:40 -04:00
Yolanda M. Davis 13418ccb91
NIFI-7437 - created separate thread for preloading predictions, refactors for performance
NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4274
2020-05-18 14:57:54 -04:00
Mark Payne c51b9051a8 NIFI-7460: Avoid NPE when a VersionedProcessor has a null value for autoTerminatedRelationships. Added additional logging and improved error handling around syncing with invalid flows 2020-05-15 16:16:39 -04:00
Nathan Gough 302a42185c
NIFI-7321 - Allow NiFi admins to configure whether Jetty will send the Jetty server version in responses.
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.

This closes #4192.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-05-12 14:19:23 -07:00
mtien 647a9a60b7
NIFI-7423 Upgraded jquery dependency version.
NIFI-7423 Upgraded jquery dependency version to latest 3.5.1.

This closes #4258
2020-05-08 13:13:38 -04:00
Mark Payne 0448e23a96
NIFI-7414: Escape user-defined values that contain invalid XML characters before writing flow.xml.gz
NIFI-7414: Updated StandardFlowSerializerTest to include testing for variable names and values being filtered

This closes #4244
2020-05-04 14:49:15 -04:00
Matt Gilman e81960f8e8 NIFI-7170:
- Adding a flag to nifi.properties to disable anonymous authentication.

NIFI-7170:
- Fixing checkstyle issues.

NIFI-7170:
- Adding missing license header.

NIFI-7170:
- Initial PR feedback.

NIFI-7170:
- Fixing broken integration tests.
- Creating new integration tests for verifying allowing and preventing anonymous access.

NIFI-7170:
- Ensuring the new anonymous authentication property is considered for proxied requests.

NIFI-7170 - Fixed comment.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4099.
2020-04-30 15:20:19 -04:00
Matthew Burgess 1259bd5bd1
NIFI-7379: Support multiple instances of Prometheus registries/metrics (#4229)
* NIFI-7379: Support multiple instances of Prometheus registries/metrics

* NIFI-7379: Refactored Prometheus objects to support multiple instances
2020-04-28 16:56:49 -04:00
Andy LoPresto 148537d64a
NIFI-7377 Cleaned up nifi-stateless logs.
Refactored masking logic to CipherUtility and indicated masking with label and Base64 output.
Added JSON masking logic to nifi-stateless module.
Added argument masking functionality to Program.
Moved groovy unit tests to proper Maven directory structure.
Modified plain argument output to use filtering/masking methods in provided utility.
Refactored utility methods.
Updated unit tests.

This closes #4222.

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 17:29:42 -07:00
Sushil Kumar 996688b419
NIFI-7389 Makes Missable heartbeat counts configurable
This closes #4236.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 13:21:21 -07:00
Sushil Kumar 6c2701abef Do not update status for stopping a deleted node 2020-04-25 11:01:08 -04:00
Rob Fellows c263daf20b
[NIFI-7358] - Fix: Sorting on 'Estimated Time to Back Pressure' in the Connection summary table does not work properly
- fix style issues
- review feedback

This closes #4208
2020-04-15 10:06:38 -04:00
Andy LoPresto 07a8311b4c
NIFI-7319 Add walkthrough document (#4193)
* NIFI-7319 Added first draft of walkthroughs doc.

* NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance.

* NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate.

* NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates.

* NIFI-7319 Added instructions and screenshots for building NiFi from source.

* NIFI-7319 [WIP] Converting secure cluster instructions to match format.
Fixed instructions regarding embedded ZooKeeper configuration.

* NIFI-7319 Completed secure cluster walkthrough.

* NIFI-7319 Added walkthroughs to documentation navigation list.

* NIFI-7319 Incorporated PR feedback on broken links.

* NIFI-7319 Removed line number helpers from update sections.

* NIFI-7319 Incorporated final PR review items.

Co-authored-by: Sandra Pius <spiusapache@gmail.com>
2020-04-10 22:25:40 -07:00
Mark Payne 728bdec0f4 NIFI-7339: Fixed bug that caused Write Ahead Provenance Repository not to rollover event files after specified time. Code cleanup. Updated some default properties. 2020-04-09 13:32:34 -04:00
Matthew Burgess 84968e70d2
NIFI-7087: Use FlowManager.findAllConnections() when available
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4026
2020-04-09 12:59:01 -04:00
Mark Payne 683b1d9952
NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited.
- Refactored Flow Synchronization to make code cleaner
 - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty.
 - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start
- Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state.
- Added additional log statements to aid in debugging

NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior

NIFI-6849: Addressed review feedback

NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup

This closes #3891
2020-04-09 10:42:05 -04:00
Matthew Burgess a093af2d42
NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping (#4156)
* NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping

* NIFI-7273: Changed method name, fix handling when analytics not enabled

* NIFI-7273: Removed attachment header from Prometheus metrics endpoint

* NIFI-7273: Removed unused variable
2020-04-03 16:12:53 -04:00
Bence Simon c0f5fcb484 NIFI-7188 Extending UI search with filters and refactoring existing solution
This closes #4123.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-02 21:49:40 +02:00
Troy Melhase 483f23a8aa
NIFI-7153 Adds ContentLengthFilter to enforce configurable maximum length on incoming HTTP requests.
Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second.
Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml.

This closes #4125.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-25 17:23:22 -07:00
Michael Hogue aeaf953e06
NIFI-5925: Added controller services to set of components that are searched
NIFI-5925: cleanup, add negative test

NIFI-5925: fixed checkstyle

This closes #4105

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-03-25 08:55:44 -04:00
Bryan Bende 6412097eb2
NIFI-7257 Added HadoopDBCPConnectionPool
- Updated InstanceClassLoader to resolve files that are in the instance urls or additional urls
- Updated nifi-mock to support KerberosContext and removeProperty for ControllerServices
- Added unit test for HadoopDBCPConnectionPool
- Addressing review feedback

This closes #4149.
2020-03-24 11:05:18 -04:00
Bryan Bende 7f32aa56db
NIFI-7271 Make command timeout configurable for ShellUserGroupProvider
- Changing ShellRunner to use a separate thread for reading the output of the process
- Removing unused member variable
- Addressing review feedback

This closes #4154.
2020-03-23 20:43:06 -04:00
Joe Witt 254b20fa5f
NIFI-7274 add time for test conditions to be met 2020-03-22 09:54:43 -04:00
Joe Witt f694e6464f NIFI-7187 adding missing version strings from accumulo bundle pom
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE

This closes #4086

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-20 10:07:56 -04:00
Andy LoPresto 3feb85a030
NIFI-7223 Resolved merge conflicts from additional test case for null key password.
This closes #4145.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-16 12:47:41 -04:00
Nathan Gough 69b6c231fd
NIFI-7223 - Added another test for when keyPasswd is not present. 2020-03-16 12:28:18 -04:00
Andy LoPresto f9d75056fa
NIFI-7223 [WIP] Resolved compilation issues in unit test on OpenJDK 11 by removing Sun security class references.
Added OkHttpReplicationClient#isTLSConfigured() method.
Added unit test.
NIFI-7223 Fixed remaining unit tests for TLS regression.
Renamed tests for clarity.
2020-03-16 12:28:01 -04:00
Nathan Gough 7374361b5c
NIFI-7223 - Fixed a minor issue where the OkHttpReplicationClient class loaded blank properties as empty string instead of an expected null value. Added a isNotBlank check. Added unit tests for replication client and HTTPNotificationService.
NIFI-7223 - Renamed some variables and methods.
NIFI-7223 - Removed unused dependency. Corrected security properties in administration-guide.
2020-03-16 12:26:50 -04:00
Joe Witt 97e250cdaa
NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4132.
2020-03-12 19:13:59 +01:00
Mark Payne d4a2afc25c
NIFI-7241: When updating Process Group to match VersionedProcessGroup, remove any connections before recursing into child groups. This ensures that if a Port exists in child group A and is connected to a port in child group B, if the VersionedProcessGroup indicates to remove the port, that connection will be removed before attempting to remove the port. Updating and adding connections must still be done last, after all components have been added. But missing connections can be removed earlier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4136.
2020-03-12 10:25:04 +01:00
Mark Payne d68720920f
NIFI-7242: When a Parameter is changed, any property referencing that parameter should have its #onPropertyModified method called. Also renamed Accumulo tests to integration tests because they start embedded servers and connect to them, which caused failures in my environment. Also fixed a bug in TestLengthDelimitedJournal because it was resulting in failures when building locally as well.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4134.
2020-03-11 21:00:43 +01:00
Mark Payne afad982e91
NIFI-7200: Revert "NIFI-6530 - HTTP SiteToSite server returns 201 in case no data is available"
This reverts commit f01668e66a.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-10 14:32:43 -04:00
Pierre Villard 12c8402ac3
NIFI-7210 - added PG path in bulletins for S2S Bulletin RT
Added group path to BULLETIN table for QueryNiFiReportingTask

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4100
2020-03-09 16:29:22 -04:00
Mark Bean f4b65afb64 NIFI-7231: move controller service validation out of synchronized block for enabling
This closes #4118.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 15:01:19 -05:00
Joe Witt 4cd63c99e8
NIFI-7205 NIFI-7206
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4093.
2020-02-26 14:02:49 -08:00
Joe Witt bad0f10a52
NIFI-7201 - Update build to latest apache-maven-parent and split Github Actions builds to include OSX and distribute localization
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4091.
2020-02-25 17:38:33 -08:00
Andy LoPresto 815e4cf51f
NIFI-7152 Added custom ExceptionMappers to handle invalid Remote Process Group port value - (#4085)
JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper.
Registered the custom ExceptionMappers.
Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path.
Added the Apache license to Groovy Test.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-25 13:14:52 -08:00
Joe Witt 778012412a
Revert "NIFI-6363 Refactors sensitive properties, adds additional providers."
This reverts commit 479fcfdc0b.

It does not build properly as shown in Github Actions.
2020-02-24 22:07:53 -08:00
Troy Melhase 479fcfdc0b NIFI-6363 Refactors sensitive properties, adds additional providers.
NIFI-6363 Additional fixes.

NIFI-6363 Fix Hadoop compile problem.  Add GCP IT instructions.

NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.

NIFI-6363 - Added some docs and experimental tag to the relevant classes.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4080.
2020-02-24 16:35:19 -05:00
MatthewKnight-NG acaf321af0
NIFI-5924 Labels should be searchable
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4070
2020-02-24 16:14:21 -05:00
Troy Melhase 0de89452f1
NIFI-6927 Consolidate SSL context and trust managers for OkHttp on JDK9.
Fixes name conflicts.

This closes #4047.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-19 16:19:39 -08:00
M Tien 37614d02cd
NIFI-7136 Added autocomplete="off" to login password input (#4055)
NIFI-7136 Added autocomplete="off" to login password input
Updated nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/login/login-form.jsp

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-14 20:46:08 -08:00
Joe Ferner 62606ff89a NIFI-6873: Added support for replacing a process group
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
 - added replace process group functionality in ProcessGroupResource
 - parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
 - refactored flow update methods to make use of commonality across different update types whenever possible
 - fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
 - improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import

This closes #4023.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-02-11 14:46:40 -05:00
Mark Payne 65b2a9bc2c
NIFI-7117: When SocketLoadBalancedFlowFileQueue creates its array of Queue Partitions in the constructor, it added the local partition as the first element in that list. This list should be ordered the same across all nodes in the cluster. By making the local partition the first in the array, each node had a different ordering of these partitions. As a result, Partition by Attribute strategy would constantly rebalance flowfiles that it received to other node, and Single Node always transferred data to the first partition, which was the local node, instead of whichever node should have been the first in the list. This commit addresses this issue by instead inserting the local partition intot he 'queuePartitions' array based on the local node identifier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4045.
2020-02-11 10:59:19 -08:00
nagasivanath 425ba63e66
NIFI-6968 Create Connection Model Allows Multiple Adds
This closes #4013.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-06 10:32:59 -05:00
Matt Gilman b1b06bf5e1 NIFI-6958:
- Addressing issue causing errors preventing version changes when there is a processor to enable or disable in a sub process group.
2020-02-05 13:49:06 -05:00
Bryan Bende 5d851e6a13
NIFI-7067 Allow a user and group with the same name/identity to exist
This closes #4019
2020-02-05 09:19:18 -05:00
Andy LoPresto d7c29f4637 NIFI-7079 Cleaned up formatting and comments. 2020-01-31 08:58:51 -05:00
Andy LoPresto 95746d346c NIFI-7079 Added SecureHasher interface and Argon2SecureHasher implementation.
Added unit tests.
Fixed JwtService charset references.
Changed FingerprintFactory to mask sensitive values via Argon2 hashing.
Resolved RAT checks.
2020-01-31 08:58:51 -05:00
Mark Payne 37ef10e6fb
NIFI-7076: This closes #4024. Revert "NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize()."
This reverts commit 7731609582.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-29 15:50:55 -05:00
Mark Payne be34767c8a
NIFI-7059: This closes #4007. Fixed bug that results in priorities not properly being set in the SocketLoadBalancedFlowFileQueue. Even though the queue's setPriorities method was called, the underlying may not have contained the localPartition. As a result, when setPriorities() was called, it did not properly delegate that call to . As a result, the queue knew that the Prioritizers were set but the local queue did not apply them. This happened due to a race condition between queue creation and NodeClusterCoordinator learning the Local Node Identifier. Additionally, updated NodeClusterCoordinator so that it will persist its state when it does learn the Local Node Identifier, as that was not being persisted. In testing this, also encounterd a deadlock in a particular Integration Test that was run, around AbstractFlowFileQueue and its synchronization. Because the 'synchronized' keyword synchronizes on 'this' and the concrete implementation also uses 'synchronized' and the result is the same 'this', a deadlock can occur that is difficult to understand. As a result, refactored AbstractFlowFileQueue to instead use read/write locks.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-22 23:30:41 -05:00
Joe Witt 535cafdb83
Merge branch 'NIFI-7021-RC3' bringing the Apache NiFi 1.11.0 to master 2020-01-22 21:15:12 -05:00
Bryan Bende 24b846d08a
NIFI-7051 Protect against empty group membership in ShellUserGroupProvider, and add differentiator to id seeding
NIFI-7051 Fixing issue where identity was being used instead of identifier, making a flag to control legacy id behavior, increasing timeout of shell command runner, and changing the NSS system check command to return less info

NIFI-7051 Updating command for getSystemCheck in NSS impl to use getent --version to improve performance

This closes #4003.
2020-01-22 15:51:42 -05:00
nagasivanath 24ef8ba4cb
Update nf-processor-details.js
NIFI-7032:
- Processor Details no longer appears when clicking 'View Processor Details'
- handling the review comments

This closes #3990
2020-01-21 13:54:12 -05:00
Joe Witt 3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt 633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Joe Witt 67c36f3c3e
NIFI-7021 apache parent pom and common dep maintenance for build 2020-01-19 13:32:37 -05:00
Nathan Gough 4ec9155cbc
NIFI-6770 - Set validator to Validator.VALID to allow empty password for truststores.
Added no-password keystore for tests
System NiFi truststore now allows a passwordless truststore. Added a unit test to prove this.
Forgot no-password-truststore.jks file for the unit test.
Refactored utility method from CertificateUtils to KeyStoreUtils.
Added utility methods to verify keystore and key passwords.
Added unit tests.
Implemented different keystore and truststore validation logic.
Refactored internal custom validation in StandardSSLContextService.
Added unit test resource for keystore with different key and keystore passwords.
Added unit test to generate passwordless truststore for https://nifi.apache.org for live testing.
Resolved NPE in SSLContext generation in StandardSSLContextService
Added unit test to generate passwordless truststore for localhost for InvokeHTTP testing.
Resolved TrustManagerFactoryImpl initialization error.
Fixed unit test without proper cleanup which caused RAT failures.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

This closes #3823.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-17 20:01:06 -08:00
Andy LoPresto 34f2a592df
NIFI-7023 This closes #3991. Removed SLF4J and Log4J transitive dependencies from Zookeeper so tests log correctly.
Changed template handling.
Added unit tests.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-16 13:00:08 -05:00
Joe Witt 23c8234586
NIFI-7031 updating copyright year on NOTICES 2020-01-15 16:10:31 -05:00
Shawn Weeks aa24664801
NIFI-7022 - This closes #3989. Disable Zookeeper Admin Server for Embedded Zookeeper
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-15 14:43:21 -05:00
Mark Payne 66d5ab80eb
NIFI-7011: This closes #3983. SwappablePriorityQueue contains two internal data structures: activeQueue, swapQueue. activeQueue is intended to be pulled from for processing. swapQueue is intended to hold FlowFiles that are waiting to be swapped out. SinWe want to ensure that we first swap in any data that has already been swapped out before processing the swap queue, in order to ensure that we process the data in the correct order. This fix ddresses an issue where data was being swapped out by writing the lowest priority data to a swap file, then adding the highest priority data to activeQueue and the 'middle' priority data back to swapQueue. As a result, when polling from the queue we got highest priority data, followed by lowest priority data, followed by middle priority data. This is addressed by avoiding putting anything back on swapQueue when we swap out. Instead, write data to the swap file, then push everything else to activeQueue. This way, any new data that comes in will still go to the swapQueue, as it should, but all data that didn't get written to the Swap file will be processed before the low priority data in the swap file.
NIFI-7011: Addressed corner case where data could be inserted out of order still if added while swapping was taking place
NIFI-7011: Fixed ordering issue with swap queue that can occur if data is migrated from swap queue to active queue instead of being swapped out
2020-01-13 22:02:05 -05:00
nagasivanath 4e2b61efe4
NIFI-6652 UI Fix overflowing text in variables dialog
This closes #3982.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-13 12:12:34 -08:00
Tamas Palfy b7fb94723c NIFI-6884 - Native library loading fixed/improved: NarClassLoader and InstanceClassLoader can load libraries from their own or their ancestors' NAR-INF/bundled-dependencies/native directory.
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.

This closes #3894.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-01-13 13:59:11 -05:00
mdayakar ccb85cf8b7
NIFI-6923:If FlowFile content is truncated in the Content Repository, NiFi does not throw Exception when reading the content.
NIFI-6923 fixing checkstyle
2020-01-13 12:45:59 -05:00
Shawn Weeks 2c18cf22ad
NIFI-6852 This closes #3973. Don't Validate Processors that accept any ControllerService Implementation
NIFI-6852 Add Integration Tests for Controller Service API Validation

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 11:39:39 -05:00
Andy LoPresto 2cc467eb58
NIFI-3833 Added encrypted flowfile repository implementation.
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.

Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3968.
2020-01-10 10:44:59 -08:00
mtien 0ae200252f
NIFI-6978 added code and unit test to throw IllegalArgumentException when improper value given
This closes #3960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-09 13:50:27 -08:00
Bryan Bende 08d0352ac1
NIFI-6985 Use correct versioned parameter contexts when child process groups are version controlled
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3962.
2020-01-09 17:35:25 +07:00
Mark Payne c958deb5b0 NIFI-6822: Ensure that when we manage a Map of ID -> Count, that we properly merge those maps during a checkpoint instead of overwriting existing values 2020-01-06 13:05:19 -05:00
Mark Payne 5f0f801e46
NIFI-6879: Added system tests for variables. Fixed bug that resulted in Variable Registry not being updated if a Processor in a child group referenced it
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3949.
2019-12-26 18:05:08 +01:00
Mark Payne 2277d08c8e
NIFI-6949: When a Controller Service is removed, ensure that any other service that it references is obtained via the ControllerServiceProvider to ensure that it is obtain to obtain and remove the reference, even if the service is in a higher-level Process Group
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3933.
2019-12-20 16:17:48 +00:00
Joe Ferner 058883091c
NIFI-6872:
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful

NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication

This closes #3931
2019-12-18 16:37:47 -05:00
Mark Payne e05d11c0b4 NIFI-6787: Added a comment to the UI tooltip, user guide, and javadocs to indicate that round robin may skip a node if that node is not receiving the data as fast as other nodes in the cluster, in order to maximum throughput. 2019-12-13 15:09:15 -05:00
Tamas Palfy 7731609582 NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize().
After: Go over all partitions and see if all of them are full.
Wrapping RoundRobinPartitioner in a (new) AvailableSeekingPartitioner which selects a new partition if the currently selected one is full.
2019-12-13 14:36:44 -05:00
Iván Rodriguez 65ba4a2d93 NIFI-6853 - flowFileEvent combineCounters hashmap overwritten
- Added unit test to verify behavior of contribution

This closes #3875.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-12-13 14:23:50 -05:00
Mark Payne 452ca98c29
NIFI-6924: When seeking to the appropriate offset for a content claim, ensure that if there are not enough bytes in the underlying resource claim that a ContentNotFoundException is thrown. Also cleaned up error-handling case in StandardProcessSession to ensure that we close the existing InputStream before calling handleContenttNotFoundException, since this method may itself throw an Exception
This closes #3924.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-12-10 15:27:53 -05:00
Bryan Bende dd620680b1
NIFI-6904 Moving ClassLoader creation before Authorizer instantiation
This closes #3903
2019-11-25 13:35:21 -05:00
Pierre Villard ac5bacccb8
NIFI-6839 - Upgrade jackson-databind direct dependencies
This closes #3870

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-11-25 10:58:22 -05:00
Andrew Lim 5bfc86737d
NIFI-6747 Updated images used by Swagger REST API
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3901.
2019-11-22 14:05:36 +01:00
Joe Ferner 3f3b778f24 NIFI-6893
Upgraded angular from 1.7.2 to 1.7.9

This closes #3899

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-11-21 16:45:33 -05:00
Mark Payne ea31c1fcbb
NIFI-6868: Ensure that when a property value is obtained by calling ProcessContext.getPropertyValue(...).evaluateAttributeExpressions(...).getValue() that we still retain the appropriate value even if the property references a sensitive parameter
* Added additional unit tests and updated ParameterEvaluator to make sure that it follows the same logic as the ParameterExpression in that sensitive parameters cannot be access from within EL

This closes #3889
Signed-off-by: Rob Fellows <rob.fellows@gmail.com>
2019-11-14 15:37:39 -05:00
Joe Ferner c947e57d5e
NIFI-6814:
- Restored JAXB Adapter

This closes #3880
2019-11-08 15:43:13 -05:00
Mark Payne 5f4ce8f431
NIFI-6707: Addition of nifi-system-tests module. Updates to EmbeddedNiFi, NiFi and SystemBundle to properly support classloader isolation that is needed for creating system-level tests and isolating the classpath of the tests themselves from the NiFi classpath. Fixed bootstrap listener to ensure that it always closes socket before calling shutdown(). Failure to do so can result in RunNiFi sometimes timing out while waiting for a response from NiFi
NIFI-6707: Added System Test to verify behavior when services depend on one another

NIFI-6707: Ensure that when tearing down flow after test, we wait for all processors and controller services to complete stop/disable before attempting to delete them

NIFI-6707: Fixed bug in RemoteProcessGroupIT that caused the test to fail intermittently based on timing. Improved logging for Process Group, Standard Process Group

NIFI-6707: Updated to include java11 subdirectory under lib/

NIFI-6707: Rebased against master and changed dependencies from 1.10.0-SNAPSHOT to 1.11.0-SNAPSHOT

NIFI-6707: Updates to see if it will help travis build correctly

NIFI-6707: Commenting out java commands in bootstrap.conf files

This closes #3831.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-11-08 12:33:16 -05:00
Mark Payne 49b7a7cd6b
NIFI-6846: If an Exception is thrown while a Processor is writing to a FlowFile, but that Content Claim is not yet eligible for destruction, mark it as a transient claim on the RepositoryRecord so that if it's available when the FlowFile Repository is checkpointed, then it will be cleaned up then
This closes #3872
2019-11-06 09:53:25 -05:00
Andy LoPresto 12bb284453
NIFI-6841 Fixed checkstyle error introduced in a9db5a8. 2019-11-05 13:12:40 -08:00
Mark Payne a9db5a8cb7
NIFI-6841: Fixed bug that resulted in the wrong number of 'Bytes Read' being reported by ByteCountingInputStream in the event that #skip was called between calls to #mark and #reset
This closes #3868
2019-11-05 13:06:06 -05:00
Jan Hentschel d6b4bce668 NIFI-4758 Changed 'Name' to 'Flow Name' in Import Version dialog 2019-11-04 15:26:25 -05:00
mtien 7f1886adad NIFI-4757 Capitalized 'version' in tool tip. 2019-11-04 15:22:52 -05:00
Joe Witt f8c3d877cf
NIFI-6733 updating to next release version for master branch 2019-11-04 13:31:39 -05:00
Joe Witt 4119279c03
Merge branch 'NIFI-6733-RC3' 2019-11-04 13:24:29 -05:00
Joe Witt 363b751310
Revert "NIFI-6733 bumping master to 1.11.0-SNAPSHOT"
This reverts commit a7c56cfd9a.
2019-11-04 13:24:13 -05:00
Joe Witt a7c56cfd9a
NIFI-6733 bumping master to 1.11.0-SNAPSHOT 2019-11-04 07:54:17 -05:00
Mark Payne 0b3a60ae3e NIFI-6832: When receiving a 0-byte FlowFile via site-to-site, avoid writing to the FlowFile at all 2019-11-01 16:22:17 -04:00
Kotaro Terada 245e055d5d
NIFI-6828 There is an unintentional string in "conf/logback.xml"
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3863.
2019-10-31 11:20:21 +01:00
mtien 3d56e2f26d
NIFI-4348 handled NPE in isExpressionLanguagePresent.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3857.
2019-10-30 09:58:55 +01:00
Mark Payne ce5eae5b2c NIFI-6800: Added unit test to verify behavior of contribution 2019-10-29 10:17:27 -04:00
Ivan Ezequiel Rodriguez 67f1677212 NIFI-6800 - Fix hashmap counter overwritten in highThroughputSession
This closes #3837.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-10-29 10:16:49 -04:00
Joe Witt 418179f5b2
NIFI-6733-RC3 prepare for next development iteration 2019-10-28 15:13:13 -07:00
Joe Witt b217ae20ad
NIFI-6733-RC3 prepare release nifi-1.10.0-RC3 2019-10-28 15:12:57 -07:00
Joe Witt dda29cd90b
NIFI-6733 updating zk to latest incremental 2019-10-28 12:40:03 -07:00
Jan Hentschel ccf85777c4
NIFI-6816 Removed duplicated dependency declarations
This closes #3851

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-28 12:49:08 -04:00
Matt Gilman 9364f7d2e8
NIFI-6811:
- Allowing enterGroup to complete prior to loading the controller service table.

This closes #3846.
2019-10-25 12:07:03 -04:00
Mark Payne 3f270f184c
NIFI-6807: When a Controller Service's state is transitioned to ENABLING, complete the Future successfully, even if the Controller Service is not valid. The Controller Service will remain in the ENABLING state until it is made valid, at which point it will ENABLE (unless explicitly disabled first). This allows us to Enable a Controller Service and its referencing components, even if the referencing component is still invalid due to it not yet recognizing the the referenced service has been enabled.
This closes #3841
2019-10-25 11:38:34 -04:00
Mark Payne 81fc2768c9
NIFI-6798: When checking that referenced Controller Services are valid, use ValidationContext.getProperties() rather than ComponentNode.getSupportedPropertyDescriptors() because the latter does not include dynamic properties
This closes #3836.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-25 09:52:44 -04:00
Mark Payne 8079d2531e
NIFI-6805, NIFI-6806: Updated toString() of StandardControllerServiceNode so that it includes the component's type and UUID, updated the more generic Exception Mappers to include stack traces regardless of the log level that is enabled
This closes #3840

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-24 19:27:30 -04:00
Yolanda M. Davis 147365285c
NIFI-6801 - fix to ensure unique model instance is created for each connection. Also increased default query interval.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3838
2019-10-24 08:51:50 -04:00
Matthew Burgess ace23c35b5 NIFI-6780: Add QueryNiFiReportingTask, RecordSinkService, S2S and DB impls (#3826)
* NIFI-6780: Introduce RecordSinkService to separate format and destination, refactor common S2S utils

Added QueryNiFiReportingTask to SQL query NiFi status and metrics

Add PROCESSOR_STATUS and PROCESS_GROUP_STATUS

Add CONNECTION_STATUS_PREDICTIONS

check for null predictions

Add ConnectionStatusRecursiveIterator

Fix issue w/ duplicate iterator outputs

Refactored query interfaces, fixed assembly POM

Rebased v master, fixed isBackPressureEnabled and Checkstyle/RAT errors

Add DatabaseRecordSink service (#13)

* Add DatabaseRecordSink service

* Incorporated review comments

* NIFI-6780: Add/fix docs, cleanup warnings, fixed some table definitions

* Added bundle profile, remove predictions table if not enabled

* Added doc for which tables are available when

This closes #3826.
2019-10-22 08:56:42 -04:00
Andy LoPresto d148fb1854
NIFI-3834 This closes #3821. Added encrypted content repository implementation.
Added skeleton implementation of EncryptedFileSystemRepository.
Added new impl to META-INF registry.
Added investigation comments to FileSystemRepository.
Implemented RepositoryObject block and stream encryptors.
Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor.
Refactored shared logic from AES CTR and G/CM encryptors to abstract parent.
Added working unit test for writing/reading via encrypted file system repository.
Added stream wrappers.
Added encryptor.
Added working unit test for writing/reading multiple pieces of content via encrypted file system repository.
Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository.
Implemented key management skeleton for encrypted content repository.
Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved.
Implemented validation on setting active key id.
Added content repository encryption properties to NiFiProperties.
Implemented configuration of encryption services from NiFiProperties.
Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules.
Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers.
Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods.
Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access.
Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR.
Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR.
Added unit test to encrypt & decrypt image resource.
Added smaller image resource for easier unit test debugging.
Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository.
Added text test resource for tests around exporting claim subsets.
Added exportTo methods to handle decrypting encrypted content.
Performed large unit test refactoring, moving shared logic to helper methods.
Added unit test for merged content claim with header/footer/demarcator.
Added unit test for merging content claims each encrypted with a different key.
Ignored non-deterministically failing firewall DNS test.
Added documentation to User and Admin Guide for Encrypted Content Repository.
Added image.
Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor.
Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance.
Cleaned up unused test code.
Cleaned up Javadoc and code comments.
Refactored shared logic.
Fixed checkstyle issue.
Fixed test failure due to error message change.
Added experimental warning to repository implementation classes and User Guide documentation.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-21 13:03:59 -04:00
Matt Gilman 8b23929638
NIFI-6786:
- Upgrading nimbus-jose-jwt.

This closes #3830.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-21 11:41:29 -04:00
samhjelmfelt ea1becac4f
NIFI-6703: Add Stateless NiFi to CLI
NIFI-6703: Fixed extracted nar directory and marked api as experimental
NIFI-6703: Moving nifi-stateless into nifi-framework
NIFI-6703: Refactored to fix jetty/spring issues
NIFI-6703: checkstyle fix
NIFI-6703: updated to mirror traditional NiFi's bootstrap process and java11 dependency management
NIFI-6703: minor changes
NIFI-6703: Documentation fixes

This closes #3795.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-17 15:46:32 -04:00
Mark Payne b004f1f94c
NIFI-6779: When resolving node identifier, if NodeIdentifier has a different value for hostname/port for web api or cluster protocol, assume that node is correct about itself instead of assuming that Cluster Coordinator knows best about the other node
NIFI-6779: Remove any conflicting Node Identifiers when a new Node ID is encountered

This closes #3819
2019-10-17 11:21:17 -04:00
Mark Payne 9dd0dda688
NIFI-6772: Improved the information that is emitted about Garbage Collection when a Diagnostics Dump is performed
This closes #3809
2019-10-14 11:32:47 -04:00
Joe Witt a273ff10f9
NIFI-6758 fixing checkstyle issues 2019-10-10 15:04:16 -04:00
Matt Gilman 7db6ba841f
NIFI-6768:
- Ensuring the parameter context resources are available via the /resources endpoint.

This closes #3808.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 11:49:39 -04:00
Endre Zoltan Kovacs 5238dc20b1
NIFI-6758: allow pre-unpacked nar's to be recorded in ExtensionMapping
- fixing resource leak

This closes #3806.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 11:36:29 -04:00
Matt Gilman ed1e843609
NIFI-6769:
- Ensuring the users policy listing can handle when the specified policy is unknown.

This closes #3807.
2019-10-10 11:30:14 -04:00
markap14 d64f4fa942 NIFI-6760: When writing/reading the length of a DataFrame, do so usin… (#3801)
* NIFI-6760: When writing/reading the length of a DataFrame, do so using a 4-byte integer instead of a 2-byte short. When using uncompressed data, we know that the length of the DataFrame will be no more than 64 KB so a 2-byte short is sufficient. However, if data is compresed, there's a chance that the compressed form of the data will be larger than the uncompressed form (for example, with random binary data or with encrypted data). In this situation, we can end up overflowing the 2-byte short, which causes the length that is written to be wrong. Using a 4-byte integer avoids this situation.

* NIFI-6760: Fixed unit tests
2019-10-10 10:19:21 -04:00
Matt Gilman 4e8dd6557f
NIFI-6756:
- Fixing issue opening a users policy listing when there is a policy for a specific parameter context.

This closes #3805.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-10 00:55:47 -04:00
Matt Gilman 99e9010b32
NIFI-6766:
- Ensuring policy label is properly escaped when populating the user's access policy listing.
This closes #3804.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-10 00:19:22 -04:00
Mark Payne 5414cd5016 NIFI-6759: When encountering Exception during load-balanced connection communications, after closing socket, return from method rather than continuing loop
This closes #3800.
2019-10-09 16:05:44 -04:00
Matt Gilman 9a496fe9d2
NIFI-6751:
- Fixing the identifier on the user table. In a previous task, this was changed to utilize the URI but that does not work with other code interacting with this table.

This closes #3798.
2019-10-09 12:00:22 -04:00
Mark Payne 47f45abdf9
NIFI-6748: Fixed bug in Parameter Contexts' affected components where if a Controller Service referenced a Parameter, any component that references that service should also be considered an affected component but wasn't. Also fixed a bug in how we handled stopping a Processor that was in the STARTING phase.
This closes #3794.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-09 10:20:45 -04:00
Kevin Doran 2d90145842
NIFI-6749 This closes #3792. Fix Swagger for ControllerServiceReferencingComponentDTO
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-08 11:48:10 -04:00
Joe Witt 0f02de6002
NIFI-6733 updating key apache commons dependencies and apache base dependency for build
This closes #3791.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2019-10-07 22:20:48 -04:00
Rob Fellows df90c65246 NIFI-6735 - validate components before restarting processors following parameter context update.
This closes #3782.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-10-04 15:48:09 -04:00
Mark Payne 03473b922b NIFI-6001: When adding Controller Services from a Versioned Flow, make a second iteration over any newly added Controller Services and update them again. This is done so that any linkage between the Controller Services happens properly, once all services have been created. 2019-10-03 16:41:55 -04:00
Bryan Bende 9c9f9c10a9 NIFI-5910 Retain external service reference when already set to an existing external service
This closes #3783.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-10-03 12:32:46 -04:00
Mark Payne 99cf87c330
NIFI-6736: Create thread on demand to handle incoming request from client for load balancing. This allows us to avoid situations where we don't have enough threads and we block on the server side, waiting for data, when clients are trying to send data in another connection
This closes #3784.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-02 14:29:00 -04:00
Mark Payne 2f6f852915 NIFI-6589: Addressed NPE 2019-10-02 10:55:55 -04:00
Alessandro D'Armiento 39a258dc38
NIFI-6727 Fixed the MockProcessContext decrypt bug and added unit test.
This closes #3773.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-10-01 12:53:44 -07:00
Joe Witt 8abf330328
Revert NIFI-6703 as it creates startup issues or logs a confusing message to the user. Details on JIRA
This reverts commit 61baa41e7b.
2019-10-01 15:33:07 -04:00
Koji Kawamura 6541eac625
NIFI-6598 Storing peers into managed-state
- Fixed checkstyle errors.
- Added PeerPersistence interface.
- Expose RemoteProcessGroup state via REST API
- Made stateManager transient.

This closes #3677.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-01 09:56:10 -04:00
samhjelmfelt 61baa41e7b
NIFI-6703: This closes #3762. Add Stateless NiFi to CLI
NIFI-6703: Fixed extracted nar directory and marked api as experimental

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 23:23:15 -04:00
Mark Payne fbd6200ab3
NIFI-6589: This closes #3670. Cache results from zookeeper when determining the leader
NIFI-6589: Updated CuratorLeaderElectionManager to cache results for no more than 5 seconds per review feedback

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 16:47:27 -04:00
Rob Fellows 2493665c27 NIFI-6587 - Fix: Unable save sensitive property value that equals masked value
This closes #3778

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-30 13:18:08 -04:00
Mark Payne 6dccdd586e
NIFI-6726: Updated fingerprint logic to ensure that Controller Services are fingerprinted when housed within a ProcessGroup
This closes #3775.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-30 11:55:46 -04:00
Mark Payne 5562c587fe
NIFI-6132: This closes #3776. Do not default ot G1GC because it has known issues with Java 8 and earlier versions
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 11:51:27 -04:00
Mark Payne c721a9ee5f
NIFI-6696: Ensured that callback to RemoteQueuePartition do not attempt to obtain the Partition Read Lock. The Read Lock is not necessary as long as the 'partitioner' is volatile, because it doesn't matter whether or not the actual partitions themselves change, since the only partition that would be touched is the Rebalancing Partition,on, which is fixed. Obtaining the partition read lock can lead to a deadlock as outlined in the Jira description.
This closes #3760.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-23 10:07:21 -04:00
Joe Witt 6d8968cc87
NIFI-6520 update to tika 1.22
This closes #3756

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-09-23 08:18:55 -04:00
Rob Fellows 870a9529c5
NIFI-6693 - This closes #3759. replace assertTrue with assertEquals. use isNull matcher.
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-20 15:40:29 -04:00
Scott Aslan 219a83b023
[NIFI-6689] reset private var on dialog close
This closes #3752
2019-09-20 15:21:38 -04:00
thenatog 282c271c7c NIFI-6578 - Upgraded zookeeper framework version. Some code changes required. This change also required a change to the embedded zookeeper.
- Updating tests to account for the new node /zookeeper/config
Upgraded Zookeeper and Curator versions for zookeeper migrator in toolkit. Updated tests to allow for new znode /zookeeper/config
- Added documentation changes to reflect changes to the zookeeper.properties file going foward.

This closes #3715.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-20 10:49:04 -04:00
Bryan Bende ff6a7d9561
NIFI-5816 This closes #3726. Remove unused SFTP classes that reference Jsch
NIFI-5816 Switching SFTP processors from JSCH to SSHJ
NIFI-5816 LICENSE/NOTICIE updates to reflect changing from JSch to SSHJ

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-19 10:58:53 -04:00
Matt Gilman 171e37ddda
NIFI-6400 Fixing checkstyle issue. 2019-09-18 14:01:52 -04:00
Troy Melhase 4596ef7c8a
NIFI-6400 Better options, consistent ids for ShellUserGroupProvider.
NIFI-6400 Fixes to address PR feedback.
NIFI-6400 Accepts proposed changes.

This closes #3637
2019-09-18 09:08:41 -04:00
Rob Fellows 9e2a959778 NIFI-6659 - Open create new parameter context dialog in edit mode.
This closes #3729

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-16 11:16:25 -04:00
Rob Fellows 7e9277a2bc
NIFI-6671 - sort parameters by name in the 'Reference Parameter' dropdown.
This closes #3736
2019-09-16 10:03:10 -04:00
Rob Fellows 5ca3655dbf
NIFI-6381 - Make Parameters and Parameter Contexts searchable in UI
NIFI-6381 - remove wildcard imports

This closes #3728
2019-09-13 16:52:14 -04:00
Mark Payne 7623e6f5a1
NIFI-6024: When fetching names of buckets and flows for registry, sort them alphanumerically
This closes #3709.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-13 11:21:32 -04:00
Mark Payne eb6085a31d
NIFI-6658: Implement new bin/nifi.sh diagnostics command that is responsible for obtaining diagnostic information about many different parts of nifi, the operating system, etc.
This closes #3727.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-13 10:45:53 -04:00
Koji Kawamura e659e3b606 NIFI-5952 Refactor RAW S2S from nio to socket 2019-09-13 10:30:04 -04:00
Yolanda M. Davis 5106b764da
NIFI-6649 - repaired test due to min zero fix
NIFI-6649 - added check if logging debug is enabled

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3730
2019-09-12 16:08:31 -04:00
Rob Fellows 75c47388a6 NIFI-6630 - Add a goto action in the property table for properties that reference parameters.
NIFI-6630 - base convert and goto parameter logic on any reference to a parameter contained in the text of the property value.

This closes #3718

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-12 13:23:32 -04:00
Rob Fellows 2f7448c9e5
NIFI-6634 - Indicate variables are no longer recommended and favor parameters
This closes #3721
2019-09-12 12:44:52 -04:00
Yolanda M. Davis 8e1452a3f3
NIFI-6649 - added separate query interval configuration for observation queries
NIFI-6649 - documentation update

NIFI-6649 - add debug logging for score and prediction information

NIFI-6649 - fix to ensure counts return minimum value of 0 if not infinite or NaN

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3719
2019-09-12 12:13:19 -04:00
Scott Aslan c187292fd9
[NIFI-6629] calc correct width for usage ellipsis
This closes #3708
2019-09-11 15:06:56 -04:00
Scott Aslan d9c8d0c5a7
[NIFI-6625] add ellipsis and tooltip to list of parameters in usage list
This closes #3707
2019-09-11 13:27:55 -04:00
Scott Aslan 250e1b0297
[NIFI-6639] consistent ux for checkboxes and their correspoinding field
This closes #3706
2019-09-11 13:09:32 -04:00
Rob Fellows 29708d8362 NIFI-6641 - fix issue with dialog state when adding a new parameter context. added ellipsis styling and titles to name and id fields.
This closes #3710

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-10 10:50:52 -04:00
Rob Fellows 70af2acc06
NIFI-6641 - support read only mode for parameter context dialog 2019-09-10 10:03:06 -04:00
Rob Fellows 5ddc01eb9f
NIFI-6644 - Consider param context permissions when deciding to allow property conversion to parameter.
NIFI-6644 - when deciding to show the convert prop option: enforce user has read and write permissions to the param context & the props are not for a controller service defined in the controller settings.

This closes #3713
2019-09-09 16:11:20 -04:00
Mark Payne f89ea8cf21
NIFI-6645: Fixed problem in unit test for StandardParameterContext because behavior changed; fixed issue in StandardParameterContext around adding a parameter with no description
This closes #3714.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-09 15:21:21 -04:00
Andy I. Christianson 8a8b9c1d08
NIFI-6510 - Analytics framework (#3681)
* NIFI-6510 Implement initial analytic engine

* NIFI-6510 Implemented basic linear regression model for queue counts

* NIFI-6510 Initial analytics REST endpoint and supporting objects

* NIFI-6510 Connect the dots for StatusAnalytics -> API

* NIFI-6510 Added poc engine with prediction model caching

(cherry picked from commit e013b91)

DFA-9 - updated logging and corrected logic for checking if not in backpressure

(cherry picked from commit a1f8e70)

* NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics

(cherry picked from commit 050e0fc)

(cherry picked from commit 9fd365f)

* NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id

(cherry picked from commit 14854ff)

DFA-9 - reduced snapshot interval to 1 minute

(cherry picked from commit 36abb0a)

* NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions

* NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly

* NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly"

This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1.

* NIFI-6510 Added prediction fields for use by UI, still need to be populated

* NIFI-6510 Analytics Framework Introduction (#10)

* DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model.

DFA-9 - added logging

* DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval

* DFA-9 - checkstyle fixes

* NIFI-6510 Add prediction percent values and predicted interval seconds

(cherry picked from commit e60015d)

* NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not.

(cherry picked from commit 6fae058)

* NIFI-6510 Added tests for engine

(cherry picked from commit 6d7a13b)

* NIFI-6150 Added tests for connection status analytics class, corrected variable names

(cherry picked from commit 58c7c81)

* NIFI-6150 Make checkstyle happy

(cherry picked from commit b6e35ac)

* NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing

* NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics

* NIFI-6510 Adjusted interval and incorporated R-squared check

Updates to support multiple variables for features, clearing cached regression model based on r-squared values

Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support).

Ratcheck fixes

Added test for SimpleRegression. Minor fix for OLS model

fixed test errors

fixed checkstyle errors

(cherry picked from commit fab411b)

* NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11)

* NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties

NIFI-6566 - changes to allow scoring configurations for model in nifi.properties

NIFI-6566 - added default implementation value to NiFiProperties

NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3663

* NIFI-6585 - Refactored tests to use mocked models and extract functions.  Added check in ConnectionStatusAnalytics to confirm expected model by type

* NIFI-6586 - documentation and comments

This closes NIFI-6586

Signed-off-by: Andrew I. Christianson <andy@andyic.org>

* NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI
* Add multi-line tooltips with detail for connection queue back pressure graphics.
* Add estimated time to back pressure to connections summary table.
* Add back pressure prediction ticks.
* add moment.js to format predicted time to back pressure
* tweak summary table headings to match data displayed. re-order connection summary columns

* NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment.

* NIFI-6510 - add an enable/disable property for analytics

* NIFI-6510 - documentation updates for enable/disable property

* NIFI-6510 - UI: handle the scenario where backpressure predictions are disabled (#3685)

* NIFI-6510 - admin guide updates to further describe model functionality

* NIFI-6510 - code quality fixes (if statement and constructor)

* NIFI-6510 - log warnings when properties could not be retrieved. fixed incorrect property retrieval for score threshold

* NIFI-6510 Extract out predictions into their own DTO

* NIFI-6510 Optimize imports

* NIFI-6510 Fix formatting

* NIFI-6510 Optimize imports

* NIFI-6510 Optimize imports

* NIFI-6510 - Notice updates for Commons math and Caffeine

* NIFI-6510 - UI updates to account for minor API changes for back pressure predictions (#3697)

* NIFI-6510 - Fix issue displaying estimated time to back pressure in connection summary table when only one of the predictions is known.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3705

* NIFI-6510 Rip out useless members

* NIFI-6510 - dto updates to check for -1 value

* NIFI-6510 - checkstyle fix

* NIFI-6510 - rolled back last change and applied minNonNegative method

* NIFI-6510 Rip out useless members
2019-09-09 15:37:11 +00:00
Matt Gilman 625d4a13ca NIFI-6612:
- Updating context menu to only show the Parameters menu item when the user has the appropriate permissions.
2019-09-09 10:24:28 -04:00
Matt Gilman f867e92606 NIFI-6384:
- Adding auditing of parameter contexts.

This closes #3699.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-06 15:20:24 -04:00
Rob Fellows 7af710f168
NIFI-6637 - Display Parameter Context Id on Update Parameter Context dialog.
This closes #3701
2019-09-06 14:15:38 -04:00
Rob Fellows b9c0f51356 NIFI-6632 - Add tooltip to parameter value field indication it does not support EL or parameter references.
This closes #3703

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-06 13:42:23 -04:00
Mark Payne 731df87f22
NIFI-6626: Fixed the order of arguments passed to the ComponentDetails constructor when creating the Component Details for a connection. Also noticed that null fields are being added to the map when calling ComponentDetails.toMap() and this causes the UI to display these fields even though they have no value, so avoided populating the map with fields whose values are null
This closes #3696
2019-09-06 13:24:10 -04:00
Mark Payne 4868308fd8
NIFI-6380: Fixed partial updates to parameters
This closes #3692
2019-09-06 11:28:53 -04:00
Rob Fellows 9aed19137c NIFI-6623 - enter process group before navigating to the controller service
This closes #3695

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-06 09:31:46 -04:00
Rob Fellows 710c02c673 NIFI-6623 - Support linking to unauthorized components from referencing components 2019-09-06 08:10:28 -04:00
Arpad Boda f01668e66a
NIFI-6530 - HTTP SiteToSite server returns 201 in case no data is available
This closes #3647.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-06 19:27:59 +09:00
Rob Fellows 92fd3129ba NIFI-6602 - Fix: Apply button should be greyed out if no changes have been made
This closes #3691

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-05 14:54:47 -04:00
Rob Fellows fa8a4462e7 NIFI-6506 - don't show 'Sensitive value set' for null, sensitive params'
This closes #3683

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-05 13:39:48 -04:00
Rob Fellows 3c918e25bd NIFI-6506 - better empty-string/null handling for edit parameters 2019-09-05 13:39:48 -04:00
Rob Fellows 5cbade7b1d NIFI-6505 - fix for editing parameter value. simplify logic in edit dialog for disabled state of the apply button. 2019-09-05 13:39:48 -04:00
Rob Fellows a52d17628a NIFI-6506 - review feedback 2019-09-05 13:39:47 -04:00
Rob Fellows 183ece194e NIFI-6506 - update parameter context list after successful update. 2019-09-05 13:39:47 -04:00
Rob Fellows 93d99820b7 NIFI-6506 - more review fixes 2019-09-05 13:39:47 -04:00
Rob Fellows 5936fe3487 NIFI-6506 - navigate to process group when selecting to go to controller service from referencing components on parameter context dialog. 2019-09-05 13:39:47 -04:00
Rob Fellows 385c067507 NIFI-6505 - fixing issues uncovered in code review. 2019-09-05 13:39:47 -04:00
Rob Fellows 10b4f6fd89 NIFI-6506 - review feedback
* fixed issues converting sensitive props to parameters
* only include parameter value if it has changed (even for non-sensitive params)
* do not allow null parameter values when editing parameters.
2019-09-05 13:39:47 -04:00
Rob Fellows aa07378a03 NIFI-6505 - Addressing review feedback: reuse single method for getting parameter field values. 2019-09-05 13:39:47 -04:00
Rob Fellows f3a04521c2 NIFI-6506 - Fix syntax/style issues. Apply same logic for disabled state when converting prop to param. 2019-09-05 13:39:47 -04:00
Rob Fellows eddc5624d8 NIFI-6506 - Add ability to convert properties to parameters
* open existing add param dialog from property table
* convert prop to param, update param context
* keep add parameter dialog open until update of the context completes when converting property to param
* Updating parameter context while converting property to parameter.
* Disable the Apply button while updating
* When Cancel is pressed and there is an update in progress, issue the DELETE request to cancel the update.
* Show some status in the Add Parameter dialog when parameter context is updating following a property conversion.
* enforce character restrictions on property names.
* Add convert property to parameter to controller service config.
* Properly set sensitive setting when converting sensitive properties
* Allow converting of properties from controller services configuration into parameters
* Refactor addNewParameter method to be able to share common parts with convertPropertyToParameter
2019-09-05 13:39:47 -04:00
Brandon 78d9f13ded
NIFI-6416: This closes #3654. Removing swap locations on SWAP_IN
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-04 17:47:59 -04:00
Mark Payne a2edc2aa87
NIFI-6025: Fixed bug that caused Flows to indicate that they have been locally modified (when they hadn't been) when running against versions of the registry older than 0.5.0
This closes #3687.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-03 13:48:05 -04:00
Scott Aslan 459ef24e99
[NIFI-6606] disable PC for which the user does not have read permissions when configuring a PG. Also, sort the list of PCs for the combo dropdown in the PG configuration
This closes #3690
2019-09-03 13:28:06 -04:00
Mark Payne 4851e00f1d
NIFI-6607: Ensure that when creating ParameterContextUpdateRequestDTO that we populate the AffectedComponentEntities with the most up-to-date version of the component
This closes #3689
2019-09-03 12:55:55 -04:00
Matt Gilman 07e1c11520 NIFI-6608:
- Conditionally support parameters in controller service configuration.

This closes #3688

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-03 12:08:19 -04:00
Bryan Bende 180a188096 NIFI-6028 Protect against null VersionedComponent when finding relevant process group
This closes #3686.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-03 11:32:03 -04:00
Mark Payne 9ba280680f
NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occur with site-to-site. Fixed bug that caused site-to-site listener not to accept connections if there are no input/output ports on the root group - this used to be a valid check and was done to prevent spawning extra threads and doing extra work if no ports exist but now that we have site-to-site ports outside of the root group it's no longer a reasonable condition to check.
This closes #3658.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-02 11:51:03 +09:00
Matt Gilman bfbf3b7532 NIFI-6290:
- Addressing review feedback.

This closes #3636

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-08-30 13:53:12 -04:00
Matt Gilman c6e500dfa4 NIFI-6290:
- Ensuring the first allowable value is selected.
2019-08-30 13:51:37 -04:00
Matt Gilman f318a8c685 NIFI-6290:
- Ensuring the first allowable value is selected.
2019-08-30 13:51:37 -04:00
Matt Gilman e5850fa1a3 NIFI-6290:
- Addressing timing issue when parameters listing request is not immediately resolved.
2019-08-30 13:51:37 -04:00
Matt Gilman 5b409d6fac NIFI-6418:
- Updating property table editors to not support parameter referencing when necessary.
2019-08-30 13:51:37 -04:00
Matt Gilman d18c98ce78 NIFI-6290:
- Updating comboEditor to allow for parameter reference.
2019-08-30 13:51:37 -04:00
Matt Gilman d93ae47afc NIFI-6346:
- Updating nfel to support referencing parameters.
2019-08-30 13:51:37 -04:00
Matt Gilman 7228496801 NIFI-6345:
- Updating property table editor to distinguish between available modes (el vs parameter reference).
2019-08-30 13:51:37 -04:00
Matt Gilman 75e452b45c NIFI-6277:
- Allow parameters to be referenced from properties that do not support EL.
2019-08-30 13:51:37 -04:00
Mark Payne 254a84d74d
NIFI-6371: Fixed bug that caused root-level Controller Services to stay cached and therefore continually be validated, even after being deleted. Also cleaned up code that looked for specific text in Exceptions instead of of catching specific Exceptions
This closes #3614.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-29 16:24:40 -04:00
Scott Aslan f678c75d70
[NIFI-6282] manage parameters and parameter contexts
[NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option
[NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again.
[NIFI-6282] address pr review comments
[NIFI-6282] remove es6 let
[NIFI-6282] update marshall parameters logic and add comments
[NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const
[NIFI-6282] update regex and
[NIFI-6282] update parameter loading/serialization/marshalling
[NIFI-6282] use referencingComponents instead of affectedComponents
[NIFI-6282] activate Apply button for sensitive parameter set empty string change
[NIFI-6282] fix bug with PG parameters context menu enable
[NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity
[NIFI-6282] display referencing components during parameter management as well as during the parameter context update
[NIFI-6282] display no value set in parameter table when parameter value is null
[NIFI-6282]
- Add ellipsis to referencing component names.
- Addressing issues canceling update requests.
- Addressing issues with incorrect service scope.
- Addressing issue showing the affected parameters.
2019-08-29 14:48:53 -04:00
Mark Payne 4ae1fec78a
NIFI-6025: Include Processor 'scheduled state' (i.e., Enabled or Disabled) in the VersionedProcessor when pushing to Flow Registry and take into account when updating flows on the NiFi side
NIFI-6025: Include difference in Scheduled State as a Local Flow Difference

This closes #3546.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-29 10:35:03 -04:00
Bryan Bende b78aeb67a1 NIFI-6382 Fix verifyUpdate for parameter contexts to account for deletions
This closes #3661.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-08-28 11:29:00 -04:00
Mark Payne 59734dad88
NIFI-6028: Updates to ignore changes between local version of a flow and a remote version of a flow if the difference is the addition of a new property that has the default value
NIFI-6028: Code refactoring to address review feedback

This closes #3544.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-28 10:38:56 -04:00
Andy LoPresto fbd201f116
NIFI-6298 Upgraded versions of Spring Framework in nifi-spring-processors and nifi-email-processors.
NIFI-6412 Upgraded Spring Security version in nifi-framework-bundle pom.xml.
NIFI-6412 Updated version for spring.security.version in nifi-kerberos-iaa-providers and nifi-ldap-iaa-providers pom.xml files.

This closes #3564.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
2019-08-27 18:58:04 -04:00
Tamas Palfy 3de81d618e
NIFI-6509 - Date related issue in unit test VolatileComponentStatusRepositoryTest
As NIFI-6508 has already solved the original issue, turned this PR into a refactor within in the affected Test class.

This closes #3621.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-26 11:02:35 +09:00
Bryan Bende cc88dd428f
NIFI-6158 Fix conversion of Avro fixed type with logicalType decimal
This closes #3665.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-26 10:57:28 +09:00
Peter Wicks 0b74795578
NIFI-6562 Expose ExecutionNode in ProcessContext
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3656.
2019-08-20 11:13:55 +02:00
Malthe Borch 00f0f84856
NIFI-6508 - Fix test failure caused by timezone and/or DST difference
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-20 11:55:54 +09:00
Koji Kawamura 1cb1b00c09
NIFI-6477 Delete operate policy when its component is deleted
+1 from markobean.

This closes #3653.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-20 10:59:00 +09:00
Matt Gilman 326bc91405
NIFI-5839:
- Introducing case insensitive group membership.
- Ensuring user identity and group name mapping is applied when the user identity or group name is inferred through group membership decisions.

This closes #3657.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-19 17:04:05 -04:00
Koji Kawamura e32689a602 NIFI-6436 Fix NPE at StandardPublicPort
Removing unnecessary ProcessGroup from PublicPort constructor.

This closes #3580.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-08-19 15:33:26 -04:00
Mark Payne bd8342c5e0
NIFI-6382: On flow import and version change, create parameter context where necessary and add parameters where necessary
NIFI-6382: Fixed bugs reported in code review: sensitive properties referencing parameters should be exported to registry; when importing a versioned flow, any new parameters should be added to the parameter context

NIFI-6382: Updated logic for handling parameter references with sensitive properties when integrating with nifi registry

NIFI-6382: If user does not have permissions to READ a parameter context with a given name when importing a flow or changing flow version, ensure that NiFi properly generates an AccessDenied messages instead of throwing NullPointerException

This closes #3642.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-16 16:47:49 -04:00
Mark Payne bcf373a049
NIFI-6382: Allow Parameters to have null values
Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-16 16:47:27 -04:00
Brandon Devries 5f696482b0 NIFI-4775: Updating documentation and updating rocksdb version
NIFI-4775: Rolling back fix for NIFI-6416

NIFI-4775: Documentation improvements

+1 from markobean

This closes #3648

Signed-off-by: Brandon <devriesb@apache.org>
2019-08-15 15:28:48 +00:00
Jeff Storck 33187f8058 NIFI-5176 NiFI builds on Java 11
Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api.
Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+
Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites
Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters.
Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build.  Needs more investigation.
Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString
Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited
Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension
Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes
Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy
Updated maven-compiler-plugin version to 3.8.1
TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4
HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory
  Removed unnecessary throws declaration from getSslSocketFactory method
Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module
Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11
Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable.
  Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs.
  URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders.

NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service

NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11

This closes #3404.
2019-08-13 18:41:39 -04:00
Brandon Devries 7d77b464cc NIFI-4775: FlowFile Repository implementation based on RocksDB
+l from markobean.

This closes #3638.

Signed-off-by: Brandon <devriesb@apache.org>
2019-08-12 19:03:38 +00:00
Alessandro D'Armiento 35d79eaf0f NIFI-6420 Controller Service references not properly tracked if multiple references from same component
<h1>Changes</h1>

-  Instead of keeping a Set of referencing `ComponentNode`, keep instead a Map which stores, for each referencing component, also the name of the property for which the service is referenced
   -  This is done to keep the `addReference()` idempotent
- `StandardControllerService.referencingComponents` changed from HashSet to HashMap
    -  `Key` is the combination of `ComponentNode.hashCode()` and property name
    -  `Value` is the referencing `ComponentNode`
- `ControllerServiceNode.addReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.removeReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.getReferences()` signature is left untouched, when it's called, the values of the `referencingComponents` map are turned into a Set, meaning a referencing component will keep being returned by this method until at least one of its properties still reference the `ControllerService`
- `StandardSchedulingContext.leaseControllerService()` uses a dedicated constant
- `FrameworkIntegrationTest` has a test which create a processor with two optional controller service properties and then verifies that having them referencing and dereferencing the same controller service doesn't cause the reported bug anymore

Removed SchedulingContext from ClojureScriptEngineConfigurator

Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3600.
2019-08-09 15:22:14 -04:00
Mark Payne 164e0c4186
NIFI-6380: Fixed NPE that is thrown when clustered and using parameters from Controller Services
This closes #3635
2019-08-06 10:19:20 -04:00
Matt Gilman 944d256d31
NIFI-6380:
- Fixing checkstyle issue.
2019-08-02 14:21:56 -04:00
Mark Payne 4f50f30ad7
NIFI-6380: Introduced the notion of Parameters and Parameter Contexts to the code base.
- Added nifi-parameter module
- Added Parameter to nifi-api
- Added ParameterContext and ParameterLookup
- Updated EL to support Parameters
- Updated backend so that any property can make use of Parameters
- Added web endpoint for Parameter Contexts
- Updated Templates to make use of Parameter Contexts
- Updated Versioned Flows to make use of Parameter Contexts
- Updated Stateless NiFi to support Parameters and take Parameters as part of the configuration, not Variables.

NIFI-6380: Addressed review feedback and fixed bugs; added additional unit and integration tests to verify

NIFI-6380: Added Description to Parameter Context

NIFI-6380: Fixed checkstyle violations

NIFI-6380: Fixed bug that caused updating a Parameter Context to fail if the name is provided and unchanged

NIFI-6380: If parameter is being deleted, don't worry about its sensitivity flag. This addresses a bug where the deletion of a Sensitive Parameter would require that the Parameter be submitted with a value of null and a sensitivity flag of true; else it would provide an error indicating that the parameter can't be changed from sensitive to non-sensitive. Now, the sensitivity flag is ignored.

NIFI-6380: Fixed bug around unsetting Process Group's Parameter Context

NIFI-6380: Moved lastRefreshed timestamp from ParameterContextDTO to ParameterContextsEntity and renamed to currentTime to match the pattern of ControllerServicesEntity rather than FlowHistory. Added parameterContextId to ProcessGroupFlowDTO

NIFI-6380: Added additional integration tests around escaped parameter references

NIFI-6380: Additional tests and bug fix for referencing EL from within another EL Expression

NIFI-6380: Created ParameterEntity to house a 'canWrite' flag for parameters and updated ParameterContext to use it. Updated ParameterContextUpdateRequestDTO to include a Set<AffectedComponentEntity> indicating the components that are affected by the update

NIFI-6380: Addressed review feedback

NIFI-6380: Addressed additional review feedback, mostly around code cleanup

NIFI-6380: Bug fix

NIFI-6380: Addressed more review feedback; fixed a couple of minor bugs encountered when testing

NIFI-6380: Bug fix around Parameter escaping for properties that support Expression Language. Fixed inconcsistency in Authorization hierarchy between /parameter-contexts/1234 and /parameter-contexts  to ensure that the parent of /parameter-contexts is /controller, regardless of how we arrive at the /parameter-contexts resource

NIFI-6380: Fixed but around using ProcessContext#newPropertyValue(String) that previously resulted in causing the Parameters to be evaluated a second time, thereby ignoring escape characters. Updated Integration Test to verify behavior. Also fixed bug in LoadBalanceQueueIT as it was a one-liner and noticed it while running tests to verify other behavior

NIFI-6380: Ensure that if Processor or Controller Services references or de-references a Parameter that user making the change has READ policy on the Parameter Context

NIFI-6380: Verify permissions on Parameter Context when creating processor/controller service and when moving between groups, insantiating template, reverting/change flow version

NIFI-6380: Addressed problem that resulted from rebasing against master; fixed bug in showing the Affected Components for a particular Parameter Context Update

NIFI-6380: Fixed NPE that occurs when attempting to update controller-level Controller Service

NIFI-6380: Updated validation error text when referencing parameter from controller-level controller service or reporting task

NIFI-6380: Ensuring permissions for copy/paste when components reference parameters.

NIFI-6380: Do not require READ policy on Parameter Context in order to un-reference a parameter, only to reference one.

NIFI-6380: Restricted parameter names to match those in the Feature Proposal. Fixed bug that allowed copy & paste of a PG for which user does not have READ policy for a child/descendant PG's parameter context

This closes #3536
2019-08-02 10:45:39 -04:00
Mark Payne a9a4b765b1
NIFI-6517: Ensure that if IOException is thrown from LoadBalanceSession that we properly catch the Exception, mark session as complete, and then re-throw. There was one condition in which this was not occurring. This commit addresses that situation.
This closes #3626.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-01 15:21:53 -04:00
Mark Payne f8e93186f5
NIFI-5856: When exporting a flow that references a Controller Service, also include the ExternalControllerServiceReference's in the snapshot. When importing/updating a flow, if a Controller Service that is inherited is referenced, attempt to find any one ControllerService that gets inherited whose name matches the reference and if found link to that Controller Service.
NIFI-5856: Fixed checkstyle violation

NIFI-5856: Addressed review feedback

NIFI-5856: When assigning a matching Controller Service at a higher level for Process Group import/update, ensure that we consider the User's read permissions when selecting the Controller Service

This closes #3553.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-07-29 10:28:53 -04:00
Mark Owens d1c30d1477 NIFI-6433 Update getStatusHistory to honor filter values.
This modification updates the getStatusHistory method of
VolatileComponentStatusRepository to utilize the start, end, and
preferredDataPoints parameters when retrieving status histories.

When calling the various get<XXXX>StatusHistory methods of
VolatileComponentStatusRepository, they are all passed start, end,
and preferredDataPoints values to allow the filtering of histories
according to the provided parameters. But the follow-on method calls ignore
those values completely. This ticket updates those methods to honor the
parameters and filter the returned Dates accordingly.

This closes #3579.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-07-26 11:27:06 -04:00
Rob Fellows 7dda04e895 NIFI-6455 - Fix issue preventing scrolling all the way to the bottom of certain tables. (#3591)
This closes #3591.

Signed-off-by: Peter Wicks <patricker@gmail.com>
2019-07-25 13:54:12 -06:00
Peter Wicks 2b0b96e59e NIFI-6439 - Revert Spring Framework HTTPS XSD (#3590) 2019-07-18 11:08:07 -07:00
Jeff Storck 4783b12a9c
NIFI-5254 Upgraded Groovy to version 2.5.4
Updated pom files regarding groovy dependencies.  groovy-all was changed to a pom type dependency
Added Groovy Bintray as a plugin repository in root pom
Upgraded spock-core to version 1.3-groovy-2.5 and added exclusion of groovy dependencies to force spock to use the version from dependencyManagement
Updated groovy-eclipse-batch to use the groovy.version property to determine the version
Updated groovy-eclipse-compiler to 3.4.0-01
Updated maven-compiler-plugin to use maven.compiler.source and maven.compiler.target properties for source and target configuration properties
Removed configuration and dependencies sections from nifi-toolkit-admin and nifi-toolkit-encrypt-config maven-compiler-plugin configurations so that the configuration from the root pom's maven-compiler-plugin is inherited
Removed dependencyManagement from nifi-jetty module, version of groovy modules will be inherited from root pom
Removed maven-compiler-plugin configuration from nifi-toolkit-api so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated spock-core dependencyManagement to 1.3-groovy-2.5
Fixed AESSensitivePropertyProviderTest issue with Groovy creating KEY_256_HEX slice of the wrong size due to BigDecimal being used as the result of the division; using intdiv to force an integer result creates the correctly sized array
Added groovy-json test dependency to nifi-web-security
Removed maven-compiler-plugin configuration from nifi-lookup-services, nifi-mock-record-utils, and nifi-web-utils so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated root pom pluginManagement to specify version 3.8.0 of maven-compiler-plugin
Added maven-compiler-plugin config to nifi-toolkit-admin and nifi-toolkit-encrypt-config to use groovy-eclipse-compiler during the compile phase so that the groovy-based tools are compiled
Addressed deprecated CliBuilder and OptionAccessor usage in nifi-toolkit-encrypt-config, those classes were moved from groovy.util to groovy.cli.commons
Removed getInner() usage from nifi-toolkit-encrypt-config, method no longer exists causing the tests to crash
Updated CryptographicHashAttributeTest to use java.time classes instead of java.util.Date
Updated nifi root POM's groovy-test dependency to be test-scoped
Added properties for specifying groovy versions for several modules: nifi, nifi-groovyx-bundle, nifi-scripting-bundle, nifi-toolkit
Established dependency management for groovy-all:pom:2.5.4, and added that dependency to several modules: nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-admin, nifi-toolkit-encrypt-config
Added groovy version property usage to several modules that established a dependency on a groovy submodule that was not listed in its own or inherited dependency management
Removed unused build-helper-maven-plugin from nifi-toolkit-api's POM
Removed unnecessary groovy-eclipse-compiler build plugin config from nifi-web-utils' POM to use the inherited config for that plugin
Updated several modules' NOTICEs to include appropriate Groovy NOTICE content
Updated to list groovy-all:pom:2.5.4 and its transitive submodule dependencies to nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, and nifi-toolkit-assembly NOTICEs
Added missing groovy-all🫙2.1.6 NOTICE to nifi-hive-nar and nifi-hive_1_1-nar NOTICEs
Added missing groovy-all🫙2.4.11 NOTICE to nifi-hive3-nar NOTICE
Updated to list groovy-all🫙2.4.16 nifi-other-graph-services-nar NOTICE
Removed Groovy NOTICE content from nifi-record-serialization-services-nar NOTICE, no Groovy modules are included in the NAR
NIFI-5254 Updated several modules' NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with Groovy modules which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-hive-nar, nifi-hive3-nar, nifi-hive_1_1-nar, nifi-other-graph-services-nar, nifi-scripting-nar, nifi-toolkit-assembly
NIFI-5254 Updated NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with modules used by Groovy 2.5.4 which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-assembly.
The following modules' LICENSE and NOTICE content were added: Apache Ant, Apache Commons CLI, JLine, JUnit Platform/Jupiter

This closes #3547

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-07-11 14:17:35 -04:00
thenatog 95f5b2278c
Removed guava test dependency as it doesn't look like any of the test code requires it.
NIFI-5562 - Removed unnecessary usage of Guava

NIFI-5562 - Updated Guava version to 28.0-jre

NIFI-5562 - Upgraded Guava to 28.0-jre and Curator to 4.2.0

This closes #3577

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-07-09 12:08:27 -04:00
Kemix Koo 5127cb209a
NIFI-6392: Avoid accessing the dir for /nifi-docs
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-07-06 18:39:29 -04:00
Rob Fellows 846c9b0a3f
NIFI-6401 - Long List of Users Overflows onto Date
This closes #3567
2019-07-02 16:02:39 -04:00
Troy Melhase e973cacb2f
NIFI-5973 Adds ShellUserGroupProvider.
NIFI-5973 More comments and better defaults for the shell provider.
NIFI-5973 Fixed bug where user was being retrieved by identifier when identity was provided.
NIFI-5973 Fixed a formatting string in the OS X shell commands.
Updated testing conditions to run IT in OS X environment.
Changed unit test to provide identity rather than identifier.

This closes #3537.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-20 16:22:42 -07:00
Jeff Storck 1d560e2b02 NIFI-6360 Updated Mockito to 2.28.2, PowerMock to 2.0.2
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated.  Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests.  Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version.  Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup

This closes #3533

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-06-17 12:21:07 -04:00
thenatog 0e1a37fcb9
NIFI-6362 - Upgraded com.puppycrawl.tools:checkstyle to 8.18. Made small checkstyle changes to nifi-kafka.
NIFI-6362 - Upgraded both checkstyle plugins to latest. Added a checkstyleCache file and property.

NIFI-6362 - Downgraded maven-checkstyle-plugin to 3.0.0 instead as there was an issue with the latest version and @throws Javadoc checkstyle module. Fixed some checkstyle issues. Changed checkstyleCache file to .checkstyleCache. Added to .gitignore.

NIFI-6362 - Changed checkstyleCache file location into target/, and is no longer a hidden file. Removed .checkstyleCache from .gitignore as it is no longer required.

NIFI-6362 - Grep pattern no longer required.

This closes #3522.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-17 17:19:09 +09:00
Bryan Bende 2102d8a0bd
NIFI-6328 Allowing force-commit when in LOCALLY_MODIFIED_AND_STALE state
NIFI-6328 Addressing review feedback
NIFI-6328 Additional validation on action

This closes #3499
2019-06-13 10:42:13 -04:00
Mark Payne 8bed249f37
NIFI-6353: If LoadBalanceSession determines remote node's queue is full, mark session complete and penalize the Partition, rather than just penalizing the session. This allows the client to move on to another session, which avoids an issue where backpressure on one queue can prevent data from being transferred between nodes in another queue
This closes #3529.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-12 11:32:41 -04:00
Mark Payne 2465f67dfe
NIFI-6339: Only call @OnConfigurationRestored method on components when they are created if FlowController has been initialized
This closes #3530.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-12 10:56:04 -04:00
Peter Wicks 2d3fce81c1
NIFI-6350 Fixed misleading flow controller startup error message.
This closes #3516.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-05 19:25:13 -07:00
Mark Payne 60b5c13ce9
NIFI-6341: Addressed bug that can result in a deadlock if Thread 1 adds a Controller Service to a Process Group (which obtains group's write lock and AbstractComponent's Lock) while at the same time Thread 2 attempts to peform validation on the service (which obtains Controller Service's write lock, then AbstractComponent's Lock)
This closes #3510.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-04 09:55:26 -04:00
Bryan Bende e7f4af6801 NIFI-6315 Ensuring remote ports get tracked correctly when saving/retrieving versioned flows
NIFI-6315 Renaming public ports during import/change-version when name alreadye exists, and ignorning name change from flow diffs

NIFI-6315 Ensuring public ports get renamed during copy/paste

NIFI-6315 Set max concurrent tasks on Port based on value from VersionedPort

This closes #3488.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-03 11:25:04 +09:00
Andy LoPresto a9d1dd7b08 NIFI-6337 Resolved groovy unit test execution problems in nifi-properties-loader and fixed failing unit tests.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3508
2019-05-31 08:44:17 -04:00
Matt Gilman 999cfdcac9
NIFI-6302:
- Updating integration tests to verify pruned results.

This closes #3487.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-29 14:45:00 -07:00
Andy LoPresto e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
Rob Fellows 0a31c38790
NIFI-6253 - Remove invalid/unnecessary argument in call to nfCanvasUtil.reload
This closes #3491
2019-05-24 11:02:09 -04:00
Rob Fellows 410854f53e
NIFI-6316 - Upgrade jQuery to the latest version (3.4.1).
This closes #3489
2019-05-24 09:52:02 -04:00
thenatog fe68d43e1d
NIFI-6280 - Broke out the matching for /access/knox/** and /access/oidc/** to allow the Jetty security filters to be applied in the /access/oidc/logout and /access/knox/logout cases.
NIFI-6280 - Updated terminology in JwtAuthenticationFilter to authentication instead of authorization. Added stricter token parsing using an explicit regex pattern. Added tests.
NIFI-6280 - Updated terminology from Authorization to Authentication.
NIFI-6280 - Updated the access logout method to use getNiFiUserIdentity(). Updated javascript logout method to handle errors.
NIFI-6280 - Fixing checkstyle issues.
NIFI-6280 - Added some javadoc comments and logging. Renamed some variables for clarity. Fixed handling of exception when JWT does not match expected format.
NIFI-6280 - Cleaned up checkstyle, increased log severity level for logout action, and cleaned up Groovy syntax in test.

This closes #3482.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-22 17:05:06 -07:00
Koji Kawamura 8a50cb10b2 NIFI-2933 Remote input/output ports at any PG
Specify remote access at port creation.
Incorporated comments, and finished refactoring.
Renamed RootGroupPort to PublicPort.
Fix error message for creating a connection from a child PG having only PublicPorts.
Enhanced ProcessGroup instances rendered in the parent ProcessGroup
Loosen Port move check, allow moving public port between PG.
Show 'Remote NiFi Instance' info on Connection dialogs
Make labels narrative.
'Within Remote Group'.
Fixed DTO (de)serialization.
Return null only if all values are null.

This closes #3351.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-22 15:44:16 -04:00
Matt Gilman a97766d62f NIFI-6302:
- Ensuring Process Group contents are pruned when appropriate.

This closes #3477.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-22 14:12:50 -04:00
Mark Payne cbafd8f834 NIFI-5922: Addressed checkstyle violations; added to README Updates to nifi-fn proposal:
- Separated into nifi-fn-core, nifi-fn-bootstrap, nifi-fn-nar, nifi-fn-assembly; fully unpacks nars and runs flows
- Rebased against master and updated to version 1.10.0-SNAPSHOT
- Removed dependency on nifi-framework-core
- Added LICENSE/NOTICE files
2019-05-21 11:19:17 -04:00
Mark Payne f08c2ee43f
NIFI-6285: Addressed issue that resulted in swapped data not being swapped back in if load balancing strategy changed while data was swapped out; added integration tests for swapping. In testing, also encountered an issue with data being swapped out while swap files were being recovered causing the queue size to be wrong and causing errors about not being able to swap data in, because it attempted to swap the data in twice.
This closes #3473.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-05-14 15:31:15 -04:00
Scott Aslan 95fd14866e
[NIFI-6292] update variables slickgrid table row ids
This closes #3469
2019-05-13 09:57:19 -04:00
Mark Bean a2bacde62c
NIFI-6225: bug fix for listing relationships in sorted order
This closes #3459
2019-05-10 15:00:45 -04:00
Jeff Storck 65c41ab917 NIFI-6196: Upgrade Jetty version to 9.4.15.v20190215
Updated NOTICE with current copyright year for Jetty dependencies
Updated Jetty SSLContextFactory usage, invoke setEndpointIdentificationAlgorithm(null) on server SslContextFactory instances
Updated TestInvokeHttpSSL to provide a separate client keystore, rather than reusing the server's keystore
Regenerated nifi-standard-processors keystore and truststore, added client keystore
Updated ITestHandleHttpRequest, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use a separate client keystore instead of reusing the server's keystore.  Also updated the tests to separately test one-way and two-way SSL

NIFI-6196 - Setting the endpoint identification algorithm to null for sockets to ensure certificates without SANs applied correctly still work. We can change this in a new NiFi release with other potentially breaking changes.

This closes #3426.
2019-05-09 12:05:12 -04:00
thenatog cf6f517250
NIFI-6085 - Added /access/logout endpoint to allow JWT auth tokens to be removed correctly. Added some tests. Found an error in the KeyDAO which did not allow key deletion.
NIFI-6085 - Updated logOut method to use NiFiUserUtils and updated tests.
NIFI-6085 - Added some more integration tests.
NIFI-6085 Suppressed stacktrace when token is used after being invalidated.

This closes #3362.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-08 17:45:39 -07:00
Jeff Storck 376c344edb NIFI-5574 Removed usage of Paths.get() due to TEST_RES_PATH being resolved to a string from a URI, which results in platform-specific path information (C:\) when tests are run on Windows.
This closes #2988

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-08 07:38:40 -04:00
Rob Fellows 4484565df1
NIFI-6253 - Fix: UI - Process Group ID display in Operate palette can get out of sync when process groups are removed from different sessions.
NIFI-6253 - simplifying loading of a new process group, handle failure.
NIFI-6253 - Expose nfCanvas.loadProcessGroup functionality through nfCanvasUtils.loadProcessGroup
NIFI-6253 - optional groupId approach to reload

This closes #3458
2019-05-07 15:34:12 -04:00
simonl 0650521eb2
NIFI-6171 always send email scope for OIDC
NIFI-6171 re-added lookupEmail() as fallback
NIFI-6171 additional OIDC scopes via nifi.properties
NIFI-6171 alternative user identification (instead of email)  via nifi.properties
NIFI-6171 changed lookupEmail() so that any configured claim can be fetched fro the UserInfo endpoint

This closes #3398
This closes #2346
2019-04-26 16:40:37 -04:00
Alex Aversa c5c9425989
NIFI-5986 Adding "Stop & Configure" button functionality to Processor Details modal.
NIFI-5986 Refactored Stop & Configure feature to enable bulletin/thread notifications and terminate capability in processor dialogs. Also added feature as a menu item to the canvas context menu.
NIFI-5986 Refactored Stop & Configure feature to enable a status bar in a dialog that conveys bulletin/thread notifications and buttons.
NIFI-5986 Refactored Stop & Configure feature to decouple status bar from modal component, updated styling and revised graph synchronization process.
NIFI-5986 Refactored Stop & Configure feature to improve status bar button hide/show functionality.
NIFI-5986: Rebased and resolved conflicts.
NIFI-5986 - Refactored Stop & Configure statusbar observer, as well as processor dialogs to remove duplicative code.

This closes #3281
2019-04-26 12:07:02 -04:00
Koji Kawamura 31463c5dad
NIFI-6160 Apply config error handling convention
What this PR changed:
- Standardized required name error message to 'The name of the xxxx must
be specified'
- Added nfErrorHandler.handleConfigurationUpdateAjaxError
    - Replaced existing custom error handling for 400 status code
    - Replaced handleAjxError with this where appropriate
- Standardized config error dialog title to 'Configuration Error'
- Close the configuration dialog when configuration is successfully
updated

Above convention applied to following components:
- ProcessGroup
- Input/OutputPort
- RemoteProcessGroup
    - RemoteGroupPort
- Template
- Connection
- ControllerService
- Label
- Processor
- ReportingTask
- NiFi Settings/GENERAL tab
- Variable
- User

NIFI-6170 Provide consistent UX at Connection config
Fixed Label.

Incorporated review comments
Removed unnecessary else block.
Disable canvas when the message pane is shown

This closes #3401
2019-04-24 16:47:11 -04:00
Peter Wicks 2b83b7d9e8
NIFI-6223 Expose Cluster Node Type to Controller Services
NIFI-6223 Enhance Mock Controller Service to allow specifying node type

This closes #3444.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-04-24 13:20:55 -04:00
Andrew Lim e8174fb99f NIFI-6002 Corrected duplicated CSS class and capitalized REST API 2019-04-24 10:05:21 -04:00
Mark Payne 5e2559db42
NIFI-6236: Update VolatileFlowFileRepository to decrement claimant counts when FlowFiles are deleted
This closes #3451.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-04-23 13:10:14 -04:00
Mark Payne 055b3cac54 NIFI-6220: If FlowFile is created by cloning a relationship, do not create an ATTRIBUTES_MODIFIED provenance event for it.
NIFI-6220: Updated test name and fixed typo

This closes #3438.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-04-23 17:08:26 +09:00
Koji Kawamura c5d0643d1d NIFI-5792: Remember created versioned flow information
- Before this fix, NiFi loses information about created versioned flow in case of subsequent snapshot creation failure, and NiFi API returned an error response
- This commit makes:
  - The created versioned Flow information is stored even if subsequent snapshot creation fails
  - NiFi API to return a successful 200 response in that case, but return versioned flow status as SYNC_FAILURE with an explanation. NiFi UI shows a popup error dialog with the explanation.
  - Versioned flow status will be LOCALLY_MODIFIED if the latest version is 0.

This closes #3134.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-04-22 13:27:40 -04:00
aeaversa 512555dbb0 NIFI-6221 Adding 'Download' and 'View' content buttons to List Queue (#3440)
NIFI-6221 Adding 'Download' and 'View' content icon buttons to slick grid actions column to improve end user navigation.
2019-04-19 16:25:15 -06:00
Andy LoPresto 22563fcb9a
NIFI-6201 Suppressed stacktraces for expected error conditions (connection timeout, missing component bundle, etc.) in application startup when the stacktrace provides no additional helpful data.
NIFI-6201 Fixed existing checkstyle complaint on missing Javadoc element.

NIFI-6201 Restored "connection refused" text to relevant registry sync error message.

NIFI-6201 Simplified error logging for bundle not found issues and removed unused bundle coordinate methods.

This closes #3427.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2019-04-15 16:52:13 -04:00
Mark Payne 7b945182af NIFI-6215: Ensure that when we are swapping FlowFiles back into nifi, that we consider forward slashes or back slashes as the same path separator when determining filename
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3434
2019-04-15 15:54:41 -04:00
Brandon Devries cbc07eb663 NIFI-6157: Allowing configuration of concurrent tasks and transferred FlowFile count to Funnels and LocalPorts
This closes #3412
Signed-off-by: Brandon <devriesb@apache.org>
2019-04-15 14:13:11 +00:00
Rob Fellows b5ff622118
NIFI-4230 - Use a better default location when pasting via keystokes and the original copied item is not fully in view.
- handle multi-select copy/paste better.
- remove dimension info from getOrigin function
- handle pasting better when pasted items should be centered and not offset from the original.
- fixed strict mode and removed console.logs
- Fixing pasting logic when trying to center the items on the canvas.

This closes #3383
2019-04-12 15:27:26 -04:00
Mark Payne 91b2283f49
NIFI-6183: Implemented base class for Framework-level integration tests that is responsible for instantiating the key components for integration tests, including FlowController, WriteAheadFlowFileRepository, WriteAheadProvenanceRepository, FileSystemRepository, FileSystemSwapManager, StandardFlowFileQueue, StandardProcessScheduler, etc. Implemented several integration tests to span different parts of the framework
This closes #3423.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-04-10 16:47:08 -04:00
Mark Payne 0f10010683
NIFI-6199: Fixed problematic unit tests that did not properly wait for async tasks to complete before making assertions
This closes #3421.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-04-09 11:24:58 -04:00
Mark Payne bf7e70e4c3
NIFI-6194: Fixed verification logic to determine whether or not a variable's value can be changed. Previously, the logic had a bug that resulted in a failure if any child group contained a running processor that references an overridden variable and the user attempts to change the overridden variable at the higher level. We should not include any components of descendent groups if the descendent group overrides the variable.
This closes #3420.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-04-09 11:15:45 -04:00
Mark Payne 38db4e97cd
NIFI-5464: Consider ports invalid unless they have both incoming and outgoing connections
This closes #3417.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-04-09 10:19:24 -04:00
Rob Fellows fb9f2af04a
NIFI-6136 - fix: UI - User and Template bug when page popped out
NIFI-6136 - return the appropriate value (false) when verifyDisconnectedCluster is called from a non-clustered environment.

This closes #3390
2019-04-08 15:45:19 -04:00
Alex Aversa 96279415a7 NIFI-6188 - Added 1s delay on Search control autocomplete query event to prevent premature results retrieval
This closes #3408.
Signed-off-by: Brandon Devries <devriesb@apache.org>
2019-04-05 12:32:40 -04:00
Rob Fellows e374d2cc30
NIFI-6140 - Incorrect title on the read only version of the Reporting Task detail dialog
This closes #3385
2019-04-04 13:40:14 -04:00
Rob Fellows cb4fdc2ae1
NIFI-6126 - throttle requests to refresh the canvas to prevent too many server requests when holding down CTRL-R
* Added lodash utility library to leverage its throttle capability (and many other useful functions in the future).
* Made lodash available in all JSP's so it could be imported into nf-common (or any component for that matter).
* Added a throttle function to nf-common that just wraps _.throttle

This closes #3393
2019-04-04 09:51:41 -04:00
Mark Payne 76392ee862
NIFI-6155: Ensure that any task submitted to FlowEngine catches Throwable so that the task doesn't die just die silently in the case of an unexpected error/exception
This closes #3395.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-29 10:09:35 -04:00
Mark Payne d3d43262db
NIFI-6150: Moved logic of decrementing content claims from ProcessSession.commit to the FlowFile Repository. Also updated load-balancing protocol and queue drop logic to no longer decrement content claims since the flowfile repo will handle the logic. This allows us ts to ensure that the claimant counts are decremented only if the repo is successfully updated and still before checking if the claim is still in use or not.
This closes #3391.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-27 10:22:01 -04:00
Joe Witt 1bdab73201 NIFI-6121 Updated the easy dependencies to update
This closes #3374

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-22 09:01:21 -04:00
Matt Gilman d35d15cdda
NIFI-6027
- Allowing user or group existence enforcement to be parameterized.
- Fixing error handling when loading user groups which may have resulted in stack trace leaking.

This closes #3377.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2019-03-20 13:38:46 -04:00
Bryan Bende 9d21a106ad NIFI-6120 Move nifi.registry.version property to root pom so it can be shared between nifi-framework and nifi-toolkit
This closes #3369.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-03-20 10:56:10 +09:00
Rob Fellows 0cb15cfb1a
NIFI-6118 - Fix: UI - Not all calls to nfCanvasUtil.reloadConnectionSourceAndDestination are made with the proper component IDs
This closes #3368
2019-03-18 14:48:41 -04:00
Rob Fellows 6401e32a33
NIFI-6126 - Change nf-canvas to respond to keyup events rather than keydown
This closes #3376
2019-03-18 14:08:56 -04:00
Ryan Bower 160ade9f9d
NIFI-5018
- Added snap alignment for nf-label, label resize events, and nf-connection
- Shift key now disables snap alignment during the drag event.
- nf-connection load-balance-icon updated

This closes #3335
2019-03-11 13:52:09 -04:00
Mark Payne c84f40ee36
NIFI-6111: Fixed bugs in the Status History values. If metrics are not available yet from all nodes for the last point on the graph, leave the cluster aggregate value off for the last point to prevent it from dropping significantly at the end of the chart. Fixed bug where counter values were not properly summed together in cluster view. Addressed issue with Average Task Duration
This closes #3361.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-08 17:27:49 -05:00
Mark Payne 58a25cfa5a
NIFI-6110: Updated StandardProcessSession such that if we fail to update the FlowFile Repository, we do not decrement claimant counts for any FlowFiles that were removed. Doing so can cause an issue where a FlowFile is removed, then the FlowFileRepo update fails, resulting in the flowfile being rolled back, but after its claimant count is decremented. It will then be processed again, which can result in the same thing, and we'll end up decrementing the claimant count repeatedly. Also updated LengthDelimitedJournal so that if the overflow directory already exists, it does not fail when trying to create the directory and instead just moves on. Updated unit tests to test both of these new fixes and updated DummyRecordSerde to be thread-safe because the TestLengthDelimitedJournal now needs it to be thread safe due to the new unit test that was added.
This closes #3358.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-08 10:32:38 -05:00
Kevin Doran 1d06044e3b
NIFI-6108: Fix port names when imported from registry
This closes #3356.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-06 17:07:18 -05:00
thenatog 36bbc77723 NIFI-6097 - Upgraded the fasterxml jackson version to 2.9.8. Ensure that the version is consistent across modules using a maven property defined in the root pom.
This closes #3347

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-04 19:30:35 -05:00
Bryan Bende be6cd4f0f2
NIFI-6069 Fixing checkstyle error 2019-02-25 16:52:41 -05:00
Bryan Bende 4b45e85327 NIFI-6055 Improving error handling during auto-loading of NARs and ensuring only one Jetty NAR is loaded
NIFI-6056 Fixing cassandra bundle so nifi-mock is a test scoped and removing incorrect usage of ProviderCreation exception from nifi-framework-api

This closes #3322.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 12:19:12 -05:00
Bryan Bende 26c00f1080 NIFI-6069 Fixing fingerprinting of connection relationships to be properly sorted
This closes #3325.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 12:13:13 -05:00
Mark Payne 414bfc94fc NIFI-6068: Fixed unit test 2019-02-22 11:55:25 -05:00