1657 Commits

Author SHA1 Message Date
Mark Payne
0448e23a96
NIFI-7414: Escape user-defined values that contain invalid XML characters before writing flow.xml.gz
NIFI-7414: Updated StandardFlowSerializerTest to include testing for variable names and values being filtered

This closes #4244
2020-05-04 14:49:15 -04:00
Matt Gilman
e81960f8e8 NIFI-7170:
- Adding a flag to nifi.properties to disable anonymous authentication.

NIFI-7170:
- Fixing checkstyle issues.

NIFI-7170:
- Adding missing license header.

NIFI-7170:
- Initial PR feedback.

NIFI-7170:
- Fixing broken integration tests.
- Creating new integration tests for verifying allowing and preventing anonymous access.

NIFI-7170:
- Ensuring the new anonymous authentication property is considered for proxied requests.

NIFI-7170 - Fixed comment.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4099.
2020-04-30 15:20:19 -04:00
Matthew Burgess
1259bd5bd1
NIFI-7379: Support multiple instances of Prometheus registries/metrics (#4229)
* NIFI-7379: Support multiple instances of Prometheus registries/metrics

* NIFI-7379: Refactored Prometheus objects to support multiple instances
2020-04-28 16:56:49 -04:00
Andy LoPresto
148537d64a
NIFI-7377 Cleaned up nifi-stateless logs.
Refactored masking logic to CipherUtility and indicated masking with label and Base64 output.
Added JSON masking logic to nifi-stateless module.
Added argument masking functionality to Program.
Moved groovy unit tests to proper Maven directory structure.
Modified plain argument output to use filtering/masking methods in provided utility.
Refactored utility methods.
Updated unit tests.

This closes #4222.

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 17:29:42 -07:00
Sushil Kumar
996688b419
NIFI-7389 Makes Missable heartbeat counts configurable
This closes #4236.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 13:21:21 -07:00
Sushil Kumar
6c2701abef Do not update status for stopping a deleted node 2020-04-25 11:01:08 -04:00
Rob Fellows
c263daf20b
[NIFI-7358] - Fix: Sorting on 'Estimated Time to Back Pressure' in the Connection summary table does not work properly
- fix style issues
- review feedback

This closes #4208
2020-04-15 10:06:38 -04:00
Andy LoPresto
07a8311b4c
NIFI-7319 Add walkthrough document (#4193)
* NIFI-7319 Added first draft of walkthroughs doc.

* NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance.

* NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate.

* NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates.

* NIFI-7319 Added instructions and screenshots for building NiFi from source.

* NIFI-7319 [WIP] Converting secure cluster instructions to match format.
Fixed instructions regarding embedded ZooKeeper configuration.

* NIFI-7319 Completed secure cluster walkthrough.

* NIFI-7319 Added walkthroughs to documentation navigation list.

* NIFI-7319 Incorporated PR feedback on broken links.

* NIFI-7319 Removed line number helpers from update sections.

* NIFI-7319 Incorporated final PR review items.

Co-authored-by: Sandra Pius <spiusapache@gmail.com>
2020-04-10 22:25:40 -07:00
Mark Payne
728bdec0f4 NIFI-7339: Fixed bug that caused Write Ahead Provenance Repository not to rollover event files after specified time. Code cleanup. Updated some default properties. 2020-04-09 13:32:34 -04:00
Matthew Burgess
84968e70d2
NIFI-7087: Use FlowManager.findAllConnections() when available
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4026
2020-04-09 12:59:01 -04:00
Mark Payne
683b1d9952
NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited.
- Refactored Flow Synchronization to make code cleaner
 - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty.
 - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start
- Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state.
- Added additional log statements to aid in debugging

NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior

NIFI-6849: Addressed review feedback

NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup

This closes #3891
2020-04-09 10:42:05 -04:00
Matthew Burgess
a093af2d42
NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping (#4156)
* NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping

* NIFI-7273: Changed method name, fix handling when analytics not enabled

* NIFI-7273: Removed attachment header from Prometheus metrics endpoint

* NIFI-7273: Removed unused variable
2020-04-03 16:12:53 -04:00
Bence Simon
c0f5fcb484 NIFI-7188 Extending UI search with filters and refactoring existing solution
This closes #4123.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-02 21:49:40 +02:00
Troy Melhase
483f23a8aa
NIFI-7153 Adds ContentLengthFilter to enforce configurable maximum length on incoming HTTP requests.
Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second.
Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml.

This closes #4125.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-25 17:23:22 -07:00
Michael Hogue
aeaf953e06
NIFI-5925: Added controller services to set of components that are searched
NIFI-5925: cleanup, add negative test

NIFI-5925: fixed checkstyle

This closes #4105

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-03-25 08:55:44 -04:00
Bryan Bende
6412097eb2
NIFI-7257 Added HadoopDBCPConnectionPool
- Updated InstanceClassLoader to resolve files that are in the instance urls or additional urls
- Updated nifi-mock to support KerberosContext and removeProperty for ControllerServices
- Added unit test for HadoopDBCPConnectionPool
- Addressing review feedback

This closes #4149.
2020-03-24 11:05:18 -04:00
Bryan Bende
7f32aa56db
NIFI-7271 Make command timeout configurable for ShellUserGroupProvider
- Changing ShellRunner to use a separate thread for reading the output of the process
- Removing unused member variable
- Addressing review feedback

This closes #4154.
2020-03-23 20:43:06 -04:00
Joe Witt
254b20fa5f
NIFI-7274 add time for test conditions to be met 2020-03-22 09:54:43 -04:00
Joe Witt
f694e6464f NIFI-7187 adding missing version strings from accumulo bundle pom
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE

This closes #4086

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-20 10:07:56 -04:00
Andy LoPresto
3feb85a030
NIFI-7223 Resolved merge conflicts from additional test case for null key password.
This closes #4145.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-16 12:47:41 -04:00
Nathan Gough
69b6c231fd
NIFI-7223 - Added another test for when keyPasswd is not present. 2020-03-16 12:28:18 -04:00
Andy LoPresto
f9d75056fa
NIFI-7223 [WIP] Resolved compilation issues in unit test on OpenJDK 11 by removing Sun security class references.
Added OkHttpReplicationClient#isTLSConfigured() method.
Added unit test.
NIFI-7223 Fixed remaining unit tests for TLS regression.
Renamed tests for clarity.
2020-03-16 12:28:01 -04:00
Nathan Gough
7374361b5c
NIFI-7223 - Fixed a minor issue where the OkHttpReplicationClient class loaded blank properties as empty string instead of an expected null value. Added a isNotBlank check. Added unit tests for replication client and HTTPNotificationService.
NIFI-7223 - Renamed some variables and methods.
NIFI-7223 - Removed unused dependency. Corrected security properties in administration-guide.
2020-03-16 12:26:50 -04:00
Joe Witt
97e250cdaa
NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4132.
2020-03-12 19:13:59 +01:00
Mark Payne
d4a2afc25c
NIFI-7241: When updating Process Group to match VersionedProcessGroup, remove any connections before recursing into child groups. This ensures that if a Port exists in child group A and is connected to a port in child group B, if the VersionedProcessGroup indicates to remove the port, that connection will be removed before attempting to remove the port. Updating and adding connections must still be done last, after all components have been added. But missing connections can be removed earlier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4136.
2020-03-12 10:25:04 +01:00
Mark Payne
d68720920f
NIFI-7242: When a Parameter is changed, any property referencing that parameter should have its #onPropertyModified method called. Also renamed Accumulo tests to integration tests because they start embedded servers and connect to them, which caused failures in my environment. Also fixed a bug in TestLengthDelimitedJournal because it was resulting in failures when building locally as well.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4134.
2020-03-11 21:00:43 +01:00
Mark Payne
afad982e91
NIFI-7200: Revert "NIFI-6530 - HTTP SiteToSite server returns 201 in case no data is available"
This reverts commit f01668e66ad2e45197915769e966a4be27e1592e.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-10 14:32:43 -04:00
Pierre Villard
12c8402ac3
NIFI-7210 - added PG path in bulletins for S2S Bulletin RT
Added group path to BULLETIN table for QueryNiFiReportingTask

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4100
2020-03-09 16:29:22 -04:00
Mark Bean
f4b65afb64 NIFI-7231: move controller service validation out of synchronized block for enabling
This closes #4118.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 15:01:19 -05:00
Joe Witt
4cd63c99e8
NIFI-7205 NIFI-7206
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4093.
2020-02-26 14:02:49 -08:00
Joe Witt
bad0f10a52
NIFI-7201 - Update build to latest apache-maven-parent and split Github Actions builds to include OSX and distribute localization
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4091.
2020-02-25 17:38:33 -08:00
Andy LoPresto
815e4cf51f
NIFI-7152 Added custom ExceptionMappers to handle invalid Remote Process Group port value - (#4085)
JsonContentConversionExceptionMapper, JsonMappingExceptionMapper, JsonParseExceptionMapper.
Registered the custom ExceptionMappers.
Added unit tests to throw Exception for string port value and sanitize script input. Handled null or empty JsonMappingException reference path.
Added the Apache license to Groovy Test.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-25 13:14:52 -08:00
Joe Witt
778012412a
Revert "NIFI-6363 Refactors sensitive properties, adds additional providers."
This reverts commit 479fcfdc0bbd97dd0635bbec0273cc25fcf9cc03.

It does not build properly as shown in Github Actions.
2020-02-24 22:07:53 -08:00
Troy Melhase
479fcfdc0b NIFI-6363 Refactors sensitive properties, adds additional providers.
NIFI-6363 Additional fixes.

NIFI-6363 Fix Hadoop compile problem.  Add GCP IT instructions.

NIFI-6363 - Removed GCP provider due to dependency conflicts with GRPC processors. Fixed unit test to match master branch after rebase.

NIFI-6363 - Added some docs and experimental tag to the relevant classes.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4080.
2020-02-24 16:35:19 -05:00
MatthewKnight-NG
acaf321af0
NIFI-5924 Labels should be searchable
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4070
2020-02-24 16:14:21 -05:00
Troy Melhase
0de89452f1
NIFI-6927 Consolidate SSL context and trust managers for OkHttp on JDK9.
Fixes name conflicts.

This closes #4047.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-19 16:19:39 -08:00
M Tien
37614d02cd
NIFI-7136 Added autocomplete="off" to login password input (#4055)
NIFI-7136 Added autocomplete="off" to login password input
Updated nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui/src/main/webapp/WEB-INF/partials/login/login-form.jsp

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-14 20:46:08 -08:00
Joe Ferner
62606ff89a NIFI-6873: Added support for replacing a process group
- decoupled flow update request behavior from VersionsResource into new abstract FlowUpdateResource
 - added replace process group functionality in ProcessGroupResource
 - parameterized FlowUpdateResource and created entity hierarchies to allow for maximum code sharing across different update types
 - refactored flow update methods to make use of commonality across different update types whenever possible
 - fixed issues in StandardProcessGroup verify update methods where same components existed in different ancestry chains but were considered a match when they shouldn't be
 - improved StandardProcessGroup to properly match up components on update using generated versioned component ids, when necessary to allow for update flow to efficiently match common components on flow import

This closes #4023.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-02-11 14:46:40 -05:00
Mark Payne
65b2a9bc2c
NIFI-7117: When SocketLoadBalancedFlowFileQueue creates its array of Queue Partitions in the constructor, it added the local partition as the first element in that list. This list should be ordered the same across all nodes in the cluster. By making the local partition the first in the array, each node had a different ordering of these partitions. As a result, Partition by Attribute strategy would constantly rebalance flowfiles that it received to other node, and Single Node always transferred data to the first partition, which was the local node, instead of whichever node should have been the first in the list. This commit addresses this issue by instead inserting the local partition intot he 'queuePartitions' array based on the local node identifier.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4045.
2020-02-11 10:59:19 -08:00
nagasivanath
425ba63e66
NIFI-6968 Create Connection Model Allows Multiple Adds
This closes #4013.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-06 10:32:59 -05:00
Matt Gilman
b1b06bf5e1 NIFI-6958:
- Addressing issue causing errors preventing version changes when there is a processor to enable or disable in a sub process group.
2020-02-05 13:49:06 -05:00
Bryan Bende
5d851e6a13
NIFI-7067 Allow a user and group with the same name/identity to exist
This closes #4019
2020-02-05 09:19:18 -05:00
Andy LoPresto
d7c29f4637 NIFI-7079 Cleaned up formatting and comments. 2020-01-31 08:58:51 -05:00
Andy LoPresto
95746d346c NIFI-7079 Added SecureHasher interface and Argon2SecureHasher implementation.
Added unit tests.
Fixed JwtService charset references.
Changed FingerprintFactory to mask sensitive values via Argon2 hashing.
Resolved RAT checks.
2020-01-31 08:58:51 -05:00
Mark Payne
37ef10e6fb
NIFI-7076: This closes #4024. Revert "NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize()."
This reverts commit 773160958209a8a173b4f741517c4bda63a28e82.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-29 15:50:55 -05:00
Mark Payne
be34767c8a
NIFI-7059: This closes #4007. Fixed bug that results in priorities not properly being set in the SocketLoadBalancedFlowFileQueue. Even though the queue's setPriorities method was called, the underlying may not have contained the localPartition. As a result, when setPriorities() was called, it did not properly delegate that call to . As a result, the queue knew that the Prioritizers were set but the local queue did not apply them. This happened due to a race condition between queue creation and NodeClusterCoordinator learning the Local Node Identifier. Additionally, updated NodeClusterCoordinator so that it will persist its state when it does learn the Local Node Identifier, as that was not being persisted. In testing this, also encounterd a deadlock in a particular Integration Test that was run, around AbstractFlowFileQueue and its synchronization. Because the 'synchronized' keyword synchronizes on 'this' and the concrete implementation also uses 'synchronized' and the result is the same 'this', a deadlock can occur that is difficult to understand. As a result, refactored AbstractFlowFileQueue to instead use read/write locks.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-22 23:30:41 -05:00
Joe Witt
535cafdb83
Merge branch 'NIFI-7021-RC3' bringing the Apache NiFi 1.11.0 to master 2020-01-22 21:15:12 -05:00
Bryan Bende
24b846d08a
NIFI-7051 Protect against empty group membership in ShellUserGroupProvider, and add differentiator to id seeding
NIFI-7051 Fixing issue where identity was being used instead of identifier, making a flag to control legacy id behavior, increasing timeout of shell command runner, and changing the NSS system check command to return less info

NIFI-7051 Updating command for getSystemCheck in NSS impl to use getent --version to improve performance

This closes #4003.
2020-01-22 15:51:42 -05:00
nagasivanath
24ef8ba4cb
Update nf-processor-details.js
NIFI-7032:
- Processor Details no longer appears when clicking 'View Processor Details'
- handling the review comments

This closes #3990
2020-01-21 13:54:12 -05:00
Joe Witt
3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00