1527 Commits

Author SHA1 Message Date
Mark Payne
5561c29ed3 NIFI-5736: Removed 'synchronized' keyword from SwappablePriorityQueue#getProperties - the method should not be synchronized, as it guards access to member variables via a ReadLock. I am guessing that the synchronized keyword is the result of refactoring from 'synchronized' to a read lock and inadvertently leaving the 'synchronized' keyword in.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3101
2018-10-22 12:22:05 -04:00
Matt Gilman
87c6b3aa7c
NIFI-5715: Updating the allowable values for the runStatus.
This closes #3087.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-10-17 10:33:53 -04:00
Bryan Bende
51ed618cf0 NIFI-5708 Fixing the creation of ValidationContextFactory for controller services so it uses the ComponentVariableRegistry and not the file-based instance
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3081
2018-10-16 15:38:32 -04:00
Mark Payne
4d21f9b34e
NIFI-5709, NIFI-5710: Addressed issue that causes NiFi to not be able to read provenance events when the a new FlowFile is created and then auto-terminated in the same session; minor bug fixes outlined in NIFI-5710
This closes #3083
2018-10-16 15:25:29 -04:00
Matt Gilman
11fd67cd1e
NIFI-375:
- Ensuring the run status endpoints are properly merged.

This closes #3082.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-10-16 14:59:27 -04:00
Mark Payne
32db43b306 NIFI-5686: Updated StandardProcessScheduler so that if it fails to schedule a Reporting Task, it re-schedules the @OnScheduled task instead of looping and calling Thread.sleep. As it was, the single-threaded Process Scheduler was, when calling ProcessScheduler.unschedule(), the unschedule task was not executing because the schedule task was using the only thread. But switching the logic to schedule the task for later and return, instead of calling Thread.sleep and looping, we are able to avoid blocking the one thread in the thread pool. Also, performed some trivial code cleanup and updated erroneous links in Java-docs.
NIFI-5686: Fixed unit test in TestSocketLoadBalancedFlowFileQueue; renamed TestProcessorLifecycle to ProcessorLifecycleIT as it is testing integration between many components and largely focuses on high numbers of concurrent tasks to see if it can trigger any threading bugs that may get introduced

NIFI-5686: Extended unit test timeouts
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3062
2018-10-15 15:21:05 -04:00
Jeff Storck
7cb39d636c NIFI-5696 Update references to default value for nifi.cluster.node.load.load.balance.port
This closes #3071.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-15 10:43:31 +09:00
Mark Payne
270ce8570d
NIFI-5695: Fixed bug that caused ports to not properly map to their correct child group on Flow Import if the child group is independently versioned
This closes #3070.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-10-12 16:26:59 -04:00
Bryan Bende
02e0a16a68 NIFI-5680 Handling trailing slashes on URLs of registry clients
This closes #3065.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-10-12 14:18:42 -04:00
Mark Payne
8398ea77bc
NIFI-5688: Ensure that when we map our flow to a VersionedProcessGroup that we include the connections' Load Balance Compression flag
This closes #3064
2018-10-11 15:06:06 -04:00
Matt Gilman
8da403ce9e
NIFI-5661:
- Allowing load balance settings to be applied during creation.
- Clearing the load balance settings when the dialog is closed.
2018-10-11 13:21:20 -04:00
Matt Gilman
79c03caf4f
NIFI-5661:
- Allowing the load balance configuration to be shown/edited in both clustered and standalone mode.
2018-10-11 12:23:53 -04:00
thenatog
64de5c70e1
NIFI-5479 - Supressed the AnnotationParser logs using the logback.xml. Dependency changes can be look at in future.
NIFI-5479 - Updated comment.

This closes #3034
2018-10-11 12:11:44 -04:00
Koji Kawamura
8a751e8018
NIFI-5661: Adding Load Balance config UI
Incorporated review comments.
Move combo options to a common place.

This closes #3046
2018-10-11 11:36:28 -04:00
Mark Payne
97afa4e7ba NIFI-5585: Addressed bug in calculating swap size of a queue partition when rebalancing
This closes #3010.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-10-11 09:52:16 -04:00
Mark Payne
a1a4c99763 NIFI-5585: Adjustments to the Connection Load Balancing to ensure that node offloading works smoothly
Signed-off-by: Jeff Storck <jtswork@gmail.com>
2018-10-11 09:23:01 -04:00
Jeff Storck
01e2098d24 NIFI-5585 A node that was previously offloaded can now be reconnected to the cluster and queue flowfiles again
Added Spock test for NonLocalPartitionPartitioner
Updated NOTICE files for FontAwesome with the updated version (4.7.0) and URL to the free license
Updated package-lock.json with the updated version of FontAwesome (4.7.0)
Added method to FlowFileQueue interface to reset an offloaded queue
Queues that are now immediately have the offloaded status reset once offloading finishes
SocketLoadBalancedFlowFileQueue now ignores back-pressure when offloading flowfiles
Cleaned up javascript in nf-cluster-table.js when creating markup for the node operation icons
Fixed incorrect handling of a heartbeat from an offloaded node.  Heartbeats from offloading or offloaded nodes will now be reported as an event, the heartbeat will be removed and ignored.
Added unit tests and integration tests to cover offloading nodes
Updated Cluster integration test class with accessor for the current cluster coordinator
Updated Node integration test class's custom NiFiProperties implementation to return the load balancing port and a method to assert an offloaded node
Added exclusion to top-level pom for ITSpec.class
2018-10-11 09:23:01 -04:00
Mark Payne
be2c24cfaf NIFI-5585: Fixed bug that arised when multiple nodes were decommissioning at same time; could get into state where the nodes queued up data for one another so the data just stayed put 2018-10-11 09:23:00 -04:00
Jeff Storck
04d8da8f46 NIFI-5585 Added capability to offload a node that is disconnected from the cluster.
Updated NodeClusterCoordinator to allow idempotent requests to offload a cluster
Added capability to connect/delete/disconnect/offload a node from the cluster to the Toolkit CLI
Added capability to get the status of nodes from the cluster to the Toolkit CLI
Upgraded FontAwesome to 4.7.0 (from 4.6.1)
Added icon "fa-upload" for offloading nodes in the cluster table UI
2018-10-11 09:23:00 -04:00
Matt Gilman
dd50322749 NIFI-5600: Recalculating the available columns for the queue listing and component state because they contain conditions which need to be re-evaluated.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3055.
2018-10-10 09:32:54 +02:00
Mark Payne
9dfc6683ee
NIFI-5672: Do not compare Load Balancing address/port for logical equivalence of Node Identifiers. Added more details to logging of Node Identifiers
This closes #3054
2018-10-09 15:24:43 -04:00
joewitt
77edddd988
NIFI-5666 Updated all usages of Spring, beanutils, collections to move beyond deps with cves
This closes #3052
2018-10-09 14:14:48 -04:00
Mark Payne
c425bd2880 NIFI-5533: Be more efficient with heap utilization
- Updated FlowFile Repo / Write Ahead Log so that any update that writes more than 1 MB of data is written to a file inside the FlowFile Repo rather than being buffered in memory
 - Update SplitText so that it does not hold FlowFiles that are not the latest version in heap. Doing them from being garbage collected, so while the Process Session is holding the latest version of the FlowFile, SplitText is holding an older version, and this results in two copies of the same FlowFile object

NIFI-5533: Checkpoint

NIFI-5533: Bug Fixes

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2974
2018-10-09 09:18:02 -04:00
Mark Payne
c87d791938 NIFI-5663: Ensure that when sort Node Identifiers that we use both the node's API Address as well as API Port, in case 2 nodes are running on same host. Also ensure that when Local Node ID is determined that we update all Load Balancing Partitions, if necessary
This closes #3048.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-09 21:14:31 +09:00
thenatog
246c090526
NIFI-5595 - Added the CORS filter to the templates/upload endpoint using a URL matcher.
Explicitly allow methods GET, HEAD. These are the Spring defaults when the allowedMethods is empty but now it is explicit. This will require other methods like POST etc to be from the same origin (for the template/upload URL).

This closes #3024.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-10-06 16:24:17 -07:00
Mark Payne
619f1ffe8f NIFI-5516: Implement Load-Balanced Connections
Refactoring StandardFlowFileQueue to have an AbstractFlowFileQueue
Refactored more into AbstractFlowFileQueue
Added documentation, cleaned up code some
Refactored FlowFileQueue so that there is SwappablePriorityQueue
Several unit tests written
Added REST API Endpoint to allow PUT to update connection to use load balancing or not. When enabling load balancing, though, I saw the queue size go from 9 to 18. Then was only able to process 9 FlowFiles.
Bug fixes
Code refactoring
Added integration tests, bug fixes
Refactored clients to use NIO
Bug fixes. Appears to finally be working with NIO Client!!!!!
NIFI-5516: Refactored some code from NioAsyncLoadBalanceClient to LoadBalanceSession
Bug fixes and allowed load balancing socket connections to be reused
Implemented ability to compress Nothing, Attributes, or Content + Attributes when performing load-balancing
Added flag to ConnectionDTO to indicate Load Balance Status
Updated Diagnostics DTO for connections
Store state about cluster topology in NodeClusterCoordinator so that the state is known upon restart
Code cleanup
Fixed checkstyle and unit tests
NIFI-5516: Updating logic for Cluster Node Firewall so that the node's identity comes from its certificate, not from whatever it says it is.
NIFI-5516: FIxed missing License headers
NIFI-5516: Some minor code cleanup
NIFI-5516: Adddressed review feedback; Bug fixes; some code cleanup. Changed dependency on nifi-registry from SNAPSHOT to official 0.3.0 release
NIFI-5516: Take backpressure configuration into account
NIFI-5516: Fixed ConnectionDiagnosticsSnapshot to include node identifier
NIFI-5516: Addressed review feedback

This closes #2947
2018-10-04 16:11:05 -04:00
Koji Kawamura
8f4d13eeac
NIFI-5581: Fix replicate request timeout
This closes #3044

- Revert 87cf474e542ef16601a86cc66c624fb8902c9fc2 to enable connection
pooling
- Changes the expected HTTP status code for the 1st request of a
two-phase commit transaction from 150 (NiFi custom) to 202 Accepted
- Corrected RevisionManager Javadoc about revision varidation protocol
2018-10-04 10:25:42 -04:00
Andy LoPresto
f65286be83
NIFI-5622 Updated test resource keystores and truststores with SubjectAlternativeNames to be compliant with RFC 6125.
Refactored some test code to be clearer.
Renamed some resources to be consistent across modules.
Changed passwords to meet new minimum length requirements.

This closes #3018
2018-10-04 09:50:09 -04:00
joewitt
8e233ca2ef
NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin
This closes #3028
2018-10-04 09:32:04 -04:00
pepov
de685a7a74
NIFI-5656 Handly empty "Node Group" property in FileAccessPolicyProvider consistently, add some logs to help with debugging, add test for the invalid group name and for the empty case.
This closes #3043.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-10-03 11:12:19 -04:00
Andy LoPresto
748cf74562
NIFI-5628 Added content length check to OkHttpReplicationClient.
Added unit tests.

This closes #3035
2018-09-28 14:34:03 -04:00
Mark Payne
2e1005e884 NIFI-5640: Improved efficiency of Avro Reader and some methods of AvroTypeUtil. Also switched ServiceStateTransition to using read/write locks instead of synchronized blocks because profiling showed that significant time was spent in determining state of a Controller Service when attempting to use it. Switching to a ReadLock should provide better performance there.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3036
2018-09-27 15:38:47 -04:00
Mark Payne
ad4c886fbf
NIFI-5634: When merging RPG entities, ensure that we only send back the ports that are common to all nodes - even if that means sending back no ports
This closes #3030
2018-09-27 10:11:15 -04:00
Mark Payne
030129c7ce NIFI-5618: Avoid NPE when viewing Provenance Event details on a disconnected node
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3027.
2018-09-25 12:28:58 +02:00
Mark Payne
4b4c9e14cb NIFI-5630: Ensure that we include counters in Status History when present
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3026.
2018-09-25 12:24:17 +02:00
Andrew I. Christianson
be2ed060a2
NIFI-5542 Added support for node groups to FileAccessPolicyProvider
This closes #2970.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-09-20 10:27:28 -04:00
Mark Payne
3dd548e807 NIFI-5609: Fixed NullPointer when attempting to view status history for a component that has not yet run
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3012
2018-09-19 16:56:42 -04:00
Koji Kawamura
f570cb980d
NIFI-375: Added operation policy
The operation policy allows that a user to operate components even if they does not have direct READ/WRITE
permission of the component.

Following operations are controlled by the new operate policy:
- Start/stop/enable/disable Processors, ControllerServices,
ReportingTasks, Input/OuputPorts
- Enable/disable transmission of RemoteInput/OutputPorts and
RemoteProcessGroups
- Terminate Processor threads

Refactored what API exposes

The previous commit let API exposes few fields in DTO. But we should
avoid returning partial DTO as it complicates authorization logic.

Instead, this commit adds StatusDTO for ReportingTaskEntity and
ControllerServiceEntity, so that it can be returned regardless of having
READ permission. Component DTO can only be returned with a READ
permission.

Refactor RPG same as ControllerService.

WIP incorporating review comments.

Incorporated review comments

- Cleaned up merger classes
- Recreate DTO instance at each function during two phase commmit

Restrict enabling ControllerService without read permission

Revert the last commit.

Fix review comments.

- Renamed confusing static method names and its parameters
- Removed unnecessary permission checks from UI condition

Fixed delete action display condition.

Fixed NPE at Summary.

Apply operation policy to activateControllerServices.

Removed OperationPermissible from ComponentEntity.

This closes #2990
2018-09-19 15:28:41 -04:00
Bryan Bende
14729be837 NIFI-5577 Fixing update method in NaiveRevisionManager so that failed updates don't change the revision
This closes #2995.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-09-14 11:33:53 -04:00
Mark Payne
87cf474e54
NIFI-5581: Disable connection pooling for OkHttpReplicationClient. We should revisit this in the future, but for now, it appears that Jetty is having problems with the connections if they are reused. By disabling the Connection Pooling, we address the concern, but for secure connections this means that every request results in a TLS handshake - and for a mutable request, both the verification and the 'performance' stages require the TLS handshake. But it's better than timing out, which is the currently observed behavior
This closes #2996
2018-09-10 09:43:53 -04:00
thenatog
fc1461298a
NIFI-5366 - Added ContentSecurityPolicyFilter which stops framing of NiFi resources. It applies the Content-Security-Policy header. This protects against clickjacking.
NIFI-5366 - Added unit test. Added single quotes around 'self' for frame-ancestors CSP header.
NIFI-5366 - Fixed dependencies.

This closes #2989.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-09-05 17:38:53 -07:00
Andy LoPresto
e83ea1f9ae
NIFI-5558 Fixed unit test to avoid contamination from System property "nifi.properties.file.path" when creating test NiFiProperties instance.
This closes #2972.

Co-authored-by: thenatog <thenatog@gmail.com>
Signed-off-by: Marc Parisi <phrocker@apache.org>
2018-08-29 14:52:07 -07:00
Matt Gilman
f04cd8681d NIFI-5479: Using the SUN provider when the keystore type is JKS.
This closes #2961.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-28 15:28:30 +09:00
joewitt
a27ccd8a56 NIFI-5479 Upgraded Jetty. Moved where we unpack bundled deps to so we can avoid a new jetty bug with META-INF loading logic.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-28 15:28:11 +09:00
Mark Bean
58cb900b90
NIFI-5543: move bendPointDrag below mousedown.selection in nf-connection.js
This closes #2957
2018-08-27 11:04:07 -04:00
Mark Payne
aac2c6a60e
NIFI-5482: Made WriteAheadProvenanceRepository the default implementation
This closes #2960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-08-23 16:49:55 -07:00
Andy LoPresto
744b15b4a7
NIFI-5540 Added unit test to demonstrate missing default sensitive properties key in flow election encryptor creation.
NIFI-5540 Added failing unit test and ignored regression test to demonstrate missing default sensitive properties key in flow election encryptor creation.

NIFI-5540 Added equality logic to StringEncryptor and utility equality methods to CryptoUtils.

NIFI-5540 Added default sensitive properties key population logic and log warning to StringEncryptor.

NIFI-5540 Cleaned up formatting.

NIFI-5540 Cleaned up boolean logic.

NIFI-5540 Added Javadoc to StringEncryptor.

NIFI-5540 Added unit test for StringEncryptor#equals().

NIFI-5540 Added performance benchmarking unit tests for constantTimeEquals methods for String, byte[], and char[].

NIFI-5540 Fixed checkstyle issue.

NIFI-5540 Fixed unit tests for default key population.

This closes #2959.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-08-22 13:56:16 -04:00
Mark Payne
5b57935bae NIFI-5527: Refactored unit test to make member variables for 'delay tolerances' so that they are easier to update in the future. Increased delay tolerances so that tests pass on heavily loaded system.
This closes #2955.
2018-08-22 10:51:47 -04:00
Mark Payne
7bbb5a823a NIFI-5466: Keep a running total of stats for each component. Refactored FlowFileEvent and repository in order to provide more efficient storage of objects on Java heap by allowing the same 'EMPTY' object to be reused
- Refactored VolatileComponentStatusRepository to avoid holding on to ProcessorStatus objects, etc, and only keep what they need
 - Updated VolatileComponentStatusRepository to ensure that we are efficiently storing metrics for processors, etc. that are not running

This closes #2939

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-08-21 11:44:25 -04:00
patricker
5106dc0af9
NIFI-4535 Only update Page Title to root flow name when user has permission.
This closes #2899
2018-08-13 12:46:16 -04:00