Commit Graph

1364 Commits

Author SHA1 Message Date
Mark Payne fea17d0ca8
NIFI-5919: Addressed a race condition that can exist if adding FlowFiles to a FlowFileQueue before adjusting the size of the queue to account for the FlowFiles
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3238.
2019-01-03 11:35:23 +01:00
Koji Kawamura 4b70465ea5 NIFI-5905 Secure S2S fails with NullPointerException
The constructor arguments should be set to instance variables so that
subsequent code can refer "this.*".
2018-12-18 15:03:31 +09:00
Jeff Storck cc47a8c0e1
NIFI-5748 Improved Proxy Header Support
- Fixed proxy header support to use X-Forwarded-Host instead of X-ForwardedServer
- Added support for the context path header used by Traefik when proxying a service (X-Forwarded-Prefix)
- Added tests to ApplicationResourceTest for X-Forwarded-Context and X-Forwarded-Prefix
- Updated administration doc to include X-Forwarded-Prefix
- Added NIFI_WEB_PROXY_CONTEXT_PATH env var to dockerhub and dockermaven start.sh scripts
- Added documentation for NIFI_WEB_PROXY_CONTEXT_PATH to dockerhub README.md
- Updated ApplicationResource to handle a port specified in X-ProxyPort and X-Forwarded-Port headers

This closes #3129.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-12-17 17:32:06 -05:00
Mark Payne 1ac5b93144
NIFI-5859: Added XML-based documentation writer that can be used to document a component.
- Found several instances of nifi-framework-api's ProviderException being thrown from processors. Changed those to IllegalStateException, as ProviderException is not an appropriate Exception in those cases, and extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller Service API's/specs into the implementations. Adding to the Service API results in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad practice. The 'main' service api nar should not have a dependency on a util class.

NIFI-5859: Added javadocs. Fixed pom.xml that was left pointing to snapshot version of nar maven plugin

NIFI-5859: Addressing review feedback: adding component type, multiple additional details into separate file(s)

This closes #3192.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-12-10 11:16:01 -05:00
Kemix Koo e76c54234d NIFI-5881: Enable to export the template for non-ascii name
This closes #3210.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-12-10 09:51:46 +09:00
Koji Kawamura 8ebb4d1974
NIFI-5863 Fixed AbstractFlowFileQueue logging
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3199.
2018-12-05 11:29:50 +01:00
Kotaro Terada 72662f0b2c
NIFI-5846: Redirect URL is incorrect after logout
This closes #3185.

Signed-off-by: Peter Wicks <patricker@gmail.com>
2018-11-29 08:16:39 -07:00
Andy LoPresto f6b171d5f7
NIFI-5833 This closes #3180. Marked GetTwitter Consumer Key and Access Token processor properties as sensitive.
NIFI-5833 Added unit test to demonstrate arbitrary decryption of sensitive values regardless of processor property sensitive status.
NIFI-5833 Updated GetTwitter documentation with note about 1.9.0+ marking Consumer Key and Access Token as sensitive.

Signed-off-by: joewitt <joewitt@apache.org>
2018-11-21 10:02:25 -05:00
Mark Payne be0949570a
NIFI-5824: Added unit test to FlowController to ensure that the ProcessScheduler that it creates is properly initialized. Also updated the properties file used by TestFlowController to use a VolatileContentRepository instead of FileSystemRepository, and fixed EventDrivenWorkerQueue to return if calls to poll() are interrupted (via Thread.interrupt) - making these minor fixes resulted in the unit test TestFlowController running in 2 seconds instead of 30+ seconds on my machine
This closes #3173.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-11-15 15:01:56 -05:00
Mark Payne 76b0065a67
NIFI-5822: Ensure that we don't call FlowController.getControllerServiceProvider() before the ControllerServiceProvider has been initialized
This closes #3171.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-11-15 13:25:33 -05:00
Kotaro Terada 13232c7413 NIFI-5752: Load balancing fails with wildcard certs
NIFI-5752: Remove an unnecessary String.format

NIFI-5752: Remove an unnecessary block

This closes #3110.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-07 17:16:20 +09:00
Mark Payne da1f9eaf6a
NIFI-5796 Addressed bug in subtract() method for keeping running total of counters for status history
This closes #3136.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-11-06 17:55:21 -05:00
Mark Payne 4069d75550
NIFI-5796: Addressed issue that caused Counters' values to show the wrong value in Status History
Signed-off-by: Bryan Bende <bbende@apache.org>
2018-11-06 17:55:09 -05:00
Mark Payne 931bb0bc3b
NIFI-5769: Refactored FlowController to use Composition over Inheritance
- Ensure that when root group is set, that we register its ID in FlowManager

This closes #3132.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-11-06 11:23:33 -05:00
Andrew Lim 59e102ad47 NIFI-5767 Added NiFi Toolkit Guide to docs
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3124.
2018-11-06 11:23:54 +01:00
Bryan Bende fdd8cdbb31 NIFI-5673 Set up property/assembly for new auto-load directory
- Set up NarAutoLoader to watch directory for new files
- Move NarAutoLoader to JettyServer since it will need access to ExtensionManager
- Created NarLoader to shared between NarAutoLoader and the framework
- Created nifi-framework-nar-loading-utils so we can use nifi-documentation to call DocGenerator
- Add additional bundles to overall map in NarClassLoaders as they are loaded
- Added handling of skipped NARs to include them in next iteration
- Added check of last modified timestamp on NARs
- Refactored JettyServer so we can load additional web contexts while the application is running
- Setting up unit tests
- Remove static use of ExtensionManager
- Adding unit tests for NarLoader
- Extracting interface for ExtensionManager and splitting discovery into it's own interface

This closes #3119.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-11-01 14:41:10 -04:00
Mark Payne 4c10b47e60
NIFI-5771: Ensure that we only increment claimant count for content claim if we have a FlowFile that references it
This closes #3118.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-10-31 15:25:24 -04:00
Jeff Storck 4f14e517db Merge branch 'NIFI-5720-RC3' as part of the NiFi 1.8.0 release process 2018-10-26 13:56:23 -04:00
Mark Payne 234ddb0fe1 NIFI-5745: When determining if backpressure should be applied across nodes for load balancing, only consider if the local partition has reached the threshold limits instead of considering the size of the entire queue
This closes #3108.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-26 16:04:46 +09:00
Mark Payne c7ff2fc5db NIFI-5746: Use Node Identifier's node address instead of getting from socket for RECEIVE prov events; make SEND prov events match syntax of RECEIVE prov events
NIFI-5746: Addressed issue found in review process

This closes #3109.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-26 10:24:12 +09:00
Jeff Storck c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00
Jeff Storck 98aabf2c50 NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 2018-10-22 22:16:23 -04:00
Matt Gilman 02261311b3 NIFI-5737:
- Removing needClientAuth property since cluster comms now requires two way ssl. Jetty client auth settings are based on configured features.
- Removing dead code.
- Updating documentation.
- Removing references to needClientAuth property in all test resources.
- Removing overloaded util method with strict parameter.

This closes #3102.
2018-10-22 20:02:47 -04:00
Mark Payne 5561c29ed3 NIFI-5736: Removed 'synchronized' keyword from SwappablePriorityQueue#getProperties - the method should not be synchronized, as it guards access to member variables via a ReadLock. I am guessing that the synchronized keyword is the result of refactoring from 'synchronized' to a read lock and inadvertently leaving the 'synchronized' keyword in.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3101
2018-10-22 12:22:05 -04:00
Matt Gilman 87c6b3aa7c
NIFI-5715: Updating the allowable values for the runStatus.
This closes #3087.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-10-17 10:33:53 -04:00
Bryan Bende 51ed618cf0 NIFI-5708 Fixing the creation of ValidationContextFactory for controller services so it uses the ComponentVariableRegistry and not the file-based instance
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3081
2018-10-16 15:38:32 -04:00
Mark Payne 4d21f9b34e
NIFI-5709, NIFI-5710: Addressed issue that causes NiFi to not be able to read provenance events when the a new FlowFile is created and then auto-terminated in the same session; minor bug fixes outlined in NIFI-5710
This closes #3083
2018-10-16 15:25:29 -04:00
Matt Gilman 11fd67cd1e
NIFI-375:
- Ensuring the run status endpoints are properly merged.

This closes #3082.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-10-16 14:59:27 -04:00
Mark Payne 32db43b306 NIFI-5686: Updated StandardProcessScheduler so that if it fails to schedule a Reporting Task, it re-schedules the @OnScheduled task instead of looping and calling Thread.sleep. As it was, the single-threaded Process Scheduler was, when calling ProcessScheduler.unschedule(), the unschedule task was not executing because the schedule task was using the only thread. But switching the logic to schedule the task for later and return, instead of calling Thread.sleep and looping, we are able to avoid blocking the one thread in the thread pool. Also, performed some trivial code cleanup and updated erroneous links in Java-docs.
NIFI-5686: Fixed unit test in TestSocketLoadBalancedFlowFileQueue; renamed TestProcessorLifecycle to ProcessorLifecycleIT as it is testing integration between many components and largely focuses on high numbers of concurrent tasks to see if it can trigger any threading bugs that may get introduced

NIFI-5686: Extended unit test timeouts
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3062
2018-10-15 15:21:05 -04:00
Matt Gilman f55204cb69
NIFI-4806:
- Bumping to tika 1.19.1.

This closes #3069
2018-10-15 13:52:49 -04:00
Jeff Storck 7cb39d636c NIFI-5696 Update references to default value for nifi.cluster.node.load.load.balance.port
This closes #3071.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-15 10:43:31 +09:00
Mark Payne 270ce8570d
NIFI-5695: Fixed bug that caused ports to not properly map to their correct child group on Flow Import if the child group is independently versioned
This closes #3070.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-10-12 16:26:59 -04:00
thenatog 5eb5e96b16
NIFI-5665 - Changed netty versions to more closely match the original netty dependency version.
NIFI-5665 - Fixed version for nifi-spark-bundle.
NIFI-5665 - Fixing copy and paste error.

This closes #3067
2018-10-12 15:56:52 -04:00
Bryan Bende 02e0a16a68 NIFI-5680 Handling trailing slashes on URLs of registry clients
This closes #3065.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-10-12 14:18:42 -04:00
Mark Payne 8398ea77bc
NIFI-5688: Ensure that when we map our flow to a VersionedProcessGroup that we include the connections' Load Balance Compression flag
This closes #3064
2018-10-11 15:06:06 -04:00
Matt Gilman 8da403ce9e
NIFI-5661:
- Allowing load balance settings to be applied during creation.
- Clearing the load balance settings when the dialog is closed.
2018-10-11 13:21:20 -04:00
Matt Gilman 79c03caf4f
NIFI-5661:
- Allowing the load balance configuration to be shown/edited in both clustered and standalone mode.
2018-10-11 12:23:53 -04:00
thenatog 64de5c70e1
NIFI-5479 - Supressed the AnnotationParser logs using the logback.xml. Dependency changes can be look at in future.
NIFI-5479 - Updated comment.

This closes #3034
2018-10-11 12:11:44 -04:00
Koji Kawamura 8a751e8018
NIFI-5661: Adding Load Balance config UI
Incorporated review comments.
Move combo options to a common place.

This closes #3046
2018-10-11 11:36:28 -04:00
Mark Payne 97afa4e7ba NIFI-5585: Addressed bug in calculating swap size of a queue partition when rebalancing
This closes #3010.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-10-11 09:52:16 -04:00
Mark Payne a1a4c99763 NIFI-5585: Adjustments to the Connection Load Balancing to ensure that node offloading works smoothly
Signed-off-by: Jeff Storck <jtswork@gmail.com>
2018-10-11 09:23:01 -04:00
Jeff Storck 01e2098d24 NIFI-5585 A node that was previously offloaded can now be reconnected to the cluster and queue flowfiles again
Added Spock test for NonLocalPartitionPartitioner
Updated NOTICE files for FontAwesome with the updated version (4.7.0) and URL to the free license
Updated package-lock.json with the updated version of FontAwesome (4.7.0)
Added method to FlowFileQueue interface to reset an offloaded queue
Queues that are now immediately have the offloaded status reset once offloading finishes
SocketLoadBalancedFlowFileQueue now ignores back-pressure when offloading flowfiles
Cleaned up javascript in nf-cluster-table.js when creating markup for the node operation icons
Fixed incorrect handling of a heartbeat from an offloaded node.  Heartbeats from offloading or offloaded nodes will now be reported as an event, the heartbeat will be removed and ignored.
Added unit tests and integration tests to cover offloading nodes
Updated Cluster integration test class with accessor for the current cluster coordinator
Updated Node integration test class's custom NiFiProperties implementation to return the load balancing port and a method to assert an offloaded node
Added exclusion to top-level pom for ITSpec.class
2018-10-11 09:23:01 -04:00
Mark Payne be2c24cfaf NIFI-5585: Fixed bug that arised when multiple nodes were decommissioning at same time; could get into state where the nodes queued up data for one another so the data just stayed put 2018-10-11 09:23:00 -04:00
Jeff Storck 04d8da8f46 NIFI-5585 Added capability to offload a node that is disconnected from the cluster.
Updated NodeClusterCoordinator to allow idempotent requests to offload a cluster
Added capability to connect/delete/disconnect/offload a node from the cluster to the Toolkit CLI
Added capability to get the status of nodes from the cluster to the Toolkit CLI
Upgraded FontAwesome to 4.7.0 (from 4.6.1)
Added icon "fa-upload" for offloading nodes in the cluster table UI
2018-10-11 09:23:00 -04:00
Matt Gilman dd50322749 NIFI-5600: Recalculating the available columns for the queue listing and component state because they contain conditions which need to be re-evaluated.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3055.
2018-10-10 09:32:54 +02:00
Mark Payne 9dfc6683ee
NIFI-5672: Do not compare Load Balancing address/port for logical equivalence of Node Identifiers. Added more details to logging of Node Identifiers
This closes #3054
2018-10-09 15:24:43 -04:00
joewitt 77edddd988
NIFI-5666 Updated all usages of Spring, beanutils, collections to move beyond deps with cves
This closes #3052
2018-10-09 14:14:48 -04:00
Mark Payne c425bd2880 NIFI-5533: Be more efficient with heap utilization
- Updated FlowFile Repo / Write Ahead Log so that any update that writes more than 1 MB of data is written to a file inside the FlowFile Repo rather than being buffered in memory
 - Update SplitText so that it does not hold FlowFiles that are not the latest version in heap. Doing them from being garbage collected, so while the Process Session is holding the latest version of the FlowFile, SplitText is holding an older version, and this results in two copies of the same FlowFile object

NIFI-5533: Checkpoint

NIFI-5533: Bug Fixes

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2974
2018-10-09 09:18:02 -04:00
Mark Payne c87d791938 NIFI-5663: Ensure that when sort Node Identifiers that we use both the node's API Address as well as API Port, in case 2 nodes are running on same host. Also ensure that when Local Node ID is determined that we update all Load Balancing Partitions, if necessary
This closes #3048.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-09 21:14:31 +09:00
thenatog 246c090526
NIFI-5595 - Added the CORS filter to the templates/upload endpoint using a URL matcher.
Explicitly allow methods GET, HEAD. These are the Spring defaults when the allowedMethods is empty but now it is explicit. This will require other methods like POST etc to be from the same origin (for the template/upload URL).

This closes #3024.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-10-06 16:24:17 -07:00