Commit Graph

4735 Commits

Author SHA1 Message Date
Jeff Storck 01e2098d24 NIFI-5585 A node that was previously offloaded can now be reconnected to the cluster and queue flowfiles again
Added Spock test for NonLocalPartitionPartitioner
Updated NOTICE files for FontAwesome with the updated version (4.7.0) and URL to the free license
Updated package-lock.json with the updated version of FontAwesome (4.7.0)
Added method to FlowFileQueue interface to reset an offloaded queue
Queues that are now immediately have the offloaded status reset once offloading finishes
SocketLoadBalancedFlowFileQueue now ignores back-pressure when offloading flowfiles
Cleaned up javascript in nf-cluster-table.js when creating markup for the node operation icons
Fixed incorrect handling of a heartbeat from an offloaded node.  Heartbeats from offloading or offloaded nodes will now be reported as an event, the heartbeat will be removed and ignored.
Added unit tests and integration tests to cover offloading nodes
Updated Cluster integration test class with accessor for the current cluster coordinator
Updated Node integration test class's custom NiFiProperties implementation to return the load balancing port and a method to assert an offloaded node
Added exclusion to top-level pom for ITSpec.class
2018-10-11 09:23:01 -04:00
Mark Payne be2c24cfaf NIFI-5585: Fixed bug that arised when multiple nodes were decommissioning at same time; could get into state where the nodes queued up data for one another so the data just stayed put 2018-10-11 09:23:00 -04:00
Jeff Storck 04d8da8f46 NIFI-5585 Added capability to offload a node that is disconnected from the cluster.
Updated NodeClusterCoordinator to allow idempotent requests to offload a cluster
Added capability to connect/delete/disconnect/offload a node from the cluster to the Toolkit CLI
Added capability to get the status of nodes from the cluster to the Toolkit CLI
Upgraded FontAwesome to 4.7.0 (from 4.6.1)
Added icon "fa-upload" for offloading nodes in the cluster table UI
2018-10-11 09:23:00 -04:00
Kotaro Terada 83ca676492 NIFI-5681: Fix a locale-dependent test in TestVersionedFlowSnapshotMetadataResult
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3061.
2018-10-11 11:47:39 +02:00
Kotaro Terada 6c17685cbe NIFI-5675: Fix some locale-dependent tests in ConvertExcelToCSVProcessorTest
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3058.
2018-10-10 09:44:00 +02:00
Kotaro Terada fc5c8baeb3 NIFI-5676: Fix a timezone-dependent test in PutORCTest
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3059.
2018-10-10 09:38:06 +02:00
Matt Gilman dd50322749 NIFI-5600: Recalculating the available columns for the queue listing and component state because they contain conditions which need to be re-evaluated.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3055.
2018-10-10 09:32:54 +02:00
Mark Payne 9dfc6683ee
NIFI-5672: Do not compare Load Balancing address/port for logical equivalence of Node Identifiers. Added more details to logging of Node Identifiers
This closes #3054
2018-10-09 15:24:43 -04:00
joewitt 77edddd988
NIFI-5666 Updated all usages of Spring, beanutils, collections to move beyond deps with cves
This closes #3052
2018-10-09 14:14:48 -04:00
Mark Payne 117e60c8e3 Empty commit to force Github sync 2018-10-09 12:23:44 -04:00
Mark Payne c425bd2880 NIFI-5533: Be more efficient with heap utilization
- Updated FlowFile Repo / Write Ahead Log so that any update that writes more than 1 MB of data is written to a file inside the FlowFile Repo rather than being buffered in memory
 - Update SplitText so that it does not hold FlowFiles that are not the latest version in heap. Doing them from being garbage collected, so while the Process Session is holding the latest version of the FlowFile, SplitText is holding an older version, and this results in two copies of the same FlowFile object

NIFI-5533: Checkpoint

NIFI-5533: Bug Fixes

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2974
2018-10-09 09:18:02 -04:00
Mark Payne c87d791938 NIFI-5663: Ensure that when sort Node Identifiers that we use both the node's API Address as well as API Port, in case 2 nodes are running on same host. Also ensure that when Local Node ID is determined that we update all Load Balancing Partitions, if necessary
This closes #3048.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-09 21:14:31 +09:00
Pierre Villard 768bcfb509 NIFI-5635 - Description PutEmail properties with multiple senders/recipients
This closes #3031

Signed-off-by: Mike Moser <mosermw@apache.org>
2018-10-07 17:21:01 -04:00
thenatog 246c090526
NIFI-5595 - Added the CORS filter to the templates/upload endpoint using a URL matcher.
Explicitly allow methods GET, HEAD. These are the Spring defaults when the allowedMethods is empty but now it is explicit. This will require other methods like POST etc to be from the same origin (for the template/upload URL).

This closes #3024.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-10-06 16:24:17 -07:00
Matthew Burgess c6572f042b NIFI-4517: Added ExecuteSQLRecord and QueryDatabaseTableRecord processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2945.
2018-10-06 10:53:11 +02:00
Mark Payne b4810b8dd7 Empty commit to force sync with mirrors 2018-10-05 12:08:55 -04:00
Mark Payne 619f1ffe8f NIFI-5516: Implement Load-Balanced Connections
Refactoring StandardFlowFileQueue to have an AbstractFlowFileQueue
Refactored more into AbstractFlowFileQueue
Added documentation, cleaned up code some
Refactored FlowFileQueue so that there is SwappablePriorityQueue
Several unit tests written
Added REST API Endpoint to allow PUT to update connection to use load balancing or not. When enabling load balancing, though, I saw the queue size go from 9 to 18. Then was only able to process 9 FlowFiles.
Bug fixes
Code refactoring
Added integration tests, bug fixes
Refactored clients to use NIO
Bug fixes. Appears to finally be working with NIO Client!!!!!
NIFI-5516: Refactored some code from NioAsyncLoadBalanceClient to LoadBalanceSession
Bug fixes and allowed load balancing socket connections to be reused
Implemented ability to compress Nothing, Attributes, or Content + Attributes when performing load-balancing
Added flag to ConnectionDTO to indicate Load Balance Status
Updated Diagnostics DTO for connections
Store state about cluster topology in NodeClusterCoordinator so that the state is known upon restart
Code cleanup
Fixed checkstyle and unit tests
NIFI-5516: Updating logic for Cluster Node Firewall so that the node's identity comes from its certificate, not from whatever it says it is.
NIFI-5516: FIxed missing License headers
NIFI-5516: Some minor code cleanup
NIFI-5516: Adddressed review feedback; Bug fixes; some code cleanup. Changed dependency on nifi-registry from SNAPSHOT to official 0.3.0 release
NIFI-5516: Take backpressure configuration into account
NIFI-5516: Fixed ConnectionDiagnosticsSnapshot to include node identifier
NIFI-5516: Addressed review feedback

This closes #2947
2018-10-04 16:11:05 -04:00
Mark Payne 5872eb3c4a NIFI-5331: When checkpointing SequentialAccessWriteAheadLog, if the journal is not healthy, ensure that we roll it over and ensure that if an Exception is thrown when attempting to fsync() or close() the journal, we continue creating a new one.
This closes #2952.
Signed-off-by: Brandon Devries <devriesb@apache.org>
2018-10-04 15:25:53 -04:00
Koji Kawamura 8f4d13eeac
NIFI-5581: Fix replicate request timeout
This closes #3044

- Revert 87cf474e54 to enable connection
pooling
- Changes the expected HTTP status code for the 1st request of a
two-phase commit transaction from 150 (NiFi custom) to 202 Accepted
- Corrected RevisionManager Javadoc about revision varidation protocol
2018-10-04 10:25:42 -04:00
Andy LoPresto f65286be83
NIFI-5622 Updated test resource keystores and truststores with SubjectAlternativeNames to be compliant with RFC 6125.
Refactored some test code to be clearer.
Renamed some resources to be consistent across modules.
Changed passwords to meet new minimum length requirements.

This closes #3018
2018-10-04 09:50:09 -04:00
joewitt 8e233ca2ef
NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin
This closes #3028
2018-10-04 09:32:04 -04:00
pepov de685a7a74
NIFI-5656 Handly empty "Node Group" property in FileAccessPolicyProvider consistently, add some logs to help with debugging, add test for the invalid group name and for the empty case.
This closes #3043.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-10-03 11:12:19 -04:00
Brandon Devries b4c8e0179b Merge branch 'pr2931' 2018-10-02 11:08:43 -04:00
Mike Moser 76a9f98d7f NIFI-3531 Catch and rethrow generic Exception to handle RuntimeExceptions, and allow test to pass
This closes #2931.
Signed-off-by: Brandon Devries <devriesb@apache.org>
2018-10-02 11:06:51 -04:00
Brandon Devries 895323f3c2 Merge branch 'pr2949' 2018-10-02 09:40:36 -04:00
Mike Moser 4f538f1ecb NIFI-3672 updated PublishJMS message property docs
This closes #2949

Signed-off-by: Brandon Devries <devriesb@apache.org>
2018-10-02 09:39:17 -04:00
Matthew Burgess 813cc1f6a2 NIFI-5650: Added Xerces to scripting bundle for Jython 2.7.1
This closes #3042

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-10-01 10:57:58 -04:00
Mike Thomsen b1478cdb19 NIFI-5051 Created ElasticSearch lookup service.
NIFI-5051 Fixed checkstyle issue.

NIFI-5051 Converted ES lookup service to use a SchemaRegistry.

NIFI-5051 Cleaned up POM and added a simple unit test that uses a mock client service.

NIFI-5051 Added change; waiting for feedback.

NIFI-5051 Changed query setup based on code review. Changed tests to Groovy to make them easier to read with all of the inline JSON.

NIFI-5051 fixed a checkstyle issue.

NIFI-5051 Rebased to cleanup merge issues

NIFI-5051 Added changes from a code review.

NIFI-5051 Fixed a checkstyle issue.

NIFI-5051 Added coverage generator for tests.

Rebased.

NIFI-5051 Updated service and switched it over to JsonInferenceSchemaRegistryService.

NIFI-5051 Removed dead code.

NIFI-5051 Fixed checkstyle errors.

NIFI-5051 Refactored query builder.

NIFI-5051 Added placeholder gitignore to force test compile.

NIFI-5051 Added note explaining why the .gitignore file was needed.

NIFI-5051 Made constructor public.

NIFI-5051 Fixed path issue in client service integration tests.

NIFI-5051 Added additional mapping capabilities to let users massage the result set into the fields they want.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2615
2018-10-01 10:48:45 -04:00
Andy LoPresto 748cf74562
NIFI-5628 Added content length check to OkHttpReplicationClient.
Added unit tests.

This closes #3035
2018-09-28 14:34:03 -04:00
Colin Dean 0dd382370b NIFI-5612: Support JDBC drivers that return Long for unsigned ints
Refactors tests in order to share code repeated in tests and to enable
some parameterized testing.

MySQL Connector/J 5.1.x in conjunction with MySQL 5.0.x will return
a Long for ResultSet#getObject when the SQL type is an unsigned integer.
This change prevents that error from occurring while implementing a more
informational exception describing what the failing object's POJO type
is in addition to its string value.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3032
2018-09-28 13:46:04 -04:00
Jeff Storck e24388aa7f NIFI-5557 Added test in PutHDFSTest for IOException with a nested GSSException
Resolved most of the code warnings in PutHDFSTest

This closes #2971.
2018-09-27 16:24:31 -04:00
Endre Zoltan Kovacs 0f55cbfb9f NIFI-5557: handling expired ticket by rollback and penalization 2018-09-27 16:24:31 -04:00
Mark Payne 2e1005e884 NIFI-5640: Improved efficiency of Avro Reader and some methods of AvroTypeUtil. Also switched ServiceStateTransition to using read/write locks instead of synchronized blocks because profiling showed that significant time was spent in determining state of a Controller Service when attempting to use it. Switching to a ReadLock should provide better performance there.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3036
2018-09-27 15:38:47 -04:00
Mark Payne ad4c886fbf
NIFI-5634: When merging RPG entities, ensure that we only send back the ports that are common to all nodes - even if that means sending back no ports
This closes #3030
2018-09-27 10:11:15 -04:00
Pierre Villard ca70dbbb36 This closes #1661. 2018-09-26 13:05:22 +02:00
Pierre Villard db023f368b This closes #1142. 2018-09-25 23:47:51 +02:00
Nick Lewis db645ec475 NIFI-3344 Added property to JoltTransformJSON allowing the user to specify pretty print, defaults to false
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2987.
2018-09-25 23:07:24 +02:00
Pierre Villard 5d3558a79d NIFI-5633 - allDelineatedValues can throw ArrayIndexOutOfBoundsException
This closes #3029.
2018-09-25 14:07:00 -04:00
Pierre Villard 78c4e223fc NIFI-5591 - Added avro compression format to ExecuteSQL
This closes #3023

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-25 09:25:13 -04:00
Mark Payne 030129c7ce NIFI-5618: Avoid NPE when viewing Provenance Event details on a disconnected node
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3027.
2018-09-25 12:28:58 +02:00
Mark Payne 4b4c9e14cb NIFI-5630: Ensure that we include counters in Status History when present
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3026.
2018-09-25 12:24:17 +02:00
Pierre Villard c4d3b5e94f NIFI-5588 - Fix max wait time in DBCP Connection Pool
This closes #3022

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-09-25 12:19:54 +05:30
Mark Payne 2a964681ec NIFI-5514: Fixed bugs in MergeRecord around minimum thresholds not being honored and validation not being performed to ensure that minimum threshold is smaller than max threshold (would previously allow min record = 100, max records = 2 as a valid configuration)
NIFI-5514: Do not rely on ProcessSession.getQueueSize() to return a queue size of 0 objects because if the processor is holding onto data, the queue size won't be 0.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2954.
2018-09-24 22:36:02 +02:00
Andrew Lim f5e9ea680a NIFI-5578 Correct errors in EL Guide including Date Manipulation and Escaping EL examples
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3017.
2018-09-21 11:03:36 +02:00
Andrew I. Christianson be2ed060a2
NIFI-5542 Added support for node groups to FileAccessPolicyProvider
This closes #2970.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-09-20 10:27:28 -04:00
Matthew Burgess 2ee0af9663 NIFI-5614: Update commons-dbcp to commons-dbcp2 for DBCPConnectionPool
This closes #3014

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-20 08:17:52 -04:00
Mark Payne 73a7c8c6b4 NIFI-5613: Upgraded version of geoip2 library to version 2.12.0; fixed unit tests
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3013.
2018-09-20 13:55:13 +02:00
Kemix Koo 8cf0662e5b NIFI-5615: set the avro-binary MIME type for converting processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3016.
2018-09-20 12:07:43 +02:00
Pierre Villard 854ee3021c NIFI-5599 - Bump Kafka versions
This closes #3006

Signed-off-by: zenfenan <sivaprasanna246@gmail.com>
2018-09-20 14:33:23 +05:30
Mark Payne 3dd548e807 NIFI-5609: Fixed NullPointer when attempting to view status history for a component that has not yet run
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3012
2018-09-19 16:56:42 -04:00