Commit Graph

5566 Commits

Author SHA1 Message Date
Bryan Bende 9c9f9c10a9 NIFI-5910 Retain external service reference when already set to an existing external service
This closes #3783.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-10-03 12:32:46 -04:00
James Cheng f1d35f46ce NIFI-6684 Add more property to Hive3ConnectionPool (#3763)
* NiFi-6684 Add more property to Hive3ConnectionPool

signed-off by: Peter Wicks <patricker@gmail.com>
2019-10-03 07:24:29 -06:00
Pierre Villard b12a9ad446
NIFI-6159 - Add BigQuery processor using the Streaming API
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3394.
2019-10-03 11:37:00 +02:00
Kotaro Terada a0a66839c4
NIFI-6715: Metrics of removed/renamed components continues to remain in PrometheusReportingTask
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3766.
2019-10-03 11:33:49 +02:00
Kotaro Terada 261a395992
NIFI-6716: Prometheus Metrics Server returns 500 when a Remote Process Group used
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3767.
2019-10-03 11:07:58 +02:00
Mark Payne 99cf87c330
NIFI-6736: Create thread on demand to handle incoming request from client for load balancing. This allows us to avoid situations where we don't have enough threads and we block on the server side, waiting for data, when clients are trying to send data in another connection
This closes #3784.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-02 14:29:00 -04:00
Mark Payne 2f6f852915 NIFI-6589: Addressed NPE 2019-10-02 10:55:55 -04:00
Jeff Storck 8d748223ff
NIFI-6275 ListHDFS now ignores scheme and authority when uses "Full Path" filter mode
Updated description for "Full Path" filter mode to state that it will ignore scheme and authority
Added tests to TestListHDFS for listing an empty and nonexistent dirs
Updated TestListHDFS' mock file system to track state properly when FileStatus instances are added, and updated listStatus to work properly with the underlying Map that contains FileStatus instances
Updated ListHDFS' additional details to document "Full Path" filter mode ignoring scheme and authority, with an example
Updated TestRunners, StandardProcessorTestRunner, MockProcessorInitializationContext to support passing in a logger.

NIFI-6275 Updated the "Full Path" filter mode to check the full path of a file with and without its scheme and authority against the filter regex
Added additional documentation for how ListHDFS handles scheme and authority when "Full Path" filter mode is used
Added test case for "Full Path" filter mode with a regular expression that includes scheme and authority

This closes #3483.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-10-02 17:08:37 +09:00
Alessandro D'Armiento 39a258dc38
NIFI-6727 Fixed the MockProcessContext decrypt bug and added unit test.
This closes #3773.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-10-01 12:53:44 -07:00
Joe Witt 8abf330328
Revert NIFI-6703 as it creates startup issues or logs a confusing message to the user. Details on JIRA
This reverts commit 61baa41e7b.
2019-10-01 15:33:07 -04:00
Andy LoPresto c2746fac5f
NIFI-4573 This closes #3460. Refactored logic handling flow XML encryption migration.
Added unit tests.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-01 13:26:05 -04:00
Troy Melhase c0de26b8d6
NIFI-4573 Catch exceptions during flow content migration. 2019-10-01 13:06:45 -04:00
Koji Kawamura 6541eac625
NIFI-6598 Storing peers into managed-state
- Fixed checkstyle errors.
- Added PeerPersistence interface.
- Expose RemoteProcessGroup state via REST API
- Made stateManager transient.

This closes #3677.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-01 09:56:10 -04:00
Jeff Storck 2f3ed5c40c NIFI-6694 Removed erroneous inclusion of the Hive 3 component on the JDK 8 ja_JP locale build stage on Travis
This closes #3781
2019-10-01 00:06:58 -04:00
samhjelmfelt 61baa41e7b
NIFI-6703: This closes #3762. Add Stateless NiFi to CLI
NIFI-6703: Fixed extracted nar directory and marked api as experimental

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 23:23:15 -04:00
Mark Payne fbd6200ab3
NIFI-6589: This closes #3670. Cache results from zookeeper when determining the leader
NIFI-6589: Updated CuratorLeaderElectionManager to cache results for no more than 5 seconds per review feedback

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 16:47:27 -04:00
Bryan Bende 7ac6e4c50e
NIFI-6711 Clear secret key when auto restarting in order to obtain new secret key from the NiFi process
This closes #3780.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2019-09-30 15:31:57 -04:00
samhjelmfelt 2ab99970b7
NIFI-6662: Adding Kudu Lookup Service
NIFI-6662: Cleaning up Kudu logic
NIFI-6662: Minor enhancements and build fixes
NIFI-6662: This closes #3732.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 14:38:33 -04:00
Rob Fellows 2493665c27 NIFI-6587 - Fix: Unable save sensitive property value that equals masked value
This closes #3778

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-30 13:18:08 -04:00
Jeff Storck 8132d182c7
NIFI-6694 This closes #3779. Exclude Hive 3 components from caching and build stages
Added exclusion of nifi-hive3-nar and nifi-hive3-processors from the caching stages
Added "*hive3*" to the find command used to filter (remove) directories from the build stages

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 12:38:36 -04:00
Mark Payne 6dccdd586e
NIFI-6726: Updated fingerprint logic to ensure that Controller Services are fingerprinted when housed within a ProcessGroup
This closes #3775.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-30 11:55:46 -04:00
Mark Payne 5562c587fe
NIFI-6132: This closes #3776. Do not default ot G1GC because it has known issues with Java 8 and earlier versions
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 11:51:27 -04:00
David Mollitor 48759be974
Updated license information dnsjava 2019-09-30 10:34:02 -04:00
David Mollitor d5a7fc5daa
NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3.2.0 2019-09-30 10:33:50 -04:00
Pierre Villard 7cb87328c8 NIFI-6709 - Upgrade jackson-databind direct dependencies to 2.9.10
This closes #3765.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
2019-09-27 16:07:59 -04:00
Peter Turcsanyi a4c3ca50dd
NIFI-6550: Create controller service for Azure Storage Credentials
Added AzureStorageCredentialsService controller service interface
with 2 implementations:
 - AzureStorageCredentialsControllerService: holds the credential properties
   (account name, account key, sas token)
 - AzureStorageCredentialsControllerServiceLookup: can be used to dynamically
   look up another AzureStorageCredentialsService (similar to
   DBCPConnectionPoolLookup)

The controller service can be used by the Azure Storage processors:
{List|Fetch|Put|Delete}AzureBlobStorage, {Get|Put}AzureQueueStorage

NIFI-6550: Review changes.

NIFI-6550: Review changes #2.

This closes #3742.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-27 01:36:28 +09:00
Andrew Lim bb1018baa7
NIFI-6700 Updated Contributor's Guide link in Developer's Guide to open in new browser tab
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3764.
2019-09-24 20:54:42 +02:00
archongum aba450d658
NIFI-6695: ExtractEmailAttachments/ExtractEmailHeaders should not validate 'SendDate'
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3757.
2019-09-24 20:51:57 +02:00
eduardofontes c7a2885c0c
NIFI-6679 - MoveHDFS removes original file when destination exists
- Include testPutWhenAlreadyExisting test
- Resolve dependencies

This closes #3746.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-24 14:29:25 -04:00
Mark Payne c721a9ee5f
NIFI-6696: Ensured that callback to RemoteQueuePartition do not attempt to obtain the Partition Read Lock. The Read Lock is not necessary as long as the 'partitioner' is volatile, because it doesn't matter whether or not the actual partitions themselves change, since the only partition that would be touched is the Rebalancing Partition,on, which is fixed. Obtaining the partition read lock can lead to a deadlock as outlined in the Jira description.
This closes #3760.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-23 10:07:21 -04:00
Joe Witt 6d8968cc87
NIFI-6520 update to tika 1.22
This closes #3756

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-09-23 08:18:55 -04:00
samhjelmfelt 17a16ba68b
NIFI-6692: Fixing NullPointerException when not using Kerberos
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3755.
2019-09-21 21:37:25 +02:00
Rob Fellows 870a9529c5
NIFI-6693 - This closes #3759. replace assertTrue with assertEquals. use isNull matcher.
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-20 15:40:29 -04:00
Andrew Lim 4cf961690a
NIFI-6558 Improved import/export versioned flows section and also edited managing local changes section
This closes #3753
2019-09-20 15:29:30 -04:00
Scott Aslan 219a83b023
[NIFI-6689] reset private var on dialog close
This closes #3752
2019-09-20 15:21:38 -04:00
thenatog 282c271c7c NIFI-6578 - Upgraded zookeeper framework version. Some code changes required. This change also required a change to the embedded zookeeper.
- Updating tests to account for the new node /zookeeper/config
Upgraded Zookeeper and Curator versions for zookeeper migrator in toolkit. Updated tests to allow for new znode /zookeeper/config
- Added documentation changes to reflect changes to the zookeeper.properties file going foward.

This closes #3715.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-20 10:49:04 -04:00
Aldrin Piri be1acd07c5
NIFI-6604: This closes #3682. Removing large NARs from generated assemblies to reduce size.
Removed items consist of:
* nifi-kafka-0-8-nar
* nifi-flume-nar
* nifi-media-nar
* nifi-druid-nar
* nifi-druid-controller-service-api-nar
* nifi-other-graph-services-nar
* nifi-kite-nar
2019-09-19 23:39:03 -04:00
Mark Payne e18d9ce1e8
NIFI-6369: This closes #3560. Updated JSON Readers to convert String values to Date/Time/Timestamp objects when appropriate according to the schema and the configured pattern
NIFI-6939: Fix to WriteJsonRecord to deal with date/time/timestamp when no format is explicitly set
2019-09-19 23:16:34 -04:00
Mark Payne d0371a5ef1
NIFI-6691: This closes #3754. Fixed bug in MapRecord's algorithm for incorporating Inactive Fields, by recursing into any 'child fields' that contain records; also fixed bug in UpdateRecord that caused it to incorrectly map the first result to all elements if the key of the property points to an array 2019-09-19 22:53:39 -04:00
Joe Witt 13381c2254
NIFI-5816 converting test to integration test as it is test environment specific 2019-09-19 21:46:06 -04:00
mans2singh 79e4f360f4
NIFI-6675 - This closes #3741. Added support for JsonPath put operation 2019-09-19 21:19:18 -04:00
Tamas Palfy 34112519c2 NIFI-6640 - UNION/CHOICE types not handled correctly
3 important changes:
1. FieldTypeInference had a bug when dealing with multiple datatypes for
 the same field where some (but not all) were in a wider-than-the-other
 relationship.
 Before: Some datatypes could be lost. String was wider than any other.
 After: Consistent behaviour. String is NOT wider than any other.
2. Choosing a datatype for a value from a ChoiceDataType:
 Before it chose the first compatible datatype as the basis of conversion.
 After change it tries to find the most suitable datatype.
3. Conversion of a value of avro union type:
 Before it chose the first compatible datatype as the basis of conversion.
 After change it tries to find the most suitable datatype.

Change: In the RecordFieldType enum moved TIMESTAMP ahead of DATE.

This closes #3724.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-19 11:02:30 -04:00
Bryan Bende ff6a7d9561
NIFI-5816 This closes #3726. Remove unused SFTP classes that reference Jsch
NIFI-5816 Switching SFTP processors from JSCH to SSHJ
NIFI-5816 LICENSE/NOTICIE updates to reflect changing from JSch to SSHJ

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-19 10:58:53 -04:00
FaizFizy Rosle f6842a5200
Fix syntax error in schema example
This closes #3745

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-09-19 07:56:42 -04:00
Kevin Doran 3686ee2a16
NIFI-6653 Change bootstrap port command handling
This closes #3722.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-09-18 16:54:09 -07:00
Matt Gilman 171e37ddda
NIFI-6400 Fixing checkstyle issue. 2019-09-18 14:01:52 -04:00
Troy Melhase 4596ef7c8a
NIFI-6400 Better options, consistent ids for ShellUserGroupProvider.
NIFI-6400 Fixes to address PR feedback.
NIFI-6400 Accepts proposed changes.

This closes #3637
2019-09-18 09:08:41 -04:00
Andrew Lim a1b02ec113
NIFI-6558 Added Parameters to User Guide and Sys Admin Guide
NIFI-6558 Edited/Improved new Parameters Context

This closes #3725
2019-09-17 12:07:32 -04:00
Mark Payne fd3f0707c6 NIFI-6677: Update ListHDFS to clear state (when appropriate) in an @OnScheduled method, just as AbstractListProcessor does, instead of doing it in onTrigger. Doing it in onTrigger is problematic because in a cluster, the Primary Node may run for some period of time, perhaps days or months. Then, when the Primary Node chagnes, onTrigger gets called for the first time on the new Primary Node, and this triggers the processor to clear state. 2019-09-17 10:02:06 -04:00
Rob Fellows 9e2a959778 NIFI-6659 - Open create new parameter context dialog in edit mode.
This closes #3729

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-16 11:16:25 -04:00