3615 Commits

Author SHA1 Message Date
Koji Kawamura
6541eac625
NIFI-6598 Storing peers into managed-state
- Fixed checkstyle errors.
- Added PeerPersistence interface.
- Expose RemoteProcessGroup state via REST API
- Made stateManager transient.

This closes #3677.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-01 09:56:10 -04:00
samhjelmfelt
61baa41e7b
NIFI-6703: This closes #3762. Add Stateless NiFi to CLI
NIFI-6703: Fixed extracted nar directory and marked api as experimental

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 23:23:15 -04:00
Mark Payne
fbd6200ab3
NIFI-6589: This closes #3670. Cache results from zookeeper when determining the leader
NIFI-6589: Updated CuratorLeaderElectionManager to cache results for no more than 5 seconds per review feedback

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 16:47:27 -04:00
samhjelmfelt
2ab99970b7
NIFI-6662: Adding Kudu Lookup Service
NIFI-6662: Cleaning up Kudu logic
NIFI-6662: Minor enhancements and build fixes
NIFI-6662: This closes #3732.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 14:38:33 -04:00
Rob Fellows
2493665c27 NIFI-6587 - Fix: Unable save sensitive property value that equals masked value
This closes #3778

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-30 13:18:08 -04:00
Mark Payne
6dccdd586e
NIFI-6726: Updated fingerprint logic to ensure that Controller Services are fingerprinted when housed within a ProcessGroup
This closes #3775.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-30 11:55:46 -04:00
Mark Payne
5562c587fe
NIFI-6132: This closes #3776. Do not default ot G1GC because it has known issues with Java 8 and earlier versions
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 11:51:27 -04:00
David Mollitor
48759be974
Updated license information dnsjava 2019-09-30 10:34:02 -04:00
David Mollitor
d5a7fc5daa
NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3.2.0 2019-09-30 10:33:50 -04:00
Peter Turcsanyi
a4c3ca50dd
NIFI-6550: Create controller service for Azure Storage Credentials
Added AzureStorageCredentialsService controller service interface
with 2 implementations:
 - AzureStorageCredentialsControllerService: holds the credential properties
   (account name, account key, sas token)
 - AzureStorageCredentialsControllerServiceLookup: can be used to dynamically
   look up another AzureStorageCredentialsService (similar to
   DBCPConnectionPoolLookup)

The controller service can be used by the Azure Storage processors:
{List|Fetch|Put|Delete}AzureBlobStorage, {Get|Put}AzureQueueStorage

NIFI-6550: Review changes.

NIFI-6550: Review changes #2.

This closes #3742.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-27 01:36:28 +09:00
archongum
aba450d658
NIFI-6695: ExtractEmailAttachments/ExtractEmailHeaders should not validate 'SendDate'
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3757.
2019-09-24 20:51:57 +02:00
eduardofontes
c7a2885c0c
NIFI-6679 - MoveHDFS removes original file when destination exists
- Include testPutWhenAlreadyExisting test
- Resolve dependencies

This closes #3746.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-24 14:29:25 -04:00
Mark Payne
c721a9ee5f
NIFI-6696: Ensured that callback to RemoteQueuePartition do not attempt to obtain the Partition Read Lock. The Read Lock is not necessary as long as the 'partitioner' is volatile, because it doesn't matter whether or not the actual partitions themselves change, since the only partition that would be touched is the Rebalancing Partition,on, which is fixed. Obtaining the partition read lock can lead to a deadlock as outlined in the Jira description.
This closes #3760.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-23 10:07:21 -04:00
Joe Witt
6d8968cc87
NIFI-6520 update to tika 1.22
This closes #3756

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-09-23 08:18:55 -04:00
samhjelmfelt
17a16ba68b
NIFI-6692: Fixing NullPointerException when not using Kerberos
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3755.
2019-09-21 21:37:25 +02:00
Rob Fellows
870a9529c5
NIFI-6693 - This closes #3759. replace assertTrue with assertEquals. use isNull matcher.
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-20 15:40:29 -04:00
Scott Aslan
219a83b023
[NIFI-6689] reset private var on dialog close
This closes #3752
2019-09-20 15:21:38 -04:00
thenatog
282c271c7c NIFI-6578 - Upgraded zookeeper framework version. Some code changes required. This change also required a change to the embedded zookeeper.
- Updating tests to account for the new node /zookeeper/config
Upgraded Zookeeper and Curator versions for zookeeper migrator in toolkit. Updated tests to allow for new znode /zookeeper/config
- Added documentation changes to reflect changes to the zookeeper.properties file going foward.

This closes #3715.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-20 10:49:04 -04:00
Mark Payne
e18d9ce1e8
NIFI-6369: This closes #3560. Updated JSON Readers to convert String values to Date/Time/Timestamp objects when appropriate according to the schema and the configured pattern
NIFI-6939: Fix to WriteJsonRecord to deal with date/time/timestamp when no format is explicitly set
2019-09-19 23:16:34 -04:00
Mark Payne
d0371a5ef1
NIFI-6691: This closes #3754. Fixed bug in MapRecord's algorithm for incorporating Inactive Fields, by recursing into any 'child fields' that contain records; also fixed bug in UpdateRecord that caused it to incorrectly map the first result to all elements if the key of the property points to an array 2019-09-19 22:53:39 -04:00
Joe Witt
13381c2254
NIFI-5816 converting test to integration test as it is test environment specific 2019-09-19 21:46:06 -04:00
Tamas Palfy
34112519c2 NIFI-6640 - UNION/CHOICE types not handled correctly
3 important changes:
1. FieldTypeInference had a bug when dealing with multiple datatypes for
 the same field where some (but not all) were in a wider-than-the-other
 relationship.
 Before: Some datatypes could be lost. String was wider than any other.
 After: Consistent behaviour. String is NOT wider than any other.
2. Choosing a datatype for a value from a ChoiceDataType:
 Before it chose the first compatible datatype as the basis of conversion.
 After change it tries to find the most suitable datatype.
3. Conversion of a value of avro union type:
 Before it chose the first compatible datatype as the basis of conversion.
 After change it tries to find the most suitable datatype.

Change: In the RecordFieldType enum moved TIMESTAMP ahead of DATE.

This closes #3724.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-19 11:02:30 -04:00
Bryan Bende
ff6a7d9561
NIFI-5816 This closes #3726. Remove unused SFTP classes that reference Jsch
NIFI-5816 Switching SFTP processors from JSCH to SSHJ
NIFI-5816 LICENSE/NOTICIE updates to reflect changing from JSch to SSHJ

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-19 10:58:53 -04:00
FaizFizy Rosle
f6842a5200
Fix syntax error in schema example
This closes #3745

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-09-19 07:56:42 -04:00
Matt Gilman
171e37ddda
NIFI-6400 Fixing checkstyle issue. 2019-09-18 14:01:52 -04:00
Troy Melhase
4596ef7c8a
NIFI-6400 Better options, consistent ids for ShellUserGroupProvider.
NIFI-6400 Fixes to address PR feedback.
NIFI-6400 Accepts proposed changes.

This closes #3637
2019-09-18 09:08:41 -04:00
Mark Payne
fd3f0707c6 NIFI-6677: Update ListHDFS to clear state (when appropriate) in an @OnScheduled method, just as AbstractListProcessor does, instead of doing it in onTrigger. Doing it in onTrigger is problematic because in a cluster, the Primary Node may run for some period of time, perhaps days or months. Then, when the Primary Node chagnes, onTrigger gets called for the first time on the new Primary Node, and this triggers the processor to clear state. 2019-09-17 10:02:06 -04:00
Rob Fellows
9e2a959778 NIFI-6659 - Open create new parameter context dialog in edit mode.
This closes #3729

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-16 11:16:25 -04:00
Rob Fellows
7e9277a2bc
NIFI-6671 - sort parameters by name in the 'Reference Parameter' dropdown.
This closes #3736
2019-09-16 10:03:10 -04:00
Rob Fellows
5ca3655dbf
NIFI-6381 - Make Parameters and Parameter Contexts searchable in UI
NIFI-6381 - remove wildcard imports

This closes #3728
2019-09-13 16:52:14 -04:00
archon
8a8852e73d
NIFI-6480: PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true
This closes #3599.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-13 12:08:27 -04:00
Mark Payne
7623e6f5a1
NIFI-6024: When fetching names of buckets and flows for registry, sort them alphanumerically
This closes #3709.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-13 11:21:32 -04:00
Mark Payne
eb6085a31d
NIFI-6658: Implement new bin/nifi.sh diagnostics command that is responsible for obtaining diagnostic information about many different parts of nifi, the operating system, etc.
This closes #3727.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-13 10:45:53 -04:00
Koji Kawamura
e659e3b606 NIFI-5952 Refactor RAW S2S from nio to socket 2019-09-13 10:30:04 -04:00
Mark Payne
af94b035cb NIFI-6660: Fixed ordering of directory creation and timestamp gathering
This closes #3731.
2019-09-13 09:34:13 -04:00
korir
3d1bb09ff8
fix deserialization issues with NiFiRecordSerDe for hive3streaming
NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types

NIFI-6295: Incorporated review comments

NIFI-6295: Refactored unit tests for NiFiRecordSerDe
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3684
2019-09-13 09:08:57 -04:00
Troy Melhase
93e6f195d9
NIFI-6596 Moves AmazonS3EncryptionService interface
to `nifi-aws-service-api` package.

This closes #3694.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-13 08:55:05 +09:00
Yolanda M. Davis
5106b764da
NIFI-6649 - repaired test due to min zero fix
NIFI-6649 - added check if logging debug is enabled

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3730
2019-09-12 16:08:31 -04:00
Rob Fellows
75c47388a6 NIFI-6630 - Add a goto action in the property table for properties that reference parameters.
NIFI-6630 - base convert and goto parameter logic on any reference to a parameter contained in the text of the property value.

This closes #3718

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-12 13:23:32 -04:00
Rob Fellows
2f7448c9e5
NIFI-6634 - Indicate variables are no longer recommended and favor parameters
This closes #3721
2019-09-12 12:44:52 -04:00
Yolanda M. Davis
8e1452a3f3
NIFI-6649 - added separate query interval configuration for observation queries
NIFI-6649 - documentation update

NIFI-6649 - add debug logging for score and prediction information

NIFI-6649 - fix to ensure counts return minimum value of 0 if not infinite or NaN

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3719
2019-09-12 12:13:19 -04:00
Scott Aslan
c187292fd9
[NIFI-6629] calc correct width for usage ellipsis
This closes #3708
2019-09-11 15:06:56 -04:00
Scott Aslan
d9c8d0c5a7
[NIFI-6625] add ellipsis and tooltip to list of parameters in usage list
This closes #3707
2019-09-11 13:27:55 -04:00
Scott Aslan
250e1b0297
[NIFI-6639] consistent ux for checkboxes and their correspoinding field
This closes #3706
2019-09-11 13:09:32 -04:00
Hsin-Ying Lee
758035b964 NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some cir… (#3673)
* NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some circumstances

signed-off by: Peter Wicks <patricker@gmail.com>
2019-09-11 09:27:52 -06:00
Rob Fellows
29708d8362 NIFI-6641 - fix issue with dialog state when adding a new parameter context. added ellipsis styling and titles to name and id fields.
This closes #3710

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-10 10:50:52 -04:00
Rob Fellows
70af2acc06
NIFI-6641 - support read only mode for parameter context dialog 2019-09-10 10:03:06 -04:00
Rob Fellows
5ddc01eb9f
NIFI-6644 - Consider param context permissions when deciding to allow property conversion to parameter.
NIFI-6644 - when deciding to show the convert prop option: enforce user has read and write permissions to the param context & the props are not for a controller service defined in the controller settings.

This closes #3713
2019-09-09 16:11:20 -04:00
Mark Payne
f89ea8cf21
NIFI-6645: Fixed problem in unit test for StandardParameterContext because behavior changed; fixed issue in StandardParameterContext around adding a parameter with no description
This closes #3714.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-09 15:21:21 -04:00
Peter Turcsanyi
21a27c8bb0 NIFI-6636: Fixed ListGCSBucket file duplication error
ListGCSBucket duplicated files if they arrived not in alphabetical order.
The set storing the name of the latest blob (which was loaded with the highest
timestamp during the previous run of the processor) was cleared too early.

Also changed the state persisting logic: it is now saved only once at the end
of onTrigger() (similar to ListS3). Some inconsistent state (only blob names
without the timestamp) was also saved earlier.

This closes #3702.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-09 13:32:17 -04:00