Commit Graph

6773 Commits

Author SHA1 Message Date
Jon Kessler 2309d75d3d NIFI-8126
- Added totalActiveQueuedDuration and maxActiveQueuedDuration to the ConnectionStatus object
- Updated FlowFileQueue implementations and supporting code to properly calculate and provide the totalActiveQueuedDuration and maxActiveQueuedDuration for their active queues
- Fixing failing unit test. In examining this it appears that the unit test only accidentally passed in the past and that the object mocked to always throw an exception was not actually being used in the test.
- Adding UI component via ConnectionStatusDescriptor along with updates based on PR comments.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-05-18 17:09:53 -04:00
Matthew Burgess f812dfdfc0
NIFI-8535: Better error message in PutDatabaseRecord when table does not exist (#5070)
* NIFI-8535: Better error message in PutDatabaseRecord when table does not exist
2021-05-14 15:13:00 -04:00
Joe Gresock ed591e0f22
NIFI-8445: Implemented HashiCorpVaultCommunicationService in nifi-vault-utils
This closes #5034

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-14 12:27:01 -05:00
Bence Simon 51aae5bcf6 NIFI-8519 Adding HDFS support for NAR autoload
- Refining classloader management with the help of @markap14

This closes #5059

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-05-14 13:14:06 -04:00
Mark Payne 7c08fbc4d4
NIFI-8542: When returning content via TriggerResult.readContent(FlowFile), ensure that we take into account the content claim offset and length
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5076
2021-05-14 12:57:01 -04:00
Nathan Gough 7e54ef9421
NIFI-8343 - Updated solr from 8.4.1 to 8.8.2. Small code changes were required.
This closes #5074

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-14 11:38:07 -05:00
exceptionfactory 6776765a92
NIFI-8538 Upgraded Apache Commons IO to 2.8.0
- Upgraded direct dependencies from 2.6 to 2.8.0
- Added dependency management configuration to use 2.8.0 for some modules
- Updated scripted Groovy tests to avoid copying unnecessary files

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5073
2021-05-14 09:31:48 -04:00
Mohammed Nadeem 73a2283c27
NIFI-8536: PrometheusReportingTask and PrometheusRecordSink do not support variable registry for port property
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5075
2021-05-13 14:05:30 -04:00
Mark Payne 97ed0efbc2
NIFI-8537: Check that a property's dependencies are satisfied before checking if the referenced controller service valid/enabled
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5071
2021-05-13 11:37:57 -04:00
Chris Sampson a274c12bbb NIFI-2892 Implement AWS Kinesis Stream Consume Processor
This closes #4822.

Co-authored-by: uday <udaygkale@gmail.com>

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-12 20:03:28 +02:00
Mark Payne ab8b7444b5
NIFI-8457: Fixed bug in load balanced connections that can result in the node never completing OFFLOAD action. Also fixed issue in which data destined for a disconnected/offloaded node was never rebalanced even for partitioning strategies that call for rebalancing on failure
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5019.
2021-05-12 13:05:48 +02:00
exceptionfactory 09e54c1dad
NIFI-8246 Set NIFI_PBKDF2_AES_GCM_256 as default properties encryption algorithm
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5055.
2021-05-12 12:58:50 +02:00
exceptionfactory 4db50f2f40
NIFI-8502 Upgraded Spring Framework to 5.3.6
- Upgraded Spring Framework references from version 4.3.30 to 5.3.6
- Upgraded Spring Security from version 4.2.20 to 5.4.6
- Upgraded Spring Data Redis from 2.1.16 to 2.5.0
- Upgraded Jedis from 2.9.0 to 3.6.0 to match Spring Data Redis 2.5.0
- Upgraded Easy Rules from 3.4.0 to 4.1.0 to support Spring 5
- Upgraded Hortonworks Schema Registry Client from 0.8.1 to 0.9.1 to support Spring 5
- Refactored ThreadPoolRequestReplicatorFactoryBean to implement DisposableBean to handle executor shutdown

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5066.
2021-05-12 12:49:01 +02:00
Sandro Tolaini fdda3582e3
NIFI-8470: Fix DataTypeUtils for CHAR() sequences
NIFI-8470: Added unit test

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5027
2021-05-11 17:46:38 -04:00
Matthew Burgess a4d435c995 NIFI-8330: Fix compilation of Jython scripts for scripting components
- Add onPropertyModified to ExecuteScript to recompile on change

This closes #4903.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-05-11 16:51:23 -04:00
exceptionfactory 340b13033f NIFI-8367 Removed joda-time from nifi-standard-processors 2021-05-11 14:50:49 -04:00
matthewlannan 86d5b08d23 Adding addititonal documentation to EvalluateJsonPath per https://issues.apache.org/jira/browse/NIFI-8518 2021-05-11 14:40:12 -04:00
Sushil Kumar 48a9054731 NIFI-4913 - Added "-E" with sudo to read environment variables set in current shell environment while running nifi with bootstrap property "run.as" set as to some user
NIFI-4913 - Updated administration-guide.adoc

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #3750.
2021-05-11 11:48:41 -04:00
s9514171 1791f4cc35
NIFI-8417 emoji in processor group name cause nifi to not start up
- Excluded Xalan dependency from spring-security-saml2-core

This closes #5012

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-11 08:27:49 -05:00
Mark Payne 1645886e5a NIFI-8477: If interrupted while waiting for Node Status Update to be replicated to other nodes, do not throw ProtocolException; instead just log a warning and return.
This closes #5039
2021-05-10 16:15:00 -04:00
Mark Bean 5c2df42be3
NIFI-8431: remove redundant validation of dynamic properties
removed unused variable

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5045
2021-05-10 13:54:18 -04:00
exceptionfactory 5062f2e54e
NIFI-8515 Upgraded Apache Tika to 1.26
- Upgraded tika-core from 1.24 to 1.26 in nifi-framework-bundle and nifi-standard-bundle
- Upgraded tika-parsers from 1.24.1 to 1.26 in nifi-media-processors

NIFI-8515 Removed jsr-275 exclusion and confirmed it is no longer included

- TIKA-2535 Replaced the jsr-275 dependency to resolve licensing issues

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5056
2021-05-10 11:13:31 -04:00
Matthew Burgess 284322feed
MINIFI-554: Move OSUtils to nifi-bootstrap-utils for MiNiFi and NiFi
This closes #5062

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-10 08:39:10 -05:00
Mark Bean 1e75b7ef06
NIFI-8496: eliminate 500 error when uploading templates
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5053
2021-05-07 11:26:21 -04:00
Mark Payne 2895bac2c0
NIFI-8512: When converting to/from Avro UNION type, we can be more efficient when the UNION consists of a Null type and one other type by determinine the non-null type and just using that. Also eliminated a call to List.stream() and related .collect() call by using an existing method that performs the logic without the very expensive call to stream()
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5051
2021-05-06 16:06:23 -04:00
Matthew Burgess 30fc26647e
NIFI-6061: Fix CLOB/BLOB handling in PutDatabaseRecord
NIFI-6061: Force getBytes() in BLOB handling to use UTF-8 charset

NIFI-6061: Use setClob(), added unit tests, incorporated review comments

This closes #5049

Co-authored-by: zhangcheng <zhangchengk@foxmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-04 14:58:14 -05:00
exceptionfactory 328b9e540b
NIFI-8513 Upgraded Spring Framework to 4.3.30
- Upgraded direct Spring Framework references to 4.3.30.RELEASE
- Upgraded direct Spring Security references to 4.2.20.RELEASE

NIFI-8513 Updated Spring Framework and Security Notices with major version

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5054.
2021-05-04 18:45:26 +02:00
Tamas Palfy 5bcfcf42bb NIFI-7788 Created a new endpoint in RemoteProcessGroupResource to allow updating run statuses/transmission state of all remote process groups within a process group.
When selecting run/stop on a process group/canvas/selection, it will try to enable/disable transmission of all involved remote process groups.

NIFI-7788 Supplied same functionality missed when selecting a process group.
NIFI-7788 Updated endpoint URL paths.
NIFI-7788 No need to return list of remote process groups when updating en masse.
NIFI-7788 Added some null checks in RemoteProcessGroupsEndpointMerger.merge.
NIFI-7788 Fix checkstyle violation.

This closes #4516.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-04 16:49:21 +02:00
exceptionfactory 13d5be622b NIFI-8230 Removed default Sensitive Properties Key and added random generation
- Retained legacy default Sensitive Properties Key in ConfigEncryptionTool to support migration
- Streamlined default file path and moved key generation conditional
- Refactored with getDefaultProperties()
- Cleared System Property in ConfigEncryptionToolTest
- Added checking and error handling for clustered status
- Added set-sensitive-properties-key command
- Refactored PropertyEncryptor classes to nifi-property-encryptor
- Added nifi-flow-encryptor
- Refactored ConfigEncryptionTool to use FlowEncryptor for supporting AEAD algorithms
- Added Admin Guide section Updating the Sensitive Properties Key

This closes #4857.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-05-04 08:49:34 -04:00
Paul Grey a74c17616d
NIFI-8478 - This closes #5052. handle undefined system property 'user.region'
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-05-03 16:55:37 -07:00
Guillaume Schaer 24d3d4325c NIFI-8506 Fix Azure Storage integration tests
This closes #5047

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-05-03 13:19:31 -07:00
Paul Grey cc554a6b11
NIFI-8478 Addressed i18n test issues; CI supplies locale to surefire
This closes #5040

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-03 13:40:01 -05:00
Nathan Gough e1697a8d58
NIFI-8510 - Added a more specific CSRF cookie request matcher to avoid issues with CSRF 403 response.
This closes #5050

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-03 13:30:08 -05:00
Mark Payne 935566ba23 NIFI-8433 Added ability to decommission a node in a cluster
This closes #5004

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-05-01 12:45:48 -07:00
Nathan Gough 1090a9748a
NIFI-7870 Resolved access to extension resources when using JWT
- Added SameSite Session Cookie __Host-Authorization-Bearer for sending JWT
- Configured Spring Security CSRF Filter comparing Authorization header and Cookie JWT
- Implemented BearerTokenResolver for retrieving JWT

This closes #4988

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-30 22:29:55 -05:00
Matthew Burgess 3963f66dff
NIFI-8320: Fix column mismatch in PutDatabaseRecord
This closes #5024

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-30 10:41:36 -05:00
Matthew Burgess e1c99e3a5c
MINIFI-553: Fix timeout-based tests in StatusLoggerTest
This closes #5041

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-29 10:45:43 -05:00
Joe Gresock 54a0e27c93
NIFI-7134: Adding auto-reloading of Keystore and Truststore
- NIFI-7261 Included TrustStoreScanner for auto-reloading of truststore

This closes #4991

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-29 08:29:42 -05:00
herbert 93dcf25d33
MINIFI-552: Fix new SSL SAN Behaviour from new Jetty Version
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5036
2021-04-28 11:47:04 -04:00
Matthew Burgess a0655bff1a
MINIFI-422: Incorporate MiNiFi Java into NiFi
- Initial copy of MiNiFi Java into NiFi
- Checkpoint for basic integration, load empty flow without startup errors
- Refactor provenance repositories, replace some MiNiFi NARs with NiFi NARs
- Remove MiNiFi LICENSE, NOTICE, and KEYS, change file permissions on build.properties
- Updated MiNiFi LICENSE and NOTICE files
- Fix headless NiFi config, set krb5 file in MiNiFi for Kerberos support
- Removed commented-out lines from POMs, added nifi-kerberos-credentials-service NAR to MiNiFi assembly
- Update Git repo URLs, replace NiFi version property

This closes #4933.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2021-04-27 21:06:56 -04:00
exceptionfactory 3fb76400db
NIFI-8485 Upgraded Jetty to 9.4.40
This closes #5035

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-27 17:49:23 -05:00
Paul Kelly 0ed3534524 NIFI-7886 Added read by range to Azure, GCS, S3 fetch processors
This closes #4576

Co-authored-by: Joey Frazee <jfrazee@apache.org>
Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-27 12:08:06 -07:00
Tamas Palfy 16dc61e151 NIFI-8458 TailFile - Fix some bugs. Add more tests.
- During cleanup keep "tailingPostRollover" in the updated state.
- Skipping tests that can't run on Windows.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-04-27 09:44:39 -04:00
Joe Gresock 90c7d03ed3
NIFI-8403: Generating Self-signed cert on startup when applicable (#4986)
* NIFI-8403: Implementing auto-generated certificates for secure startup

* Adding check for passwords in SecureNiFiConfigUtil
2021-04-27 09:34:44 -04:00
Peter Turcsanyi 525bfe00b3
NIFI-8474: Added new Replacement Strategy for variable substitution in ReplaceText.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5032.
2021-04-27 13:03:21 +02:00
exceptionfactory f9d3bb7f69
NIFI-8473 Added Parameter Contexts section to REST API documentation
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5030.
2021-04-27 12:23:33 +02:00
Paul Grey 835f50c83a
NIFI-8465 Handle bcrypt legacy decrypt failures in testing
This closes #5029

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-26 11:34:20 -05:00
Denes Arvay 9f45b48866 NIFI-8437 RecordReader 'Infer Schema' for large records may throw BufferedInputStream error
This closes #5011.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-04-23 13:46:41 -04:00
exceptionfactory 92bdc23adb NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors
- Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations
- NIFI-7396 EncryptContentPGP writes encryption metadata attributes
- NIFI-6708 Controller Services support ElGamal Public and Private Keys
- NIFI-5346 Controller Services support Keyring Files and ASCII Key properties
- NIFI-5335 Controller Services support multiple public or private keys from keyrings
- NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing
- NIFI-1694 Controller Services support individual key files or keyrings

NIFI-8251 Refactored Public Key ID Property to Public Key Search

NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP

- Added unit tests for encryption and decryption with both password-based and public key
- Added PGP NAR dependencies to nifi-assembly

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4842.
2021-04-22 15:48:49 -04:00
Paul Grey a1597ced89
NIFI-8460 Remove unused BouncyCastle reference in test class
This closes #5022

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-22 12:01:44 -05:00