Commit Graph

6436 Commits

Author SHA1 Message Date
Joe Gresock 54a0e27c93
NIFI-7134: Adding auto-reloading of Keystore and Truststore
- NIFI-7261 Included TrustStoreScanner for auto-reloading of truststore

This closes #4991

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-29 08:29:42 -05:00
herbert 93dcf25d33
MINIFI-552: Fix new SSL SAN Behaviour from new Jetty Version
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5036
2021-04-28 11:47:04 -04:00
Matthew Burgess a0655bff1a
MINIFI-422: Incorporate MiNiFi Java into NiFi
- Initial copy of MiNiFi Java into NiFi
- Checkpoint for basic integration, load empty flow without startup errors
- Refactor provenance repositories, replace some MiNiFi NARs with NiFi NARs
- Remove MiNiFi LICENSE, NOTICE, and KEYS, change file permissions on build.properties
- Updated MiNiFi LICENSE and NOTICE files
- Fix headless NiFi config, set krb5 file in MiNiFi for Kerberos support
- Removed commented-out lines from POMs, added nifi-kerberos-credentials-service NAR to MiNiFi assembly
- Update Git repo URLs, replace NiFi version property

This closes #4933.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2021-04-27 21:06:56 -04:00
exceptionfactory 3fb76400db
NIFI-8485 Upgraded Jetty to 9.4.40
This closes #5035

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-27 17:49:23 -05:00
Paul Kelly 0ed3534524 NIFI-7886 Added read by range to Azure, GCS, S3 fetch processors
This closes #4576

Co-authored-by: Joey Frazee <jfrazee@apache.org>
Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-27 12:08:06 -07:00
Tamas Palfy 16dc61e151 NIFI-8458 TailFile - Fix some bugs. Add more tests.
- During cleanup keep "tailingPostRollover" in the updated state.
- Skipping tests that can't run on Windows.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-04-27 09:44:39 -04:00
Joe Gresock 90c7d03ed3
NIFI-8403: Generating Self-signed cert on startup when applicable (#4986)
* NIFI-8403: Implementing auto-generated certificates for secure startup

* Adding check for passwords in SecureNiFiConfigUtil
2021-04-27 09:34:44 -04:00
Peter Turcsanyi 525bfe00b3
NIFI-8474: Added new Replacement Strategy for variable substitution in ReplaceText.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5032.
2021-04-27 13:03:21 +02:00
exceptionfactory f9d3bb7f69
NIFI-8473 Added Parameter Contexts section to REST API documentation
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5030.
2021-04-27 12:23:33 +02:00
Paul Grey 835f50c83a
NIFI-8465 Handle bcrypt legacy decrypt failures in testing
This closes #5029

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-26 11:34:20 -05:00
Denes Arvay 9f45b48866 NIFI-8437 RecordReader 'Infer Schema' for large records may throw BufferedInputStream error
This closes #5011.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-04-23 13:46:41 -04:00
exceptionfactory 92bdc23adb NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors
- Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations
- NIFI-7396 EncryptContentPGP writes encryption metadata attributes
- NIFI-6708 Controller Services support ElGamal Public and Private Keys
- NIFI-5346 Controller Services support Keyring Files and ASCII Key properties
- NIFI-5335 Controller Services support multiple public or private keys from keyrings
- NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing
- NIFI-1694 Controller Services support individual key files or keyrings

NIFI-8251 Refactored Public Key ID Property to Public Key Search

NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP

- Added unit tests for encryption and decryption with both password-based and public key
- Added PGP NAR dependencies to nifi-assembly

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4842.
2021-04-22 15:48:49 -04:00
Paul Grey a1597ced89
NIFI-8460 Remove unused BouncyCastle reference in test class
This closes #5022

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-22 12:01:44 -05:00
Paul Grey 17fa0cf3c1
NIFI-3580 Add configurable TLS Cipher Suite properties
This closes #5018

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-22 11:31:19 -05:00
Paul Kelly aedacdf86f
NIFI-8454: Allow InvokeHTTP to output final URL from response request property
This closes #5016

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-21 18:18:25 -05:00
Joe Witt 8207c9db20
NIFI-8456 This closes #5017. Ensure we have only three builds and we maximally cover zulu vs adopt, linux vs windows vs osx, and java 8 vs 11 and EN vs FR vs JP
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-21 14:30:24 -07:00
exceptionfactory ed6d5bacba NIFI-8451 Updated KeyStoreUtils to use KeyStore.getInstance() with provider
- Refactored and consolidated KeyStoreUtils unit tests
- Corrected KeyStoreUtils.loadEmptyKeyStore() to use KeyStoreUtils.getKeyStore()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5015.
2021-04-21 13:45:50 -04:00
Mark Payne 1ef1905461 NIFI-8344: Addressed corner case in which we didn't handle a case where we tailed a file after being rolled over and then processor encounters a newline followed by NUL characters
This closes #5009.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-04-20 14:51:24 +02:00
Joe Gresock be716c1621 NIFI-3862: Adding Issuer DN support to ListenHTTP
This closes #4976.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-20 13:09:53 +02:00
Arpad Boda 1497a6ea0d
Update KEYS
Adding public keys of fgerlits and adamdebreceni
2021-04-20 11:47:16 +02:00
Otto Fowler a44ab0d683
NIFI-8397 update to simple-syslog-5424 0.0.16
resolves issue working with syslog with BOM markers

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4978.
2021-04-20 09:28:29 +02:00
exceptionfactory bbe75e81dd
NIFI-8441 Added timeout-minutes for GitHub workflow on Windows
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5010.
2021-04-20 09:23:17 +02:00
Tamas Palfy a50957161c NIFI-8365 Fix JSON AbstractJsonRowRecordReader to handle deep CHOICE-typed records properly: change the logic that selects the first compatible schema which can have missing fields compared to the real value and search for a more strict match first and fallback to the existing logic only if not one found.
- AbstractJsonRowRecordReader - Handle (meaning log a warning and not fail completely) multi-array CHOICE type when data has extra fields (not defined by the schema) and can't determine correct type.
- AvroTypeUtil - Allow multiple different record types in avro union type. Minor refactors. Added documentation fro EqualsWrapper.
2021-04-19 12:56:09 -04:00
Joe Witt 77c353219b
NIFI-8421 This closes #4996. increase commit frequency to ensure we commit often during these tests to minimize race conditions
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-19 07:03:58 -07:00
Joe Witt 173daae664
NIFI-8420 update to latest jetty on 9.4.x line 2021-04-19 07:03:40 -07:00
Timea Barna 78c03795f6 NIFI-8372 Refactoring ScanAccumulo unit test and add test for value in result
This closes #4998.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-19 08:33:38 +02:00
Bence Simon f0ca63b2ab NIFI-8429 Changing DBCPConnectionPool driver manager in order to prevent from leaking
NIFI-8429 Refining error handling based on review comment

This closes #5003.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-16 09:26:50 +02:00
Pierre Villard 0f8789b8b0
NIFI-8409 - ConsumeMQTT Processor Broker URI and Username Expression Language
This closes #4984

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-14 16:08:44 -05:00
Pierre Villard 4fb86d79e4
NIFI-8428 - increased actions column width for controller services
This closes #5000

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-14 15:56:06 -05:00
Peter Turcsanyi 2fb05cd8dc
NIFI-8430: Close Atlas client in order to free up resources
This closes #5002

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-14 15:30:08 -05:00
Joe Witt 8c66049d1d
NIFI-8411 update to use latest java action and specific distribution so we can validate azul and zulu
This closes #4995

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-04-14 08:43:43 -04:00
Joe Witt f0611f31da
NIFI-8422 ensure stale workflow for github ci is there and update asf license header missing from a few files
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4997.
2021-04-14 09:26:30 +02:00
Paul Grey de3e1a0d0d
NIFI-8416 - Removed dl.bintray.com repository for Groovy
- Upgraded Groovy from 2.5.4 to 2.5.6
- Unit test updates correcting issues with Groovy/Spock and Java 11

This closes #4999

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-13 12:23:33 -05:00
Mohammed Nadeem 7d5903c323
NIFI-8412: This closes #4993. Use proper log messages in PublishKafka(Record) processors
incase of 'Rollback on Failure'

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-13 07:25:50 -07:00
exceptionfactory 06ebb761a2
NIFI-5541 This closes #4971. Added OWASP Maven build profile for dependency checking
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-12 20:46:51 -07:00
Bryan Bende 1ab4cf5307
NIFI-8419 This closes #4994. Fix issues with contoller services during parameter context update in a secure instance
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-12 20:35:44 -07:00
Wouter de Vries 6309ab96d1
NIFI-8295: This closes #4946. upgrade cassandra driver to 3.11.0
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-12 20:31:53 -07:00
exceptionfactory ef794c56b6
NIFI-8398 Updated dependencies to build using Apache Maven 3.8.1
- Updated ParCEFone to 1.2.8
- Added conjars.org HTTPS repository for Hive transitive dependencies

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4989
2021-04-12 14:45:02 -04:00
Andrew Lim 55236e6fd7
NIFI-8396 Update incorrect case in references to ZooKeeper in Admin and Toolkit guides
This closes #4985

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-12 11:50:05 -05:00
Joe Gresock 2298953f90
NIFI-8410: Enabling TLS in RedisStateProvider
This closes #4990

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-12 11:38:47 -05:00
Mark Payne 7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
exceptionfactory 052c60dc53
NIFI-8363 Added Single User Login Identity Provider and Authorizer (#4968)
* NIFI-8363 Added Single User Login Identity Provider and Authorizer

- Reads and writes username and hashed password in login-identity-providers.xml
- Generates random username using java.util.UUID.randomUUID()
- Generates random password using java.security.SecureRandom with Base64 encoding
- Writes generated password hash using bcrypt

* NIFI-8363 Updated SingleUserAuthorizer to require SingleUserLoginIdentityProvider

* NIFI-8363 Added handling of null login identity provider property
2021-04-12 08:48:20 -04:00
Mark Payne 74531fa1ca NIFI-8348: Updated system tests to reference the version of jersey client used elsewhere, as it was missed in the previous commit for NIFI-8348 and system tests are failing 2021-04-10 10:33:16 -04:00
sjyang18 365dde2d11 NIFI-7573 Added Azure Active Directory User Group Provider
This closes #4367

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-09 15:06:14 -07:00
nabmoh123 d0045c13b1
NIFI-8341 Support Multi Hosts in AMQP Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4939.
2021-04-09 17:42:24 +02:00
Joe Gresock 76f33e42c6
NIFI-7198: Adding optional SSL Context Service to RedisConnectionPoolService
This closes #4981

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-09 10:36:31 -05:00
Timea Barna bfd964b9c7 NIFI-8176 - Move the timeout check after we process the response from Splunk to make sure we poll for acknowledgement at least once. (No need for flag.)
This closes #4824.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-04-09 17:22:56 +02:00
Mark Payne 14e6dc3dc6
NIFI-8405: Added debug logging around how long it takes to establish connections/query dns/read and write headers and body when replication requests; added additional timing around Ranger audits and authorizations and monitoring of long-running tasks because those run often and frequently show up in the logs at the same time as the long requests
This closes #4983

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-08 15:41:44 -05:00
Mark Payne 78b69f10dc NIFI-8404: Added capability to rollback on failure for PublishKafka(Record)_2_0 and _2_6
NIFI-8404: Added unit tests for new Failure Strategy property

This closes #4982.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 17:47:13 +02:00
Matthew Burgess 33ec8c8427 NIFI-8400: Use longs in SystemDiagnostics, add null checks
This closes #4980.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 10:06:26 +02:00