Commit Graph

4363 Commits

Author SHA1 Message Date
Joe Gresock 54a0e27c93
NIFI-7134: Adding auto-reloading of Keystore and Truststore
- NIFI-7261 Included TrustStoreScanner for auto-reloading of truststore

This closes #4991

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-29 08:29:42 -05:00
Matthew Burgess a0655bff1a
MINIFI-422: Incorporate MiNiFi Java into NiFi
- Initial copy of MiNiFi Java into NiFi
- Checkpoint for basic integration, load empty flow without startup errors
- Refactor provenance repositories, replace some MiNiFi NARs with NiFi NARs
- Remove MiNiFi LICENSE, NOTICE, and KEYS, change file permissions on build.properties
- Updated MiNiFi LICENSE and NOTICE files
- Fix headless NiFi config, set krb5 file in MiNiFi for Kerberos support
- Removed commented-out lines from POMs, added nifi-kerberos-credentials-service NAR to MiNiFi assembly
- Update Git repo URLs, replace NiFi version property

This closes #4933.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2021-04-27 21:06:56 -04:00
Paul Kelly 0ed3534524 NIFI-7886 Added read by range to Azure, GCS, S3 fetch processors
This closes #4576

Co-authored-by: Joey Frazee <jfrazee@apache.org>
Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-27 12:08:06 -07:00
Tamas Palfy 16dc61e151 NIFI-8458 TailFile - Fix some bugs. Add more tests.
- During cleanup keep "tailingPostRollover" in the updated state.
- Skipping tests that can't run on Windows.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-04-27 09:44:39 -04:00
Joe Gresock 90c7d03ed3
NIFI-8403: Generating Self-signed cert on startup when applicable (#4986)
* NIFI-8403: Implementing auto-generated certificates for secure startup

* Adding check for passwords in SecureNiFiConfigUtil
2021-04-27 09:34:44 -04:00
Peter Turcsanyi 525bfe00b3
NIFI-8474: Added new Replacement Strategy for variable substitution in ReplaceText.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5032.
2021-04-27 13:03:21 +02:00
exceptionfactory f9d3bb7f69
NIFI-8473 Added Parameter Contexts section to REST API documentation
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5030.
2021-04-27 12:23:33 +02:00
Paul Grey 835f50c83a
NIFI-8465 Handle bcrypt legacy decrypt failures in testing
This closes #5029

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-26 11:34:20 -05:00
Denes Arvay 9f45b48866 NIFI-8437 RecordReader 'Infer Schema' for large records may throw BufferedInputStream error
This closes #5011.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-04-23 13:46:41 -04:00
exceptionfactory 92bdc23adb NIFI-8251 Added EncryptContentPGP and DecryptContentPGP Processors
- Added PGPPrivateKeyService and PGPPublicKeyService interfaces with standard implementations
- NIFI-7396 EncryptContentPGP writes encryption metadata attributes
- NIFI-6708 Controller Services support ElGamal Public and Private Keys
- NIFI-5346 Controller Services support Keyring Files and ASCII Key properties
- NIFI-5335 Controller Services support multiple public or private keys from keyrings
- NIFI-2983 DecryptContentPGP finds and decrypts Encrypted Data Packets regardless of signing
- NIFI-1694 Controller Services support individual key files or keyrings

NIFI-8251 Refactored Public Key ID Property to Public Key Search

NIFI-8251 Corrected handling of multiple Encrypted Data packets in DecryptContentPGP

- Added unit tests for encryption and decryption with both password-based and public key
- Added PGP NAR dependencies to nifi-assembly

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4842.
2021-04-22 15:48:49 -04:00
Paul Grey a1597ced89
NIFI-8460 Remove unused BouncyCastle reference in test class
This closes #5022

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-22 12:01:44 -05:00
Paul Grey 17fa0cf3c1
NIFI-3580 Add configurable TLS Cipher Suite properties
This closes #5018

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-22 11:31:19 -05:00
Paul Kelly aedacdf86f
NIFI-8454: Allow InvokeHTTP to output final URL from response request property
This closes #5016

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-21 18:18:25 -05:00
Mark Payne 1ef1905461 NIFI-8344: Addressed corner case in which we didn't handle a case where we tailed a file after being rolled over and then processor encounters a newline followed by NUL characters
This closes #5009.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-04-20 14:51:24 +02:00
Joe Gresock be716c1621 NIFI-3862: Adding Issuer DN support to ListenHTTP
This closes #4976.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-20 13:09:53 +02:00
Otto Fowler a44ab0d683
NIFI-8397 update to simple-syslog-5424 0.0.16
resolves issue working with syslog with BOM markers

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4978.
2021-04-20 09:28:29 +02:00
Tamas Palfy a50957161c NIFI-8365 Fix JSON AbstractJsonRowRecordReader to handle deep CHOICE-typed records properly: change the logic that selects the first compatible schema which can have missing fields compared to the real value and search for a more strict match first and fallback to the existing logic only if not one found.
- AbstractJsonRowRecordReader - Handle (meaning log a warning and not fail completely) multi-array CHOICE type when data has extra fields (not defined by the schema) and can't determine correct type.
- AvroTypeUtil - Allow multiple different record types in avro union type. Minor refactors. Added documentation fro EqualsWrapper.
2021-04-19 12:56:09 -04:00
Joe Witt 77c353219b
NIFI-8421 This closes #4996. increase commit frequency to ensure we commit often during these tests to minimize race conditions
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-19 07:03:58 -07:00
Timea Barna 78c03795f6 NIFI-8372 Refactoring ScanAccumulo unit test and add test for value in result
This closes #4998.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-19 08:33:38 +02:00
Bence Simon f0ca63b2ab NIFI-8429 Changing DBCPConnectionPool driver manager in order to prevent from leaking
NIFI-8429 Refining error handling based on review comment

This closes #5003.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-16 09:26:50 +02:00
Pierre Villard 0f8789b8b0
NIFI-8409 - ConsumeMQTT Processor Broker URI and Username Expression Language
This closes #4984

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-14 16:08:44 -05:00
Pierre Villard 4fb86d79e4
NIFI-8428 - increased actions column width for controller services
This closes #5000

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-14 15:56:06 -05:00
Peter Turcsanyi 2fb05cd8dc
NIFI-8430: Close Atlas client in order to free up resources
This closes #5002

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-14 15:30:08 -05:00
Mohammed Nadeem 7d5903c323
NIFI-8412: This closes #4993. Use proper log messages in PublishKafka(Record) processors
incase of 'Rollback on Failure'

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-13 07:25:50 -07:00
Bryan Bende 1ab4cf5307
NIFI-8419 This closes #4994. Fix issues with contoller services during parameter context update in a secure instance
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-12 20:35:44 -07:00
Wouter de Vries 6309ab96d1
NIFI-8295: This closes #4946. upgrade cassandra driver to 3.11.0
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-12 20:31:53 -07:00
exceptionfactory ef794c56b6
NIFI-8398 Updated dependencies to build using Apache Maven 3.8.1
- Updated ParCEFone to 1.2.8
- Added conjars.org HTTPS repository for Hive transitive dependencies

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4989
2021-04-12 14:45:02 -04:00
Joe Gresock 2298953f90
NIFI-8410: Enabling TLS in RedisStateProvider
This closes #4990

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-12 11:38:47 -05:00
Mark Payne 7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
exceptionfactory 052c60dc53
NIFI-8363 Added Single User Login Identity Provider and Authorizer (#4968)
* NIFI-8363 Added Single User Login Identity Provider and Authorizer

- Reads and writes username and hashed password in login-identity-providers.xml
- Generates random username using java.util.UUID.randomUUID()
- Generates random password using java.security.SecureRandom with Base64 encoding
- Writes generated password hash using bcrypt

* NIFI-8363 Updated SingleUserAuthorizer to require SingleUserLoginIdentityProvider

* NIFI-8363 Added handling of null login identity provider property
2021-04-12 08:48:20 -04:00
sjyang18 365dde2d11 NIFI-7573 Added Azure Active Directory User Group Provider
This closes #4367

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-09 15:06:14 -07:00
nabmoh123 d0045c13b1
NIFI-8341 Support Multi Hosts in AMQP Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4939.
2021-04-09 17:42:24 +02:00
Joe Gresock 76f33e42c6
NIFI-7198: Adding optional SSL Context Service to RedisConnectionPoolService
This closes #4981

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-09 10:36:31 -05:00
Timea Barna bfd964b9c7 NIFI-8176 - Move the timeout check after we process the response from Splunk to make sure we poll for acknowledgement at least once. (No need for flag.)
This closes #4824.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-04-09 17:22:56 +02:00
Mark Payne 14e6dc3dc6
NIFI-8405: Added debug logging around how long it takes to establish connections/query dns/read and write headers and body when replication requests; added additional timing around Ranger audits and authorizations and monitoring of long-running tasks because those run often and frequently show up in the logs at the same time as the long requests
This closes #4983

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-08 15:41:44 -05:00
Mark Payne 78b69f10dc NIFI-8404: Added capability to rollback on failure for PublishKafka(Record)_2_0 and _2_6
NIFI-8404: Added unit tests for new Failure Strategy property

This closes #4982.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 17:47:13 +02:00
Matthew Burgess 33ec8c8427 NIFI-8400: Use longs in SystemDiagnostics, add null checks
This closes #4980.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 10:06:26 +02:00
Mohammed Nadeem 77a73a603b NIFI-8394 Fix NPE error when NiFi connects to external zookeeper insecurely
This closes #4974

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-07 11:23:25 -07:00
Nathan Gough 9da3b1ec01
NIFI-7912 - Added properties to configure DoSFilter timeout and whitelisted addresses
- Added nifi.web.request.ip.whitelist property to set DoSFilter.ipWhitelist
- Added nifi.web.request.timeout property to set DoSFilter.maxRequestMs with default of 60 seconds

This closes #4972

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-07 07:40:03 -05:00
Mark Payne b79987918a
NIFI-8380: Allow for an extensions.directory property to specify where to place downloaded files. Also fixed an issue that was encountered, when a Source Processor is scheduled for Primary Node Only but more than 1 task is set. In that case, even though only a single task will should be scheduled, an Exception was getting thrown because @OnScheduled methods of Processors were still called. To avoid this, moved the initialization of the dataflow outside of the creation of the dataflow so that initialization can be triggered only when appropriate.
NIFI-8380: Removed requirement in validation for working directory and extensions directory to exist; removed auto-creation of directories in validation

NIFI-8380: Fixed a few thrading bugs, so that if we have multiple threads trying to download/unpack extensions, we properly synchronize the unpacking and unpack into the correct sub-directory under the working directory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4950.
2021-04-07 13:28:12 +02:00
Pierre Villard f4ad658fae NIFI-8389 - Grok Reader - raw data strategy when no match
added allowable value in property
added docs in additionalDetails

This closes #4966.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-07 09:11:44 +02:00
Peter Turcsanyi 4b852ba7c8
NIFI-8390: Handle HBase namespaces in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4977.
2021-04-07 08:57:29 +02:00
Jose Luis Pedrosa fa3da2aa1c
There was some minor changes required to stop using deprecated APIs byu accumulo, inspecting the deprecated makes the fixes really simple
* writerFactory.createWriter receives an extra parameteres with a map, in the implementation it sends an empty one if not passed.
* tests were relying on  getConnector, that is replaced by createAccumuloClient

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4942.
2021-04-06 11:10:01 +02:00
exceptionfactory 64b12176b2
NIFI-8388 Upgrade Hazelcast from 4.0.1 to 4.2
- Removes dependency on Guava using ByteBuffer methods for serializing revision

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4967.
2021-04-05 11:46:32 +02:00
Jose Luis Pedrosa f21c1be60f
NIFI-8374 Refactor readOnly to accessMode in ApiModelProperty usages
This closes #4945

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-04-02 14:22:07 -04:00
Rob Fellows 87b561f026
[NIFI-8387] - Use the bulletins from the referencing components rathe… (#4969)
* [NIFI-8387] - Use the bulletins from the referencing components rather than making extraneous http calls to get them.

* Moving the spinner next to the Parameter/Variable lables

This closes #4969
2021-04-02 09:59:34 -04:00
Mark Payne 7db1b8d564
NIFI-8386: Ensure that we set (and merge) bulletins when creating AffectedComponent entities and ControllerService Reference entities
NIFI-8386: Addressed review feedback: removed unused call to determine permissions, null out bulletins in standalone mode if permissions not allowed. Also fixed automated tests that were failing due to changes

This closes #4955
2021-04-01 14:11:32 -04:00
Mark Payne 2edf5514b7 NIFI-8344: If configured to tail a file for some period of time post-rollover, ensure that we only consume lines that are fully written (i.e., end with a newline). Once we stop tailing that file post-rollover, consume any data from that file that has not yet been consumed, up to the end of the file, even if there is no newline.
This closes #4937.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-31 13:40:08 +02:00
exceptionfactory 89dda0b6a4
NIFI-8364 This closes #4953. Set static date for TestQuerySolr
- Static date for TestQuerySolr avoids the potential for missing values in facet range queries

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-30 16:05:04 -07:00
exceptionfactory 02ea103c6b
NIFI-8379 This closes #4949. Improved reliability and performance of DBCPServiceTest
- Removed unreliable H2 network connection test methods
- Removed duplicate and ignored test methods
- Reduced duplication of setting common service properties

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-30 07:24:35 -07:00