Commit Graph

5947 Commits

Author SHA1 Message Date
Joe Witt 536dbb72bb
NIFI-7703 updated all commons codec references to 1.14
This closes #4448.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-08-04 12:11:57 -07:00
shreeju 24d4ad348a
NIFI-7702 This closes #4447. validate requestentity not null
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-08-04 09:48:54 -07:00
Mike Thomsen 0861b2f632
NIFI-7605 Removed user-agent default value so no header will be sent by default.
Added and updated unit tests.

This closes #4428.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-08-03 17:23:56 -07:00
Peter Turcsanyi c980b64bf5
NIFI-7340: Adding ListAzureDataLakeStorage
Also added validator for Directory Name property in AbstractAzureDataLakeStorageProcessor
Fix Tracking Entities strategy: use milliseconds for lastModified

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4438.
2020-08-03 19:27:21 +02:00
Peter Turcsanyi 7b4cce9e21
NIFI-6312: Improved connection handling in AMQP processors
Disable connection automatic recovery which can lead to uncontrolled/stale threads. Handle the recovery in the processors instead.
Use poisoning in case of errors, then discarding and recreating the poisoned consumer/publisher.
NIFI-6312: Use conventional exception handling instead of poisoning
Use component logger in workers.
Remove basicNack()/basicReject() calls as they are not needed because all unacknowledged messages will be redelivered.
NIFI-6312: Further improve exception handling and error logging.
NIFI-6312: Fix consumer closing in previous commit
NIFI-6312: Use custom executor with a single thread (no more is used by the processor)

Reviewed by tamas palfy and simon bence

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-08-03 09:43:25 -07:00
Kent Nguyen ddb304a927 NIFI-7664 Add Content Disposition property to PutS3Object processor
Add new property 'Content Disposition' to allow user
to set the content-disposition http header on the S3 object.

Allowed values are 'inline' (default) and 'attachment'.
If 'attachment' is selected, the filename will be set to the S3 Object key.

Remove default value and keep backward compatibility
Update fetchS3Object filename attribute settin
Update constant names
Update order of if-else condition
NIFI-7664 Update condition in FetchS3Processor
NIFI-7664 Undo the unexpected indent
NIFI-7664 Update international chars unit test
NIFI-7664 Set fetchS3 file path name
NIFI-7664 Update code style

This closes #4423.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-08-03 18:27:35 +02:00
Mike Thomsen 255d0c0f19
NIFI-7604 Moved nifi-elasticsearch-5 NAR to a profile to save space.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4389
2020-07-31 16:12:20 -04:00
Karthikeyan Singaravelan 6a497aa8c3 NIFI-7691: Validate request object to be not null. 2020-07-31 15:31:32 -04:00
Mark Payne 2de4962ff1 NIFI-7592: Fixed typo in comment in pom.xml and changed headless profile to not active by default.
This closes #4376.
2020-07-31 14:11:48 -04:00
Matthew Burgess 1b91a6caf3 NIFI-7592: Allow NiFi to be started without a GUI/REST interface
- Rename MiNiFi references to Headless NiFi, added profile to build headless assembly

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-07-31 14:09:12 -04:00
Jan Hentschel 84586a6772 NIFI-7315 Fixed typo in bootstrap.conf for 17th argument 2020-07-31 11:17:17 -04:00
Ruben Laguna 30b328ceab
NIFI-7694 - Move misplaced text related to oidc
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4445.
2020-07-31 11:00:14 +02:00
Mike Thomsen 83fd4ca291
NIFI-7526 Updated code to use the new helper library for OkHttp.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4331.
2020-07-31 10:51:15 +02:00
Joey Frazee 633fd8883d
NIFI-7693: Bump aspectjweaver version to resolve BeanCreationException on Java 11 2018-09-25
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4444.
2020-07-31 10:44:47 +02:00
Andy LoPresto ac60bf6507
NIFI-7687 Improved JVM version handling in RunNiFi and OSUtils.
Added unit tests.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4442.
2020-07-30 09:47:14 +02:00
Kent Nguyen 0cff54097e NIFI-6332: Add Cache Control property to PutS3Object processor
Add new property 'Cache Control' to allow user to
set the cache-control http header on the S3 object.

This property is not required, and has no default value.

The implementation is similar to the Content-Type property,
except that this property does not allow Expression Language.

Update property description

Add support EL for cache-control property

This closes #4422.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-30 08:37:10 +02:00
Tamás Bunth 455f48fce4
NIFI-7640 Add documentation: temporary directory (#4414)
NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
2020-07-29 13:57:34 -07:00
Andy LoPresto 716ba992f5
NIFI-7669 Changed custom PBE AEAD algorithm to derive key once rather than on every encrypt/decrypt operation, leading to substantial performance gains.
Updated documentation.
Added unit tests.

NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.

NIFI-7669 Fixed failing test due to nifi.properties initialization.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4435.
2020-07-29 09:29:00 +02:00
Andy LoPresto 7f0416ee8b
NIFI-7680 Added convenience methods for creating XML DocumentBuilder instances.
Added unit tests.

NIFI-7680 Duplicated DocumentBuilder creation method in NotificationServiceManager to avoid nifi-bootstrap dependency on nifi-security-utils.
Explicitly added commons-lang3 to lib/bootstrap/ directory in nifi-assembly.

NIFI-7680 Reverted unnecessary dependency changes.
Added explicit dependencies where necessary.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4436
2020-07-28 17:02:42 -04:00
Karthikeyan Singaravelan cee6cfddf2
NIFI-7682: Check for request object to be null.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4437.
2020-07-28 21:17:46 +02:00
Mark Payne e926a17968
NIFI-7683: Fixed bug that resulted in a byte[] being allocated and held onto by a member variable that was unprotected. This caused the byte array to be modified by two different threads concurrently, which can potentially cause corruption of FlowFile's data
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4439
2020-07-28 14:52:57 -04:00
Andy LoPresto 90c9db8208
NIFI-7679 Changed flow fingerprint masking logic to derive a static key once (slowly) from nifi.sensitive.props.key and use a (fast) HMAC/SHA-256 operation during fingerprinting to resolve unacceptable delays.
Added unit tests.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4434.
2020-07-28 10:22:24 +02:00
Mark Payne 0b9f2fbe3b
NIFI-7678: Fixed memory leak, improved some logic that constantly created collections unnecessarily. Changed types of Collections being used to much more efficient implementations. Removed using Streams for performance-critical parts of application
NIFI-7678: Check if debug is enabled before logging message about processor.onTrigger because obtaining class name is expensive

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4431.
2020-07-27 13:28:15 +02:00
Peter Turcsanyi cdd766d649
NIFI-7671: Support Message Selector in ConsumeJMS processor
Also fixed some display names and variable names.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4432.
2020-07-27 12:50:44 +02:00
Andy LoPresto 7d20c03f89
NIFI-7638 Implemented custom nifi.sensitive.props.algorithm for AES-G/CM with Argon2 KDF.
Added documentation for encryption of flow sensitive values.
Added unit tests.

This closes #4427.
2020-07-24 18:11:37 -07:00
Bryan Bende 5cb8d24689
NIFI-7622 Use param context name from inside nested versioned PG when importing
This closes #4401.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-07-24 14:15:57 -07:00
Nathan Gough d846a74730
NIFI-7568 - Applied Kerberos mappings to authentication requests. Kerberos mappings should now be applied correctly in H2 database for username/password based login. Added tests.
Logout now deletes signing key by key ID rather than identity.
Validate token expiration now uses mapped identity instead, which allows logging of the mapped identity.
Updated delete key to expect only 0 or 1 keys deleted.

This closes #4416.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-07-24 13:28:01 -07:00
Andy LoPresto eeeda84474
NIFI-7122 Introduced improvements to EncryptContent processor.
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.

This closes #4421.

Co-authored-by: mtien <mtien.apache@gmail.com>
2020-07-24 12:31:39 -07:00
Pierre Villard 4b0d349a49
NIFI-7667 - add variable support to GetHBase properties (#4424) 2020-07-23 09:29:44 -04:00
Bryan Bende e9a443fe2e
NIFI-7521 Remove additional unused property from nifi proeprties
This closes #4327.
2020-07-23 08:14:39 -04:00
Mark Payne e371f4ac7c
NIFI-7521: Removed unused properties from nifi properties. Updated the defaults to use for File System's archive cleanup, flowfile checkpoint interval 2020-07-23 08:14:21 -04:00
Bryan Bende 99b3780ad9
NIFI-7373 Setting TTL on keys in Redis putAll
This closes #4217.
2020-07-23 07:39:15 -04:00
Mike Thomsen 0e61dbc9a0
NIFI-7373
Added new methods to DistributedMapCacheClient for bulk get and put. Updated HBase 1.1.2 clients.
Added HBase 2 support.
Added Redis support.
2020-07-23 07:38:59 -04:00
Pierre Villard 7f0b188be4 NIFI-7531 - changed RW to RO access requirement on autoload NARs directory 2020-07-22 14:31:48 -04:00
Bryan Bende 2c45a80ef4
NIFI-7517 Fixing unit test, checkstyle issue, and duplicate property
This closes #4324.
2020-07-22 14:20:09 -04:00
Mark Payne 43c6ecd3eb
NIFI-7517: When mapping a ProcessGroup to a VersionedProcessGroup, take into account FlowFile Concurrency and FlowFile Outbound Policy 2020-07-22 14:19:51 -04:00
Andy LoPresto 78d88b46ef
NIFI-7657 Introduced AccessTokenUnnecessaryException for authentication calls to HTTP NiFi server.
Added exception mapper.

NIFI-7657 Renamed exception & exception mapper to reflect scope of authentication not supported.
Registered exception mapper.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4418.
2020-07-22 14:35:09 +02:00
Andy LoPresto f80b844907
NIFI-7660 This closes #4419. Resolved failing unit test on Azul Zulu JDK 8 update 262.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-07-21 16:46:46 -07:00
Andy LoPresto 6297310a90
NIFI-7658 Lowered log severity to DEBUG when web request max content length is blank.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4415.
2020-07-21 13:42:43 +02:00
Mark Payne a473fc0373
NIFI-7633: Added FlowFileConcurrency of SINGLE_BATCH_PER_NODE to allow data to be pulled into a ProcessGroup as a batch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4405.
2020-07-20 12:02:54 +02:00
Pierre Villard 2e803e0e8f
NIFI-7552 - fix checkstyle 2020-07-17 17:03:47 +02:00
zhangcheng e18b4f0c75
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4243.
2020-07-17 14:39:30 +02:00
Joey Frazee a72c3d685c
NIFI-7372 Added test for ZooKeeperStateServer
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4241.
2020-07-17 14:25:56 +02:00
Pierre Villard 5da596ea8d
NIFI-7552 - fixed checkstyle issue 2020-07-17 13:53:13 +02:00
Pierre Villard 958d9c4cb6
NIFI-7493 - fix checkstyle issue 2020-07-17 12:22:47 +02:00
Vasily Makarov 950437b2d7
NIFI-6603 Add cancel button to the variables.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4366.
2020-07-17 11:54:57 +02:00
Mike Thomsen b8d51333d5
NIFI-7582 Added profile builds for kafka 0.9 and 0.10 so they can be removed from the convenience binary.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4364.
2020-07-17 11:38:42 +02:00
Mark Payne 7e09e0db33
NIFI-7493: When inferring schema for XML data, if we find a text element that also has attributes, infer it as a Record type, in order to match how the data will be read when using the XML Reader
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4375.
2020-07-17 11:35:10 +02:00
Peter Turcsanyi 21c3085d15
NIFI-7654: Deprecated Client Auth property on AMQP processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4412.
2020-07-17 11:29:49 +02:00
Bryan Bende f6c2824f74
NIFI-7608 Add CLI command to replace contents of a process group
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4392.
2020-07-17 11:21:28 +02:00