Commit Graph

3967 Commits

Author SHA1 Message Date
bibistroc 51091a5fca
NIFI-7338 This closes #4169. Added comment about com.hierynomus.smbj.SMBClient syncronicity
- added file.size attribute on GetSmbFile

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-08-10 09:44:52 -07:00
si 71d155e874
NIFI-7338 fixed provenance, top level dir,
chatty smbj log, pw validation
update accessmask to be more restrictive

Author:    si <si.sun@roche.com>

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-08-10 09:43:43 -07:00
Joe Witt 58e324e9f1
NIFI-7522 bumping from Apache Kafka 2.5 to Apache Kafka 2.6
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4462.
2020-08-10 09:55:53 +02:00
Mark Payne 0f18c5966e NIFI-7592: Updated nifi system test to no longer include nifi-server-nar and also changed a unit test to an integration test because it starts an embedded server on a specific port
This closes #4459.
2020-08-07 12:51:17 -04:00
Matthew Burgess 55adb11f06 Revert "NIFI-7592: Allow NiFi to be started without a GUI/REST interface"
This reverts commit 1b91a6caf3.
2020-08-07 12:50:48 -04:00
Pierre Villard 27e7d0b02a
NIFI-7707 - This closes #4451. fixed Publish GCP PubSub processor for binary data
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-08-07 08:14:19 -07:00
Tamas Palfy 3ef566f7da NIFI-7663 Added option for emptying all queues in a process group. Available from context menu.
NIFI-7663 Minor changes (variable name refactor, javadoc, GUI message). Merging Drop All Flowfiles responses across all nodes in a cluster.

NIFI-7663 Reloading the canvas after completing a Drop All Flowfiles request.

NIFI-7663 Fixed typos.

This closes #4425.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-08-07 16:23:00 +02:00
Matthew Burgess 6488db1376
NIFI-7572: Force ScriptedTransformRecord to recompile the script when started
This closes #4458.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-08-06 17:40:12 -07:00
Peter Gyori 5cc7c0485a NIFI-7708: Add Azure SDK for Java (and transitive dependencies) to LICENCE and NOTICE files
This closes #4452

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-08-06 10:55:31 -07:00
Mark Payne eca7f153d0
NIFI-7706, NIFI-5702: Allow NiFi to keep FlowFiles if their queue is unknown. This way, if a Flow is inadvertently removed, updated, etc., and NiFi is restarted, the data will not be dropped by default. The old mechanism of dropping data is exposed via a property
This closes #4454.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-08-06 10:55:56 -04:00
Scott Aslan 6596fb1f87
[NIFI-7705] update fronted deps
make variable and parameters tables position relative

Signed-off-by: Rob Fellows <rob.fellows@gmail.com>

This closes #4449
2020-08-06 10:48:24 -04:00
Bryan Bende 93e5abf274
NIFI-7522 Updating LICENSE and NOTICE for Kafka 2.5 NAR
This closes #4383.
2020-08-04 16:59:54 -04:00
Mathieu 1df7025ae8
NIFI-7522 : Add Kafka 2.5 Processors set 2020-08-04 16:59:39 -04:00
Joe Witt 536dbb72bb
NIFI-7703 updated all commons codec references to 1.14
This closes #4448.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-08-04 12:11:57 -07:00
shreeju 24d4ad348a
NIFI-7702 This closes #4447. validate requestentity not null
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-08-04 09:48:54 -07:00
Mike Thomsen 0861b2f632
NIFI-7605 Removed user-agent default value so no header will be sent by default.
Added and updated unit tests.

This closes #4428.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-08-03 17:23:56 -07:00
Peter Turcsanyi c980b64bf5
NIFI-7340: Adding ListAzureDataLakeStorage
Also added validator for Directory Name property in AbstractAzureDataLakeStorageProcessor
Fix Tracking Entities strategy: use milliseconds for lastModified

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4438.
2020-08-03 19:27:21 +02:00
Peter Turcsanyi 7b4cce9e21
NIFI-6312: Improved connection handling in AMQP processors
Disable connection automatic recovery which can lead to uncontrolled/stale threads. Handle the recovery in the processors instead.
Use poisoning in case of errors, then discarding and recreating the poisoned consumer/publisher.
NIFI-6312: Use conventional exception handling instead of poisoning
Use component logger in workers.
Remove basicNack()/basicReject() calls as they are not needed because all unacknowledged messages will be redelivered.
NIFI-6312: Further improve exception handling and error logging.
NIFI-6312: Fix consumer closing in previous commit
NIFI-6312: Use custom executor with a single thread (no more is used by the processor)

Reviewed by tamas palfy and simon bence

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-08-03 09:43:25 -07:00
Kent Nguyen ddb304a927 NIFI-7664 Add Content Disposition property to PutS3Object processor
Add new property 'Content Disposition' to allow user
to set the content-disposition http header on the S3 object.

Allowed values are 'inline' (default) and 'attachment'.
If 'attachment' is selected, the filename will be set to the S3 Object key.

Remove default value and keep backward compatibility
Update fetchS3Object filename attribute settin
Update constant names
Update order of if-else condition
NIFI-7664 Update condition in FetchS3Processor
NIFI-7664 Undo the unexpected indent
NIFI-7664 Update international chars unit test
NIFI-7664 Set fetchS3 file path name
NIFI-7664 Update code style

This closes #4423.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-08-03 18:27:35 +02:00
Karthikeyan Singaravelan 6a497aa8c3 NIFI-7691: Validate request object to be not null. 2020-07-31 15:31:32 -04:00
Matthew Burgess 1b91a6caf3 NIFI-7592: Allow NiFi to be started without a GUI/REST interface
- Rename MiNiFi references to Headless NiFi, added profile to build headless assembly

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-07-31 14:09:12 -04:00
Jan Hentschel 84586a6772 NIFI-7315 Fixed typo in bootstrap.conf for 17th argument 2020-07-31 11:17:17 -04:00
Mike Thomsen 83fd4ca291
NIFI-7526 Updated code to use the new helper library for OkHttp.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4331.
2020-07-31 10:51:15 +02:00
Joey Frazee 633fd8883d
NIFI-7693: Bump aspectjweaver version to resolve BeanCreationException on Java 11 2018-09-25
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4444.
2020-07-31 10:44:47 +02:00
Kent Nguyen 0cff54097e NIFI-6332: Add Cache Control property to PutS3Object processor
Add new property 'Cache Control' to allow user to
set the cache-control http header on the S3 object.

This property is not required, and has no default value.

The implementation is similar to the Content-Type property,
except that this property does not allow Expression Language.

Update property description

Add support EL for cache-control property

This closes #4422.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-30 08:37:10 +02:00
Tamás Bunth 455f48fce4
NIFI-7640 Add documentation: temporary directory (#4414)
NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
2020-07-29 13:57:34 -07:00
Andy LoPresto 716ba992f5
NIFI-7669 Changed custom PBE AEAD algorithm to derive key once rather than on every encrypt/decrypt operation, leading to substantial performance gains.
Updated documentation.
Added unit tests.

NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.

NIFI-7669 Fixed failing test due to nifi.properties initialization.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4435.
2020-07-29 09:29:00 +02:00
Andy LoPresto 7f0416ee8b
NIFI-7680 Added convenience methods for creating XML DocumentBuilder instances.
Added unit tests.

NIFI-7680 Duplicated DocumentBuilder creation method in NotificationServiceManager to avoid nifi-bootstrap dependency on nifi-security-utils.
Explicitly added commons-lang3 to lib/bootstrap/ directory in nifi-assembly.

NIFI-7680 Reverted unnecessary dependency changes.
Added explicit dependencies where necessary.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4436
2020-07-28 17:02:42 -04:00
Karthikeyan Singaravelan cee6cfddf2
NIFI-7682: Check for request object to be null.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4437.
2020-07-28 21:17:46 +02:00
Mark Payne e926a17968
NIFI-7683: Fixed bug that resulted in a byte[] being allocated and held onto by a member variable that was unprotected. This caused the byte array to be modified by two different threads concurrently, which can potentially cause corruption of FlowFile's data
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4439
2020-07-28 14:52:57 -04:00
Andy LoPresto 90c9db8208
NIFI-7679 Changed flow fingerprint masking logic to derive a static key once (slowly) from nifi.sensitive.props.key and use a (fast) HMAC/SHA-256 operation during fingerprinting to resolve unacceptable delays.
Added unit tests.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4434.
2020-07-28 10:22:24 +02:00
Mark Payne 0b9f2fbe3b
NIFI-7678: Fixed memory leak, improved some logic that constantly created collections unnecessarily. Changed types of Collections being used to much more efficient implementations. Removed using Streams for performance-critical parts of application
NIFI-7678: Check if debug is enabled before logging message about processor.onTrigger because obtaining class name is expensive

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4431.
2020-07-27 13:28:15 +02:00
Peter Turcsanyi cdd766d649
NIFI-7671: Support Message Selector in ConsumeJMS processor
Also fixed some display names and variable names.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4432.
2020-07-27 12:50:44 +02:00
Andy LoPresto 7d20c03f89
NIFI-7638 Implemented custom nifi.sensitive.props.algorithm for AES-G/CM with Argon2 KDF.
Added documentation for encryption of flow sensitive values.
Added unit tests.

This closes #4427.
2020-07-24 18:11:37 -07:00
Bryan Bende 5cb8d24689
NIFI-7622 Use param context name from inside nested versioned PG when importing
This closes #4401.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-07-24 14:15:57 -07:00
Nathan Gough d846a74730
NIFI-7568 - Applied Kerberos mappings to authentication requests. Kerberos mappings should now be applied correctly in H2 database for username/password based login. Added tests.
Logout now deletes signing key by key ID rather than identity.
Validate token expiration now uses mapped identity instead, which allows logging of the mapped identity.
Updated delete key to expect only 0 or 1 keys deleted.

This closes #4416.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-07-24 13:28:01 -07:00
Andy LoPresto eeeda84474
NIFI-7122 Introduced improvements to EncryptContent processor.
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.

This closes #4421.

Co-authored-by: mtien <mtien.apache@gmail.com>
2020-07-24 12:31:39 -07:00
Pierre Villard 4b0d349a49
NIFI-7667 - add variable support to GetHBase properties (#4424) 2020-07-23 09:29:44 -04:00
Bryan Bende e9a443fe2e
NIFI-7521 Remove additional unused property from nifi proeprties
This closes #4327.
2020-07-23 08:14:39 -04:00
Mark Payne e371f4ac7c
NIFI-7521: Removed unused properties from nifi properties. Updated the defaults to use for File System's archive cleanup, flowfile checkpoint interval 2020-07-23 08:14:21 -04:00
Bryan Bende 99b3780ad9
NIFI-7373 Setting TTL on keys in Redis putAll
This closes #4217.
2020-07-23 07:39:15 -04:00
Mike Thomsen 0e61dbc9a0
NIFI-7373
Added new methods to DistributedMapCacheClient for bulk get and put. Updated HBase 1.1.2 clients.
Added HBase 2 support.
Added Redis support.
2020-07-23 07:38:59 -04:00
Pierre Villard 7f0b188be4 NIFI-7531 - changed RW to RO access requirement on autoload NARs directory 2020-07-22 14:31:48 -04:00
Bryan Bende 2c45a80ef4
NIFI-7517 Fixing unit test, checkstyle issue, and duplicate property
This closes #4324.
2020-07-22 14:20:09 -04:00
Mark Payne 43c6ecd3eb
NIFI-7517: When mapping a ProcessGroup to a VersionedProcessGroup, take into account FlowFile Concurrency and FlowFile Outbound Policy 2020-07-22 14:19:51 -04:00
Andy LoPresto 78d88b46ef
NIFI-7657 Introduced AccessTokenUnnecessaryException for authentication calls to HTTP NiFi server.
Added exception mapper.

NIFI-7657 Renamed exception & exception mapper to reflect scope of authentication not supported.
Registered exception mapper.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4418.
2020-07-22 14:35:09 +02:00
Andy LoPresto f80b844907
NIFI-7660 This closes #4419. Resolved failing unit test on Azul Zulu JDK 8 update 262.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-07-21 16:46:46 -07:00
Andy LoPresto 6297310a90
NIFI-7658 Lowered log severity to DEBUG when web request max content length is blank.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4415.
2020-07-21 13:42:43 +02:00
Mark Payne a473fc0373
NIFI-7633: Added FlowFileConcurrency of SINGLE_BATCH_PER_NODE to allow data to be pulled into a ProcessGroup as a batch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4405.
2020-07-20 12:02:54 +02:00
zhangcheng e18b4f0c75
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4243.
2020-07-17 14:39:30 +02:00