Commit Graph

5932 Commits

Author SHA1 Message Date
Kent Nguyen 0cff54097e NIFI-6332: Add Cache Control property to PutS3Object processor
Add new property 'Cache Control' to allow user to
set the cache-control http header on the S3 object.

This property is not required, and has no default value.

The implementation is similar to the Content-Type property,
except that this property does not allow Expression Language.

Update property description

Add support EL for cache-control property

This closes #4422.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-30 08:37:10 +02:00
Tamás Bunth 455f48fce4
NIFI-7640 Add documentation: temporary directory (#4414)
NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
2020-07-29 13:57:34 -07:00
Andy LoPresto 716ba992f5
NIFI-7669 Changed custom PBE AEAD algorithm to derive key once rather than on every encrypt/decrypt operation, leading to substantial performance gains.
Updated documentation.
Added unit tests.

NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.

NIFI-7669 Fixed failing test due to nifi.properties initialization.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4435.
2020-07-29 09:29:00 +02:00
Andy LoPresto 7f0416ee8b
NIFI-7680 Added convenience methods for creating XML DocumentBuilder instances.
Added unit tests.

NIFI-7680 Duplicated DocumentBuilder creation method in NotificationServiceManager to avoid nifi-bootstrap dependency on nifi-security-utils.
Explicitly added commons-lang3 to lib/bootstrap/ directory in nifi-assembly.

NIFI-7680 Reverted unnecessary dependency changes.
Added explicit dependencies where necessary.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4436
2020-07-28 17:02:42 -04:00
Karthikeyan Singaravelan cee6cfddf2
NIFI-7682: Check for request object to be null.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4437.
2020-07-28 21:17:46 +02:00
Mark Payne e926a17968
NIFI-7683: Fixed bug that resulted in a byte[] being allocated and held onto by a member variable that was unprotected. This caused the byte array to be modified by two different threads concurrently, which can potentially cause corruption of FlowFile's data
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4439
2020-07-28 14:52:57 -04:00
Andy LoPresto 90c9db8208
NIFI-7679 Changed flow fingerprint masking logic to derive a static key once (slowly) from nifi.sensitive.props.key and use a (fast) HMAC/SHA-256 operation during fingerprinting to resolve unacceptable delays.
Added unit tests.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4434.
2020-07-28 10:22:24 +02:00
Mark Payne 0b9f2fbe3b
NIFI-7678: Fixed memory leak, improved some logic that constantly created collections unnecessarily. Changed types of Collections being used to much more efficient implementations. Removed using Streams for performance-critical parts of application
NIFI-7678: Check if debug is enabled before logging message about processor.onTrigger because obtaining class name is expensive

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4431.
2020-07-27 13:28:15 +02:00
Peter Turcsanyi cdd766d649
NIFI-7671: Support Message Selector in ConsumeJMS processor
Also fixed some display names and variable names.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4432.
2020-07-27 12:50:44 +02:00
Andy LoPresto 7d20c03f89
NIFI-7638 Implemented custom nifi.sensitive.props.algorithm for AES-G/CM with Argon2 KDF.
Added documentation for encryption of flow sensitive values.
Added unit tests.

This closes #4427.
2020-07-24 18:11:37 -07:00
Bryan Bende 5cb8d24689
NIFI-7622 Use param context name from inside nested versioned PG when importing
This closes #4401.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-07-24 14:15:57 -07:00
Nathan Gough d846a74730
NIFI-7568 - Applied Kerberos mappings to authentication requests. Kerberos mappings should now be applied correctly in H2 database for username/password based login. Added tests.
Logout now deletes signing key by key ID rather than identity.
Validate token expiration now uses mapped identity instead, which allows logging of the mapped identity.
Updated delete key to expect only 0 or 1 keys deleted.

This closes #4416.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-07-24 13:28:01 -07:00
Andy LoPresto eeeda84474
NIFI-7122 Introduced improvements to EncryptContent processor.
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.

This closes #4421.

Co-authored-by: mtien <mtien.apache@gmail.com>
2020-07-24 12:31:39 -07:00
Pierre Villard 4b0d349a49
NIFI-7667 - add variable support to GetHBase properties (#4424) 2020-07-23 09:29:44 -04:00
Bryan Bende e9a443fe2e
NIFI-7521 Remove additional unused property from nifi proeprties
This closes #4327.
2020-07-23 08:14:39 -04:00
Mark Payne e371f4ac7c
NIFI-7521: Removed unused properties from nifi properties. Updated the defaults to use for File System's archive cleanup, flowfile checkpoint interval 2020-07-23 08:14:21 -04:00
Bryan Bende 99b3780ad9
NIFI-7373 Setting TTL on keys in Redis putAll
This closes #4217.
2020-07-23 07:39:15 -04:00
Mike Thomsen 0e61dbc9a0
NIFI-7373
Added new methods to DistributedMapCacheClient for bulk get and put. Updated HBase 1.1.2 clients.
Added HBase 2 support.
Added Redis support.
2020-07-23 07:38:59 -04:00
Pierre Villard 7f0b188be4 NIFI-7531 - changed RW to RO access requirement on autoload NARs directory 2020-07-22 14:31:48 -04:00
Bryan Bende 2c45a80ef4
NIFI-7517 Fixing unit test, checkstyle issue, and duplicate property
This closes #4324.
2020-07-22 14:20:09 -04:00
Mark Payne 43c6ecd3eb
NIFI-7517: When mapping a ProcessGroup to a VersionedProcessGroup, take into account FlowFile Concurrency and FlowFile Outbound Policy 2020-07-22 14:19:51 -04:00
Andy LoPresto 78d88b46ef
NIFI-7657 Introduced AccessTokenUnnecessaryException for authentication calls to HTTP NiFi server.
Added exception mapper.

NIFI-7657 Renamed exception & exception mapper to reflect scope of authentication not supported.
Registered exception mapper.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4418.
2020-07-22 14:35:09 +02:00
Andy LoPresto f80b844907
NIFI-7660 This closes #4419. Resolved failing unit test on Azul Zulu JDK 8 update 262.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-07-21 16:46:46 -07:00
Andy LoPresto 6297310a90
NIFI-7658 Lowered log severity to DEBUG when web request max content length is blank.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4415.
2020-07-21 13:42:43 +02:00
Mark Payne a473fc0373
NIFI-7633: Added FlowFileConcurrency of SINGLE_BATCH_PER_NODE to allow data to be pulled into a ProcessGroup as a batch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4405.
2020-07-20 12:02:54 +02:00
Pierre Villard 2e803e0e8f
NIFI-7552 - fix checkstyle 2020-07-17 17:03:47 +02:00
zhangcheng e18b4f0c75
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4243.
2020-07-17 14:39:30 +02:00
Joey Frazee a72c3d685c
NIFI-7372 Added test for ZooKeeperStateServer
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4241.
2020-07-17 14:25:56 +02:00
Pierre Villard 5da596ea8d
NIFI-7552 - fixed checkstyle issue 2020-07-17 13:53:13 +02:00
Pierre Villard 958d9c4cb6
NIFI-7493 - fix checkstyle issue 2020-07-17 12:22:47 +02:00
Vasily Makarov 950437b2d7
NIFI-6603 Add cancel button to the variables.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4366.
2020-07-17 11:54:57 +02:00
Mike Thomsen b8d51333d5
NIFI-7582 Added profile builds for kafka 0.9 and 0.10 so they can be removed from the convenience binary.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4364.
2020-07-17 11:38:42 +02:00
Mark Payne 7e09e0db33
NIFI-7493: When inferring schema for XML data, if we find a text element that also has attributes, infer it as a Record type, in order to match how the data will be read when using the XML Reader
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4375.
2020-07-17 11:35:10 +02:00
Peter Turcsanyi 21c3085d15
NIFI-7654: Deprecated Client Auth property on AMQP processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4412.
2020-07-17 11:29:49 +02:00
Bryan Bende f6c2824f74
NIFI-7608 Add CLI command to replace contents of a process group
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4392.
2020-07-17 11:21:28 +02:00
Mark Payne 44fc4d9f27
NIFI-7552: When Process Group is configured to transfer data in batch, add an attribute to each outbound FlowFile that indicates how many FlowFiles went to each port. Updated user guide to explain the new attributes.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4345.
2020-07-17 10:33:59 +02:00
Joe Witt 77078a85d9
NIFI-7304 resolving a contrib check issue with line length 2020-07-14 13:30:53 -07:00
markap14 f8a730568f
NIFI-7627: Fixed typo that showed the wrong case for Run Statuses in the ProcessorRunStatusDetailsDTO (#4400) 2020-07-14 15:44:52 -04:00
Andy LoPresto dbee774c5b
NIFI-7304 Removed default value for nifi.web.max.content.size.
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.

Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-07-14 10:42:00 -07:00
Andy LoPresto 0fa8776f4d
NIFI-7621 & NIFI-7614 Updated terminology throughout codebase and documentation.
Fixed unit test failures introduced from static imports during refactoring.

Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-07-14 10:39:28 -07:00
Peter Gyori 4f11e36260 NIFI-7635: StandardConfigurationContext.getProperty() gets the property descriptor and its default value from the component itself
This closes #4408.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-07-14 13:20:59 -04:00
Tamás Bunth b1e77019ac NIFI-6128 UnpackContent: Store unpacked file data
Tar format allows us to archive files with their original permission,
owner, group name and last modification time.

When unpacking with Tar unpacker, these information are stored in new
attributes with names: "file.inner.*". This way, it preserves backward
compatibility when using parallel with GetFile processor (which stores
information in "file.*").

NIFI-6128 Tar unpackContent: assert date of last modification of content is a valid date format.

NIFI-6128 UnpackContent: use original attributes

In case of tar format:
- Use "file.*" attributes instead of "file.inner.*" (which eventually
  lead to overwrite if tar had been fetched with GetFile)
- Store file permission in "rwx" format, instead of integer
  representation.
- Also replace SimpleDateFormat with DateTimeFormatter.

Replace generic error with IllegalArgumentException
Also refactor: move permission string to top of file as static constant.

Update test

Remove 'file systems' text

Unpackcontent: Fill file.creationTime attribute
which always holds the very same value as file.lastModifiedTime.

This closes #4370.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-14 18:04:56 +02:00
Tamás Bunth 72666a9c7e NIFI-7589 Fix path value when unpacking tar
Refactor: remove variable reassignment

This closes #4371.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-14 17:32:51 +02:00
Mike bb5e9fb00b
NIFI-7497 Removed a few style check bugs that crept up in the last commit. (#4402) 2020-07-13 09:34:24 -04:00
Cuong Nguyen c7a235f854 NIFI-7629: Make property PROJECT_ID evaluate EL on processors ConsumeGCPubSub & PublishGCPubSub 2020-07-13 08:54:41 -04:00
Tamas Palfy 19a84dd3dd NIFI-7581 Added ControllerService-based credential settings support for ADLS processors
NIFI-7581 Separated Controller Service for providing Azure credentials for ADLS (ADLSCredentialsControllerService) form the one that does the same for Blob storages (AzureStorageCredentialsDetails). (This was done due to the considerable difference in the APIs of the libraries used to connect to both.)
NIFI-7581 Fix: Register controller service in META-INF. Minor fixes.
NIFI-7581 Minor changes (documentation, type etc.)
NIFI-7581 Updated integration tests.
NIFI-7581 Minor changes (renaming).

This closes #4369.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-12 11:56:31 +02:00
Mike Thomsen 8d53f5d0c9
NIFI-7497 Removed a few style check bugs that crept up in the last commit. 2020-07-10 17:12:27 -04:00
Peter Turcsanyi ae877b9908
NIFI-7591: Allow PutS3Object to post to AWS Snowball
Added properties to enable/disable chunked encoding and path-style access
for endpoints that do not support chunked encoding / only support path-style access.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4386.
2020-07-10 18:24:44 +02:00
Mark Payne 1fc25db47c
NIFI-7469: Updated RepositoryRecord to include flag indicating whether or not the content of a FlowFile was modified. This allows us to explicitly keep track of this state rather than implying it (potentially incorrectly).
This closes #4399.
2020-07-10 10:44:51 -04:00
neptunesalt ee91341ec3
NIFI-7497 Adding support for AWS Credentials Assume Role to be able to set the STS Endpoint
NIFI-7497 Updating property description per comment

This closes #4309

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-07-10 09:19:58 -04:00