3947 Commits

Author SHA1 Message Date
Matthew Burgess
1b91a6caf3 NIFI-7592: Allow NiFi to be started without a GUI/REST interface
- Rename MiNiFi references to Headless NiFi, added profile to build headless assembly

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-07-31 14:09:12 -04:00
Jan Hentschel
84586a6772 NIFI-7315 Fixed typo in bootstrap.conf for 17th argument 2020-07-31 11:17:17 -04:00
Mike Thomsen
83fd4ca291
NIFI-7526 Updated code to use the new helper library for OkHttp.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4331.
2020-07-31 10:51:15 +02:00
Joey Frazee
633fd8883d
NIFI-7693: Bump aspectjweaver version to resolve BeanCreationException on Java 11 2018-09-25
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4444.
2020-07-31 10:44:47 +02:00
Kent Nguyen
0cff54097e NIFI-6332: Add Cache Control property to PutS3Object processor
Add new property 'Cache Control' to allow user to
set the cache-control http header on the S3 object.

This property is not required, and has no default value.

The implementation is similar to the Content-Type property,
except that this property does not allow Expression Language.

Update property description

Add support EL for cache-control property

This closes #4422.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-30 08:37:10 +02:00
Tamás Bunth
455f48fce4
NIFI-7640 Add documentation: temporary directory (#4414)
NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
2020-07-29 13:57:34 -07:00
Andy LoPresto
716ba992f5
NIFI-7669 Changed custom PBE AEAD algorithm to derive key once rather than on every encrypt/decrypt operation, leading to substantial performance gains.
Updated documentation.
Added unit tests.

NIFI-7669 Moved time-based encryption tests to integration tests to avoid running during CI builds.

NIFI-7669 Fixed failing test due to nifi.properties initialization.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4435.
2020-07-29 09:29:00 +02:00
Andy LoPresto
7f0416ee8b
NIFI-7680 Added convenience methods for creating XML DocumentBuilder instances.
Added unit tests.

NIFI-7680 Duplicated DocumentBuilder creation method in NotificationServiceManager to avoid nifi-bootstrap dependency on nifi-security-utils.
Explicitly added commons-lang3 to lib/bootstrap/ directory in nifi-assembly.

NIFI-7680 Reverted unnecessary dependency changes.
Added explicit dependencies where necessary.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4436
2020-07-28 17:02:42 -04:00
Karthikeyan Singaravelan
cee6cfddf2
NIFI-7682: Check for request object to be null.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4437.
2020-07-28 21:17:46 +02:00
Mark Payne
e926a17968
NIFI-7683: Fixed bug that resulted in a byte[] being allocated and held onto by a member variable that was unprotected. This caused the byte array to be modified by two different threads concurrently, which can potentially cause corruption of FlowFile's data
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4439
2020-07-28 14:52:57 -04:00
Andy LoPresto
90c9db8208
NIFI-7679 Changed flow fingerprint masking logic to derive a static key once (slowly) from nifi.sensitive.props.key and use a (fast) HMAC/SHA-256 operation during fingerprinting to resolve unacceptable delays.
Added unit tests.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4434.
2020-07-28 10:22:24 +02:00
Mark Payne
0b9f2fbe3b
NIFI-7678: Fixed memory leak, improved some logic that constantly created collections unnecessarily. Changed types of Collections being used to much more efficient implementations. Removed using Streams for performance-critical parts of application
NIFI-7678: Check if debug is enabled before logging message about processor.onTrigger because obtaining class name is expensive

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4431.
2020-07-27 13:28:15 +02:00
Peter Turcsanyi
cdd766d649
NIFI-7671: Support Message Selector in ConsumeJMS processor
Also fixed some display names and variable names.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4432.
2020-07-27 12:50:44 +02:00
Andy LoPresto
7d20c03f89
NIFI-7638 Implemented custom nifi.sensitive.props.algorithm for AES-G/CM with Argon2 KDF.
Added documentation for encryption of flow sensitive values.
Added unit tests.

This closes #4427.
2020-07-24 18:11:37 -07:00
Bryan Bende
5cb8d24689
NIFI-7622 Use param context name from inside nested versioned PG when importing
This closes #4401.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-07-24 14:15:57 -07:00
Nathan Gough
d846a74730
NIFI-7568 - Applied Kerberos mappings to authentication requests. Kerberos mappings should now be applied correctly in H2 database for username/password based login. Added tests.
Logout now deletes signing key by key ID rather than identity.
Validate token expiration now uses mapped identity instead, which allows logging of the mapped identity.
Updated delete key to expect only 0 or 1 keys deleted.

This closes #4416.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-07-24 13:28:01 -07:00
Andy LoPresto
eeeda84474
NIFI-7122 Introduced improvements to EncryptContent processor.
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.

This closes #4421.

Co-authored-by: mtien <mtien.apache@gmail.com>
2020-07-24 12:31:39 -07:00
Pierre Villard
4b0d349a49
NIFI-7667 - add variable support to GetHBase properties (#4424) 2020-07-23 09:29:44 -04:00
Bryan Bende
e9a443fe2e
NIFI-7521 Remove additional unused property from nifi proeprties
This closes #4327.
2020-07-23 08:14:39 -04:00
Mark Payne
e371f4ac7c
NIFI-7521: Removed unused properties from nifi properties. Updated the defaults to use for File System's archive cleanup, flowfile checkpoint interval 2020-07-23 08:14:21 -04:00
Bryan Bende
99b3780ad9
NIFI-7373 Setting TTL on keys in Redis putAll
This closes #4217.
2020-07-23 07:39:15 -04:00
Mike Thomsen
0e61dbc9a0
NIFI-7373
Added new methods to DistributedMapCacheClient for bulk get and put. Updated HBase 1.1.2 clients.
Added HBase 2 support.
Added Redis support.
2020-07-23 07:38:59 -04:00
Pierre Villard
7f0b188be4 NIFI-7531 - changed RW to RO access requirement on autoload NARs directory 2020-07-22 14:31:48 -04:00
Bryan Bende
2c45a80ef4
NIFI-7517 Fixing unit test, checkstyle issue, and duplicate property
This closes #4324.
2020-07-22 14:20:09 -04:00
Mark Payne
43c6ecd3eb
NIFI-7517: When mapping a ProcessGroup to a VersionedProcessGroup, take into account FlowFile Concurrency and FlowFile Outbound Policy 2020-07-22 14:19:51 -04:00
Andy LoPresto
78d88b46ef
NIFI-7657 Introduced AccessTokenUnnecessaryException for authentication calls to HTTP NiFi server.
Added exception mapper.

NIFI-7657 Renamed exception & exception mapper to reflect scope of authentication not supported.
Registered exception mapper.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4418.
2020-07-22 14:35:09 +02:00
Andy LoPresto
f80b844907
NIFI-7660 This closes #4419. Resolved failing unit test on Azul Zulu JDK 8 update 262.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-07-21 16:46:46 -07:00
Andy LoPresto
6297310a90
NIFI-7658 Lowered log severity to DEBUG when web request max content length is blank.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4415.
2020-07-21 13:42:43 +02:00
Mark Payne
a473fc0373
NIFI-7633: Added FlowFileConcurrency of SINGLE_BATCH_PER_NODE to allow data to be pulled into a ProcessGroup as a batch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4405.
2020-07-20 12:02:54 +02:00
zhangcheng
e18b4f0c75
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the DataTypeUtils.toString(clob/nClob,(String) null, StandardCharsets.UTF_8)) method to get the String
NIFI-7410 Update JdbcCommon.java when javaSqlType is CLOB or NCLOB in convertToAvroStream method, use the CharacterStream rto read the value of CLOB
NIFI-7410 Add a unit test. validate if it's unreadable when the clob value is Chinese, Japanese, and Korean.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4243.
2020-07-17 14:39:30 +02:00
Joey Frazee
a72c3d685c
NIFI-7372 Added test for ZooKeeperStateServer
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4241.
2020-07-17 14:25:56 +02:00
Pierre Villard
5da596ea8d
NIFI-7552 - fixed checkstyle issue 2020-07-17 13:53:13 +02:00
Pierre Villard
958d9c4cb6
NIFI-7493 - fix checkstyle issue 2020-07-17 12:22:47 +02:00
Vasily Makarov
950437b2d7
NIFI-6603 Add cancel button to the variables.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4366.
2020-07-17 11:54:57 +02:00
Mark Payne
7e09e0db33
NIFI-7493: When inferring schema for XML data, if we find a text element that also has attributes, infer it as a Record type, in order to match how the data will be read when using the XML Reader
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4375.
2020-07-17 11:35:10 +02:00
Peter Turcsanyi
21c3085d15
NIFI-7654: Deprecated Client Auth property on AMQP processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4412.
2020-07-17 11:29:49 +02:00
Mark Payne
44fc4d9f27
NIFI-7552: When Process Group is configured to transfer data in batch, add an attribute to each outbound FlowFile that indicates how many FlowFiles went to each port. Updated user guide to explain the new attributes.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4345.
2020-07-17 10:33:59 +02:00
Joe Witt
77078a85d9
NIFI-7304 resolving a contrib check issue with line length 2020-07-14 13:30:53 -07:00
markap14
f8a730568f
NIFI-7627: Fixed typo that showed the wrong case for Run Statuses in the ProcessorRunStatusDetailsDTO (#4400) 2020-07-14 15:44:52 -04:00
Andy LoPresto
dbee774c5b
NIFI-7304 Removed default value for nifi.web.max.content.size.
Added Bundle#toString() method.
Refactored implementation of filter addition logic.
Added logging.
Added unit tests to check for filter enablement.
Introduced content-length exception handling in StandardPublicPort.
Added filter bypass functionality for framework requests in ContentLengthFilter.
Updated property documentation in Admin Guide.
Renamed methods & added Javadoc to clarify purpose of filters in JettyServer.
Cleaned up conditional logic in StandardPublicPort.
Moved ContentLengthFilterTest to correct module.
Refactored unit tests for accuracy and clarity.
Fixed remaining merge conflict due to method renaming.

Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-07-14 10:42:00 -07:00
Andy LoPresto
0fa8776f4d
NIFI-7621 & NIFI-7614 Updated terminology throughout codebase and documentation.
Fixed unit test failures introduced from static imports during refactoring.

Signed-off-by: Joe Witt <joe.witt@gmail.com>
2020-07-14 10:39:28 -07:00
Peter Gyori
4f11e36260 NIFI-7635: StandardConfigurationContext.getProperty() gets the property descriptor and its default value from the component itself
This closes #4408.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-07-14 13:20:59 -04:00
Tamás Bunth
b1e77019ac NIFI-6128 UnpackContent: Store unpacked file data
Tar format allows us to archive files with their original permission,
owner, group name and last modification time.

When unpacking with Tar unpacker, these information are stored in new
attributes with names: "file.inner.*". This way, it preserves backward
compatibility when using parallel with GetFile processor (which stores
information in "file.*").

NIFI-6128 Tar unpackContent: assert date of last modification of content is a valid date format.

NIFI-6128 UnpackContent: use original attributes

In case of tar format:
- Use "file.*" attributes instead of "file.inner.*" (which eventually
  lead to overwrite if tar had been fetched with GetFile)
- Store file permission in "rwx" format, instead of integer
  representation.
- Also replace SimpleDateFormat with DateTimeFormatter.

Replace generic error with IllegalArgumentException
Also refactor: move permission string to top of file as static constant.

Update test

Remove 'file systems' text

Unpackcontent: Fill file.creationTime attribute
which always holds the very same value as file.lastModifiedTime.

This closes #4370.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-14 18:04:56 +02:00
Tamás Bunth
72666a9c7e NIFI-7589 Fix path value when unpacking tar
Refactor: remove variable reassignment

This closes #4371.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-14 17:32:51 +02:00
Cuong Nguyen
c7a235f854 NIFI-7629: Make property PROJECT_ID evaluate EL on processors ConsumeGCPubSub & PublishGCPubSub 2020-07-13 08:54:41 -04:00
Tamas Palfy
19a84dd3dd NIFI-7581 Added ControllerService-based credential settings support for ADLS processors
NIFI-7581 Separated Controller Service for providing Azure credentials for ADLS (ADLSCredentialsControllerService) form the one that does the same for Blob storages (AzureStorageCredentialsDetails). (This was done due to the considerable difference in the APIs of the libraries used to connect to both.)
NIFI-7581 Fix: Register controller service in META-INF. Minor fixes.
NIFI-7581 Minor changes (documentation, type etc.)
NIFI-7581 Updated integration tests.
NIFI-7581 Minor changes (renaming).

This closes #4369.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-12 11:56:31 +02:00
Mike Thomsen
8d53f5d0c9
NIFI-7497 Removed a few style check bugs that crept up in the last commit. 2020-07-10 17:12:27 -04:00
Peter Turcsanyi
ae877b9908
NIFI-7591: Allow PutS3Object to post to AWS Snowball
Added properties to enable/disable chunked encoding and path-style access
for endpoints that do not support chunked encoding / only support path-style access.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4386.
2020-07-10 18:24:44 +02:00
Mark Payne
1fc25db47c
NIFI-7469: Updated RepositoryRecord to include flag indicating whether or not the content of a FlowFile was modified. This allows us to explicitly keep track of this state rather than implying it (potentially incorrectly).
This closes #4399.
2020-07-10 10:44:51 -04:00
neptunesalt
ee91341ec3
NIFI-7497 Adding support for AWS Credentials Assume Role to be able to set the STS Endpoint
NIFI-7497 Updating property description per comment

This closes #4309

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-07-10 09:19:58 -04:00