NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP: Adjusted Time Window. User can specify the time zone or time difference (compared to where NiFi runs) of the system hosting the files and based on the calculates the current time there. Lists files modified before this adjusted current time (and after the last listing).
NIFI-8081 'Time Adjustment' validated not to be set if listing strategy is not 'Adjusted Time Window'. Extracted validator to a separate class. Added more tests. Minor refactor. Typo fix.
NIFI-8081 Improved validation.
NIFI-8081 'Time Adjustment' is not necessary - in fact it can cause problems. SFTP (and usually FTP - which has a more general bug at the moment) returns a timestamp that doesn't really need adjustment. (SFTP in particular returns the an 'epoch' time.) Everything remains the same - the new listing strategy relies on a sliding time window, but without the unnecessary option to adjust for the modification time.
NIFI-8081 Resolved conflicts after rebasing to main.
NIFI-8081 Renamed 'AbstractListProcessor.listByAdjustedSlidingTimeWindow' to 'listByTimeWindow'. Post main rebase correction.
NIFI-8081 Updated user doc for the BY_TIME_WINDOW strategy to warn user on it's reliance of accurate time.
This closes#4721.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
Updated TestInvokeHTTP, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use new Keystore functionality.
NIFI-1355 Refactored and removed unnecessary unit tests in KeyStoreUtilsGroovyTest.
NIFI-1355 Added a password requirement when creating a new truststore.
Handled exception when loading a passwordless truststore type of Bouncy Castle PKCS12.
This closes#4801
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- Removed references to StandardSSLContextService from nifi-standard-processors
- Removed TestGetHTTPGroovy and TestPostHTTPGroovy since these are testing deprecated processors
- Optimized HandleHttpRequest, GetHTTP, PostHTTP to use SSLContextService.createContext()
NIFI-8178 Changed TestGetHTTP to ITGetHTTP since GetHTTP is deprecated
NIFI-8178 Changed TestPostHTTP to ITPostHTTP since PostHTTP is deprecated
Signed-off-by: Joe Witt <joewitt@apache.org>
NIFI-8171 Increased response and idle timeouts for HTTP unit tests
NIFI-8171 Increased TestServer idle timeout to 45 seconds for HTTP unit tests
NIFI-8171 Adjusted timeout and sleep on TestPutTCPCommon.testPruneSenders
NIFI-8171 Increased TestServer idle timeout to 60 seconds and removed 500ms Thread.sleep() in TestInvokeHttpSSL
NIFI-8171 Optimized OkHttpClientUtils to avoid reading trust store twice during initialization
NIFI-8171 Added static variable for server startup sleep
NIFI-8171 Increased TestInvokeHTTP Connect Timeout and TestListenHTTP Response Timeout to 30 seconds
NIFI-8171 Refactored unit tests for InvokeHTTP and ListenHTTP to optimize SSLContext creation
NIFI-8171 Updated TestListenHTTP for static creation of SSLContext
NIFI-8171 Added started check for ListenHTTP Server in TestListenHTTP
NIFI-8171 Refactored TestPutTCP classes to optimize SSLContext creation
NIFI-8171 Increased TestListenHTTP timeout for server start to 120 seconds and added exception when not connected
NIFI-8171 Increased Connect and Read Timeouts for InvokeHTTP SSL unit tests
Signed-off-by: Joe Witt <joewitt@apache.org>
NIFI-8146: Updated PutDatabaseRecord to avoid using the functional style framework that had previously been used. Doing so resulted in code that was difficult to understand and maintain. Added additional unit tests and improved MockRecordParser so that it could throw configurable types of exceptions
NIFI-8146: Fixed checkstyle issues
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4763
- Added default property value for automatic determination of Client Authentication Policy based on SSLContextService Trust Store properties
- Added new ClientAuthentication enum with values specific to ListenHTTP
This closes#4749.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Added SSLContextService.createContext() and refactored referencing components
- Removed references to ClientAuth from SslContextFactory methods
This closes#4737.
Signed-off-by: Mark Payne <markap14@hotmail.com>
When using MonitorActivity, it would be interesting to use variables in "Threshold Duration", which will help maintain Version Control in ProcessGroups (Avoiding "breaking" versions when including the value manually).
I've included the option expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY), so we can use variables in "Threshold Duration".
- Force the TailFile processor to recreate and reposition the reader
by setting it to null in case of a NulCharacterEncounteredException
- Updated the TestTailFile.testNULContent() to not initialize the
processor when calling the second run()
Update the reader's position instead of abandoning it.
This closes#4736.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-8039 Review findings; refining thread pool to be able to scale down properly when not under load
NIFI-8039 Answers to PR comments
This closes#4689.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Refactored nifi-stateless to make use of nifi-framework-components
- Removed requirement for nifi-framework-nar to be provided.
- Refactored stateless nifi into api, engine, nar, and bootstrap modules, with a parent 'bundle' module
- Creation of nifi-stateless-system-tests
- Added unit tests and logging
- Changed flow configuration to use properties file instead of json
- Allow for -p parameter to specify parameters on command line
- Moved implementations of Authorizer, NiFiUser, and UserGroupProviders to new module named nifi-framework-authorization-providers so that those that depend on nifi-framework-authorization don't have to bring in the providers. This way, we can have stateless not bring in those providers, as we otherwise get warnings on startup about the provider already being registered. Additionally, it avoids needing dependencies on spring-security-core
- Updated bin/nifi.sh script to run new stateless bootstrap
- Added Reporting Tasks to stateless.
- Download bundles as necessary on stateless nifi startup
NIFI-7897: Addressing review feedback
NIFI-7897: Fixed typos in README and also addressed issue that caused parameters with spaces not to be parsed properly
This closes#4669.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-7922: Fixes based on review comments
ListenHTTP: changed if(!sslRequired) to if(sslRequired) so that the positive case comes first.
HealthCheckServlet: response body for GET contains "OK".
ContentAcknowledgmentServlet: super.doDelete() is called when a DELETE should be rejected because of port mismatch.
NIFI-7922: Refactored, based on review comments
NIFI-7922: Fixed a checkstyle violation (organized the imports)
This closes#4603.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
- Make reporting in clustered scope to dependent of expected cluster state in order to prevent unexpected flow file emission
This closes#4642.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added a 'dependent' attribute to determine whether or not to save dependent property values
Co-authored-by: Scott Aslan <scottyaslan@gmail.com>
Signed-off-by: Bryan Bende <bbende@apache.org>
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.
* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.
* NIFI-7804 Resolved failing unit test.
* NIFI-7804 Removed legacy dependency.
* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
The TikaInputStream and FileInputStream instances utilized in IdentifyMimeType are now explicitly closed. The FileInputStream is additionally wrapped by a BufferedInputStream.
Signed-off-by: Joe Witt <joewitt@apache.org>
Cleaned up JettyServer code.
Changed test logging severity to include debug statements.
Added test resources.
This closes#4498.
Co-authored-by: Kotaro Terada <kotarot@apache.org>
Because even though permissions are only the 7 least significant bits of the file
mode but the file mode can be wider and can contain further info (like the
sticky bit).
Extend unit test for converting file mode with sticky bit into 'rwx' style
permission string.
Remove old test cases
This closes#4490.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NiFi uses the Java IO temporary directory for storing HTTP multipart
files when using HandleHttpRequest processor. The directory can be
overwritten with Java command line parameter.
Added unit tests.
NIFI-7680 Duplicated DocumentBuilder creation method in NotificationServiceManager to avoid nifi-bootstrap dependency on nifi-security-utils.
Explicitly added commons-lang3 to lib/bootstrap/ directory in nifi-assembly.
NIFI-7680 Reverted unnecessary dependency changes.
Added explicit dependencies where necessary.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4436
Added unit tests.
Refactored shared logic from various algorithm-specific secure hasher implementations to AbstractSecureHasher.
Introduced secure hasher implementations for various KDFs.
Added custom validation to EncryptContent processor.
Implemented logic for EncryptContent to write operational metadata to flowfile attributes.
Added encryption metadata attribute annotations to EncryptContent.
Added Argon2 KDF documentation and Bcrypt key derivation change notes to Admin Guide.
Updated unit tests to calculate default/recommended cost parameters for Argon2.
This closes#4421.
Co-authored-by: mtien <mtien.apache@gmail.com>
Tar format allows us to archive files with their original permission,
owner, group name and last modification time.
When unpacking with Tar unpacker, these information are stored in new
attributes with names: "file.inner.*". This way, it preserves backward
compatibility when using parallel with GetFile processor (which stores
information in "file.*").
NIFI-6128 Tar unpackContent: assert date of last modification of content is a valid date format.
NIFI-6128 UnpackContent: use original attributes
In case of tar format:
- Use "file.*" attributes instead of "file.inner.*" (which eventually
lead to overwrite if tar had been fetched with GetFile)
- Store file permission in "rwx" format, instead of integer
representation.
- Also replace SimpleDateFormat with DateTimeFormatter.
Replace generic error with IllegalArgumentException
Also refactor: move permission string to top of file as static constant.
Update test
Remove 'file systems' text
Unpackcontent: Fill file.creationTime attribute
which always holds the very same value as file.lastModifiedTime.
This closes#4370.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-6934 Added more documentation and unit tests.
NIFI-6934 Added missing license for new test class.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4350
ListenHTTP processor now binds port and creates a HTTP connection only
if one of the following conditions apply:
- Primary node execution is 'false'
- Primary node execution is 'true' and node is elected as primary node.
Changes:
- Connection is established in 'onTrigger' annotated method instead of
'onSchedule'. (This is similar to how handleHTTPRequest processor
handles connections.)
- 'onPrimaryNodeStateChange' annotated method is introduced to tear down
server on reelection of primary node
This closes#4356.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-7542 Override additional jackson-databind versions.
NIFI-7542 Upgrade jackson-databind dependency to 2.9.10.5 in the root pom.xml.
This closes#4343
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
Matthew Burgess
Pierre Villard
exceptionfactory
Joey Frazee
Joe Witt
Nathan Gough
Peter Turcsanyi
Tamas Palfy
Denes Arvay
mtien
exceptionfactory
snorlaxa
Mark Payne
tlsmith
Otto Fowler
Kevin Barranco
Mike Thomsen
Bence Simon
r65535
Peter Gyori
Dan Kim
Denes Arvay
Mohammed Nadeem
Andy LoPresto
noedetore
Cameron E. Tidd
Kotaro Terada
Tamás Bunth
Kyle Miracle
Andy LoPresto