Commit Graph

1035 Commits

Author SHA1 Message Date
exceptionfactory 340b13033f NIFI-8367 Removed joda-time from nifi-standard-processors 2021-05-11 14:50:49 -04:00
matthewlannan 86d5b08d23 Adding addititonal documentation to EvalluateJsonPath per https://issues.apache.org/jira/browse/NIFI-8518 2021-05-11 14:40:12 -04:00
Matthew Burgess 30fc26647e
NIFI-6061: Fix CLOB/BLOB handling in PutDatabaseRecord
NIFI-6061: Force getBytes() in BLOB handling to use UTF-8 charset

NIFI-6061: Use setClob(), added unit tests, incorporated review comments

This closes #5049

Co-authored-by: zhangcheng <zhangchengk@foxmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-04 14:58:14 -05:00
Paul Grey cc554a6b11
NIFI-8478 Addressed i18n test issues; CI supplies locale to surefire
This closes #5040

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-03 13:40:01 -05:00
Matthew Burgess 3963f66dff
NIFI-8320: Fix column mismatch in PutDatabaseRecord
This closes #5024

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-30 10:41:36 -05:00
Tamas Palfy 16dc61e151 NIFI-8458 TailFile - Fix some bugs. Add more tests.
- During cleanup keep "tailingPostRollover" in the updated state.
- Skipping tests that can't run on Windows.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-04-27 09:44:39 -04:00
Peter Turcsanyi 525bfe00b3
NIFI-8474: Added new Replacement Strategy for variable substitution in ReplaceText.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5032.
2021-04-27 13:03:21 +02:00
Paul Grey 835f50c83a
NIFI-8465 Handle bcrypt legacy decrypt failures in testing
This closes #5029

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-26 11:34:20 -05:00
Paul Kelly aedacdf86f
NIFI-8454: Allow InvokeHTTP to output final URL from response request property
This closes #5016

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-21 18:18:25 -05:00
Mark Payne 1ef1905461 NIFI-8344: Addressed corner case in which we didn't handle a case where we tailed a file after being rolled over and then processor encounters a newline followed by NUL characters
This closes #5009.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-04-20 14:51:24 +02:00
Joe Gresock be716c1621 NIFI-3862: Adding Issuer DN support to ListenHTTP
This closes #4976.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-20 13:09:53 +02:00
exceptionfactory ef794c56b6
NIFI-8398 Updated dependencies to build using Apache Maven 3.8.1
- Updated ParCEFone to 1.2.8
- Added conjars.org HTTPS repository for Hive transitive dependencies

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4989
2021-04-12 14:45:02 -04:00
Mark Payne 7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
Mark Payne 2edf5514b7 NIFI-8344: If configured to tail a file for some period of time post-rollover, ensure that we only consume lines that are fully written (i.e., end with a newline). Once we stop tailing that file post-rollover, consume any data from that file that has not yet been consumed, up to the end of the file, even if there is no newline.
This closes #4937.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-31 13:40:08 +02:00
exceptionfactory 1cd3fbb4eb NIFI-8288 Removed OkHttpClientUtils to reduce reliance on nifi-security-utils
- Added createTrustManager() on SSLContextService
- Removed nifi-security-utils and okhttp dependencies from nifi-web-utils

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4869.
2021-03-25 15:38:24 -04:00
Hsin-Ying Lee 0f28702b47
NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file… (#4923)
* NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file if the process start failed
* NIFI-8354 Record the log when delete file failed


This closes #4923

Signed-off-by: Otto Fowler <otto@apache.org>
2021-03-24 12:26:56 -04:00
exceptionfactory e505e8b42d
NIFI-8352 This closes #4922. Changed assertServerConnections to loop and sleep while evaluating
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-23 07:15:48 -07:00
Mark Payne e1b9548ab6 NIFI-8344: Introduced new Rollover Tail Period property
This closes #4916.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-22 18:22:00 +01:00
exceptionfactory a7ba5bb4f8
NIFI-8319 Added AES/CBC/NoPadding for decryption in EncryptContent Processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4894.
2021-03-18 01:40:18 +04:00
exceptionfactory 8202bffc98
NIFI-8304 This closes #4907. Replaced HttpsURLConnection with OkHttpClient in TestListenHTTP
NIFI-8304 Updated TestPutTCP to shutdown server before checking connections
NIFI-8304 Changed TestListenTCP to send messages in one byte array
NIFI-8304 Added check for expected jdk.tls.disabledAlgorithms
2021-03-17 14:15:25 -07:00
exceptionfactory be7233d252
NIFI-8304 Changed TestListenTCPRecord to loop until files found
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4904.
2021-03-17 11:41:21 +04:00
exceptionfactory 2ad88bbfff
NIFI-8304 This closes #4900. Improved Socket test reliability for several Processors
- Refactored TestPutTCP to single class
- Improved TestListenRELP
- Improved TestListenTCP
- Improved TestListenUDP
- Improved TestListenTCPRecord
- Changed OnUnscheduled to OnStopped in AbstractListenEventProcessor

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-16 15:30:11 -07:00
exceptionfactory b1f513a2ca
NIFI-8304 Refactored InvokeHTTP unit tests using OkHttp MockWebServer
- Replaced Joda Time with java.time for date formatting
- Replaced Guava Files with java.nio.file.Files for cache directory
- Updated PutTCP test server to close connection when testing connection per FlowFile

NIFI-8304 Removed Thread wrapper for TestListenHTTP client requests

NIFI-8304 Disabled InvokeHTTP Connection Pooling for testing

NIFI-8304 Set 60 second timeout for testing TLS connections

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4892.
2021-03-15 21:28:40 +04:00
exceptionfactory 09692c5f69
NIFI-8313 Upgraded net.lingala.zip4j:zip4j to 2.7.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4889.
2021-03-12 12:21:07 +04:00
Otto Fowler 9f0b47af77
NIFI-2702 Support named captures in ExtractText
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4384.
2021-03-12 12:11:09 +04:00
Mark Payne b523698534
NIFI-8212: Refactored StandardExtensionDiscoveringManager to avoid using ServiceLoader
Instead, it will look at the ServiceLoader file and read the names of the classes but avoid instantiating all of the objects or loading the classes into memory.
- Updated Doc Generation so that if the documentation for a given NAR already exists, it doesn't delete it and re-generate it. This was necessary because we are no longer instantiating an instance of each component and instead lazily creating the components as necessary.
- Removed stateless version of extension registry because it's no longer necessary

This closes #4852

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-08 12:21:19 -06:00
Peter Gyori 4a27b23b1f
NIFI-8263: Maximum Thread Pool Size property introduced in ListenHTTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4847.
2021-03-01 20:14:34 +04:00
Peter Gyori a54f739229 NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
NIFI-7969: Documentation update
Clarified that the Force Types From Schema property applies to the data read, whereas the Strict Type Checking property applies to the validation.
NIFI-7969: Documentation update - updated the property name in additionalDetails.html

This closes #4825.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-01 16:04:11 +01:00
Matthew Burgess 29cc3886fc NIFI-8259: Infer CSV field types as strings if no records are present 2021-02-25 12:45:12 -05:00
Matthew Burgess 34fc94454f NIFI-8249: Fixed issue with error during multiple FlowFile results in ExecuteSQL processors 2021-02-24 10:10:32 -05:00
Matthew Burgess 2dedd8bf0f
NIFI-8253: Restore call to session.commit() in GenerateTableFetch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4840.
2021-02-24 15:57:16 +04:00
Peter Gyori d049bca794
NIFI-8205: Documentation improvements for the Wait processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4808.
2021-02-23 21:22:19 +04:00
Matthew Burgess 200c04c6d0
NIFI-8237: This closes #4835. Added missing SQL types to getDataTypeFromSQLTypeValue(), added defensive code
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-23 08:08:09 -07:00
Pierre Villard 9101160cbf
NIFI-8250 - This closes #4838. UTF8 encoding for FTP processors
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-23 08:05:32 -07:00
Joe Witt 88fab00e29
NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT 2021-02-15 12:09:32 -07:00
Joe Witt 4afb2ba743
NIFI-7873-RC4 prepare for next development iteration 2021-02-15 12:09:31 -07:00
Joe Witt 487280bee9
NIFI-7873-RC4 prepare release nifi-1.13.0-RC4 2021-02-15 12:09:30 -07:00
Matthew Burgess d08f02428d
NIFI-8223: This closes #4819. Use column datatype in PutDatabaseRecord when calling setObject()
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-10 18:01:43 -07:00
Nathan Gough 4ebeb4e146
NIFI-8181 - Added Disable HTTP/2 property to InvokeHTTP
This closes #4804

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-08 14:53:35 -06:00
Peter Turcsanyi 40d8c41656
NIFI-8193: Fixed PutSQL reordering FlowFiles
Also removed sorting of transferred MockFlowFiles in test framework.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4803
2021-02-05 16:17:10 -05:00
Matthew Burgess b77dbd5030
NIFI-8172: Provide schema name to getPrimaryKeys call in PutDatabaseRecord
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4782.
2021-02-06 00:00:58 +04:00
Tamas Palfy b55998afc1 NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP: Time Window
NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP: Adjusted Time Window. User can specify the time zone or time difference (compared to where NiFi runs) of the system hosting the files and based on the calculates the current time there. Lists files modified before this adjusted current time (and after the last listing).
NIFI-8081 'Time Adjustment' validated not to be set if listing strategy is not 'Adjusted Time Window'. Extracted validator to a separate class. Added more tests. Minor refactor. Typo fix.
NIFI-8081 Improved validation.
NIFI-8081 'Time Adjustment' is not necessary - in fact it can cause problems. SFTP (and usually FTP - which has a more general bug at the moment) returns a timestamp that doesn't really need adjustment. (SFTP in particular returns the an 'epoch' time.) Everything remains the same - the new listing strategy relies on a sliding time window, but without the unnecessary option to adjust for the modification time.
NIFI-8081 Resolved conflicts after rebasing to main.
NIFI-8081 Renamed 'AbstractListProcessor.listByAdjustedSlidingTimeWindow' to 'listByTimeWindow'. Post main rebase correction.
NIFI-8081 Updated user doc for the BY_TIME_WINDOW strategy to warn user on it's reliance of accurate time.

This closes #4721.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-02-04 17:04:37 +01:00
Denes Arvay c1f88ec740 NIFI-8183 TailFile intermittently creates records with ascii NULL after rollover
This closes #4792.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-03 13:09:00 -05:00
mtien 6e1f737c53
NIFI-1355 Implemented new methods in KeyStoreUtils to programmatically-generate certificates, Keystores, and Truststores and return it wrapped in a TLS configuration.
Updated TestInvokeHTTP, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use new Keystore functionality.

NIFI-1355 Refactored and removed unnecessary unit tests in KeyStoreUtilsGroovyTest.

NIFI-1355 Added a password requirement when creating a new truststore.
Handled exception when loading a passwordless truststore type of Bouncy Castle PKCS12.

This closes #4801

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-03 08:53:47 -06:00
Peter Turcsanyi 77eb8af275
NIFI-8187: Fixed PutSQL duplicating provenance events
Also fixed provenance event mocking in test framework.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4796
2021-02-02 19:02:02 -05:00
exceptionfactory 11e9ff3773
NIFI-8178 This closes #4787. Replaced StandardSSLContextService in unit tests with SslContextUtils
- Removed references to StandardSSLContextService from nifi-standard-processors
- Removed TestGetHTTPGroovy and TestPostHTTPGroovy since these are testing deprecated processors
- Optimized HandleHttpRequest, GetHTTP, PostHTTP to use SSLContextService.createContext()
NIFI-8178 Changed TestGetHTTP to ITGetHTTP since GetHTTP is deprecated
NIFI-8178 Changed TestPostHTTP to ITPostHTTP since PostHTTP is deprecated

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-27 17:48:06 -07:00
exceptionfactory 8c1fe84f62
NIFI-8177 This closes #4786. Added timeout to Sender Pool polling to avoid blocking indefinitely
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-27 15:04:23 -07:00
snorlaxa 04fb1aca47
NIFI-8142: Add "Insert Ignore" option to PutDatabaseRecord
NIFI-8142: Fix Checkstyle error

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4778
2021-01-27 11:50:40 -05:00
exceptionfactory abb6ed3128
NIFI-8171 This closes #4779. Upgraded Bouncy Castle libraries to 1.68 and centralized dependency version
NIFI-8171 Increased response and idle timeouts for HTTP unit tests
NIFI-8171 Increased TestServer idle timeout to 45 seconds for HTTP unit tests
NIFI-8171 Adjusted timeout and sleep on TestPutTCPCommon.testPruneSenders
NIFI-8171 Increased TestServer idle timeout to 60 seconds and removed 500ms Thread.sleep() in TestInvokeHttpSSL
NIFI-8171 Optimized OkHttpClientUtils to avoid reading trust store twice during initialization
NIFI-8171 Added static variable for server startup sleep
NIFI-8171 Increased TestInvokeHTTP Connect Timeout and TestListenHTTP Response Timeout to 30 seconds
NIFI-8171 Refactored unit tests for InvokeHTTP and ListenHTTP to optimize SSLContext creation
NIFI-8171 Updated TestListenHTTP for static creation of SSLContext
NIFI-8171 Added started check for ListenHTTP Server in TestListenHTTP
NIFI-8171 Refactored TestPutTCP classes to optimize SSLContext creation
NIFI-8171 Increased TestListenHTTP timeout for server start to 120 seconds and added exception when not connected
NIFI-8171 Increased Connect and Read Timeouts for InvokeHTTP SSL unit tests

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-26 21:24:07 -07:00
exceptionfactory 2cdb0fb6a3 NIFI-8088 Removed deprecation warning log for PKCS12 trust stores
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4727.
2021-01-26 16:50:33 -05:00