1081 Commits

Author SHA1 Message Date
Otto Fowler
9f0b47af77
NIFI-2702 Support named captures in ExtractText
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4384.
2021-03-12 12:11:09 +04:00
Mark Payne
b523698534
NIFI-8212: Refactored StandardExtensionDiscoveringManager to avoid using ServiceLoader
Instead, it will look at the ServiceLoader file and read the names of the classes but avoid instantiating all of the objects or loading the classes into memory.
- Updated Doc Generation so that if the documentation for a given NAR already exists, it doesn't delete it and re-generate it. This was necessary because we are no longer instantiating an instance of each component and instead lazily creating the components as necessary.
- Removed stateless version of extension registry because it's no longer necessary

This closes #4852

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-08 12:21:19 -06:00
Peter Gyori
4a27b23b1f
NIFI-8263: Maximum Thread Pool Size property introduced in ListenHTTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4847.
2021-03-01 20:14:34 +04:00
Peter Gyori
a54f739229 NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
NIFI-7969: Documentation update
Clarified that the Force Types From Schema property applies to the data read, whereas the Strict Type Checking property applies to the validation.
NIFI-7969: Documentation update - updated the property name in additionalDetails.html

This closes #4825.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-01 16:04:11 +01:00
Matthew Burgess
29cc3886fc NIFI-8259: Infer CSV field types as strings if no records are present 2021-02-25 12:45:12 -05:00
Matthew Burgess
34fc94454f NIFI-8249: Fixed issue with error during multiple FlowFile results in ExecuteSQL processors 2021-02-24 10:10:32 -05:00
Matthew Burgess
2dedd8bf0f
NIFI-8253: Restore call to session.commit() in GenerateTableFetch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4840.
2021-02-24 15:57:16 +04:00
Peter Gyori
d049bca794
NIFI-8205: Documentation improvements for the Wait processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4808.
2021-02-23 21:22:19 +04:00
Matthew Burgess
200c04c6d0
NIFI-8237: This closes #4835. Added missing SQL types to getDataTypeFromSQLTypeValue(), added defensive code
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-23 08:08:09 -07:00
Pierre Villard
9101160cbf
NIFI-8250 - This closes #4838. UTF8 encoding for FTP processors
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-23 08:05:32 -07:00
exceptionfactory
f532b3ae1d
NIFI-5623 Upgraded OkHttp3 to 4.9.1 and updated unit tests
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4826.
2021-02-19 14:42:16 +04:00
Joey Frazee
042b34cb43 NIFI-8231 Clarify behavior of allowed values for Memory Pools in MonitorMemory 2021-02-18 09:58:48 -05:00
Matthew Burgess
fd50d6f042 NIFI-3383 Check if usage operations are supported in MonitorMemory
This closes #4790

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-02-16 17:25:28 -06:00
Joe Witt
88fab00e29
NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT 2021-02-15 12:09:32 -07:00
Joe Witt
4afb2ba743
NIFI-7873-RC4 prepare for next development iteration 2021-02-15 12:09:31 -07:00
Joe Witt
487280bee9
NIFI-7873-RC4 prepare release nifi-1.13.0-RC4 2021-02-15 12:09:30 -07:00
Matthew Burgess
d08f02428d
NIFI-8223: This closes #4819. Use column datatype in PutDatabaseRecord when calling setObject()
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-10 18:01:43 -07:00
Nathan Gough
4ebeb4e146
NIFI-8181 - Added Disable HTTP/2 property to InvokeHTTP
This closes #4804

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-08 14:53:35 -06:00
Peter Turcsanyi
40d8c41656
NIFI-8193: Fixed PutSQL reordering FlowFiles
Also removed sorting of transferred MockFlowFiles in test framework.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4803
2021-02-05 16:17:10 -05:00
Matthew Burgess
b77dbd5030
NIFI-8172: Provide schema name to getPrimaryKeys call in PutDatabaseRecord
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4782.
2021-02-06 00:00:58 +04:00
Tamas Palfy
b55998afc1 NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP: Time Window
NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP: Adjusted Time Window. User can specify the time zone or time difference (compared to where NiFi runs) of the system hosting the files and based on the calculates the current time there. Lists files modified before this adjusted current time (and after the last listing).
NIFI-8081 'Time Adjustment' validated not to be set if listing strategy is not 'Adjusted Time Window'. Extracted validator to a separate class. Added more tests. Minor refactor. Typo fix.
NIFI-8081 Improved validation.
NIFI-8081 'Time Adjustment' is not necessary - in fact it can cause problems. SFTP (and usually FTP - which has a more general bug at the moment) returns a timestamp that doesn't really need adjustment. (SFTP in particular returns the an 'epoch' time.) Everything remains the same - the new listing strategy relies on a sliding time window, but without the unnecessary option to adjust for the modification time.
NIFI-8081 Resolved conflicts after rebasing to main.
NIFI-8081 Renamed 'AbstractListProcessor.listByAdjustedSlidingTimeWindow' to 'listByTimeWindow'. Post main rebase correction.
NIFI-8081 Updated user doc for the BY_TIME_WINDOW strategy to warn user on it's reliance of accurate time.

This closes #4721.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-02-04 17:04:37 +01:00
Denes Arvay
c1f88ec740 NIFI-8183 TailFile intermittently creates records with ascii NULL after rollover
This closes #4792.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-03 13:09:00 -05:00
mtien
6e1f737c53
NIFI-1355 Implemented new methods in KeyStoreUtils to programmatically-generate certificates, Keystores, and Truststores and return it wrapped in a TLS configuration.
Updated TestInvokeHTTP, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use new Keystore functionality.

NIFI-1355 Refactored and removed unnecessary unit tests in KeyStoreUtilsGroovyTest.

NIFI-1355 Added a password requirement when creating a new truststore.
Handled exception when loading a passwordless truststore type of Bouncy Castle PKCS12.

This closes #4801

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-03 08:53:47 -06:00
Joe Witt
d826416217
NIFI-8192 updating Copyright years for nifi things to include 2021
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4802.
2021-02-03 12:56:28 +04:00
Peter Turcsanyi
77eb8af275
NIFI-8187: Fixed PutSQL duplicating provenance events
Also fixed provenance event mocking in test framework.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4796
2021-02-02 19:02:02 -05:00
exceptionfactory
11e9ff3773
NIFI-8178 This closes #4787. Replaced StandardSSLContextService in unit tests with SslContextUtils
- Removed references to StandardSSLContextService from nifi-standard-processors
- Removed TestGetHTTPGroovy and TestPostHTTPGroovy since these are testing deprecated processors
- Optimized HandleHttpRequest, GetHTTP, PostHTTP to use SSLContextService.createContext()
NIFI-8178 Changed TestGetHTTP to ITGetHTTP since GetHTTP is deprecated
NIFI-8178 Changed TestPostHTTP to ITPostHTTP since PostHTTP is deprecated

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-27 17:48:06 -07:00
exceptionfactory
8c1fe84f62
NIFI-8177 This closes #4786. Added timeout to Sender Pool polling to avoid blocking indefinitely
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-27 15:04:23 -07:00
snorlaxa
04fb1aca47
NIFI-8142: Add "Insert Ignore" option to PutDatabaseRecord
NIFI-8142: Fix Checkstyle error

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4778
2021-01-27 11:50:40 -05:00
exceptionfactory
abb6ed3128
NIFI-8171 This closes #4779. Upgraded Bouncy Castle libraries to 1.68 and centralized dependency version
NIFI-8171 Increased response and idle timeouts for HTTP unit tests
NIFI-8171 Increased TestServer idle timeout to 45 seconds for HTTP unit tests
NIFI-8171 Adjusted timeout and sleep on TestPutTCPCommon.testPruneSenders
NIFI-8171 Increased TestServer idle timeout to 60 seconds and removed 500ms Thread.sleep() in TestInvokeHttpSSL
NIFI-8171 Optimized OkHttpClientUtils to avoid reading trust store twice during initialization
NIFI-8171 Added static variable for server startup sleep
NIFI-8171 Increased TestInvokeHTTP Connect Timeout and TestListenHTTP Response Timeout to 30 seconds
NIFI-8171 Refactored unit tests for InvokeHTTP and ListenHTTP to optimize SSLContext creation
NIFI-8171 Updated TestListenHTTP for static creation of SSLContext
NIFI-8171 Added started check for ListenHTTP Server in TestListenHTTP
NIFI-8171 Refactored TestPutTCP classes to optimize SSLContext creation
NIFI-8171 Increased TestListenHTTP timeout for server start to 120 seconds and added exception when not connected
NIFI-8171 Increased Connect and Read Timeouts for InvokeHTTP SSL unit tests

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-26 21:24:07 -07:00
exceptionfactory
2cdb0fb6a3 NIFI-8088 Removed deprecation warning log for PKCS12 trust stores
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4727.
2021-01-26 16:50:33 -05:00
Peter Turcsanyi
67d06003b7 NIFI-8023: Convert java.sql.Date between UTC/local time zone normalized forms before/after database operations
This closes #4781

Signed-off-by: David Handermann <exceptionfactory@gmail.com>
2021-01-26 14:39:02 -06:00
exceptionfactory
8c49f0688b
NIFI-7243 Upgraded com.hierynomus.sshj to 0.30.0 to resolve authentication race condition in version 0.27.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4774.
2021-01-21 21:21:47 +04:00
Matthew Burgess
fb2a8b5820
NIFI-8043: Quote update key column names in PutDatabaseRecord
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4772.
2021-01-21 11:17:13 +04:00
Mark Payne
803ba882aa
NIFI-8146: Ensure that we close the Connection/Statement/PreparedStatement objects in finally blocks or try-with-resources
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4770
2021-01-20 13:48:57 -05:00
Mark Payne
25ab050ed7
NIFI-8149: Updated Apache Calcite version from 1.21.0 to 1.26.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4765.
2021-01-20 13:04:08 +04:00
Mark Payne
930e8d9e0e
NIFI-7698: Added a merge.reason attribute for flowfiles indicating why they were merged by MergeContent. Also updated logs to indicate the reason and added additional documentation for processor
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4764
2021-01-19 17:20:06 -05:00
Mark Payne
f02079c2fa
NIFI-8146: Added RecordPath for Statement Type and to specify where to find the data within the incoming Record. Also performed some minor code refactoring to avoid lots of large lambdas, as it results in code that is more readable and more maintainable. Updated some properties to use dependsOn() notation now that this is available.
NIFI-8146: Updated PutDatabaseRecord to avoid using the functional style framework that had previously been used. Doing so resulted in code that was difficult to understand and maintain. Added additional unit tests and improved MockRecordParser so that it could throw configurable types of exceptions

NIFI-8146: Fixed checkstyle issues

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4763
2021-01-19 13:23:54 -05:00
Mark Payne
097edf4f7c
NIFI-8136: Added getState/setState/replaceState/clearState methods to ProcessSession, updated processors to use these methods instead of StateManager version where appropriate
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4757
2021-01-15 10:02:09 -05:00
Matthew Burgess
382439c1d0 NIFI-7973: Add default precision and scale properties to remaining SQL-based components
This closes #4682.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-13 08:41:12 -05:00
exceptionfactory
d41f2e1d0a NIFI-8121 Updated ListenHTTP with inferred Client Authentication Policy
- Added default property value for automatic determination of Client Authentication Policy based on SSLContextService Trust Store properties
- Added new ClientAuthentication enum with values specific to ListenHTTP

This closes #4749.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-11 18:02:09 +01:00
exceptionfactory
a4027e8e77 NIFI-8120 Added RuntimeException handling on HttpContextMap.complete()
NIFI-8120 Renamed exception variable and reordered log statements

This closes #4747.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-08 17:13:10 +00:00
exceptionfactory
817f621d6f NIFI-8096 Deprecated ClientAuth references in SSLContextService
- Added SSLContextService.createContext() and refactored referencing components
- Removed references to ClientAuth from SslContextFactory methods

This closes #4737.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-06 16:24:26 -05:00
tlsmith
7214dc0f85 NIFI-6242 PutFileTransfer generating incorrect provenance event 2021-01-06 12:10:20 -05:00
Otto Fowler
ceb9dff3b9
NIFI-7761 Allow HandleHttpRequest to add specified form data to FlowFile attributes
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4513.
2021-01-05 09:42:41 +01:00
exceptionfactory
e7c6bdad42 NIFI-7937 Added StandardFlowFileMediaType enum to replace string references to FlowFile Media Types 2021-01-04 14:20:05 -05:00
Kevin Barranco
0f667cb4f3 MonitorActivity THRESHOLD to use Variables
When using MonitorActivity, it would be interesting to use variables in "Threshold Duration", which will help maintain Version Control in ProcessGroups (Avoiding "breaking" versions when including the value manually). 

I've included the option expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY), so we can use variables in "Threshold Duration".
2021-01-04 14:14:29 -05:00
tlsmith
112b893bf2 NIFI-7225 FetchSFTP processor routing to not.found error given when Private Key Path property is invalid 2021-01-04 13:41:03 -05:00
Denes Arvay
31c5dc5ab4 NIFI-8102 TailFile can skip characters if tailing a file on NFS mount
- Force the TailFile processor to recreate and reposition the reader
  by setting it to null in case of a NulCharacterEncounteredException
- Updated the TestTailFile.testNULContent() to not initialize the
  processor when calling the second run()

Update the reader's position instead of abandoning it.

This closes #4736.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-12-18 14:20:41 -05:00
Tamas Palfy
989287adaf
NIFI-8027 In AbstractExecuteSQL make 'SQL Pre-Query' and 'SQL Post-Query' properties support escaping semicolons to allow them be part of statements.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4732
2020-12-16 14:57:59 -05:00
exceptionfactory
28ca7478d6 NIFI-8019 Added TlsPlatform to provide runtime TLS protocol configuration
NIFI-8019 Renamed getDefaultProtocols() to getSupportedProtocols()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4673.
2020-12-14 19:34:58 -05:00