Commit Graph

970 Commits

Author SHA1 Message Date
Joey c18b27af18
NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment (#4341)
* NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment

This also fixes NIFI-4760.

* NIFI-7540: Remove duplicate mail.smtp.starttls.enable from TestListenSMTP

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-06-16 17:45:37 -07:00
wanghq-c 3c757b6ba8
NIFI-7313:fix bug on 'Quote Table Identifiers'
NIFI-7313:add test by wanghongqi
NIFI-7313:edit test

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4185
2020-06-09 16:02:25 -04:00
Wouter de Vries f2368a0dd1 NIFI-7393: Add max idle time and idle connections to InvokeHTTP
This closes #4233.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-06-08 09:29:10 -05:00
Mark Payne e0dd6d466e NIFI-7369: Consider DECIMAL type as a numeric type when using a CHOICE type in QueryRecord
This closes #4223.
2020-06-02 15:13:14 -04:00
Bence Simon 5c2bfcf7d3 NIFI-7369 Adding decimal support for record handling in order to avoid missing precision when reading in records
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-06-02 15:13:14 -04:00
jahenaor 04711ab466
NIFI-7477 Optionally adding validation details as a new attribute of the flowfile
NIFI-7477 Improving description and unit test now verifies attribute content

NIFI-7477: Fixed checkstyle errors

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4301
2020-06-01 14:08:08 -04:00
zhangcheng 1ec6675f38
NIFI-7403:Add a function that adjust if the result is failed before we call the onFailed or onCompleted function. If the result is failed, return true and do sth
NIFI-7403:Add an extension point to adjust the result, if the result is failed then process onFailed function

NIFI-7403:Implement the AdjustFailed Function, if PutSQL set the SUPPORT_TRANSACTIONS true, then check whether the result contains REL_RETRY or REL_FAILURE.If it contains that, reroute the result and return true.

NIFI-7403: fix reroute logic in AdjustFailed function

NIFI-7403:Add and modify some unit test for PutSQL's SUPPORT_TRANSACTIONS property

NIFI-7403:Update for PR recheck

NIFI-7403:Add documentation on the Support Fragmented Transactions property to indicate the transactions rollback behavior

NIFI-7403: Fix Checkstyle issue

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4266
2020-06-01 11:33:10 -04:00
Niels Basjes 2a82efc417
NIFI-6666 Add Useragent Header to InvokeHTTP requests
This closes #3734

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-29 21:39:14 -04:00
Pierre Villard af58714606 NIFI-7430 - LookupRecord change coordinate key for in-place replacement 2020-05-29 13:54:11 -04:00
zhangcheng a1f277fdf7
NIFI-7484:fix ListFTP and FetchFTP docs. Change 'SFTP' to 'FTP' in description
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4294.
2020-05-29 10:41:19 +02:00
zhangcheng e04e6793ca
NIFI-7483 - Remove description about 'Rolling strategy' in TailFile's docs
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4293.
2020-05-29 10:39:04 +02:00
r65535 4ced1775b5
NIFI-7487 - Added batch support and displayName to ModifyBytes processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4302.
2020-05-29 10:28:37 +02:00
adyoun2 101387bfaa
NIFI-6785 Support Deflate Compression
NIFI-6785 Remove unused imports

This closes #3822

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-25 16:22:08 -04:00
Andy LoPresto 97a919a3be
NIFI-7482 Changed InvokeHTTP to be extensible.
Added unit test.

This closes #4291.

Signed-off-by: Arpad Boda <aboda@apache.org>
2020-05-22 11:44:53 -07:00
pcgrenier b6ef7e13bf
NIFI-7462: This adds a way to convert or cast a choice object into a valid type for use with calcite query functions
NIFI-7462: Update to allow FlowFile Table's schema to be more intelligent when using CHOICE types

NIFI-7462: Fixed checkstyle violation, removed documentation around the CAST functions that were no longer needed

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4282
2020-05-21 13:13:12 -04:00
Andy LoPresto 441781cec5
NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in shared constant).
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.

This closes #4263.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-19 12:56:59 -07:00
Otto Fowler f22cd26cf4 NIFI-7420 remove the http.param attribute. It contained both the query parameters already captured in http.query.param _and_ the multipart form data names and values, which are captured in the part data, and could be very very large
This closes #4251.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-06 17:20:08 -04:00
Alan Jackoway effa930151
NIFI-7425 Log Message for ReplaceText Over Buffer Size
Adds a log message when ReplaceText sends a flowfile to the failure relationship because
it is larger than the max buffer size.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4255.
2020-05-06 09:40:46 +02:00
Otto Fowler 659a383723 NIFI-7394: Add support for sending Multipart/FORM data to InvokeHTTP.
By using dynamic properties with a prefix naming scheme, allow
definition of the parts, including the name to give the Flowfile content
part, and optionally it's file name.
After review:
- change so that we can send just the form content or just form data
  without the flowfile
- change the content name and content file name from dynamic properties
  to properties
- change the dynamic name to be an invalid http header "post:form:xxxx"
- add validation and more tests

This closes #4234.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-04-29 13:01:06 -04:00
EndzeitBegins 8e3f42051f
NIFI-7348 Wait - Removes WAIT_START_TIMESTAMP after expiration
This closes #4201.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2020-04-16 15:56:40 +09:00
Bence Simon 83400789f6
NIFI-7292 Preventing file listing from fail because of insufficient privileges
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4195.
2020-04-15 18:54:51 +02:00
Mark Payne a61a4c2b58
NIFI-7347: Fixed NullPointerException that can happen if a bin is merged due to timeout and has no records
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4210.
2020-04-15 12:39:13 +02:00
Tamas Palfy 09cece8e99 NIFI-7314 HandleHttpRequest stops Jetty in OnUnscheduled instead of OnStopped. Also reject pending request and clean their queue when shutting down.
NIFI-7314 In HandleHttpRequest returning 503 when rejecting pending requests before shutdown.
NIFI-7314 In HandleHttpRequest add logs and better response message during cleanup.

This closes #4191.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-07 20:12:18 +02:00
Tamas Palfy c79473baf5 NIFI-7286 ListenTCPRecord cleanup changed from @OnStopped to @OnUnscheduled 2020-03-26 11:45:09 -04:00
Joe Witt f694e6464f NIFI-7187 adding missing version strings from accumulo bundle pom
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE

This closes #4086

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-20 10:07:56 -04:00
M Tien f91d6c420d
NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… (#4151)
* NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library.
Updated LICENSE and NOTICE files to reflect changes.
Updated unit tests.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7268 Fixed typo in Javadoc.

Co-authored-by: Andy LoPresto <alopresto@apache.org>
2020-03-17 19:49:15 -07:00
Matthew Burgess ace5754524 NIFI-7208: Restore default timezone in JdbcCommon 2020-03-13 15:00:24 -04:00
Mark Payne 0ad58200af
NIFI-7256: This closes #4142. Fixed thresholds in unit test. Instead of assuming that multiple runs of the processor will occur within 100 milliseconds, allowed the multiple runs to occur within 3 mins of one another.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-13 14:05:40 -04:00
Joe Witt 561be89a21
NIFI-7250 activate user.timezone appropriate to each region
NIFI-7250 fix a test which appears brittle at least on windows builds on slow environments
NIFI-7250 activated a timezone run for AU Australia/Melbourne which exposed a poor magic number and needless assertion but interesting results worth keeping

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4140.
2020-03-13 17:30:19 +01:00
Joe Witt 97e250cdaa
NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4132.
2020-03-12 19:13:59 +01:00
Mark Payne 67676ba5b4 Fixed bug in JsonRowRecordReader when reading a 'raw' record with a schema that indicates that a field should be a Map. Also updated unit test to explicitly define schema, since schema inference will never return a Map but rather a Record 2020-03-11 14:29:44 -04:00
Pierre Villard cd83e70b91 unit test reproducing the issue 2020-03-11 14:29:44 -04:00
Pierre Villard a679e88b6f
NIFI-4970 - Add a property to deal with empty GZIP HTTP response
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4127.
2020-03-11 10:08:51 +01:00
Pierre Villard 578430c9d9 NIFI-7197 - In-place replacement in LookupRecord processor
This closes #4088

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 15:35:59 -05:00
Joe Witt 20dda05f26
NIFI-7232 if users do not supply a remote path PutSFTP with conflict resolution will fail with an NPE 2020-03-06 12:46:02 -05:00
Joe Witt 040c8a0af9
NIFI-7222 Cleaned up API for FTP/SFTP remote file retrieval and ensure we close remote file resources for SFTP pulls in particular
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4115.
2020-03-06 12:48:15 +01:00
Matthew Burgess 74b1b2fc59 NIFI-7208: Fixed PutSQL/JdbcCommon handling of timestamps (nanoseconds, e.g.) 2020-03-04 11:46:47 -05:00
Eduardo Fontes bad254ec5b
NIFI-4970 - EOF Exception in InvokeHttp when body's response is empty with gzip
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4109.
2020-03-04 10:36:06 +01:00
karthik.kadajji 2cf4fde686
NIFI-5644 Fixed typo in getWrappedQuery method of AbstractDatabaseFetchProcessor class
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4106.
2020-03-03 16:43:56 +01:00
Joe Witt 4cd63c99e8
NIFI-7205 NIFI-7206
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4093.
2020-02-26 14:02:49 -08:00
Shawn Weeks 0bb8ce7438
NIFI-7139 Add record.error.message on failure of a record reader or writer
Handle scenario where message might be null.

Update to test case that was failing because adding attributes modified a flow file even if you don't change the contents.

Fixed Style Issues and Updated WritesAttributes.

Added Test Case for Error Message

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4052
2020-02-26 08:50:32 -05:00
Pierre Villard 4f315edc6e
NIFI-7133 - This closes #4049. Clarification of EnforceOrder description
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-23 11:59:49 -08:00
Pierre Villard fd7c01453b
NIFI-7184 - This closes #4074. Added mime type property to GenerateFlowFile
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-23 11:54:28 -08:00
Pierre Villard 9a8a551e03
NIFI-7183 - This closes #4073. Improve ReplaceText when removing FF's content
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-23 11:50:34 -08:00
Joe Witt f8a5d1295c
NIFI-7175 removed appveyor and travis builds which have become unreliable. Added Github Actions based CI build for pull requests and pushes. Removed unit tests which make unreadable or unaccesible dirs. Includes windows build as well as ubuntu. we can do localization based builds later as this is all proven stable.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4058.
2020-02-20 16:36:51 -08:00
Troy Melhase 0de89452f1
NIFI-6927 Consolidate SSL context and trust managers for OkHttp on JDK9.
Fixes name conflicts.

This closes #4047.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-19 16:19:39 -08:00
David Savage 5e964fbc47
NIFI-4792: Add support for querying array fields in QueryRecord
Work in progress adding support for array based queries
updated calcite dependency

tidy up unused imports highlighted by checkstyle in travis build

tidy up }s highlighted by checkstyle in travis build

Add test for use case referenced in NIFI-4792

Bumped Calcite version to 1.21.0

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4015
2020-02-11 10:59:20 -05:00
John Highcock f738e19a75
NIFI-2537: Add custom MIME type configuration support to IdentifyMimeType
Add two new properties to IdentifyMimeType (Config File and Config Body).
Specifying one of these properties will override the default NiFi MIME
type configuration and use the configured property's MIME config instead.
Add additional runtime documentation for IdentifyMimeType's usage. The
default behavior of IdentifyMimeType is unchanged.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4016.
2020-02-06 19:34:18 -05:00
mdayakar 850869c6d2
NIFI-7049 : SFTP processors shouldn't silently try to access known hosts file of the user
Signed-off-by: Arpad Boda <aboda@apache.org>

This closes #4014
2020-01-30 15:15:43 +01:00
Joe Witt 3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00