954 Commits

Author SHA1 Message Date
Otto Fowler
f22cd26cf4 NIFI-7420 remove the http.param attribute. It contained both the query parameters already captured in http.query.param _and_ the multipart form data names and values, which are captured in the part data, and could be very very large
This closes #4251.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-06 17:20:08 -04:00
Alan Jackoway
effa930151
NIFI-7425 Log Message for ReplaceText Over Buffer Size
Adds a log message when ReplaceText sends a flowfile to the failure relationship because
it is larger than the max buffer size.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4255.
2020-05-06 09:40:46 +02:00
Otto Fowler
659a383723 NIFI-7394: Add support for sending Multipart/FORM data to InvokeHTTP.
By using dynamic properties with a prefix naming scheme, allow
definition of the parts, including the name to give the Flowfile content
part, and optionally it's file name.
After review:
- change so that we can send just the form content or just form data
  without the flowfile
- change the content name and content file name from dynamic properties
  to properties
- change the dynamic name to be an invalid http header "post:form:xxxx"
- add validation and more tests

This closes #4234.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-04-29 13:01:06 -04:00
EndzeitBegins
8e3f42051f
NIFI-7348 Wait - Removes WAIT_START_TIMESTAMP after expiration
This closes #4201.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2020-04-16 15:56:40 +09:00
Bence Simon
83400789f6
NIFI-7292 Preventing file listing from fail because of insufficient privileges
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4195.
2020-04-15 18:54:51 +02:00
Mark Payne
a61a4c2b58
NIFI-7347: Fixed NullPointerException that can happen if a bin is merged due to timeout and has no records
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4210.
2020-04-15 12:39:13 +02:00
Tamas Palfy
09cece8e99 NIFI-7314 HandleHttpRequest stops Jetty in OnUnscheduled instead of OnStopped. Also reject pending request and clean their queue when shutting down.
NIFI-7314 In HandleHttpRequest returning 503 when rejecting pending requests before shutdown.
NIFI-7314 In HandleHttpRequest add logs and better response message during cleanup.

This closes #4191.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-07 20:12:18 +02:00
Tamas Palfy
c79473baf5 NIFI-7286 ListenTCPRecord cleanup changed from @OnStopped to @OnUnscheduled 2020-03-26 11:45:09 -04:00
Joe Witt
f694e6464f NIFI-7187 adding missing version strings from accumulo bundle pom
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE

This closes #4086

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-20 10:07:56 -04:00
M Tien
f91d6c420d
NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… (#4151)
* NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library.
Updated LICENSE and NOTICE files to reflect changes.
Updated unit tests.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7268 Fixed typo in Javadoc.

Co-authored-by: Andy LoPresto <alopresto@apache.org>
2020-03-17 19:49:15 -07:00
Matthew Burgess
ace5754524 NIFI-7208: Restore default timezone in JdbcCommon 2020-03-13 15:00:24 -04:00
Mark Payne
0ad58200af
NIFI-7256: This closes #4142. Fixed thresholds in unit test. Instead of assuming that multiple runs of the processor will occur within 100 milliseconds, allowed the multiple runs to occur within 3 mins of one another.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-13 14:05:40 -04:00
Joe Witt
561be89a21
NIFI-7250 activate user.timezone appropriate to each region
NIFI-7250 fix a test which appears brittle at least on windows builds on slow environments
NIFI-7250 activated a timezone run for AU Australia/Melbourne which exposed a poor magic number and needless assertion but interesting results worth keeping

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4140.
2020-03-13 17:30:19 +01:00
Joe Witt
97e250cdaa
NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4132.
2020-03-12 19:13:59 +01:00
Mark Payne
67676ba5b4 Fixed bug in JsonRowRecordReader when reading a 'raw' record with a schema that indicates that a field should be a Map. Also updated unit test to explicitly define schema, since schema inference will never return a Map but rather a Record 2020-03-11 14:29:44 -04:00
Pierre Villard
cd83e70b91 unit test reproducing the issue 2020-03-11 14:29:44 -04:00
Pierre Villard
a679e88b6f
NIFI-4970 - Add a property to deal with empty GZIP HTTP response
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4127.
2020-03-11 10:08:51 +01:00
Pierre Villard
578430c9d9 NIFI-7197 - In-place replacement in LookupRecord processor
This closes #4088

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-06 15:35:59 -05:00
Joe Witt
20dda05f26
NIFI-7232 if users do not supply a remote path PutSFTP with conflict resolution will fail with an NPE 2020-03-06 12:46:02 -05:00
Joe Witt
040c8a0af9
NIFI-7222 Cleaned up API for FTP/SFTP remote file retrieval and ensure we close remote file resources for SFTP pulls in particular
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4115.
2020-03-06 12:48:15 +01:00
Matthew Burgess
74b1b2fc59 NIFI-7208: Fixed PutSQL/JdbcCommon handling of timestamps (nanoseconds, e.g.) 2020-03-04 11:46:47 -05:00
Eduardo Fontes
bad254ec5b
NIFI-4970 - EOF Exception in InvokeHttp when body's response is empty with gzip
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4109.
2020-03-04 10:36:06 +01:00
karthik.kadajji
2cf4fde686
NIFI-5644 Fixed typo in getWrappedQuery method of AbstractDatabaseFetchProcessor class
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4106.
2020-03-03 16:43:56 +01:00
Joe Witt
4cd63c99e8
NIFI-7205 NIFI-7206
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4093.
2020-02-26 14:02:49 -08:00
Shawn Weeks
0bb8ce7438
NIFI-7139 Add record.error.message on failure of a record reader or writer
Handle scenario where message might be null.

Update to test case that was failing because adding attributes modified a flow file even if you don't change the contents.

Fixed Style Issues and Updated WritesAttributes.

Added Test Case for Error Message

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4052
2020-02-26 08:50:32 -05:00
Pierre Villard
4f315edc6e
NIFI-7133 - This closes #4049. Clarification of EnforceOrder description
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-23 11:59:49 -08:00
Pierre Villard
fd7c01453b
NIFI-7184 - This closes #4074. Added mime type property to GenerateFlowFile
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-23 11:54:28 -08:00
Pierre Villard
9a8a551e03
NIFI-7183 - This closes #4073. Improve ReplaceText when removing FF's content
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-02-23 11:50:34 -08:00
Joe Witt
f8a5d1295c
NIFI-7175 removed appveyor and travis builds which have become unreliable. Added Github Actions based CI build for pull requests and pushes. Removed unit tests which make unreadable or unaccesible dirs. Includes windows build as well as ubuntu. we can do localization based builds later as this is all proven stable.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4058.
2020-02-20 16:36:51 -08:00
Troy Melhase
0de89452f1
NIFI-6927 Consolidate SSL context and trust managers for OkHttp on JDK9.
Fixes name conflicts.

This closes #4047.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-02-19 16:19:39 -08:00
David Savage
5e964fbc47
NIFI-4792: Add support for querying array fields in QueryRecord
Work in progress adding support for array based queries
updated calcite dependency

tidy up unused imports highlighted by checkstyle in travis build

tidy up }s highlighted by checkstyle in travis build

Add test for use case referenced in NIFI-4792

Bumped Calcite version to 1.21.0

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4015
2020-02-11 10:59:20 -05:00
John Highcock
f738e19a75
NIFI-2537: Add custom MIME type configuration support to IdentifyMimeType
Add two new properties to IdentifyMimeType (Config File and Config Body).
Specifying one of these properties will override the default NiFi MIME
type configuration and use the configured property's MIME config instead.
Add additional runtime documentation for IdentifyMimeType's usage. The
default behavior of IdentifyMimeType is unchanged.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4016.
2020-02-06 19:34:18 -05:00
mdayakar
850869c6d2
NIFI-7049 : SFTP processors shouldn't silently try to access known hosts file of the user
Signed-off-by: Arpad Boda <aboda@apache.org>

This closes #4014
2020-01-30 15:15:43 +01:00
Joe Witt
3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt
633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Joe Witt
b205b99668
NIFI-7041 This closes #4000. Ensure that if the permissions arent set by the flowfile or processor property that we dont attempt to set perms on the remote host
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-19 08:47:58 -05:00
Joe Witt
0789ec3a6b
NIFI-6882 from plain patch file in JIRA contributed by 'Josh'
Removed tests which created dirs outside target location and which do not appear unit testable.
2020-01-18 21:02:50 -05:00
Nathan Gough
4ec9155cbc
NIFI-6770 - Set validator to Validator.VALID to allow empty password for truststores.
Added no-password keystore for tests
System NiFi truststore now allows a passwordless truststore. Added a unit test to prove this.
Forgot no-password-truststore.jks file for the unit test.
Refactored utility method from CertificateUtils to KeyStoreUtils.
Added utility methods to verify keystore and key passwords.
Added unit tests.
Implemented different keystore and truststore validation logic.
Refactored internal custom validation in StandardSSLContextService.
Added unit test resource for keystore with different key and keystore passwords.
Added unit test to generate passwordless truststore for https://nifi.apache.org for live testing.
Resolved NPE in SSLContext generation in StandardSSLContextService
Added unit test to generate passwordless truststore for localhost for InvokeHTTP testing.
Resolved TrustManagerFactoryImpl initialization error.
Fixed unit test without proper cleanup which caused RAT failures.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

This closes #3823.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-17 20:01:06 -08:00
Joe Witt
23c8234586
NIFI-7031 updating copyright year on NOTICES 2020-01-15 16:10:31 -05:00
Michael Hogue
421bfdd5ff
NIFI-6919: Added relationship attribute to DistributeLoad
NIFI-6919: Cleaned up docs

NIFI-6919: Cleanup

NIFI-6919: Cleanup

NIFI-6919: added negative unit test

NIFI-6919: Removed unnecesary feature flag

Updated attribute description

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3939
2020-01-14 16:14:48 -05:00
John Pierce
0dc6439ddb
NIFI-6964 This closes #3975. Use compression level for xz-lzma2 format of the CompressContent processor
NIFI-6965 adding resource warning on memory and highlighting this with the compression level property

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 15:08:13 -05:00
Yolanda Davis
ebd33452c4 NIFI-6962 - moved HashAlgorithm and HashService to nifi-security-utils (#3947) 2020-01-08 11:10:58 -08:00
Matthew Burgess
c604923c0b NIFI-6947: Add PutRecord processor using RecordSinkService (#3943)
* NIFI-6947: Add PutRecord processor using RecordSinkService

* NIFI-6947: Incorporated review comments
2020-01-08 04:25:14 -04:00
Matthew Burgess
f4ef45678e
NIFI-6963: Fix ValidateRecord handling of missing required array fields
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3948.
2020-01-08 02:02:04 +07:00
Shawn Weeks
b470db620b
NIFI-6979 Add record.index field to UpdateRecord
This closes #3955

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-01-03 19:25:01 -05:00
Matthew Burgess
8b3f6fa61c NIFI-6953: Add missing documentation for input.flowfile.uuid attribute of ExecuteSQL processors 2019-12-16 14:48:12 -05:00
Pierre Villard
ac5bacccb8
NIFI-6839 - Upgrade jackson-databind direct dependencies
This closes #3870

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-11-25 10:58:22 -05:00
Matthew Burgess
d617c0b96a NIFI-6865: Added Fetch Size property to ExecuteSQL processors
This closes #3888.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2019-11-19 23:33:46 +01:00
Koji Kawamura
b3880a4a06
NIFI-5970 Handle multiple input FlowFiles at Put.initConnection
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3583
2019-11-19 16:48:14 -05:00
Koji Kawamura
86cae184ff
NIFI-6599 Fix MergeRecord failure in defragment mode
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3678.
2019-11-14 14:44:01 +01:00