231 Commits

Author SHA1 Message Date
Tamas Palfy
990285ba1c NIFI-8676 Added 'Tracking Entities' listing strategy to 'ListS3' and 'ListGCSBucket'
This closes #5413.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-01-10 20:35:04 +01:00
exceptionfactory
6039095625
NIFI-9267 Replaced nifi-standard-web-test-utils with mockwebserver
- Replaced instances of custom Jetty TestServer with OkHttp MockWebServer
- Removed integration tests referencing nifi-standard-web-test-utils SslContextUtils
- Removed nifi-standard-web-test-utils

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5620.
2022-01-09 05:25:32 -05:00
Chris Sampson
bd1b2b4d69 NIFI-9520 update AWS SDK and Kinesis Client Library versions; Allow PutS3Object to use all available StorageClasses
This closes #5632.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-01-06 13:43:23 +01:00
Paul Grey
0d49371c53
NIFI-7371 Added FlowFile attributes for exceptions in S3 processors
This closes #5606

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-12-21 22:35:31 -06:00
exceptionfactory
7c6bdcb035
NIFI-9491 Excluded commons-logging and added jcl-over-slf4j references
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5608
2021-12-17 07:21:03 -05:00
Joe Gresock
f07c37285e NIFI-9353: Adding Config Verification to AWS Processors
This closes #5504
2021-12-11 14:33:23 +00:00
Joe Witt
c055895952
NIFI-9344-RC3 prepare for next development iteration 2021-11-03 08:53:37 -07:00
Joe Witt
7fdc07cccd
NIFI-9344-RC3 prepare release nifi-1.15.0-RC3 2021-11-03 08:53:32 -07:00
Joe Gresock
ca530f40d8
NIFI-9317: Updating config verification for ListS3 (#5485) 2021-10-27 13:47:58 -04:00
Lehel Boér
32c7d856c8
NIFI-9300: Fix AWSCredentialsService EL attribute evaluation. (#5456) 2021-10-14 10:46:15 -04:00
Joe Gresock
05407476b4
NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless (#5391)
NIFI-9174: Adding AWS SecretsManager ParamValueProvider for Stateless
2021-10-08 16:56:45 -04:00
Mark Payne
607480c1a0
NIFI-9239: Updated Consume/Publish Kafka processors to support Exactly Once Semantics. Added ExecuteStateless processor/bundle
- A few minor updates to Stateless in order to surface some concepts from the stateless engine up to the caller, such as bulletins, counters, etc.

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5412.
2021-10-04 15:32:21 -04:00
Mark Payne
baf29e59f1
NIFI-9009: Created VerifiableProcessor, VerifiableControllerService, VerifiableReportingTask components; implemented backend work to call the methods. Added REST APIs and created/updated data models for component configuration verification
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5288
2021-09-22 19:00:23 -04:00
ScottGiles
d97bec63ed
NIFI-8941: Adding ENDPOINT_OVERRIDE to properties List of PutLambda.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5244.
2021-09-18 15:17:06 +02:00
Alasdair Brown
50e60c812d
NIFI-8087 Adds MessageGroupID and MessageDuplicationId to PutSNS
Updated topic arn assert to match dummy topic name with .fifo suffix

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5367.
2021-09-18 15:08:02 +02:00
Joey
38a1f476e3
NIFI-9055 Added handling for 0- read range to FetchS3Object (#5317) 2021-09-13 19:19:19 +02:00
Paul Grey
cbc25aa180
NIFI-9197 - Improve AWS Surefire Run Time
This closes #5370

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-09-08 06:46:48 -04:00
Paul Grey
6ce9919a1f NIFI-9197 - Improve AWS Surefire Run Time 2021-09-08 06:46:26 -04:00
exceptionfactory
20804ff2b1
NIFI-8749 Removed implicit time zone conversion to GMT
- Updated DataTypeUtils.getDateFormat() to use system default time zone
- Updated Record Path Guide to match Expression Language Guide regarding default time zone
- Updated impacted unit tests to expect localized dates instead of dates converted to GMT

NIFI-8749 Set user.timezone in TestRecordPath to avoid unexpected date conversions

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5361
2021-08-31 16:07:53 -04:00
Matthew Burgess
dfa6592a47
NIFI-8749 Revert "Removed implicit time zone conversion to GMT"
This reverts commit 758e1850ed0e4af5063877c7e3a56628e5e7ed78.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>
2021-08-31 13:33:53 -04:00
exceptionfactory
758e1850ed
NIFI-8749 Removed implicit time zone conversion to GMT
- Updated DataTypeUtils.getDateFormat() to use system default time zone
- Updated Record Path Guide to match Expression Language Guide regarding default time zone
- Updated impacted unit tests to expect localized dates instead of dates converted to GMT

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5305
2021-08-31 10:40:57 -04:00
Joe Witt
97feacc181
NIFI-8767-RC2 prepare for next development iteration 2021-07-10 12:17:09 -07:00
Joe Witt
fcbf1d5f97
NIFI-8767-RC2 prepare release nifi-1.14.0-RC2 2021-07-10 12:17:05 -07:00
exceptionfactory
2efc2b6b8b
NIFI-8718 Upgraded Apache Commons IO to 2.10.0
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5169
2021-06-21 11:29:27 -04:00
Peter Turcsanyi
820b2cff29 NIFI-8662: Using the configured AWS region when parsing from VPCE endpoint URL fails in AbstractAWSProcessor
This closes #5140.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-06-16 21:50:25 +02:00
Mark Payne
ecacfdaa4c NIFI-8469: Introduced ProcessSession.commitAsync and updated processors to use it. Deprecated ProcessSession.commit()
- Updated Mock Framework to now fail tests that use ProcessSession.commit() unless they first call TestRunner.setAllowSynchronousSessionCommits(true)
- Updated stateless nifi in order to make use of async session commits
- Fixed bug that caused stateless to not properly handle Additional Classpath URLs and bug that caused warnings about validation to get generated when a flow that used controller services was initialized. While this is not really in scope of NIFI-8469, it was found when testing and blocked further progress so addresssed here.
- If Processor fails to progress when run from stateless, trigger from start of flow until that is no longer the case
- Introduced notion of TransactionThresholds that can limit the amount of data that a flow will bring in for a given invocation of stateless dataflow
- Several new system-level tests
2021-05-19 08:51:16 -04:00
exceptionfactory
6776765a92
NIFI-8538 Upgraded Apache Commons IO to 2.8.0
- Upgraded direct dependencies from 2.6 to 2.8.0
- Added dependency management configuration to use 2.8.0 for some modules
- Updated scripted Groovy tests to avoid copying unnecessary files

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5073
2021-05-14 09:31:48 -04:00
Chris Sampson
a274c12bbb NIFI-2892 Implement AWS Kinesis Stream Consume Processor
This closes #4822.

Co-authored-by: uday <udaygkale@gmail.com>

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-12 20:03:28 +02:00
Paul Kelly
0ed3534524 NIFI-7886 Added read by range to Azure, GCS, S3 fetch processors
This closes #4576

Co-authored-by: Joey Frazee <jfrazee@apache.org>
Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-27 12:08:06 -07:00
Mark Payne
7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
Nathan Gough
07a4966d10
NIFI-8329 - Updated dependencies with no build failures
NIFI-8329 - Removed unnecessary jackson.version from azure bundle to use the global property instead.

NIFI-8329 - Updated jackson/jackson-databind version and removed the 'jackson-databind.version' pom property in favor of 'jackson.version'

Updated dependencies include the following:

- jackson-core
- jackson-databind
- icu4j
- snakeyaml
- spring-integration-mail
- spring-core and framework modules
- activemq-client
- activemq-broker
- xercesImpl

This closes #4911

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-19 14:46:33 -05:00
Joe Witt
88fab00e29
NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT 2021-02-15 12:09:32 -07:00
Joe Witt
4afb2ba743
NIFI-7873-RC4 prepare for next development iteration 2021-02-15 12:09:31 -07:00
Joe Witt
487280bee9
NIFI-7873-RC4 prepare release nifi-1.13.0-RC4 2021-02-15 12:09:30 -07:00
Mark Payne
097edf4f7c
NIFI-8136: Added getState/setState/replaceState/clearState methods to ProcessSession, updated processors to use these methods instead of StateManager version where appropriate
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4757
2021-01-15 10:02:09 -05:00
exceptionfactory
817f621d6f NIFI-8096 Deprecated ClientAuth references in SSLContextService
- Added SSLContextService.createContext() and refactored referencing components
- Removed references to ClientAuth from SslContextFactory methods

This closes #4737.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-06 16:24:26 -05:00
Peter Turcsanyi
02665d7561
NIFI-8092: Fix ITPutS3Object test failure due to Outposts storage class
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4728.
2020-12-16 12:43:24 +01:00
Peter Turcsanyi
ec3d5f89f0
NIFI-8086: Fix file separator char in PutS3Object multipart state directory path
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4726.
2020-12-13 10:18:32 +01:00
r65535
2f99d6fce4
NIFI-4540 - Added FIFO options to PutSQS
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4705.
2020-12-07 13:11:22 +01:00
Pierre Villard
9c1656dbff NIFI-7854 - Add proxy username/password support to SQS processors
This closes #4600.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-10-27 00:22:15 +01:00
Mark Payne
5d37809d30
NIFI-7874: Fixed bug that results in buffering all S3 elements in heap before emitting any of them. Added Batch Size property to ListS3
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4566.
2020-10-02 19:12:40 +02:00
Pierre Villard
a57d38c58d
NIFI-7827 - PutS3Object - configuration of the multipart temp dir (#4541) 2020-09-29 09:15:09 -04:00
Pierre Villard
0996674522 NIFI-7834 - InvokeAWSGatewayAPI should not be annotated with @TriggerWhenEmpty 2020-09-22 15:58:16 -04:00
Andy LoPresto
9370571131
NIFI-7804 Split nifi-security-utils into sub-module for nifi-security… (#4533)
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.

* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.

* NIFI-7804 Resolved failing unit test.

* NIFI-7804 Removed legacy dependency.

* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
2020-09-17 12:52:22 -04:00
Joe Witt
8baa5c9940
NIFI-7692 updating for next dev release 1.13.0 2020-08-18 14:48:02 -07:00
Joe Witt
fb57bcbc11
NIFI-7692-RC1 prepare for next development iteration 2020-08-13 09:20:39 -07:00
Joe Witt
303d6c59ba
NIFI-7692-RC1 prepare release nifi-1.12.0-RC1 2020-08-13 09:20:36 -07:00
Kent Nguyen
ddb304a927 NIFI-7664 Add Content Disposition property to PutS3Object processor
Add new property 'Content Disposition' to allow user
to set the content-disposition http header on the S3 object.

Allowed values are 'inline' (default) and 'attachment'.
If 'attachment' is selected, the filename will be set to the S3 Object key.

Remove default value and keep backward compatibility
Update fetchS3Object filename attribute settin
Update constant names
Update order of if-else condition
NIFI-7664 Update condition in FetchS3Processor
NIFI-7664 Undo the unexpected indent
NIFI-7664 Update international chars unit test
NIFI-7664 Set fetchS3 file path name
NIFI-7664 Update code style

This closes #4423.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-08-03 18:27:35 +02:00
Kent Nguyen
0cff54097e NIFI-6332: Add Cache Control property to PutS3Object processor
Add new property 'Cache Control' to allow user to
set the cache-control http header on the S3 object.

This property is not required, and has no default value.

The implementation is similar to the Content-Type property,
except that this property does not allow Expression Language.

Update property description

Add support EL for cache-control property

This closes #4422.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-07-30 08:37:10 +02:00
Mike Thomsen
8d53f5d0c9
NIFI-7497 Removed a few style check bugs that crept up in the last commit. 2020-07-10 17:12:27 -04:00