41 Commits

Author SHA1 Message Date
Juldrixx
b04632129f
NIFI-9798 Added Proxy Support for ConsumeGCPubSub and PublishGCPubSub
This closes #5868

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-05-02 11:37:47 -05:00
Peter Turcsanyi
157a5d342a
NIFI-9898: Used CloseableHttpClient and replaced deprecated class/method usages in ProxyAwareTransportFactory in GCP module
Also bumped GCP dependencies

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5950.
2022-04-11 16:50:07 +02:00
exceptionfactory
fee7c16732
NIFI-9679 Added access-environment-credentials permission
- Applied new permission restrictions to AWSCredentialsProviderControllerService and GCPCredentialsControllerService

Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5796.
2022-03-02 12:31:09 -05:00
Mike Thomsen
13be48cfe3
NIFI-9113 Refactored nifi-grpc-bundle to use JUnit 5
- NIFI-9111 Refactored nifi-gcp-bundle to use JUnit 5
- NIFI-9110 Refactored nifi-flume-bundle to use JUnit 5
- NIFI-9108 Refactored nifi-evtx-bundle to use JUnit 5
- NIFI-9106 Refactored nifi-email-bundle to use JUnit 5
- NIFI-9105 Refactored nifi-easyrules-bundle to use JUnit 5
- NIFI-9104 Refactored nifi-druid-bundle to use JUnit 5

This closes #5790

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-24 16:59:44 -06:00
exceptionfactory
a2f6420f43
NIFI-9638 Refactored Google Guava usage in extensions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5742
2022-02-09 16:39:53 -05:00
Tamas Palfy
990285ba1c NIFI-8676 Added 'Tracking Entities' listing strategy to 'ListS3' and 'ListGCSBucket'
This closes #5413.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-01-10 20:35:04 +01:00
Joe Gresock
f3c7537d9b
NIFI-9343: Adding config verification to GCP processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5494.
2021-12-11 19:58:01 +01:00
Mark Payne
7a83c32a01
NIFI-9441: Ensure that we only update our member variable for the latest timestamp after processing all objects within the GCS Bucket
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5567.
2021-12-04 23:49:27 +01:00
Pierre Villard
37c0527a72
NIFI-9304 - Adding PublishGCPubSubLite and ConsumeGCPubSubLite processors
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5460.
2021-11-08 09:25:35 -05:00
Pierre Villard
d70359d3e8
NIFI-6717 - Add Job ID in attributes for PutBigQueryBatch
This closes #3768

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-09-18 09:08:18 -05:00
Juldrixx
0c96e573a4
NIFI-8442 Add a new test with date, timestamp and time as string & New management of date, time and timestamp
NIFI-8442 Put DateTimeFormatter as static and Add comments to explain why ZoneOffset.UTC is required

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5014.
2021-07-22 20:05:53 +02:00
Mark Payne
46b1f6755c NIFI-8631: Ensure that GCP Pub/Sub messages are not acknowledged until session has been committed, in order ot ensure that we don't have data loss
This closes #5102.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-26 21:54:31 +02:00
Mark Payne
ecacfdaa4c NIFI-8469: Introduced ProcessSession.commitAsync and updated processors to use it. Deprecated ProcessSession.commit()
- Updated Mock Framework to now fail tests that use ProcessSession.commit() unless they first call TestRunner.setAllowSynchronousSessionCommits(true)
- Updated stateless nifi in order to make use of async session commits
- Fixed bug that caused stateless to not properly handle Additional Classpath URLs and bug that caused warnings about validation to get generated when a flow that used controller services was initialized. While this is not really in scope of NIFI-8469, it was found when testing and blocked further progress so addresssed here.
- If Processor fails to progress when run from stateless, trigger from start of flow until that is no longer the case
- Introduced notion of TransactionThresholds that can limit the amount of data that a flow will bring in for a given invocation of stateless dataflow
- Several new system-level tests
2021-05-19 08:51:16 -04:00
Paul Kelly
0ed3534524 NIFI-7886 Added read by range to Azure, GCS, S3 fetch processors
This closes #4576

Co-authored-by: Joey Frazee <jfrazee@apache.org>
Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-27 12:08:06 -07:00
Mark Payne
7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
Mark Payne
097edf4f7c
NIFI-8136: Added getState/setState/replaceState/clearState methods to ProcessSession, updated processors to use these methods instead of StateManager version where appropriate
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4757
2021-01-15 10:02:09 -05:00
Pierre Villard
27e7d0b02a
NIFI-7707 - This closes #4451. fixed Publish GCP PubSub processor for binary data
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-08-07 08:14:19 -07:00
Cuong Nguyen
c7a235f854 NIFI-7629: Make property PROJECT_ID evaluate EL on processors ConsumeGCPubSub & PublishGCPubSub 2020-07-13 08:54:41 -04:00
Mark Payne
0a16002076
NIFI-7509: Added optional Record Writer property to all List* Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4315.
2020-06-19 17:30:40 +02:00
Pierre Villard
8162edfd98 NIFI-6701 - Fix for PublishGCPPubSub 2020-05-29 13:38:11 -04:00
Pierre Villard
b12a9ad446
NIFI-6159 - Add BigQuery processor using the Streaming API
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3394.
2019-10-03 11:37:00 +02:00
Peter Turcsanyi
21a27c8bb0 NIFI-6636: Fixed ListGCSBucket file duplication error
ListGCSBucket duplicated files if they arrived not in alphabetical order.
The set storing the name of the latest blob (which was loaded with the highest
timestamp during the previous run of the processor) was cleared too early.

Also changed the state persisting logic: it is now saved only once at the end
of onTrigger() (similar to ListS3). Some inconsistent state (only blob names
without the timestamp) was also saved earlier.

This closes #3702.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-09 13:32:17 -04:00
Pierre Villard
639e81e5a1
NIFI-6407 - added support for useAvroLogicalTypes in PutBigQueryBatch
fix Maven dep

This closes #3592.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-31 17:07:37 +09:00
Pierre Villard
630c651226
NIFI-6334 - fix custom validate error in PutBigqueryBatch
This closes #3589.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-19 13:49:23 +09:00
Pierre Villard
2336044b90 NIFI-6059 - Fixed NPE in GCS processors when project ID is not specified
This closes #3327.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 09:45:13 -05:00
Pierre Villard
03ef646547
NIFI-4731 This closes #3019. This closes #2682. This closes #2420.
NIFI-4933 BigQuery PR Review

Signed-off-by: joewitt <joewitt@apache.org>
2018-11-21 10:28:40 -05:00
Daniel Jimenez
444caf8a78
NIFI-4731: BQ Processors and GCP library update.
Signed-off-by: joewitt <joewitt@apache.org>
2018-11-21 10:27:41 -05:00
Koji Kawamura
5a58c9a171
NIFI-5282: Add ProxyConfigurationService to GCSProcessors
This closes #2981
2018-08-31 11:07:09 -04:00
jugi92
cdae2b14b3 NIFI-5282: GCPProcessor with HTTP Proxy with Auth
added http proxy support with authentication for GCP processors
added proxy support for Google Credential Service

This closes #2943.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-31 20:28:00 +09:00
Andy LoPresto
f60585a9b6
NIFI-5376 Removed deprecation warnings.
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.

This closes #2841.

Signed-off-by: Joe Witt <joewitt@apache.org>
2018-07-09 20:45:34 -07:00
Mike Thomsen
d2fd7e5e7d NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
This closes #2765

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-07 20:38:25 +05:30
zenfenan
504152eaa1 NIFI-5261: Added JSON_VALIDATOR to StandardValidators
This closes #2758

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-05 18:24:47 -04:00
zenfenan
1663a6c094 NIFI-5133: Implemented Google Cloud PubSub Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2724.
2018-05-30 09:47:18 +02:00
zenfenan
0973c2d8d1 NIFI-543 Added annotation to restrict processor to run only on the primary node
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly

This closes #2509.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-25 11:50:29 -04:00
zenfenan
f742a3a6ac NIFI-5156: Updated GCP SDK to latest version
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2680.
2018-05-09 18:19:17 +02:00
Pierre Villard
4c787799ff NIFI-4149 - Indicate if EL is evaluated against FFs or not
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
2018-04-06 12:40:54 -04:00
Andre F de Miranda
8d4fe38bb4 NIFI-4197 - Expose some proxy settings to GCS Processors
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2017.
2018-01-06 17:33:31 -08:00
joewitt
7467bb7b0f
NIFI-4730 restructure of nifi-gcp-bundle
This closes #2367.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-01-03 11:21:52 -05:00
Andy LoPresto
7d242076ce
NIFI-3594 Implemented encrypted provenance repository.
Added src/test/resources/logback-test.xml files resetting log level from DEBUG (in nifi-data-provenance-utils) to WARN because later tests depend on MockComponentLog recording a certain number of messages and this number is different than expected if the log level is DEBUG.

This closes #1686.

Signed-off-by: Bryan Bende, Yolanda M. Davis, and Mark Payne
2017-05-02 13:24:07 -04:00
Andre F de Miranda
cefc02c5fb NIFI-3514 - Ensures ListGCSBucket triggers serially and forbids inputs
This closes #1526.

Signed-off-by: James Wing <jvwing@gmail.com>
2017-02-22 10:24:37 -08:00
Gene Peters
897c70298a NIFI-3449: Adding Google Cloud Storage Bundle
* Credentials service with tests
* Abstract processor definitions
* GCS-themed processors and their corresponding tests

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1482.
2017-02-12 19:52:38 -08:00