Commit Graph

21 Commits

Author SHA1 Message Date
Pierre Villard b12a9ad446
NIFI-6159 - Add BigQuery processor using the Streaming API
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3394.
2019-10-03 11:37:00 +02:00
Peter Turcsanyi 21a27c8bb0 NIFI-6636: Fixed ListGCSBucket file duplication error
ListGCSBucket duplicated files if they arrived not in alphabetical order.
The set storing the name of the latest blob (which was loaded with the highest
timestamp during the previous run of the processor) was cleared too early.

Also changed the state persisting logic: it is now saved only once at the end
of onTrigger() (similar to ListS3). Some inconsistent state (only blob names
without the timestamp) was also saved earlier.

This closes #3702.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-09 13:32:17 -04:00
Pierre Villard 639e81e5a1
NIFI-6407 - added support for useAvroLogicalTypes in PutBigQueryBatch
fix Maven dep

This closes #3592.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-31 17:07:37 +09:00
Pierre Villard 630c651226
NIFI-6334 - fix custom validate error in PutBigqueryBatch
This closes #3589.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-19 13:49:23 +09:00
Pierre Villard 2336044b90 NIFI-6059 - Fixed NPE in GCS processors when project ID is not specified
This closes #3327.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-02-22 09:45:13 -05:00
Pierre Villard 03ef646547
NIFI-4731 This closes #3019. This closes #2682. This closes #2420.
NIFI-4933 BigQuery PR Review

Signed-off-by: joewitt <joewitt@apache.org>
2018-11-21 10:28:40 -05:00
Daniel Jimenez 444caf8a78
NIFI-4731: BQ Processors and GCP library update.
Signed-off-by: joewitt <joewitt@apache.org>
2018-11-21 10:27:41 -05:00
Koji Kawamura 5a58c9a171
NIFI-5282: Add ProxyConfigurationService to GCSProcessors
This closes #2981
2018-08-31 11:07:09 -04:00
jugi92 cdae2b14b3 NIFI-5282: GCPProcessor with HTTP Proxy with Auth
added http proxy support with authentication for GCP processors
added proxy support for Google Credential Service

This closes #2943.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-31 20:28:00 +09:00
Andy LoPresto f60585a9b6
NIFI-5376 Removed deprecation warnings.
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.

This closes #2841.

Signed-off-by: Joe Witt <joewitt@apache.org>
2018-07-09 20:45:34 -07:00
Mike Thomsen d2fd7e5e7d NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
This closes #2765

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-07 20:38:25 +05:30
zenfenan 504152eaa1 NIFI-5261: Added JSON_VALIDATOR to StandardValidators
This closes #2758

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-05 18:24:47 -04:00
zenfenan 1663a6c094 NIFI-5133: Implemented Google Cloud PubSub Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2724.
2018-05-30 09:47:18 +02:00
zenfenan 0973c2d8d1 NIFI-543 Added annotation to restrict processor to run only on the primary node
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly

This closes #2509.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-25 11:50:29 -04:00
zenfenan f742a3a6ac NIFI-5156: Updated GCP SDK to latest version
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2680.
2018-05-09 18:19:17 +02:00
Pierre Villard 4c787799ff NIFI-4149 - Indicate if EL is evaluated against FFs or not
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
2018-04-06 12:40:54 -04:00
Andre F de Miranda 8d4fe38bb4 NIFI-4197 - Expose some proxy settings to GCS Processors
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2017.
2018-01-06 17:33:31 -08:00
joewitt 7467bb7b0f
NIFI-4730 restructure of nifi-gcp-bundle
This closes #2367.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-01-03 11:21:52 -05:00
Andy LoPresto 7d242076ce
NIFI-3594 Implemented encrypted provenance repository.
Added src/test/resources/logback-test.xml files resetting log level from DEBUG (in nifi-data-provenance-utils) to WARN because later tests depend on MockComponentLog recording a certain number of messages and this number is different than expected if the log level is DEBUG.

This closes #1686.

Signed-off-by: Bryan Bende, Yolanda M. Davis, and Mark Payne
2017-05-02 13:24:07 -04:00
Andre F de Miranda cefc02c5fb NIFI-3514 - Ensures ListGCSBucket triggers serially and forbids inputs
This closes #1526.

Signed-off-by: James Wing <jvwing@gmail.com>
2017-02-22 10:24:37 -08:00
Gene Peters 897c70298a NIFI-3449: Adding Google Cloud Storage Bundle
* Credentials service with tests
* Abstract processor definitions
* GCS-themed processors and their corresponding tests

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1482.
2017-02-12 19:52:38 -08:00