Commit Graph

718 Commits

Author SHA1 Message Date
zenfenan 0973c2d8d1 NIFI-543 Added annotation to restrict processor to run only on the primary node
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly

This closes #2509.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-25 11:50:29 -04:00
Mark Payne 397e88c858 NIFI-4227: Fixed typo 2018-05-24 15:57:28 -04:00
Pierre Villard be0ed70423 NIFI-4227 - add a ForkRecord processor
Added split/extract modes, unit tests, and additional details

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-24 15:55:17 -04:00
Mike Thomsen 716587d09f NIFI-5169 This closes #2705. Upgrade to JSONPath 2.4
NIFI-5169 Added explicit checking for empty strings in JsonPath expression property.

Signed-off-by: joewitt <joewitt@apache.org>
2018-05-23 16:16:25 -05:00
Mark Payne cca520aab7
NIFI-5228: Make the storing of file attributes as FlowFile attributes optional for ListFile
NIFI-5228: Addressed review feedback
This closes #2733
2018-05-23 10:50:01 -04:00
Mark Payne 167014441b
NIFI-5191: Revert "NIFI-4631: Use java.nio.file.Files in ListFile to improve performance"
This reverts commit 758e44682f.
2018-05-23 10:50:01 -04:00
Andrew Psaltis 6356d7b9ee NIFI-4988 This closes #2725. Changed the exception handling so that an invliad ZIP is handled. When an invalid zip is processed, the exception is an IllegalArgumentException which was not being handled and thus the session was being rollbacked.
Signed-off-by: joewitt <joewitt@apache.org>
2018-05-22 10:24:57 -04:00
Koji Kawamura 2834fa4ce4 NIFI-4199: Added ProxyConfigurationService
- Added ProxyConfigurationService to manage centralized proxy
configurations
- Adopt ProxyConfigurationService at FTP and HTTP processors

NIFI-4175 - Add HTTP proxy support to *SFTP processors

This closes #2018.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>

NIFI-4199: Add ProxyConfigurationService to SFTP processors

- Fixed check style issue
- Use the same proxy related PropertyDescriptors from FTPTransfer and
SFTPTransfer
- Dropped FlowFile EL evaluation support to make it align with other
processors spec, Now it supports VARIABLE_REGISTRY
- Added ProxyConfigurationService to SFTP processors
- Added SOCKS proxy support to SFTP processors

NIFI-4199: Added ProxyConfigurationService to ElasticsearchHttp processors

- ElasticsearchHttp processors now support SOCKS proxy, too
- Added proxy support to PutElasticsearchHttpRecord
- Moved more common property descriptors to
AbstractElasticsearchHttpProcessor and just return static unmodifiable
property descriptor list at each implementation processors

NIFI-4196 - Expose AWS proxy authentication settings

NIFI-4196 - Fix jUnit errors

This closes #2016.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>

NIFI-4199: Add ProxyConfigService to AWS processors

- Applied ProxyConfigService to S3 processors
- Added proxy support to following processors:
  - PutKinesisFirehose, PutKinesisStream
  - PutDynamoDB, DeleteDynamoDB, GetDynamoDB
  - PutKinesisStream
- All AWS processors support HTTP proxy now

NIFI-4199: Add ProxyConfigService to Azure processors

NIFI-4199: More explicit validation and docs for Proxy spec

- Each processor has different supporting Proxy specs
- Show supported spec to ProxyConfigurationService property doc
- Validate not only Proxy type, but also with Authentication

NIFI-4199: Incorporated review comments

- Fixed TestListS3 property descriptor check
- Separate name and displayName

This closes #2016
This closes #2018
This closes #2704

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-20 17:57:47 -04:00
Mark Payne 2afbf96381
NIFI-5204: Ensure that verifyCanStop throws ISE if component is disabled
NIFI-5204: If processor joins cluster and inherits 'disabled' state but is still stopping, ensure that the state becomes disabled when the processor finishes stopping
This closes #2713
2018-05-17 13:44:52 -04:00
Mike Thomsen f5108ea839 NIFI-5197 Fixed a bunch of EL scope bugs.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2712
2018-05-17 09:57:35 -04:00
Matthew Burgess 5613bf4012 NIFI-5202: Fixed timing issue in TestListDatabaseTables
This closes #2709

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-17 07:10:35 -04:00
Mark Payne 604656fe88
NIFI-950: Make component validation asynchronous
NIFI-950: Still seeing some slow response times when instantiating a large template in cluster mode so making some minor tweaks based on the results of CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired intermediate data model that can be serialized, separate from serializing. This allows us to hold the FlowController's Read Lock only while creating the data model, not while actually serializing the data. Configured Jersey Client in ThreadPoolRequestReplicator not to look for features using the Service Loader for every request. Updated Template object to hold a DOM Node that represents the template contents instead of having to serialize the DTO, then parse the serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so that it will clean up any resource related to component lifecycle.
NIFI-950: Avoid gathering the Status objects for entire flow when we don't need them; removed unnecessary code
NIFI-950: Bug fixes
NIFI-950: Bug fix; added validation status to ProcessorDTO, ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause time to be set in the customValidate method for testing functionality
NIFI-950: Addressing test failures
NIFI-950: Bug fixes
NIFI-950: Addressing review feedback
NIFI-950: Fixed validation logic in mock framework
This closes #2693
2018-05-16 14:39:23 -04:00
Otto Fowler ccedc71c8b NIFI-5077 ExtractGrok support for `keep empty captures`
Support for the new option to keep empty captures.  I did not add to the GrokReader because I am not sure of the effect on
the schema extraction.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2698.
2018-05-15 19:24:10 +02:00
Otto Fowler 61fe493786 NIFI-5170 Upgrad Grok to version 0.1.9
This closes #2691

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-13 08:40:20 -04:00
Gardella Juan Pablo 64356e0014 NIFI-5049 Fix handling of Phonenix datetime columns
This closes #2625

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-11 09:51:24 -04:00
Matthew Burgess 099bfcdf3a NIFI-5121: Added DBCPService API method for passing in flow file attributes when available
This closes #2658

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-11 08:19:16 -04:00
patricker 0a44bad76e NIFI-5168 - ReplaceText Processor Should Use Single FlowFile Processing Instead of Batch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2687.
2018-05-09 18:46:35 +02:00
Koji Kawamura 2007c207ab NIFI-4393: Handle database specific identifier escape characters
QueryDatabaseTable and GenerateTableFetch processors were not able to
use max value state as expected, if max value column was wrapped with
escape characters. Due to a mis-match between computed state keys
and actual keys used in the managed state. State keys computed by
getStateKey method included escape characters while actual stored keys
did not. Resulted querying the same dataset again and again.

This commit added unwrapIdentifier method to DatabaseAdapter class to
remove database specific escape characters for identifiers such as table
and column names, so that max value state keys are populated correctly
even if identifiers are wrapped with escape characters.

This commit also added new DatabaseAdapter for MySQL, to handle MySQL
specific identifier escape with back-ticks.

This closes #2424

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-03 09:06:05 -04:00
Koji Kawamura 6e8b1c8f74 NIFI-5090: Create move target dir dynamically at FetchFTP and FetchSFTP
Support creating target parent directories even if directory listing is disabled.

fixed typo in doc

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2642
2018-04-27 11:48:26 -04:00
joewitt ac9944ccee
NIFI-5108 Updated all explicit refs and media nar usage of commons-compress to latest version and updated spring redis client
This closes #2651.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-04-24 15:16:04 -04:00
patricker 0390c0f196 NIFI-4561 ExecuteSQL returns no FlowFile for some queries
This closes #2243

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-04-24 10:27:44 -04:00
Matthew Burgess ba32879ec8 NIFI-5082: Added support for custom Oracle timestamp types to Avro conversion
This closes #2638

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-04-22 19:59:48 -04:00
Joe Percivall a0c9bebe24 NIFI-5085 In InvokeHttp, moving the OkHttp Response object to a try with resources 2018-04-18 13:18:30 -04:00
patricker 82ac815536 NIFI-1706: Extend QueryDatabaseTable to support arbitrary queries
- Only include Maximum Value columns in the type map.
- Squashed commits in the previous PR
- Rebased against the latest master
- Added stop method to GenerateTableFetch so that it refreshes the
column type map when it gets restarted
- Fixed whitespacing around if/for statement
- Updated expressionLanguageSupported value since it is not auto-merged
correctly

This closes #2618.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-04-13 16:30:13 +09:00
joewitt 59f625d3c2 Merge branch 'NIFI-4995-RC3' as part of NiFi 1.6.0 release process 2018-04-06 16:48:08 -07:00
Vi On b414132324 NIFI-5050 update frontend-maven-plugin
This closes #2613

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-04-06 16:07:25 -04:00
Pierre Villard 4c787799ff NIFI-4149 - Indicate if EL is evaluated against FFs or not
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
2018-04-06 12:40:54 -04:00
Marco Gaido c119547222 NIFI-5043: TailFile in Multifile mode should not open new readers in onTrigger
This closes #2606.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-04-05 13:53:01 -04:00
joewitt d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
Mark Bean c59b6fdf66 NIFI-4658 set Maximum Number of Entries to required and allow FlowFiles having fragment.count greater than Max Entries property
Signed-off-by: Mike Moser <mosermw@apache.org>

This closes #2559
2018-04-02 20:42:04 -04:00
Mark Payne 94d444abb0 NIFI-4883: Updated ValidateRecord to allow an optional Record Writer to be configured for invalid records that differs from the Record Writer used for writing valid records
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2600.
2018-04-02 18:16:40 +02:00
Mark Payne bbbe428e7b NIFI-5030: If ControlRate encounters a FlowFile that cannot be transferred, it should continue processing other FlowFiles that have different group attribute values
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2593.
2018-03-29 10:31:15 +02:00
Scott Aslan c1459825bb
[NIFI-4998] update node and npm version
This closes #2571
2018-03-28 16:49:32 -04:00
Mark Payne f394c874e1
NIFI-4895: Added backend code to give users the ability to forcibly terminate a processor if the processor does not complete after clicking Stop
NIFI-4895: Addressed review feedback; addressed checkstyle issues
This closes #2555
2018-03-28 12:02:47 -04:00
joewitt 7b5bf265a6 NIFI-4995 updating copyright year on all notices 2018-03-26 21:54:10 -04:00
Joe Percivall 56ad14ad1e
NIFI-4977 Adding expression language support to the Sender properties of PutSyslog
This closes #2547.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-26 15:49:52 -04:00
Matt Gilman 69a564e4c8
NIFI-5009:
- Fixing required permission for PutParquet.

NIFI-5008:
- Ensuring all restricted components are tagged as such.

This closes #2583.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-26 09:47:54 -04:00
Otto Fowler 98c295dfc7
NIFI-5013 create test jar for standard processors
remove unnec TestServer instances

This closes #2580.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2018-03-24 10:18:46 -04:00
Marco Gaido 758e44682f NIFI-4631: Use java.nio.file.Files in ListFile to improve performance
This closes #2565.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-23 11:00:21 -04:00
sbouchex 9c9215982f NIFI-4999: Added request URI into the "restlistener.request.uri" attribute
This closes #2574.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-21 13:17:17 -04:00
Andy LoPresto d09145e3e3 NIFI-4557 Added Expression Language support for PGP private keyring passphrase in EncryptContent.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2566.
2018-03-20 11:23:45 +01:00
Derek Straka 5bdb7cf6e7 NIFI-4912: This closes #2494. Update jackson version to latest stable version (2.9.4)
Signed-off-by: joewitt <joewitt@apache.org>
2018-03-19 10:22:50 -04:00
Pierre Villard 4687e0fb35 NIFI-4981 Allow using FF's content as message in PutEmail
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2556.
2018-03-18 11:35:43 -07:00
Mark Bean 8b2c5b7246 NIFI-4950 Defining behavior for MergeContent when more than 1 FlowFile has the same fragment.index value
Signed-off-by: Mike Moser <mosermw@apache.org>

This closes #2557.
2018-03-16 18:31:43 +00:00
Marco Gaido 86f162b611 NIFI-4959: Remove flowfiles and close connection for Bad Requests causing IOException
This closes #2544.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-16 12:24:00 -04:00
Bryan Bende de71a41bd0 NIFI-4935 Refactoring to support specifying schema branch or schema version when using schema by name strategy
This closes #2523.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-15 16:16:12 -04:00
Deon Huang 37ad48844a NIFI-4962 FlattenJson processor add unexpected backslash after flatten
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2538.
2018-03-13 22:33:53 +01:00
Pierre Villard 11a7d5c09f NIFI-4955 - Preserve columns ordering with CSV and ValidateRecord
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2528
2018-03-13 17:09:07 -04:00
m-hogue 6471f66acd NIFI-3402 - Added etag support to InvokeHTTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2150.
2018-03-13 10:00:11 +01:00