Commit Graph

762 Commits

Author SHA1 Message Date
Matthew Burgess 692943f016 NIFI-5454: Added EL support and copy.index attribute to DuplicateFlowFile
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2917.
2018-07-26 14:13:14 +02:00
Koji Kawamura 8b9d446118 NIFI-54096: Consolidating new model into existing List processors.
Adding followings:

- Use separate DistributedMapCache for tracking entities to avoid
conflict with existing code
- Added more validation
- Delete listed entities from cache if reset is needed
- Support Local scope
- Added Initial Listing Target

This closes #2876.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-07-17 13:59:57 -04:00
Otto Fowler b10220439c
NIFI-5337 Syslog 5424 Record Reader and nifi-syslog-utils
- Create nifi-syslog-utils to move syslog parsing functionalty to a central location shared by the processors and serialization/record system.
- Refactor Processors to use these utils
- Update 5424 syslog classes using simple-syslog-5424 to pick up new changes to support this work, as well as keep dependencies/types from bleeding out to the
processors or readers
- Refactor Syslog5424Event and Parser
- Create Syslog5424RecordReader
- per review, handle blank message differently from eof
- name schema per review

This closes #2816.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-07-16 11:50:55 -04:00
Marco Gaido 31bb89514c NIFI-5426: Use NIO.2 API for ListFile 2018-07-13 13:54:27 -04:00
Andy LoPresto ffbff42421 NIFI-5415 Renamed ListenSyslogGroovyTest to ITListenSyslogGroovy.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2883
2018-07-13 09:57:50 -04:00
patricker 0dd4a91a67 NIFI-5312 QueryDatabaseTable updates state when an SQLException is thrown
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2868
2018-07-10 19:03:14 -04:00
patricker e13602f752 NIFI-4279 PutDatabaseRecord and ConvertJSONToSQL stream has already been closed error on Oracle
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2874
2018-07-10 15:30:07 -04:00
patricker 624bbab8f0 NIFI-5382 ConvertRecord Add Option To Drop 0 Record Files
NIFI-5382 ConvertRecord Add Option To Drop 0 Record Files

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2853
2018-07-10 12:56:53 -04:00
Andy LoPresto f60585a9b6
NIFI-5376 Removed deprecation warnings.
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.

This closes #2841.

Signed-off-by: Joe Witt <joewitt@apache.org>
2018-07-09 20:45:34 -07:00
patricker 3826536546 NIFI-1251 ExecuteSQL Max Rows and Output Batching
NIFI-1251: Fixed doc and checkstyle issues
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2834
2018-07-09 13:36:45 -04:00
Mike Thomsen 790102d8b1 Revert "NIFI-5147 Add CalculateAttributeHash processor"
This reverts commit 7fb931889f.
2018-07-02 16:03:28 -04:00
Otto Fowler 7fb931889f NIFI-5147 Add CalculateAttributeHash processor
This closes #2802

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-07-02 06:20:21 -04:00
patricker a14dbc7dbe NIFI-5339 - Better Time Tracking for ExecuteSQL Durations
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2824.
2018-06-29 16:47:55 +02:00
patricker c1083dfb62 NIFI-5339 - Better Time Tracking for ExecuteSQL Durations
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2817.
2018-06-29 14:59:27 +02:00
Matthew Burgess e741ba1ce4 NIFI-5349: Fixed EL handling in Initial Max Value props for DB Fetch processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2822.
2018-06-29 14:51:59 +02:00
Andy LoPresto ea9b0db2f6
Merge branch 'NIFI-5323-RC1' as part of Apache NiFi 1.7.0 release process. 2018-06-25 12:03:00 -07:00
Otto Fowler a64680bfa5
NIFI-5325 A Syslog Parser that fully supports RFC 5424 Structured Data
- address review comments
- per review, add property for including body in attributes
- per review, create processor in OnScheduled

This closes #2805.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-06-21 12:38:41 -04:00
Andy LoPresto d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto 99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Bartlomiej Tartanus 235463af7b allow expression language in Schema File property
Update ValidateXml.java

Update ValidateXml.java

NIFI-5192: Added evaluateAttributeExpressions call, error handling, and unit tests
2018-06-19 16:12:52 -04:00
Mark Payne 2760b07770 NIFI-5322: Addressed bug that caused QueryRecord to fail (rollback session instead of routing to 'failure') whenever a record in the incoming FlowFile did not adhere to its schema. This happened because the InputStream for the FlowFile was not properly closed. Also updated the text of the Exception to include information from its 'cause' so user is better able to understand the underlying issue.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2803
2018-06-19 16:01:33 -04:00
Mike Moser d4d9259664 NIFI-5274 avoid rollback on uncaught errors in ReplaceText
forgot to penalize flowfiles sent to failure

NIFI-5274 avoid rollback on StackOverflowError in ReplaceText

add comment

removed penalize on failure, doesn't really apply here

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2767
2018-06-18 12:00:30 -04:00
Dustin Rodrigues 021ee6faf2 NIFI-5317 - support non-ASCII X-Mailer PutEmail header
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2800.
2018-06-18 10:45:56 +02:00
Dustin Rodrigues 2277b9411e NIFI-5252 - support arbitrary headers in PutEmail processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2787.
2018-06-15 14:29:56 +02:00
Matthew Burgess 0886dcb0b4 NIFI-5303: Fixed QueryDatabaseTable to work with DB2
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2788.
2018-06-15 13:56:52 +02:00
Mike Thomsen be63378a1e NIFI-5214 Added REST LookupService
NIFI-5214 Added support for the new ProxyConfigurationService

NIFI-5214 Integration tests added.

NIFI-5214 Added missing pom.xml and a change from a code review.

NIFI-5214 Added another tag based on code review.

NIFI-5214 Added user-defined header support.

NIFI-5214 Added Basic Auth support.

NIFI-5214 Moved documentation.

NIFI-5214 Fixed checkstyle issues; added changes requested in a review.

NIFI-5214 Added changes requested in a code review.

NIFI-5214 Added verb test and @DynamicProperties

NIFI-5214 Added templated URL support to RestLookupService.

NIFI-5214 Fixed documentation based on code review changes.

NIFI-5214 Changed RestLookupService to use a property descriptor.

NIFI-5214 Updated documentation.

NIFI-5214 Made changes requested in code review.

NIFI-5214 Renamed nifi-standard-web-utils to reflect that it is for tests.

NIFI-5214: Refactor RestLookupService.

1. Added 'Base URL' property to address environment specific part of URL.
2. Removed 'Record Path Property Name' property, because the name of
a resulted record field of a record path can be obtained by field name.
3. Lower cased HTTP method name should be used throughout.
4. Added mimeType require check when body is specified.
5. Added debug log to print HTTP response code.
6. Prepare for NIFI-5287.
7. Fixed that mime.type being used regardless of whether body is
specified or not, caused NullPointerException when 'mime.type' is not
specified when it is not required.
8. Updated documentation.

NIFI-5214 Fixed AWS processors that broke w/ change to TestServer location.

NIFI-5214 Added changes requested in a code review.

Refactored the way to evaluate EL for URL property

- Use PropertyValue instead of PreparedQuery to utilize Variable
Registry.
- Removed BASE_URL because Variable Registry can be used at URL

NIFI-5214 Rebased to use new LookupService method.

This closes #2723.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-06-15 15:18:03 +09:00
Koji Kawamura 4349a8e078 NIFI-5231: CalculateRecordCount should use 'record.count'
This closes #2792

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-14 07:02:47 -04:00
Mike Thomsen 0831059d2c NIFI-5287 Made LookupRecord able to take in flowfile attributes and combine them with lookup keys.
NIFI-5287 Removed unneeded property descriptor.

NIFI-5287 Added additional changes from a code review.

This closes #2777.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-06-14 10:54:44 +09:00
Pierre Villard 09f9260035 NIFI-5293 - add merge.uuid to MergeRecord processor
This closes #2783

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-12 19:16:17 +05:30
Pierre Villard ee18ead16c NIFI-5297 - EL support in ScanAttribute
This closes #2786

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-11 18:48:38 -04:00
Mike Thomsen 1803c15bcd NIFI-5231 Added RecordStats processor.
NIFI-5231 Minor fixes from code review.

NIFI-5231 Added documentation and made record_count a constant.

NIFI-5231 Added changes requested in a code review.

NIFI-5231 Added changes based on code review.

This closes #2737.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-06-11 13:35:01 +09:00
zenfenan cf3c666683 NIFI-5286: Updated FasterXML Jackson libraries to 2.9.5
This closes #2775

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-09 14:19:45 -04:00
Pierre Villard 05d7b6c6e7 NIFI-4262 - MergeContent - option to add merged uuid in original flow files
This closes #2056

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-08 20:54:47 +05:30
Pierre Villard 6e067734d5 NIFI-5264 - Added attribute for validation error message in ValidateCSV
This closes #2769

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-08 19:16:20 +05:30
Matthew Burgess 0e09b98b02 NIFI-5143: Initial work to support column values for paging results
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2728.
2018-06-08 09:36:27 +02:00
Joe Trite f5364875e8 Squashed commit of the following:
commit 9c31e45d3f
Author: Joe Trite <joetrite@gmail.com>
Date:   Mon Mar 13 07:55:19 2017 -0400

    NIFI-1705 Adding AttributesToCSV processor

commit 5e9afa9ccf
Merge: 3177eb185 74cbfc4b6
Author: Joe Trite <joetrite@gmail.com>
Date:   Sat Mar 4 08:12:39 2017 -0500

    Merge remote-tracking branch 'origin/master'

    # Conflicts:
    #	nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ScanAttribute.java

commit 3177eb1852
Author: Joe Trite <joetrite@gmail.com>
Date:   Sat Mar 4 08:09:06 2017 -0500

    NIFI-3497 Converted properties to use displayName, changed validator on demarcator property and created a method for duplicate code.

commit 74cbfc4b69
Merge: a974f78e0 f8cad0f8c
Author: Joe Trite <joetrite@gmail.com>
Date:   Sat Mar 4 07:47:46 2017 -0500

    Merge branch 'master' into master

commit a974f78e03
Author: Joe Trite <joetrite@gmail.com>
Date:   Sat Mar 4 07:43:02 2017 -0500

    NIFI-3497 Converted properties to use displayName, changed validator on demarcator property and created a method for duplicate code.

commit 1bfaef8e87
Merge: 65ed46de9 89ec68d14
Author: Joe Trite <joetrite@gmail.com>
Date:   Fri Mar 3 08:01:59 2017 -0500

    Merge branch 'master' of https://github.com/joetrite/nifi

commit 65ed46de9a
Author: Joe Trite <joetrite@gmail.com>
Date:   Fri Feb 24 18:09:36 2017 -0500

    NIFI-3497 - fixing Pcontrib issues

commit c5d52cf6f0
Author: Joe Trite <joetrite@gmail.com>
Date:   Thu Feb 23 10:19:01 2017 -0500

    NIFI-3497 - excluding test files

    Adding new test data files to exclude list.

commit b195934113
Author: Joe Trite <joetrite@gmail.com>
Date:   Wed Feb 22 16:48:10 2017 -0500

    NIFI-3497 - New dictionary files for test

    Adding new dictionary files to support metadata dictionary option.

commit e296268f39
Author: Joe Trite <joetrite@gmail.com>
Date:   Wed Feb 22 16:46:13 2017 -0500

    NIFI-3497 test cases for metadata updates

    Adding test cases to support metadata option update.

commit de7e348e62
Author: Joe Trite <joetrite@gmail.com>
Date:   Wed Feb 22 16:36:08 2017 -0500

    NIFI-3497 - Added metadata option

    Added optional to post additional metadata as new attributed if a match is found in the dictionary.

commit 89ec68d14b
Author: Joe Trite <joetrite@gmail.com>
Date:   Fri Feb 24 18:09:36 2017 -0500

    NIFI-3497 - fixing Pcontrib issues

commit d71426037b
Author: Joe Trite <joetrite@gmail.com>
Date:   Thu Feb 23 10:19:01 2017 -0500

    NIFI-3497 - excluding test files

    Adding new test data files to exclude list.

commit a7a7b6ace8
Author: Joe Trite <joetrite@gmail.com>
Date:   Wed Feb 22 16:48:10 2017 -0500

    NIFI-3497 - New dictionary files for test

    Adding new dictionary files to support metadata dictionary option.

commit 8eb54a5019
Author: Joe Trite <joetrite@gmail.com>
Date:   Wed Feb 22 16:46:13 2017 -0500

    NIFI-3497 test cases for metadata updates

    Adding test cases to support metadata option update.

commit f52e1f2a06
Author: Joe Trite <joetrite@gmail.com>
Date:   Wed Feb 22 16:36:08 2017 -0500

    NIFI-3497 - Added metadata option

    Added optional to post additional metadata as new attributed if a match is found in the dictionary.

NIFI-1705 Adding AttributesToCSV processor.

NIFI-1705 Adding AttributesToCSV processor.

NIFI-1705 Adding AttributesToCSV processor.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2711
2018-06-07 14:48:53 -04:00
Otto Fowler f7f809c3d3 NIFI-4272 support multiple captures when EL is present in replacement value
This closes #2748

Signed-off-by: Mike Moser <mosermw@apache.org>
2018-06-06 17:23:20 +00:00
Marco Gaido eedf1237a1 NIFI-5262 - Retrieve file attributes only once in ListFile
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2754.
2018-06-06 10:10:29 +02:00
Koji Kawamura 71cf3fd46d NIFI-5268: Fix JostTransformJSON spec validation with EL
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2762.
2018-06-06 09:37:02 +02:00
Mike Thomsen 83fccc4beb NIFI-5024 Fixed a checkstyle issue that was missed in the merge of 5024. 2018-05-31 08:35:25 -04:00
Nicolas Sanglard 498fd8f22d NIFI-5024 Resolves deadlock in ExecuteStreamCommand processor
This closes #2594

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-31 06:35:39 -04:00
Matt Gilman 9093d280f2 NIFI-5208:
- Removing test for unsupported functionality.

This closes #2740

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-30 13:41:20 -04:00
Matt Gilman 05a1d63090 NIFI-5208:
- Fixing missing message when a node is disconnected from a cluster.
- Updating endpoints to accept a flag to allow for changes to be made to a disconnected node.
- Updating custom UIs to acknowledge disconnected nodes prior to performing modifications.
2018-05-30 13:41:03 -04:00
Scott Aslan 9aa7e65f70
[NIFI-5215] Upgrade AngularJS to v1.7.0. Also, adding package-lock.json to source control
[NIFI-5215] add package-lock.json to rat excludes
This closes #2721
2018-05-29 13:20:49 -04:00
zenfenan 0973c2d8d1 NIFI-543 Added annotation to restrict processor to run only on the primary node
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly

This closes #2509.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-25 11:50:29 -04:00
Mark Payne 397e88c858 NIFI-4227: Fixed typo 2018-05-24 15:57:28 -04:00
Pierre Villard be0ed70423 NIFI-4227 - add a ForkRecord processor
Added split/extract modes, unit tests, and additional details

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-24 15:55:17 -04:00
Mike Thomsen 716587d09f NIFI-5169 This closes #2705. Upgrade to JSONPath 2.4
NIFI-5169 Added explicit checking for empty strings in JsonPath expression property.

Signed-off-by: joewitt <joewitt@apache.org>
2018-05-23 16:16:25 -05:00
Mark Payne cca520aab7
NIFI-5228: Make the storing of file attributes as FlowFile attributes optional for ListFile
NIFI-5228: Addressed review feedback
This closes #2733
2018-05-23 10:50:01 -04:00
Mark Payne 167014441b
NIFI-5191: Revert "NIFI-4631: Use java.nio.file.Files in ListFile to improve performance"
This reverts commit 758e44682f.
2018-05-23 10:50:01 -04:00