Commit Graph

730 Commits

Author SHA1 Message Date
Colin Dean 0dd382370b NIFI-5612: Support JDBC drivers that return Long for unsigned ints
Refactors tests in order to share code repeated in tests and to enable
some parameterized testing.

MySQL Connector/J 5.1.x in conjunction with MySQL 5.0.x will return
a Long for ResultSet#getObject when the SQL type is an unsigned integer.
This change prevents that error from occurring while implementing a more
informational exception describing what the failing object's POJO type
is in addition to its string value.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3032
2018-09-28 13:46:04 -04:00
Nick Lewis db645ec475 NIFI-3344 Added property to JoltTransformJSON allowing the user to specify pretty print, defaults to false
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2987.
2018-09-25 23:07:24 +02:00
Pierre Villard 78c4e223fc NIFI-5591 - Added avro compression format to ExecuteSQL
This closes #3023

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-25 09:25:13 -04:00
Mark Payne 2a964681ec NIFI-5514: Fixed bugs in MergeRecord around minimum thresholds not being honored and validation not being performed to ensure that minimum threshold is smaller than max threshold (would previously allow min record = 100, max records = 2 as a valid configuration)
NIFI-5514: Do not rely on ProcessSession.getQueueSize() to return a queue size of 0 objects because if the processor is holding onto data, the queue size won't be 0.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2954.
2018-09-24 22:36:02 +02:00
Otto Fowler 7e627f2fed
NIFI-5566 Added unit test for PR feedback.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-09-17 17:12:53 -07:00
Andy LoPresto cd6877404b
NIFI-5147, NIFI-5566 Added CryptographicHashAttribute and CryptographicHashContent processors.
Deprecated HashContent processor.
Added documentation to HashAttribute processor.
Added shared HashService and HashAlgorithm enum.
Added unit tests.
Added #clearProperties() to TestRunner, StandardProcessorTestRunner, and MockProcessContext.
Updated processor manifest.
Updated Javadoc.
Added documentation about deprecated/renamed components.
Added logic handling for UTF-16 encoding where UTF-16BE is overridden so the BOM is not inserted prior to hashing.

This closes #2983.

Co-authored-by: Otto Fowler <ottobackwards@gmail.com>
Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-09-17 17:03:57 -07:00
Andy LoPresto 7e8b7752ca NIFI-5569 Added keywords to Route* and ScanAttribute processors to improve discoverability.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2984.
2018-09-11 18:09:48 +02:00
Mark Payne 0274bd4ff3 Revert "NIFI-4272 support multiple captures when EL is present in replacement value"
This reverts commit f7f809c3d3.

NIFI-5474, NIFI-4272: When using Regex Replace with ReplaceText, and there are capturing groups, ensure that we populate the 'additionalVariables' map for each match of the regex. This allows Expression Language to reference the back-references properly even when there are multiple matches

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2951
2018-09-11 11:07:33 -04:00
Lars Francke 1b73578c48 NIFI-5408 Add documentation for NIFI-4279
This closes #2878

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-08-31 19:09:58 -04:00
Andy LoPresto 8f37b5ee10
NIFI-4426 Removed exclude from nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml. 2018-08-30 13:05:09 -07:00
Matthew Burgess 8a8a8cb072 NIFI-5553: Fixed issue with rollback on result set errors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2967.
2018-08-29 13:22:10 +02:00
Mark Payne 6ceee25859 NIFI-5551: If session.get() returns null, then return fron onTrigger instead of assuming that the FlowFile is non-null
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2964.
2018-08-27 16:52:07 +02:00
Mike Moser 8c0705cb6b NIFI-5275 PostHTTP SocketConfig setup, fixed connection pool when using HTTPS, setup idle connection checker, setup request retry handler, improved some exception handling and logging, and NIFI-1336 fix as well
This closes #2796.
Signed-off-by: Brandon Devries <devriesb@apache.org>
2018-08-24 10:07:11 -04:00
patricker 17ea29244e NIFI-5381 SFTP Unit Tests
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2846.
2018-08-10 20:17:28 +02:00
Mark Payne d1ab17580f NIFI-5465: Set the Idle Timeout on jetty connectors to the same as the Request Timeout. 2018-08-06 10:33:24 -04:00
Matthew Burgess 9742dd2fac NIFI-5471: Push table aliasing to DatabaseAdapter impls
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2926.
2018-07-31 10:14:27 +02:00
Matthew Burgess 692943f016 NIFI-5454: Added EL support and copy.index attribute to DuplicateFlowFile
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2917.
2018-07-26 14:13:14 +02:00
Koji Kawamura 8b9d446118 NIFI-54096: Consolidating new model into existing List processors.
Adding followings:

- Use separate DistributedMapCache for tracking entities to avoid
conflict with existing code
- Added more validation
- Delete listed entities from cache if reset is needed
- Support Local scope
- Added Initial Listing Target

This closes #2876.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-07-17 13:59:57 -04:00
Otto Fowler b10220439c
NIFI-5337 Syslog 5424 Record Reader and nifi-syslog-utils
- Create nifi-syslog-utils to move syslog parsing functionalty to a central location shared by the processors and serialization/record system.
- Refactor Processors to use these utils
- Update 5424 syslog classes using simple-syslog-5424 to pick up new changes to support this work, as well as keep dependencies/types from bleeding out to the
processors or readers
- Refactor Syslog5424Event and Parser
- Create Syslog5424RecordReader
- per review, handle blank message differently from eof
- name schema per review

This closes #2816.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-07-16 11:50:55 -04:00
Marco Gaido 31bb89514c NIFI-5426: Use NIO.2 API for ListFile 2018-07-13 13:54:27 -04:00
Andy LoPresto ffbff42421 NIFI-5415 Renamed ListenSyslogGroovyTest to ITListenSyslogGroovy.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2883
2018-07-13 09:57:50 -04:00
patricker 0dd4a91a67 NIFI-5312 QueryDatabaseTable updates state when an SQLException is thrown
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2868
2018-07-10 19:03:14 -04:00
patricker e13602f752 NIFI-4279 PutDatabaseRecord and ConvertJSONToSQL stream has already been closed error on Oracle
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2874
2018-07-10 15:30:07 -04:00
patricker 624bbab8f0 NIFI-5382 ConvertRecord Add Option To Drop 0 Record Files
NIFI-5382 ConvertRecord Add Option To Drop 0 Record Files

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2853
2018-07-10 12:56:53 -04:00
Andy LoPresto f60585a9b6
NIFI-5376 Removed deprecation warnings.
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.

This closes #2841.

Signed-off-by: Joe Witt <joewitt@apache.org>
2018-07-09 20:45:34 -07:00
patricker 3826536546 NIFI-1251 ExecuteSQL Max Rows and Output Batching
NIFI-1251: Fixed doc and checkstyle issues
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2834
2018-07-09 13:36:45 -04:00
Mike Thomsen 790102d8b1 Revert "NIFI-5147 Add CalculateAttributeHash processor"
This reverts commit 7fb931889f.
2018-07-02 16:03:28 -04:00
Otto Fowler 7fb931889f NIFI-5147 Add CalculateAttributeHash processor
This closes #2802

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-07-02 06:20:21 -04:00
patricker a14dbc7dbe NIFI-5339 - Better Time Tracking for ExecuteSQL Durations
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2824.
2018-06-29 16:47:55 +02:00
patricker c1083dfb62 NIFI-5339 - Better Time Tracking for ExecuteSQL Durations
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2817.
2018-06-29 14:59:27 +02:00
Matthew Burgess e741ba1ce4 NIFI-5349: Fixed EL handling in Initial Max Value props for DB Fetch processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2822.
2018-06-29 14:51:59 +02:00
Andy LoPresto ea9b0db2f6
Merge branch 'NIFI-5323-RC1' as part of Apache NiFi 1.7.0 release process. 2018-06-25 12:03:00 -07:00
Otto Fowler a64680bfa5
NIFI-5325 A Syslog Parser that fully supports RFC 5424 Structured Data
- address review comments
- per review, add property for including body in attributes
- per review, create processor in OnScheduled

This closes #2805.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-06-21 12:38:41 -04:00
Andy LoPresto d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto 99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Bartlomiej Tartanus 235463af7b allow expression language in Schema File property
Update ValidateXml.java

Update ValidateXml.java

NIFI-5192: Added evaluateAttributeExpressions call, error handling, and unit tests
2018-06-19 16:12:52 -04:00
Mark Payne 2760b07770 NIFI-5322: Addressed bug that caused QueryRecord to fail (rollback session instead of routing to 'failure') whenever a record in the incoming FlowFile did not adhere to its schema. This happened because the InputStream for the FlowFile was not properly closed. Also updated the text of the Exception to include information from its 'cause' so user is better able to understand the underlying issue.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2803
2018-06-19 16:01:33 -04:00
Mike Moser d4d9259664 NIFI-5274 avoid rollback on uncaught errors in ReplaceText
forgot to penalize flowfiles sent to failure

NIFI-5274 avoid rollback on StackOverflowError in ReplaceText

add comment

removed penalize on failure, doesn't really apply here

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2767
2018-06-18 12:00:30 -04:00
Dustin Rodrigues 021ee6faf2 NIFI-5317 - support non-ASCII X-Mailer PutEmail header
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2800.
2018-06-18 10:45:56 +02:00
Dustin Rodrigues 2277b9411e NIFI-5252 - support arbitrary headers in PutEmail processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2787.
2018-06-15 14:29:56 +02:00
Matthew Burgess 0886dcb0b4 NIFI-5303: Fixed QueryDatabaseTable to work with DB2
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2788.
2018-06-15 13:56:52 +02:00
Mike Thomsen be63378a1e NIFI-5214 Added REST LookupService
NIFI-5214 Added support for the new ProxyConfigurationService

NIFI-5214 Integration tests added.

NIFI-5214 Added missing pom.xml and a change from a code review.

NIFI-5214 Added another tag based on code review.

NIFI-5214 Added user-defined header support.

NIFI-5214 Added Basic Auth support.

NIFI-5214 Moved documentation.

NIFI-5214 Fixed checkstyle issues; added changes requested in a review.

NIFI-5214 Added changes requested in a code review.

NIFI-5214 Added verb test and @DynamicProperties

NIFI-5214 Added templated URL support to RestLookupService.

NIFI-5214 Fixed documentation based on code review changes.

NIFI-5214 Changed RestLookupService to use a property descriptor.

NIFI-5214 Updated documentation.

NIFI-5214 Made changes requested in code review.

NIFI-5214 Renamed nifi-standard-web-utils to reflect that it is for tests.

NIFI-5214: Refactor RestLookupService.

1. Added 'Base URL' property to address environment specific part of URL.
2. Removed 'Record Path Property Name' property, because the name of
a resulted record field of a record path can be obtained by field name.
3. Lower cased HTTP method name should be used throughout.
4. Added mimeType require check when body is specified.
5. Added debug log to print HTTP response code.
6. Prepare for NIFI-5287.
7. Fixed that mime.type being used regardless of whether body is
specified or not, caused NullPointerException when 'mime.type' is not
specified when it is not required.
8. Updated documentation.

NIFI-5214 Fixed AWS processors that broke w/ change to TestServer location.

NIFI-5214 Added changes requested in a code review.

Refactored the way to evaluate EL for URL property

- Use PropertyValue instead of PreparedQuery to utilize Variable
Registry.
- Removed BASE_URL because Variable Registry can be used at URL

NIFI-5214 Rebased to use new LookupService method.

This closes #2723.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-06-15 15:18:03 +09:00
Koji Kawamura 4349a8e078 NIFI-5231: CalculateRecordCount should use 'record.count'
This closes #2792

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-14 07:02:47 -04:00
Mike Thomsen 0831059d2c NIFI-5287 Made LookupRecord able to take in flowfile attributes and combine them with lookup keys.
NIFI-5287 Removed unneeded property descriptor.

NIFI-5287 Added additional changes from a code review.

This closes #2777.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-06-14 10:54:44 +09:00
Pierre Villard 09f9260035 NIFI-5293 - add merge.uuid to MergeRecord processor
This closes #2783

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-12 19:16:17 +05:30
Pierre Villard ee18ead16c NIFI-5297 - EL support in ScanAttribute
This closes #2786

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-11 18:48:38 -04:00
Mike Thomsen 1803c15bcd NIFI-5231 Added RecordStats processor.
NIFI-5231 Minor fixes from code review.

NIFI-5231 Added documentation and made record_count a constant.

NIFI-5231 Added changes requested in a code review.

NIFI-5231 Added changes based on code review.

This closes #2737.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-06-11 13:35:01 +09:00
Pierre Villard 05d7b6c6e7 NIFI-4262 - MergeContent - option to add merged uuid in original flow files
This closes #2056

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-08 20:54:47 +05:30
Pierre Villard 6e067734d5 NIFI-5264 - Added attribute for validation error message in ValidateCSV
This closes #2769

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-08 19:16:20 +05:30
Matthew Burgess 0e09b98b02 NIFI-5143: Initial work to support column values for paging results
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2728.
2018-06-08 09:36:27 +02:00