Commit Graph

845 Commits

Author SHA1 Message Date
Matthew Burgess c32ea618c5
NIFI-6040: Fixed ExecuteSQL processors when Output Batch Size is set
This closes #3355.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-06 13:44:33 -05:00
thenatog 36bbc77723 NIFI-6097 - Upgraded the fasterxml jackson version to 2.9.8. Ensure that the version is consistent across modules using a maven property defined in the root pom.
This closes #3347

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-04 19:30:35 -05:00
Matthew Burgess 32bd7ed8b4 NIFI-6062: Add support for BLOB, CLOB, NCLOB in record handling
This closes #3329

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-28 08:59:37 -05:00
Arpad Boda 2a6c3c1686
NIFI-5977 - Add "Minimum/Maximum File Age" Parameter to ListSFTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3324.
2019-02-28 09:15:59 +01:00
joewitt 25cc7b4a1e
NIFI-6029 merging nifi 1.9.0 release into master 2019-02-19 22:55:49 -05:00
Denes Arvay 76e92c8682 NIFI-6052 Update NOTICE files to reflect 2019
This closes #3319

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 18:32:15 -05:00
joewitt 0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt 45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Aleksandr Salatich 9e0687ab6b
NIFI-6018 Marking GetHTTP and PostHTTP deprecated. Adding info about deprecation to processors descriptions. Fix small mistake in developer-guide.
Suppressed deprecation warnings in TestGetHTTP, TestPostHTTP, TestGetHTTPGroovy, and TestPostHTTPGroovy. The annotation suppresses some warnings, but until Java 9, explicit FQCN imports are needed to avoid all warnings.

This closes #3312.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-02-15 14:09:26 -08:00
Mark Payne b508d6bfbc
NIFI-6033, NIFI-6034, NIFI-6035, NIFI-6036, NIFI-6037: Fixed bugs that were found during 1.9.0-RC1 validation. If multiple FlowFiles were written to same Content Claim, and a Processor attempted to read two of them wi within a single session, it would seek to the wrong part of the content or else throw a ContentNotFoundException. Updated logic for considering a processor to be 'running' / having 'active threads' if the processor is invalid upon NiFi restart but scheduled to run. Fixed NPE in FreeFormTextWriter. If MergeRecord reaches minimum number of records, flush writer after writing content out so that its minimum size can accurately be checked.
This closes #3309.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 11:59:47 -05:00
Mark Payne 82a0434901 NIFI-5903: Allow RecordPath to be used in QueryRecord processor. Also some code cleanup and improvements to the docs
NIFI-5903: Removed TODO comments that were done

NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3223
2019-02-12 16:44:17 -05:00
Mark Payne 36c0a99e91 NIFI-5938: Added ability to infer record schema on read from JsonTreeReader, JsonPathReader, XML Reader, and CSV Reader.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to  to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
 - Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
 - Updated LookupRecord to allow for Record fields to be arbitrarily added
 - Implemented ContentClaimInputStream
 - Added controller service for caching schemas
 - UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.

NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).

NIFI-5938: Added another unit test for AvroTypeUtil

NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3253
2019-02-11 12:56:50 -05:00
a.durov 8e777203a0 NIFI-5974 fix: Fragment Attributes are populated in case no split has occured.
Unit test is implemented: testNoSplitterInString

NIFI-5974: Fixed Checkstyle violations

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3275
2019-01-30 09:40:36 -05:00
erichanson5 4ef2251d74 NIFI-3611: Added ability to set Transaction Isolation Level on Database connection for the QueryDatabaseTable processor
NIFI-3611: Make TRANS_ISOLATION_LEVEL property optional

This closes #3248.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-01-21 17:19:42 +09:00
Koji Kawamura 05de73d6a0 NIFI-5951 Fix error logging with rollback on failure
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3264
2019-01-17 17:11:39 -05:00
Otto Fowler 1330b92cfa NIFI-5892 Wait timestamp lingers, potentially messing up downstream wait-notify pairs
Clear the wait timestamp when transferring to failur or success

replace explicit attribute clear with function call, refactor and integrate into existing tests per review

This closes #3233.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-12-28 10:30:02 +09:00
SavtechSolutions f9f386b0f0 NIFI-5871 ignore UUID attribute when copying flow file attributes (#3203)
NIFI-5871 ignore UUID attribute when copying flow file attributes

Signed-off-by: Peter Wicks <patricker@gmail.com>
2018-12-20 10:05:25 -07:00
Pierre Villard 590fa2063c NIFI-4892 - ValidateCSV: no doublequote escaping in invalid output
NIFI-4892 - ValidateCSV: no doublequote escaping in invalid output

NIFI-5907 - unit test

This closes #2481.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-12-19 11:20:55 +09:00
Matthew Burgess 7548d7c859 NIFI-3988: Add fragment attributes to SplitRecord
This closes #3217.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-12-18 15:07:43 +09:00
Koji Kawamura 0efddf47d5 NIFI-4579: Fix ValidateRecord type coercing
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2794
2018-12-13 16:10:41 -05:00
Pierre Villard a6f91a1975 NIFI-5872 - Added compression option to JsonRecordSetWriter
This closes #3208.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-12-10 16:25:59 +09:00
Mark Payne 72ea93a657
NIFI-5868: Added instrumentation around ListFile such that all disk accesses are timed and any unusually long listing times or disk access operations can be logged. Additionally, information is logged at a debug level including significant amounts of troubleshooting information when configured to do so
This closes #3202.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-12-05 13:01:14 -05:00
Matthew Burgess 8a12307d1a
NIFI-5855: Remove unnecessary ORDER BY clause in GenerateTableFetch when Partition Size is zero
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3191.
2018-11-30 10:54:23 +01:00
Matthew Burgess 68a49cfad0 NIFI-5845: Add support for OTHER and SQLXML JDBC types to SQL/Hive processors
NIFI-5845: Incorporated review comments

This closes #3184.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-29 09:50:21 +09:00
zenfenan e34d653ba1
NIFI-5812: Marked Database processors as PrimaryNodeOnly
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3167.
2018-11-22 09:33:01 +01:00
joewitt 2a4745c077
NIFI-5836 This closes #3181. changed the brittle timing based tests to integration tests 2018-11-21 13:11:06 -05:00
Kumara M S Hemantha 71499f774d
NIFI-5560 This closes #3000. Added Follow SYMLINK support for listFTP & listSFTP and getFTP & getSFTP processors
Signed-off-by: joewitt <joewitt@apache.org>
2018-11-21 10:53:38 -05:00
yjhyjhyjh0 3c7012ffda
NIFI-5744: Put exception message to attribute while ExecuteSQL fail
This closes #3107.

Signed-off-by: Peter Wicks <patricker@gmail.com>
2018-11-20 09:00:46 -07:00
Colin Dean 54402a1ecd NIFI-5828: Documents behavior of ExecuteSQL attrs when Max Rows Per Flow File is enabled
This closes #3177.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-19 14:18:31 +09:00
Matthew Burgess 0207d0813e
NIFI-5604: Added property to allow empty FlowFile when no SQL generated by GenerateTableFetch
co-authored by: Peter Wicks <patricker@gmail.com>
Signed-off-by: Peter Wicks <patricker@gmail.com>

This closes #3075.
2018-11-15 13:37:43 -07:00
yjhyjhyjh0 75906226a6
NIFI-5780 Add pre and post statements to ExecuteSQL and ExecuteSQLRecord
Signed-off-by: Peter Wicks <patricker@gmail.com>

This closes #3156.
2018-11-15 13:18:31 -07:00
vadimar d319a3ef2f NIFI-5788: Introduce batch size limit in PutDatabaseRecord processor
NIFI-5788: Introduce batch size limit in PutDatabaseRecord processor

        Renamed 'batch size' to 'Maximum Batch Size'.
        Changed default value of max_batch_size to zero (INFINITE)
        Fixed parameter validation.
        Added unit tests

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3128
2018-11-15 10:31:34 -05:00
Matthew Burgess 13011ac6d6 NIFI-5652: Fixed LogMessage when logging level is disabled
This closes #3170.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-15 15:14:57 +09:00
Pierre Villard 4112af013d NIFI-4130 Add lookup controller service in TransformXML to define XSLT from the UI
addressed review comments

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #1953
2018-11-14 09:45:20 -05:00
Mark Payne 08189596d2 NIFI-5809: If QueryRecord has a single-column projection and that results in a null value, do not confuse that with a null value being returned from the Record Reader
This closes #3163.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-12 14:39:27 +09:00
Mark Payne 564ad0cd71
NIFI-5718: Implemented LineDemarcator and removed NLKBufferedReader in order to improve performance 2018-11-09 14:26:20 -07:00
Arek Burdach 765df67817 NIFI-5757 Using Caffeine instead of slow synchronization on LinkedHashMap for caches - mainly avro schema caches
This closes #3111.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-11-09 14:50:24 -05:00
Vish Uma 63f55d05b4 NIFI-5724 make database connection autocommit configurable
making the database session autocommit value a configurable property

adding custom validation to PutSQL processor so as to disallow 'supports transaction' and 'rollback on failure' to be true when the autocommit value has been set to true

fixing some style issues to conform to standards

This closes #3113.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-09 17:07:27 +09:00
Mark Payne 49ba3643c7 Increase timeouts in all HandleHttpRequest unit tests from 3 seconds to 20 seconds to avoid frequent unit test failures 2018-11-08 16:20:28 -05:00
Mark Payne 50c0d1ed5d Increase timeouts in unit tests to avoid frequent failures 2018-11-08 16:17:50 -05:00
Koji Kawamura 78a1cb7c5e NIFI-5802: Add QueryRecord nullable field support
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3158.
2018-11-08 09:50:45 +01:00
Mark Payne 3ba0c0ca82 NIFI-5798: Fixed bug in FlattenJson that was escaping text as Java instead of escaping as JSON
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3138.
2018-11-08 09:27:30 +01:00
Kotaro Terada 0c9120a73f NIFI-5777: Update the log-level property of LogMessage to support EL
This closes #3122.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-11-06 16:20:40 +09:00
Matthew Burgess d8d220ccb8
NIFI-5601: Add fragment.* attributes to GenerateTableFetch
Signed-off-by: Peter Wicks <patricker@gmail.com>

This closes #3074
2018-11-01 15:14:29 -06:00
Koji Kawamura f0a7dda842 NIFI-5776: ListSFTP should log SSH_FXP_STATUS id value
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3120.
2018-11-01 13:01:40 +01:00
Kotaro Terada 2a035cc390 NIFI-1505: Add processor description for LogAttribute
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3121.
2018-11-01 12:56:01 +01:00
Gardella Juan Pablo 1f2cf4bc61 NIFI-5761 ReplaceText processor can stop processing data if it evaluates invalid expressions
NIFI-5761 Code review.
Remove startsWidth to check an exception. Added the dependency as provided.

NIFI-5761 Code review.
Remove provided.

NIFI-5761 Code review.
Improve logging.

This closes #3112.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-31 12:01:16 +09:00
Jeff Storck 4f14e517db Merge branch 'NIFI-5720-RC3' as part of the NiFi 1.8.0 release process 2018-10-26 13:56:23 -04:00
juliansniff fdbcf34281 NIFI-5747 fix NPE when redirecting from HTTPS to HTTP for InvokeHTTP
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-26 17:08:17 +09:00
Bryan Bende 10479a5a2a NIFI-5740 Ensuring permissions are restored after test completes in TestFetchFile
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3104
2018-10-23 15:26:33 -04:00