Commit Graph

146 Commits

Author SHA1 Message Date
Tamas Neumer 80e71068be NIFI-11860 fixed bug in ResultSetRecordSet affecting decimalScale when resultSetScale was zero.
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #7545
2023-08-02 12:22:55 -04:00
Chris Sampson 3f5ed23504
NIFI-9206 Added RemoveRecordField Processor
This closes #6816

Co-authored-by: Peter Gyori <peter.gyori.dev@gmail.com>
Co-authored-by: Chris Sampson <chris.sampson82@gmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-06-26 09:50:41 -05:00
Matt Burgess f3e8651522 NIFI-11647: Support UUID type in DataTypeUtils.getSQLTypeValue
This closes #7355

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2023-06-08 11:13:07 -04:00
krisztina-zsihovszki a36040fded
NIFI-11367 Database DATE value displayed incorrectly in record output if timezone is GMT-X
This closes #7103.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2023-04-02 22:47:42 +02:00
Joe Witt 43eab4c81d
NIFI-11103 prepping for 2.0.0 line 2023-02-09 15:32:53 -07:00
dan-s1 53371844a4
NIFI-11035 Replaced remaining JUnit 4 assertions in nifi-commons with JUnit 5
- Replaced Groovy asserts with JUnit 5 assertions and Groovy shouldFail method Junit 5 with assertThrow method

This closes #6880

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-01-24 15:30:53 -06:00
Lehel Boér ed6ba53724 NIFI-10832: Create PutSalesforceRecord processor
This closes #6670.

Signed-off-by: Tamas Palfy <tpalfy@apache.org>
2022-12-13 18:59:41 +01:00
Matt Burgess b744fac479
NIFI-10956: Fix inference issues with mixed arrays (#6763) 2022-12-08 09:28:19 -05:00
Joe Witt 75e7c9e47c
NIFI-10854-RC1 prepare for next development iteration 2022-11-22 19:22:47 -07:00
Joe Witt ec87bf93ad
NIFI-10854-RC1 prepare release nifi-1.19.0-RC1 2022-11-22 19:22:45 -07:00
Matthew Burgess a76abef270 NIFI-10635: Fix handling of enums in PutDatabaseRecord
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #6518.
2022-10-17 11:29:53 -04:00
Mark Bathori e87bced147
NIFI-10442: Create PutIceberg processor
This closes #6368.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2022-10-11 21:51:18 +02:00
Joe Witt c9ebdd2025
NIFI-10521-RC4 prepare for next development iteration 2022-10-03 10:59:36 -07:00
Joe Witt 109e54cd58
NIFI-10521-RC4 prepare release nifi-1.18.0-RC4 2022-10-03 10:59:34 -07:00
Csaba Bejan 35fb66f50f
NIFI-10403 Add processor supporting the new BigQuery Write API
Signed-off-by: Arpad Boda <aboda@apache.org>

This closes #6344
2022-09-19 14:32:11 +02:00
Mark Payne ec09c56e93
NIFI-10508: When inferring data types for values, allow float and double to encapsulate byte/short/int/long values
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #6421
2022-09-16 17:58:19 -04:00
Joe Witt 8c66bf948c
NIFI-10272-RC2 prepare for next development iteration 2022-07-27 13:24:09 -07:00
Joe Witt 8d256784d8
NIFI-10272-RC2 prepare release nifi-1.17.0-RC2 2022-07-27 13:24:06 -07:00
Mike Thomsen a3e8048b2d
NIFI-9981 Added support for Avro UUID types
This closes #6013

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-06-23 08:35:31 -05:00
exceptionfactory 5febd47c72
NIFI-10109 Changed HashMap to LinkedHashMap in JsonTreeRowRecordReader (#6112)
- Added unit test to confirm preservation of field ordering
2022-06-21 17:00:34 -04:00
Matthew Burgess 639ab328c9
NIFI-10057: Add support for SQL Server's DATETIMEOFFSET type to ExecuteSQL processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #6080.
2022-06-03 10:33:41 +02:00
Joe Witt 0419dc2939
NIFI-9780-RC3 prepare for next development iteration 2022-03-21 11:58:27 -07:00
Joe Witt b019a9191f
NIFI-9780-RC3 prepare release nifi-1.16.0-RC3 2022-03-21 11:58:09 -07:00
在红尘中成仙 8d5de6b292
NIFI-9818 fix flaky tests
This closes #5883

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-03-21 08:23:40 -04:00
Mark Payne 6a1c7c72d5
NIFI-9781: Fix handling when selecting array element via QueryRecord
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5853.
2022-03-09 13:07:15 -05:00
zhangcheng a85cafe771
NIFI-9064:Support Oracle timestamp when `Use Avro Logical Types` is true for ExecuteSQLRecord and QueryDatabaseTableRecord
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5807.
2022-03-02 11:18:58 -05:00
Mike Thomsen 4b41570d17
NIFI-9714 Added overloaded toMap to MapRecord that can convert sub-records into maps.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5784
2022-02-28 09:12:51 -05:00
Mark Payne ce0122bd25
NIFI-9691: Added ForkEnrichment, JoinEnrichment processors
Signed-off-by: Joe Gresock <jgresock@gmail.com>

This closes #5772.
2022-02-22 18:54:32 -05:00
Pierre Villard 90930ca197
NIFI-9476 - Fix QueryRecord when no result and with array type column
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5594
2021-12-15 17:37:24 -05:00
exceptionfactory b7ad1f924d
NIFI-9457 Support microseconds for String Timestamps in PutKudu
- Implemented override for Timestamp Record Field Type format handling to add support for optional microseconds
- Added FieldConverter and ObjectTimestampFieldConverter implementation for generalized Timestamp parsing using DateTimeFormatter
- Updated PutKudu unit tests for standard Timestamp and Timestamp with microseconds

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5589.
2021-12-10 10:31:52 +01:00
Matthew Burgess fe4161b1c2
NIFI-8392: Translate JDBC CHAR type to RecordFieldType STRING
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5506.
2021-11-14 16:03:17 +01:00
Joe Witt c055895952
NIFI-9344-RC3 prepare for next development iteration 2021-11-03 08:53:37 -07:00
Joe Witt 7fdc07cccd
NIFI-9344-RC3 prepare release nifi-1.15.0-RC3 2021-11-03 08:53:32 -07:00
Mike Thomsen 8d513c5ed3
NIFI-9080 Converted nifi-commons to use JUnit 5
This closes #5332

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-10-25 21:07:54 -05:00
Peter Gyori 38d26dafa3
NIFI-9192: ResultSetRecordSet considers value of useLogicalTypes flag when determining the object's schema.
NIFI-9192: ResultSetRecordSet consider value of useLogicalType flag when sqlType is not handled separately (default) and readerSchema is not null

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Unit tests added to test the use of useLogicalType flag

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: ResultSetRecordSet consider not using logical type when sqlType is OTHER

NIFI-9192: Fixed checkstyle violations

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Renamed JdbcCommon.getUseLogicalTypes to isUseLogicalTypes

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Added unit tests for Array types with and without using logical types

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>

NIFI-9192: Review comments considered and unit tests refactored

NIFI-9192: Added further unit tests

NIFI-9192: Minor modifications based on review comments

Signed-off-by: Peter Gyori <peter.gyori.dev@gmail.com>
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5388
2021-09-27 16:27:39 -04:00
Matthew Burgess 98e3b465c3
NIFI-8376: Gracefully handle SQL exceptions in ResultSetRecordSet
This closes #4951

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-09-08 09:11:26 -04:00
exceptionfactory 20804ff2b1
NIFI-8749 Removed implicit time zone conversion to GMT
- Updated DataTypeUtils.getDateFormat() to use system default time zone
- Updated Record Path Guide to match Expression Language Guide regarding default time zone
- Updated impacted unit tests to expect localized dates instead of dates converted to GMT

NIFI-8749 Set user.timezone in TestRecordPath to avoid unexpected date conversions

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5361
2021-08-31 16:07:53 -04:00
Matthew Burgess dfa6592a47
NIFI-8749 Revert "Removed implicit time zone conversion to GMT"
This reverts commit 758e1850ed.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>
2021-08-31 13:33:53 -04:00
exceptionfactory 758e1850ed
NIFI-8749 Removed implicit time zone conversion to GMT
- Updated DataTypeUtils.getDateFormat() to use system default time zone
- Updated Record Path Guide to match Expression Language Guide regarding default time zone
- Updated impacted unit tests to expect localized dates instead of dates converted to GMT

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5305
2021-08-31 10:40:57 -04:00
Joe Witt 7356332852
Merge branch 'NIFI-8767-RC2' 2021-07-14 09:07:22 -05:00
exceptionfactory 05d7867a16
NIFI-8768 Added toLocalDate() for convertType() handling of DATE fields
- Updated PutKudu to use DataTypeUtils.toLocalDate() for DATE fields
- Updated PutDatabaseRecord to remove convertDateToLocalTZ() since convertType() uses toLocalDate()
- Updated PutElasticsearchHttpRecord to use default time zone format for DATE fields
- Updated WriteXMLResult to use default time zone format for DATE fields
- Updated WriteJsonResult to use default time zone format for DATE fields
- Updated AvroTypeUtil to use toLocalDate() for logical DATE fields
- Updated JdbcCommon to avoid conversion to UTC for logical DATE fields
- Updated Processor and RecordReader unit tests for consistency in DATE comparison

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5210
2021-07-13 16:17:36 -04:00
Joe Witt 97feacc181
NIFI-8767-RC2 prepare for next development iteration 2021-07-10 12:17:09 -07:00
Joe Witt fcbf1d5f97
NIFI-8767-RC2 prepare release nifi-1.14.0-RC2 2021-07-10 12:17:05 -07:00
Sandro Tolaini fdda3582e3
NIFI-8470: Fix DataTypeUtils for CHAR() sequences
NIFI-8470: Added unit test

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5027
2021-05-11 17:46:38 -04:00
Matthew Burgess 30fc26647e
NIFI-6061: Fix CLOB/BLOB handling in PutDatabaseRecord
NIFI-6061: Force getBytes() in BLOB handling to use UTF-8 charset

NIFI-6061: Use setClob(), added unit tests, incorporated review comments

This closes #5049

Co-authored-by: zhangcheng <zhangchengk@foxmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-04 14:58:14 -05:00
Tamas Palfy a50957161c NIFI-8365 Fix JSON AbstractJsonRowRecordReader to handle deep CHOICE-typed records properly: change the logic that selects the first compatible schema which can have missing fields compared to the real value and search for a more strict match first and fallback to the existing logic only if not one found.
- AbstractJsonRowRecordReader - Handle (meaning log a warning and not fail completely) multi-array CHOICE type when data has extra fields (not defined by the schema) and can't determine correct type.
- AvroTypeUtil - Allow multiple different record types in avro union type. Minor refactors. Added documentation fro EqualsWrapper.
2021-04-19 12:56:09 -04:00
Matthew Burgess a5dbf56114
NIFI-8368: If decimal scale > precision, set precision = scale
This closes #4938

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-29 10:53:46 -05:00
Matthew Burgess b039606cf8
NIFI-8297: Report incompatible value if it doesn't fit in an Integer
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4886.
2021-03-12 19:40:19 +04:00
Koji Kawamura 68d38dd0a6
NIFI-6752 Add ASN.1 RecordReader
NIFI-6752 Refactored type and value conversion logic. Added support for more types. Added more tests.
Removed 'parent' from 'Recursive'. (Caused issues. The recursive nature is still there as it has a child with the same type).
Updated jasn1 1.11.2 to asn1bean 1.12.0. If an asn field name is a Java reserved keyword, the field gets a trailing "_" but the getter remains normal. In JASN1Utils adjusted logic when looking for the getter.
Added support for inherited types. OctetStrings are converted to Strings instead of byte arrays.
Service takes care of the compilation of the ASN files. Test sources are generated and removed from source control.

NIFI-6752 Removed obsolete TODOs.

NIFI-6752 Updated nifi-asn1-nar version to 1.13.0-SNAPSHOT. Fixed checkstyle violations (unused imports).

NIFI-6752 ASN.1 reader - ASN.1 bundle requires 'include-asn1' profile to be active to be part of assembly.

NIFI-6752 ASN.1 reader - Updated ASN1.xml template.

NIFI-6752 ASN.1 reader - Updated versions.

NIFI-6752 ASN.1 reader - Update example generator. Updated ASN1.xml template. Updated (fixed) nifi-asn1-nar version in pom.xml.

NIFI-6752 ASN.1 reader - Added missing license for ASN1.xml.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4577
2021-02-25 12:58:05 -05:00
Matthew Burgess 200c04c6d0
NIFI-8237: This closes #4835. Added missing SQL types to getDataTypeFromSQLTypeValue(), added defensive code
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-23 08:08:09 -07:00