- Updated PutKudu to use DataTypeUtils.toLocalDate() for DATE fields
- Updated PutDatabaseRecord to remove convertDateToLocalTZ() since convertType() uses toLocalDate()
- Updated PutElasticsearchHttpRecord to use default time zone format for DATE fields
- Updated WriteXMLResult to use default time zone format for DATE fields
- Updated WriteJsonResult to use default time zone format for DATE fields
- Updated AvroTypeUtil to use toLocalDate() for logical DATE fields
- Updated JdbcCommon to avoid conversion to UTC for logical DATE fields
- Updated Processor and RecordReader unit tests for consistency in DATE comparison
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#5210
NIFI-6061: Force getBytes() in BLOB handling to use UTF-8 charset
NIFI-6061: Use setClob(), added unit tests, incorporated review comments
This closes#5049
Co-authored-by: zhangcheng <zhangchengk@foxmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
- AbstractJsonRowRecordReader - Handle (meaning log a warning and not fail completely) multi-array CHOICE type when data has extra fields (not defined by the schema) and can't determine correct type.
- AvroTypeUtil - Allow multiple different record types in avro union type. Minor refactors. Added documentation fro EqualsWrapper.
NIFI-6752 Refactored type and value conversion logic. Added support for more types. Added more tests.
Removed 'parent' from 'Recursive'. (Caused issues. The recursive nature is still there as it has a child with the same type).
Updated jasn1 1.11.2 to asn1bean 1.12.0. If an asn field name is a Java reserved keyword, the field gets a trailing "_" but the getter remains normal. In JASN1Utils adjusted logic when looking for the getter.
Added support for inherited types. OctetStrings are converted to Strings instead of byte arrays.
Service takes care of the compilation of the ASN files. Test sources are generated and removed from source control.
NIFI-6752 Removed obsolete TODOs.
NIFI-6752 Updated nifi-asn1-nar version to 1.13.0-SNAPSHOT. Fixed checkstyle violations (unused imports).
NIFI-6752 ASN.1 reader - ASN.1 bundle requires 'include-asn1' profile to be active to be part of assembly.
NIFI-6752 ASN.1 reader - Updated ASN1.xml template.
NIFI-6752 ASN.1 reader - Updated versions.
NIFI-6752 ASN.1 reader - Update example generator. Updated ASN1.xml template. Updated (fixed) nifi-asn1-nar version in pom.xml.
NIFI-6752 ASN.1 reader - Added missing license for ASN1.xml.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4577
NIFI-7843 Recursive avro schemas fail to write with RecordWriter
Add new test case to TestSimpleRecordSchema to test the scenario
when schema name and schema namespace match.
This closes#4550.
Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
NIFI-7729: Updated docs to explain how to add new fields to Records via ScriptedTransformRecord and added example. Fixed checkstyle violation.
NIFI-7729: Always call Record.incorporateInactiveFields with ScriptedTransformRecord
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#4470
Some jdbc drivers e.g. Oracle returns java.sql.Array objects for array types, not just Lists.
This commit also handles these cases, and extracts the primitive java arrays out of this jdbc holder class.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#4034.
NIFI-7043: Account for case where MapRecord.incorporateInactiveFields is called, and there are no inactive fields, but there are updated fields (i.e., fields whose type has changed from the schema).
Signed-off-by: Joe Witt <joewitt@apache.org>
3 important changes:
1. FieldTypeInference had a bug when dealing with multiple datatypes for
the same field where some (but not all) were in a wider-than-the-other
relationship.
Before: Some datatypes could be lost. String was wider than any other.
After: Consistent behaviour. String is NOT wider than any other.
2. Choosing a datatype for a value from a ChoiceDataType:
Before it chose the first compatible datatype as the basis of conversion.
After change it tries to find the most suitable datatype.
3. Conversion of a value of avro union type:
Before it chose the first compatible datatype as the basis of conversion.
After change it tries to find the most suitable datatype.
Change: In the RecordFieldType enum moved TIMESTAMP ahead of DATE.
This closes#3724.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.
This closes#3497
- Make enriched fields nullable at LookupRecord.
- Removed unnecessary AvroConversionOptions and reader schema creation,
because ResultSetRecordSet can generate NiFi Record Schema from RS
directly. No Avro schema is needed to do that.
NIFI-5903: Removed TODO comments that were done
NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3223
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
- Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
- Updated LookupRecord to allow for Record fields to be arbitrarily added
- Implemented ContentClaimInputStream
- Added controller service for caching schemas
- UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.
NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).
NIFI-5938: Added another unit test for AvroTypeUtil
NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3253
NIFI-5141: Loosened regex for floating-point numbers to account for decimal place followed by 0 digits, such as '13.' and also added unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2679
using per-record flowfiles
NIFI-4428: Added initial L&N, fixed dependency hierarchy
NIFI-4428: Exposed additional ZK properties
NIFI-4428: More work to reduce dependencies in Druid artifacts
NIFI-4428: Removed inaccessible attribute
NIFI-4428: Fixed typo in property description
NIFI-4428: Updates to Druid NAR L&N
NIFI-4428: Updated LICENSE to refer to ICU License for icu4j
NiFi can now parse an Avro schema of a record that references an already defined record, including itself.
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#2034.
Also, updated record writers to ensure that they write the schema as appropriate if not using a RecordSet. Updated ConsumeKafkaRecord to allow for multiple schemas to be on same topic and partition
Signed-off-by: joewitt <joewitt@apache.org>
Joe Witt
exceptionfactory
Sandro Tolaini
Matthew Burgess
Tamas Palfy
Koji Kawamura
Peter Turcsanyi
Pierre Villard
Denes Arvay
Denes Arvay
Peter Wicks
Mark Payne
Bence Simon
Bryan Bende
Christian Zügner
Tamas Palfy
archon
Andy LoPresto
Mike Thomsen
Ed B
Jeff Storck
Carl Gieringer
Wesley-Lawrence
James Wing
Matt Gilman