Commit Graph

62 Commits

Author SHA1 Message Date
Andy LoPresto e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
archon 4d18eaa481 NIFI-6035: 1. Add formatWithTimeZone() and toDateWithTimeZone(); 2. Their test code and docs.
This closes #3481

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-21 07:33:45 -04:00
Matthew Burgess a5dff7b45c NIFI-6096: Fixed ValidateRecord handling of nested maps
This closes #3424

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-07 11:31:17 -04:00
Koji Kawamura cae01d555c NIFI-6117: Fix BIGINT handling in DataTypeUtils
Follow numeric type conversion convention used for other integral types.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3371
2019-03-26 10:39:35 -04:00
Matthew Burgess c2dc0910b7 NIFI-6117: Fix BIGINT handling in DataTypeUtils 2019-03-26 10:39:29 -04:00
Koji Kawamura 4db5446c87 NIFI-6082: Refactor the way to handle fields nullable
- Make enriched fields nullable at LookupRecord.
- Removed unnecessary AvroConversionOptions and reader schema creation,
because ResultSetRecordSet can generate NiFi Record Schema from RS
directly. No Avro schema is needed to do that.
2019-03-14 09:33:39 +09:00
Matthew Burgess ca76fe178c NIFI-6082: Added DatabaseRecordLookupService, refactored common DB utils
NIFI-6082: Added SimpleDatabaseLookupService

NIFI-6082: Merged Koji's improvements, incorporated review comments

This closes #3341.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-03-14 09:33:30 +09:00
Koji Kawamura 2846d3c3c6 NIFI-6105: Fix handling of arrays of records/maps in record utilities
Refactored to use the same check logic for Record and Map types

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3367
2019-03-13 11:49:00 -04:00
Matthew Burgess f91311da9d NIFI-6105: Fix handling of arrays of records/maps in record utilities
This closes #3353.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-03-13 11:47:55 -04:00
Matthew Burgess 32bd7ed8b4 NIFI-6062: Add support for BLOB, CLOB, NCLOB in record handling
This closes #3329

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-28 08:59:37 -05:00
joewitt 0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt 45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Mark Payne 319979f256
NIFI-6039: Avoid NPE if Record Writer does not provide schema
This closes #3310.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-02-15 13:06:30 -05:00
Mark Payne 82a0434901 NIFI-5903: Allow RecordPath to be used in QueryRecord processor. Also some code cleanup and improvements to the docs
NIFI-5903: Removed TODO comments that were done

NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3223
2019-02-12 16:44:17 -05:00
Mark Payne 36c0a99e91 NIFI-5938: Added ability to infer record schema on read from JsonTreeReader, JsonPathReader, XML Reader, and CSV Reader.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to  to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
 - Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
 - Updated LookupRecord to allow for Record fields to be arbitrarily added
 - Implemented ContentClaimInputStream
 - Added controller service for caching schemas
 - UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.

NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).

NIFI-5938: Added another unit test for AvroTypeUtil

NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3253
2019-02-11 12:56:50 -05:00
Mike Thomsen 13a7f16860 NIFI-5890 Added a unit test that proves that 1.9 fixes a bug in the date conversion behavior in 1.8.
This closes #3214.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-12-12 11:08:18 +09:00
Mike Thomsen e603c486f4 NIFI-5333 Added GetMongoRecord.
Signed-off-by: zenfenan <zenfenan@apache.org>

This closes #3011
2018-11-23 15:01:45 +05:30
Mark Payne a9045d54a1 NIFI-5800: Do not recursively call hashCode on child schema for Record Field Types
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3157.
2018-11-08 09:36:47 +01:00
Ed B 2812fe60a2 NIFI-5728 - XML Writer to populate record tag name properly
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3098.
2018-11-02 09:46:45 +01:00
Jeff Storck c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00
Jeff Storck 98aabf2c50 NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 2018-10-22 22:16:23 -04:00
Carl Gieringer a6b9364ebf NIFI-5664 Support ArrayList in DataTypeUtils#toArray
NIFI-5664 Generalize to handling List

This closes #3049

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-10-12 06:28:56 -04:00
Matthew Burgess c6572f042b NIFI-4517: Added ExecuteSQLRecord and QueryDatabaseTableRecord processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2945.
2018-10-06 10:53:11 +02:00
Mark Payne 2e1005e884 NIFI-5640: Improved efficiency of Avro Reader and some methods of AvroTypeUtil. Also switched ServiceStateTransition to using read/write locks instead of synchronized blocks because profiling showed that significant time was spent in determining state of a Controller Service when attempting to use it. Switching to a ReadLock should provide better performance there.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3036
2018-09-27 15:38:47 -04:00
Andy LoPresto d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto 99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Mark Payne 06d1276f09 NIFI-5141: Updated regex for doubles to allow for numbers that have no decimal
NIFI-5141: Loosened regex for floating-point numbers to account for decimal place followed by 0 digits, such as '13.' and also added unit tests

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2679
2018-06-04 12:23:57 -04:00
Matthew Burgess d79216d6b0 NIFI-5217: Allow non-nullable fields to be added to MockRecordParser
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2720
2018-05-18 15:59:12 -04:00
Mark Payne 4700b8653d NIFI-5138: Bug fix to ensure that when we have a CHOICE between two or more REOCRD types that we choose the appropriate RECORD type when creating the Record in the JSON Reader.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2670.
2018-05-09 18:38:04 +02:00
Matthew Burgess ba32879ec8 NIFI-5082: Added support for custom Oracle timestamp types to Avro conversion
This closes #2638

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-04-22 19:59:48 -04:00
Matthew Burgess b29304df79 NIFI-4857: Support String<->byte[] conversion for records
This closes #2570.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-04-09 14:53:21 -04:00
joewitt d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
Bryan Bende de71a41bd0 NIFI-4935 Refactoring to support specifying schema branch or schema version when using schema by name strategy
This closes #2523.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-15 16:16:12 -04:00
Matthew Burgess ecb8067883 NIFI-4428: Replaced JSON input with RecordReader
using per-record flowfiles
NIFI-4428: Added initial L&N, fixed dependency hierarchy

NIFI-4428: Exposed additional ZK properties

NIFI-4428: More work to reduce dependencies in Druid artifacts

NIFI-4428: Removed inaccessible attribute

NIFI-4428: Fixed typo in property description

NIFI-4428: Updates to Druid NAR L&N

NIFI-4428: Updated LICENSE to refer to ICU License for icu4j
2018-01-16 13:08:02 -05:00
joewitt 41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00
joewitt 36405e888c NIFI-4751-RC1 prepare for next development iteration 2018-01-08 23:39:49 -07:00
joewitt 46d30c7e92 NIFI-4751-RC1 prepare release nifi-1.5.0-RC1 2018-01-08 23:39:32 -07:00
Mark Payne 953e922d32 NIFI-4749: This closes #2386. Pass the record reader's schema along to the ResultSetRecordSet so that it is able to resolve the schema for Record fields
Signed-off-by: joewitt <joewitt@apache.org>
2018-01-08 19:47:18 -07:00
Mark Payne c91d99884a NIFI-4717: Several minor bug fixes and performance improvements around record-oriented processors
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2359
2017-12-29 10:43:21 -05:00
Mark Payne f772f2f093 NIFI-4671: This closes #2328. Ensure that Avro Schemas that are created properly denote fields as being nullable iff the schemas says they are, for non-top-level fields
Signed-off-by: joewitt <joewitt@apache.org>
2017-12-11 11:46:15 -05:00
Mike Thomsen c70a86eac0 NIFI-4575 Added ability to convert from a date to a timestamp within the record API.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2255
2017-11-13 12:57:39 -05:00
Jeff Storck a57911d3db NIFI-4412-RC2 prepare for next development iteration 2017-09-28 13:45:36 -04:00
Jeff Storck e6508ba7d3 NIFI-4412-RC2 prepare release nifi-1.4.0-RC2 2017-09-28 13:45:21 -04:00
Mark Payne 451f9cf124 NIFI-4142: This closes #2015. Refactored Record Reader/Writer to allow for reading/writing "raw records". Implemented ValidateRecord. Updated Record Reader to take two parameters for nextRecord: (boolean coerceTypes) and (boolean dropUnknownFields)
Signed-off-by: joewitt <joewitt@apache.org>
2017-08-11 22:01:46 -07:00
Wesley-Lawrence 40cde0466a NIFI-4215 NiFi can now parse an Avro schema of a record that references an already defined record, including itself.
This closes #2052.
2017-08-03 15:13:07 -04:00
James Wing 2502b79bae NIFI-4215 Revert Complex Avro Schema Changes
This reverts commit cf49a58ee7.
2017-08-01 21:03:04 -07:00
Wesley-Lawrence cf49a58ee7 NIFI-4215 Allow Complex Avro Schema Parsing
NiFi can now parse an Avro schema of a record that references an already defined record, including itself.

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2034.
2017-07-30 16:31:39 -07:00
Matt Gilman cc741d2be6
NIFI-3997:
- Bumping to next minor version.
2017-06-08 15:22:51 -04:00
Matt Gilman 6ee12e9b47
NIFI-3997-RC1prepare for next development iteration 2017-06-05 11:07:43 -04:00