Apache/nifi
1
0
Fork 0
mirror of https://github.com/apache/nifi.git synced 2025-02-13 13:35:20 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,970 Commits 148 Branches 203 Tags
Commit Graph

165 Commits

Author SHA1 Message Date
JohannesDaniel
04745a556a NIFI-5189 Schema name is not available for RecordSchema 
This closes #2700

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
 2018-05-14 14:35:12 -04:00
Matthew Burgess
159b64b4c8
NIFI-5123: Move SchemaRegistryService to nifi-avro-record-utils 
This closes #2661.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2018-04-26 11:50:01 -04:00
Matthew Burgess
b29304df79 NIFI-4857: Support String<->byte[] conversion for records 
This closes #2570.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-04-09 14:53:21 -04:00
joewitt
59f625d3c2 Merge branch 'NIFI-4995-RC3' as part of NiFi 1.6.0 release process 2018-04-06 16:48:08 -07:00
Pierre Villard
4c787799ff NIFI-4149 - Indicate if EL is evaluated against FFs or not 
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
 2018-04-06 12:40:54 -04:00
joewitt
d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt
f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
Derek Straka
c056ede6cc NIFI-4951: Update convertToAvroObject to use the DataTypeUtils conversion function 
The feature allows users to convert from non-integral types to the correct underlying type.  The
original behavior is maintained; however, now simple conversions take place automatically for some
logical types (date, time, and timestamp).

This closes #2526.

Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-03-16 14:08:14 -04:00
Bryan Bende
de71a41bd0 NIFI-4935 Refactoring to support specifying schema branch or schema version when using schema by name strategy 
This closes #2523.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-03-15 16:16:12 -04:00
joewitt
c71409fb5d
NIFI-4936 trying to quiet down the mvn output a bit so we dont exceed the travis-ci 4MB max 
NIFI-4936 updated dependency handling pushing down delcarations where they belong
This closes #2512
 2018-03-09 16:34:53 -05:00
gardellajuanpablo
2e4ddbf3b2 NIFI-4893 Cannot convert Avro schemas to Record schemas with default arrays 
Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-03-06 14:09:39 -05:00
Derek Straka
44bc2d41d7 NIFI-4880: Add the ability to map record based on the aliases. This closes #2474 
Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-02-28 14:19:53 -05:00
Koji Kawamura
270892f926 NIFI-4846: AvroTypeUtil to support more input types for logical decimal conversion 
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2451
 2018-02-07 19:42:26 -05:00
Koji Kawamura
2b062e211f NIFI-4844: Adjust BigDecimal scale to the target Avro schema 
- Applied the same scale adjustment not only to BigDecimal inputs, but
also to Double values.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2450
 2018-02-07 10:38:45 -05:00
Mark Payne
7c1ce17223
NIFI-4756: Updated PublishKafkaRecord processors to include attributes generated from schema write strategy into the message headers when appropriate 
This closes #2396.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2018-01-16 16:31:34 -05:00
joewitt
41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00
joewitt
42edfa75b7 Merge branch 'NIFI-4751-RC1' 2018-01-12 15:00:32 -05:00
Mark Payne
a36afe0bbe NIFI-4767 - Fixed issues with RecordPath using the wrong field name for arrays and maps. Also addressed issue where Avro Reader was returning a Record object when it should return a Map 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2398.
 2018-01-11 23:24:12 +01:00
Patrice Freydiere
5f7bd81af9 NIFI-4441: patch avro maps in union types. This closes #2207. 
Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-01-11 15:04:42 -05:00
joewitt
36405e888c NIFI-4751-RC1 prepare for next development iteration 2018-01-08 23:39:49 -07:00
joewitt
46d30c7e92 NIFI-4751-RC1 prepare release nifi-1.5.0-RC1 2018-01-08 23:39:32 -07:00
Mark Payne
c91d99884a NIFI-4717: Several minor bug fixes and performance improvements around record-oriented processors 
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2359
 2017-12-29 10:43:21 -05:00
Mark Payne
f772f2f093 NIFI-4671: This closes #2328. Ensure that Avro Schemas that are created properly denote fields as being nullable iff the schemas says they are, for non-top-level fields 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-12-11 11:46:15 -05:00
Koji Kawamura
77a51e1a9e NIFI-4544: Improve HDFS processors provenance transit URL 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2238.
 2017-11-02 10:10:03 +01:00
patricker
e3482cc772 NIFI-4534 Choose Character Set for CSV Record Read/Write streams 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2229.
 2017-10-27 10:34:15 +02:00
patricker
fd00df3d2f NIFI-4465 ConvertExcelToCSV Data Formatting and Delimiters 
This closes #2194.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-10-17 14:56:49 +09:00
Jeff Storck
a57911d3db NIFI-4412-RC2 prepare for next development iteration 2017-09-28 13:45:36 -04:00
Jeff Storck
e6508ba7d3 NIFI-4412-RC2 prepare release nifi-1.4.0-RC2 2017-09-28 13:45:21 -04:00
Koji Kawamura
1f67cbf628 NIFI-4004: Use RecordReaderFactory without FlowFile. 
- Removed FlowFile from RecordReaderFactory, RecordSetWriterFactory and SchemaAccessStrategy.
- Renamed variable 'allowableValue' to 'strategy' to represent its meaning better.
- Removed creation of temporal FlowFile to resolve Record Schema from ConsumerLease.

- Removed unnecessary 'InputStream content' argument from
  RecordSetWriterFactory.getSchema method.

This closes #1877.
 2017-09-08 12:37:40 -04:00
Bryan Bende
cf57639396 NIFI-4311 Allowing umask to get set properly before initializing the FileSystem 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2106.
 2017-08-22 22:40:26 +02:00
Mark Payne
451f9cf124 NIFI-4142: This closes #2015. Refactored Record Reader/Writer to allow for reading/writing "raw records". Implemented ValidateRecord. Updated Record Reader to take two parameters for nextRecord: (boolean coerceTypes) and (boolean dropUnknownFields) 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-08-11 22:01:46 -07:00
Bryan Bende
0029f025f8 NIFI-4152 Initial commit of ListenTCPRecord 2017-08-07 22:44:11 +02:00
Wesley-Lawrence
40cde0466a NIFI-4215 NiFi can now parse an Avro schema of a record that references an already defined record, including itself. 
This closes #2052.
 2017-08-03 15:13:07 -04:00
James Wing
2502b79bae NIFI-4215 Revert Complex Avro Schema Changes 
This reverts commit cf49a58ee75601e1d0d7512104b9ed0ca2e8ec41.
 2017-08-01 21:03:04 -07:00
Wesley-Lawrence
cf49a58ee7 NIFI-4215 Allow Complex Avro Schema Parsing 
NiFi can now parse an Avro schema of a record that references an already defined record, including itself.

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2034.
 2017-07-30 16:31:39 -07:00
Mark Payne
1d6b486b63 NIFI-4232: Ensure that we handle conversions to Avro Arrays properly. Also, if unable to convert a value to the expected object, include in the log message the (fully qualified) name of the field that is problematic 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2040.
 2017-07-27 08:57:25 +02:00
Mark Payne
b603cb955d NIFI-4060: Initial implementation of MergeRecord 
NIFI-4060: Addressed threading issue with RecordBin being updated after it is completed; fixed issue that caused mime.type attribute not to be written properly if all incoming flowfiles already have a different value for that attribute

NIFI-4060: Bug fixes; improved documentation; added a lot of debug information; updated StandardProcessSession to produce more accurate logs in case of a session being committed/rolled back with open input/output streams
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1958
 2017-07-12 16:36:48 -04:00
Mark Payne
3906d4e1d2 NIFI-1763: Initial implementation of ConfluentSchemaRegistry. 
NIFI-1763: Fixed bug where the Confluent Schema Registry Schema Access Writer was not being created

Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1938
 2017-07-06 15:52:07 -04:00
Mark Payne
e7dcb6f6c5 NIFI-3921: Allow Record Writers to inherit schema from Record 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1902
 2017-06-09 16:13:25 -04:00
Matt Gilman
cc741d2be6
NIFI-3997: 
- Bumping to next minor version.
 2017-06-08 15:22:51 -04:00
Matt Gilman
1bf0a1a849
Merge branch 'NIFI-3997-RC1' 2017-06-08 14:30:10 -04:00
Bryan Bende
b0c9428776 NIFI-4030 Populating default values on GenericRecord from Avro schema if not present in RecordSchema 
This closes #1896.
 2017-06-07 13:59:40 -04:00
Steve Champagne
45e035686f
NIFI-4029: Allow null Avro default values in HortonworksSchemaRegistry 
This closes #1894.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-07 12:03:53 -04:00
Matt Gilman
6ee12e9b47
NIFI-3997-RC1prepare for next development iteration 2017-06-05 11:07:43 -04:00
Matt Gilman
ddb73612bd
NIFI-3997-RC1prepare release nifi-1.3.0-RC1 2017-06-05 11:07:28 -04:00
Mark Payne
7035694e37
NIFI-3995: Updated Hwx Encoded Schema Ref Writer to write 13 bytes for header instead of 14; added unit test to verify 
This closes #1876.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-01 10:29:58 -04:00
Mark Payne
a0b2311ff6 NIFI-3995: This closes #1873. No longer use the 14th byte in the header for hwx content-encoded schema reference 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-31 13:41:25 -04:00
Koji Kawamura
13b59b5621 NIFI-3958: Decimal logical type with undefined precision and scale. 
- Oracle NUMBER can return 0 precision and -127 or 0 scale with variable scale NUMBER such as ROWNUM or function result
- Added 'Default Decimal Precision' and 'Default Decimal Scale' property to ExecuteSQL and QueryDatabaseTable to apply default precision and scale if those are unknown
- Coerce BigDecimal scale to field schema logical type, so that BigDecimals having different scale can be written

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1851
 2017-05-25 14:37:17 -04:00
Mark Payne
c49933f03d NIFI-3948: This closes #1834. Added flush() method to RecordWriter and call it when writing a single record to OutputStream for PublishKafkaRecord. Also removed no-longer-used class WriteAvroResult 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-19 23:05:04 -04:00
Mark Payne
ae9953db64 NIFI-3857: This closes #1825. Added PartitionRecord processor 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-19 02:08:52 -04:00