Apache/nifi
1
0
Fork 0
mirror of https://github.com/apache/nifi.git synced 2025-02-08 19:14:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,094 Commits 147 Branches 203 Tags
Commit Graph

106 Commits

Author SHA1 Message Date
Dustin Rodrigues
161e4b5763 NIFI-6231 - fix source code permissions to be non-executable 
This closes #3449.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2019-04-22 10:13:07 +09:00
Joe Witt
1bdab73201 NIFI-6121 Updated the easy dependencies to update 
This closes #3374

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
 2019-03-22 09:01:21 -04:00
thenatog
36bbc77723 NIFI-6097 - Upgraded the fasterxml jackson version to 2.9.8. Ensure that the version is consistent across modules using a maven property defined in the root pom. 
This closes #3347

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
 2019-03-04 19:30:35 -05:00
Matthew Burgess
e5fa18d63c NIFI-6088: Widen type inference for BIGINT and DOUBLE 
This closes #3342

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
 2019-02-28 05:51:33 -05:00
joewitt
0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt
45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Mark Payne
82f44155f6
NIFI-6044: This closes #3314. Retain the input data's order in the CSV Reader's inferred schema 
Signed-off-by: joewitt <joewitt@apache.org>
 2019-02-16 20:57:42 -05:00
Mark Payne
b508d6bfbc
NIFI-6033, NIFI-6034, NIFI-6035, NIFI-6036, NIFI-6037: Fixed bugs that were found during 1.9.0-RC1 validation. If multiple FlowFiles were written to same Content Claim, and a Processor attempted to read two of them wi within a single session, it would seek to the wrong part of the content or else throw a ContentNotFoundException. Updated logic for considering a processor to be 'running' / having 'active threads' if the processor is invalid upon NiFi restart but scheduled to run. Fixed NPE in FreeFormTextWriter. If MergeRecord reaches minimum number of records, flush writer after writing content out so that its minimum size can accurately be checked. 
This closes #3309.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2019-02-15 11:59:47 -05:00
Mark Payne
36c0a99e91 NIFI-5938: Added ability to infer record schema on read from JsonTreeReader, JsonPathReader, XML Reader, and CSV Reader. 
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to  to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
 - Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
 - Updated LookupRecord to allow for Record fields to be arbitrarily added
 - Implemented ContentClaimInputStream
 - Added controller service for caching schemas
 - UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.

NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).

NIFI-5938: Added another unit test for AvroTypeUtil

NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3253
 2019-02-11 12:56:50 -05:00
Endre Zoltan Kovacs
24a7d480c8 NIFI-5983: handling parse problems in recordReader implementations 
Fixed Checkstyle violation

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3282
 2019-02-06 15:54:15 -05:00
Pierre Villard
a6f91a1975 NIFI-5872 - Added compression option to JsonRecordSetWriter 
This closes #3208.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2018-12-10 16:25:59 +09:00
Mark Payne
d3b1674813 NIFI-5805: Pool the BinaryEncoders used by the WriteAvroResultWithExternalSchema writer. Unfortunately, the writer that embeds schemas does not allow for this optimization due to the Avro API 
This closes #3160.
 2018-11-12 09:32:12 -05:00
Arek Burdach
765df67817 NIFI-5757 Using Caffeine instead of slow synchronization on LinkedHashMap for caches - mainly avro schema caches 
This closes #3111.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-11-09 14:50:24 -05:00
Ed B
2812fe60a2 NIFI-5728 - XML Writer to populate record tag name properly 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3098.
 2018-11-02 09:46:45 +01:00
Joe Percivall
db966cf348 NIFI-5765 Fixing WriteJsonResult to use chosenDataType when writing an Array value 
Fixing the same bug in PutElasticsearchHttpRecord.java

This closes #3114.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2018-10-30 11:51:17 +09:00
Jeff Storck
c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00
Jeff Storck
98aabf2c50 NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 2018-10-22 22:16:23 -04:00
Pierre Villard
c6106d1d88 NIFI-5525 - CSVRecordReader fails with StringIndexOutOfBoundsException when field is a double quote 
review

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3092
 2018-10-22 16:58:36 -04:00
joewitt
8e233ca2ef
NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin 
This closes #3028
 2018-10-04 09:32:04 -04:00
Mark Payne
2e1005e884 NIFI-5640: Improved efficiency of Avro Reader and some methods of AvroTypeUtil. Also switched ServiceStateTransition to using read/write locks instead of synchronized blocks because profiling showed that significant time was spent in determining state of a Controller Service when attempting to use it. Switching to a ReadLock should provide better performance there. 
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3036
 2018-09-27 15:38:47 -04:00
Mark Payne
647152ab16
NIFI-5597: Restored FreeFormTextRecordSetWriter to the META-INF services file 
This closes #3004.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2018-09-14 09:52:39 -04:00
Vadim Arshavsky
410176ed22 NIFI-5525: CSVRecordReader fails with StringIndexOutOfBoundsException when field is a double quote 
This closes #2953.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-08-16 14:22:27 -04:00
Otto Fowler
0a493bf7fd
NIFI-5432 Add Syslog Record Reader legacy Syslog 
- Add additional details for schema

This closes #2900.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2018-07-17 13:40:47 -04:00
Otto Fowler
b10220439c
NIFI-5337 Syslog 5424 Record Reader and nifi-syslog-utils 
- Create nifi-syslog-utils to move syslog parsing functionalty to a central location shared by the processors and serialization/record system.
- Refactor Processors to use these utils
- Update 5424 syslog classes using simple-syslog-5424 to pick up new changes to support this work, as well as keep dependencies/types from bleeding out to the
processors or readers
- Refactor Syslog5424Event and Parser
- Create Syslog5424RecordReader
- per review, handle blank message differently from eof
- name schema per review

This closes #2816.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2018-07-16 11:50:55 -04:00
Mark Bean
b279624398 NIFI-5368 controller services validated prior to enabling; referenced controller services must be enabled for referencing component to be valid (mock framework) 
This closes #2873.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-07-10 11:23:40 -04:00
Andy LoPresto
d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto
99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
zenfenan
cf3c666683 NIFI-5286: Updated FasterXML Jackson libraries to 2.9.5 
This closes #2775

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
 2018-06-09 14:19:45 -04:00
Mark Payne
49228aa5dc NIFI-5281: If value is not valid according to the schema's CHOICE field, JSON Writer should write null value instead of throwing NullPointerException 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2772.
 2018-06-08 10:07:43 +02:00
Mike Thomsen
716587d09f NIFI-5169 This closes #2705. Upgrade to JSONPath 2.4 
NIFI-5169 Added explicit checking for empty strings in JsonPath expression property.

Signed-off-by: joewitt <joewitt@apache.org>
 2018-05-23 16:16:25 -05:00
JohannesDaniel
2c8c9374af NIFI-5113: Add XMLRecordSetWriter 
This closes #2675.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-05-15 11:07:19 -04:00
Otto Fowler
61fe493786 NIFI-5170 Upgrad Grok to version 0.1.9 
This closes #2691

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
 2018-05-13 08:40:20 -04:00
Mark Payne
4700b8653d NIFI-5138: Bug fix to ensure that when we have a CHOICE between two or more REOCRD types that we choose the appropriate RECORD type when creating the Record in the JSON Reader. 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2670.
 2018-05-09 18:38:04 +02:00
Matthew Burgess
0289ca7114 NIFI-4456: Support multiple JSON objects in JSON record reader/writers 
This closes #2640.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-05-02 15:22:01 -04:00
Matthew Burgess
159b64b4c8
NIFI-5123: Move SchemaRegistryService to nifi-avro-record-utils 
This closes #2661.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2018-04-26 11:50:01 -04:00
Mark Payne
0e736f59fd NIFI-4185: Minor tweaks to XML Record Reader, around documentation and error handling 
This closes #2587.
 2018-04-23 14:41:17 -04:00
JohannesDaniel
d21bd3870b NIFI-4185: Add XML Record Reader 2018-04-23 14:39:18 -04:00
joewitt
59f625d3c2 Merge branch 'NIFI-4995-RC3' as part of NiFi 1.6.0 release process 2018-04-06 16:48:08 -07:00
Pierre Villard
4c787799ff NIFI-4149 - Indicate if EL is evaluated against FFs or not 
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
 2018-04-06 12:40:54 -04:00
joewitt
d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt
f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
Derek Straka
95dd1ebffc NIFI-4882: Resolve issue with parsing custom date, time, and timestamp formats in CSV files 
Refactor the two existing CSV classes to inherit from an abstract base class that parses the data
Add unit tests for calls to nextRecord that do not coerce types

This closes #2473.

Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2018-03-22 09:18:00 +09:00
Derek Straka
5bdb7cf6e7 NIFI-4912: This closes #2494. Update jackson version to latest stable version (2.9.4) 
Signed-off-by: joewitt <joewitt@apache.org>
 2018-03-19 10:22:50 -04:00
Bryan Bende
de71a41bd0 NIFI-4935 Refactoring to support specifying schema branch or schema version when using schema by name strategy 
This closes #2523.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-03-15 16:16:12 -04:00
Pierre Villard
11a7d5c09f NIFI-4955 - Preserve columns ordering with CSV and ValidateRecord 
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2528
 2018-03-13 17:09:07 -04:00
Pierre Villard
95e543d4cc NIFI-4967 - CSVRecordReader does not read header with specific formats 
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2536
 2018-03-13 16:54:46 -04:00
Pierre Villard
c75604a564 NIFI-4966 - JacksonCSVRecordReader - NPE with some CSV formats 
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2535
 2018-03-13 16:49:47 -04:00
joewitt
c71409fb5d
NIFI-4936 trying to quiet down the mvn output a bit so we dont exceed the travis-ci 4MB max 
NIFI-4936 updated dependency handling pushing down delcarations where they belong
This closes #2512
 2018-03-09 16:34:53 -05:00
Mark Payne
7c1ce17223
NIFI-4756: Updated PublishKafkaRecord processors to include attributes generated from schema write strategy into the message headers when appropriate 
This closes #2396.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2018-01-16 16:31:34 -05:00
joewitt
41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00