Apache
/
nifi
mirror of https://github.com/apache/nifi.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,303 Commits 146 Branches 201 Tags 401 MiB
Commit Graph

140 Commits

Author SHA1 Message Date
joewitt 7b5bf265a6 NIFI-4995 updating copyright year on all notices 2018-03-26 21:54:10 -04:00
Mark Payne 0b0aebe148
NIFI-4917: Externalize Keytab and Principal configuration from Processors to a Controller Service. This gives us the ability to allow users to interact with those Keytabs/Principals to which they've been given access without allowing them access to all Keytabs and Principals 
- Addressed review feedback; rebased against master

This closes #2552.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2018-03-21 14:23:41 -04:00
joewitt c71409fb5d
NIFI-4936 trying to quiet down the mvn output a bit so we dont exceed the travis-ci 4MB max 
NIFI-4936 updated dependency handling pushing down delcarations where they belong
This closes #2512
 2018-03-09 16:34:53 -05:00
Mark Payne 7c1ce17223
NIFI-4756: Updated PublishKafkaRecord processors to include attributes generated from schema write strategy into the message headers when appropriate 
This closes #2396.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2018-01-16 16:31:34 -05:00
joewitt 41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00
joewitt 36405e888c NIFI-4751-RC1 prepare for next development iteration 2018-01-08 23:39:49 -07:00
joewitt 46d30c7e92 NIFI-4751-RC1 prepare release nifi-1.5.0-RC1 2018-01-08 23:39:32 -07:00
Pierre Villard 10e3b14433 NIFI-4515 - This closes #2224. Enabled EL on Kerberos properties for Kafka 0.10 & 0.11 & 1.0 processors 
Signed-off-by: joewitt <joewitt@apache.org>
 2018-01-08 22:07:14 -07:00
Koji Kawamura e5ed62a98f NIFI-4724: Support 0 byte message with PublishKafka 
Before this fix, PublishKafka (0.9) and PublishKafka_0_10 fail with empty incoming FlowFiles due to 'transfer relationship not specified' error.
Because the internal 'publish' method is not called as StreamDemarcator does not emit any token regardless whether demarcator is set or not.

As for PublishKafka_0_11 and PublishKafka_1_0, empty FlowFiles are transferred to 'success' relationship, however no Kafka message is sent to Kafka.

Since Kafka allows 0 byte body empty messages, NiFi should be able to send it, too.

This commit changes above current situation to the followings, with all PublishKafka_* processors:

- If demarcator is not set, then publish incoming FlowFile content as it is. This enables sending an empty Kafka message.
- If demarcator is set, send each token as a separate message.
  Even if no token is found (empty incoming FlowFile), transfer the FlowFile to 'success'.

This closes #2362.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-01-05 10:42:58 -05:00
jknulst d543cfde25 NIFI-4675 Lifted restriction on demarcator and kafka.key usage together. This closes #2326. 2017-12-14 15:15:13 -05:00
Mark Payne c138987bb4 NIFI-4656, NIFI-4680: This closes #2330. Fix error handling in consume/publish kafka processors. Address issue with HortonworksSchemaRegistry throwing RuntimeException when it should be IOException. Fixed bug in ConsumeerLease/ConsumKafkaRecord that caused it to report too many records received 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-12-08 16:01:14 -05:00
Mark Payne 113ad5ecfa NIFI-4639: Updated PublishKafka 1.0 processor to use a fresh writer for each output record as well. This closes #2292. 2017-12-08 09:14:17 -05:00
matthew-silverman c9cc76b5c8 NIFI-4639: fresh writer for each output record 2017-12-08 08:39:22 -05:00
joewitt cdc1facf39
NIFI-4664, NIFI-4662, NIFI-4660, NIFI-4659 moved tests which are timing/threading/network dependent and brittle to integration tests and un-ignored tests that are IT. Updated travis to reduce impact on infra and appveyor now skips test runs so is just to prove build works on windows. This closes #2319 
squash
 2017-12-06 10:53:09 -05:00
Mark Payne 00b11e82b7 NIFI-4600: This closes #2312. Added nifi-kafka-1-0-nar and nifi-kafka-1-0-processors modules 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-12-04 16:51:59 -05:00
Janosch Woschitz e8b2387cb2 NIFI-4623: This closes #2281. Removed obsolete instability warning in documentation of newer (>= 0_10) Kafka processors 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-11-21 12:16:54 -05:00
Matt Gilman 6baea8ccff
NIFI-4444: 
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.

This closes #2206.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-10-12 10:27:02 -07:00
Mark Payne 7ad7520150 NIFI-4437: This closes #2183. When using ConsumeKafka_0_11 and no message demarcator, ensure that we add FlowFile Attributes for any Message Header that matches the 'Headers to Add as Attributes (Regex)' property 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-10-06 15:06:32 -04:00
Mark Payne 582df7f4e8 NIFI-4008: This closes #2189. Update ConsumeKafkaRecord 0.11 so that it can consume multiple records from a single Kafka message 
NIFI-4008: Ensure that we always check if a Kafka message's value is null before dereferencing it

Signed-off-by: joewitt <joewitt@apache.org>
 2017-10-06 15:03:37 -04:00
Koji Kawamura 58e4fb576e NIFI-4008: Allow 0 or more records within a message. This closes #1891. 2017-10-02 15:40:29 -04:00
Jeff Storck 2694adcca9 Merge branch 'NIFI-4412-RC2' 2017-10-02 13:58:54 -04:00
Mark Payne b3be2459e4 NIFI-4330: Fixed checkstyle violations (tabs instead of spaces). This closes #2185. 2017-09-29 10:33:56 -04:00
gardellajuanpablo 2d5b8c7267 NIFI-4330 ConsumeKafka* throw NullPointerException if Kafka message has a null value 
It is possible null values to be stored in Kafka topics. Fixed handle this scenario.
Notice without this fix, the consumer is unable to consume more messages (at least
without removing messages from the queue).
 2017-09-29 10:57:53 -03:00
Jeff Storck a57911d3db NIFI-4412-RC2 prepare for next development iteration 2017-09-28 13:45:36 -04:00
Jeff Storck e6508ba7d3 NIFI-4412-RC2 prepare release nifi-1.4.0-RC2 2017-09-28 13:45:21 -04:00
Mark Payne 3fb704c58f NIFI-4201: This closes #2024. Implementation of processors for interacting with Kafka 0.11 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-09-22 22:08:19 -04:00
Mark Payne d698f227df NIFI-4290: Ensure that the InFlightMessageTracker is instantiated when getTracker() is called 
This closes #2083.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-09-12 10:30:44 +09:00
Koji Kawamura 1f67cbf628 NIFI-4004: Use RecordReaderFactory without FlowFile. 
- Removed FlowFile from RecordReaderFactory, RecordSetWriterFactory and SchemaAccessStrategy.
- Renamed variable 'allowableValue' to 'strategy' to represent its meaning better.
- Removed creation of temporal FlowFile to resolve Record Schema from ConsumerLease.

- Removed unnecessary 'InputStream content' argument from
  RecordSetWriterFactory.getSchema method.

This closes #1877.
 2017-09-08 12:37:40 -04:00
Mark Payne 451f9cf124 NIFI-4142: This closes #2015. Refactored Record Reader/Writer to allow for reading/writing "raw records". Implemented ValidateRecord. Updated Record Reader to take two parameters for nextRecord: (boolean coerceTypes) and (boolean dropUnknownFields) 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-08-11 22:01:46 -07:00
Mark Payne cdc154f7c8 NIFI-4046: If we are unable to parse out any records from a Kafka Mesaage with ConsumeKafkaRecord, then we should route all of the bytes received to 'parse.failure' 
NIFI-4046: Addressed issue of Record Writer failing with ConsumeKafkaRecord

This closes #1906.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-06-30 17:56:14 +09:00
Matt Gilman 490e1da5db NIFI-4019: 
- Adding support for X-Forwarded-* headers.
- Unrelated code clean up.
- Addressing additional proxying issues.
- Addressing mistyped landing page.
- Handled trailing slashes in proxy headers for context path
 2017-06-15 14:17:35 -04:00
Mark Payne e7dcb6f6c5 NIFI-3921: Allow Record Writers to inherit schema from Record 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1902
 2017-06-09 16:13:25 -04:00
Matt Gilman cc741d2be6
NIFI-3997: 
- Bumping to next minor version.
 2017-06-08 15:22:51 -04:00
Matt Gilman 6ee12e9b47
NIFI-3997-RC1prepare for next development iteration 2017-06-05 11:07:43 -04:00
Matt Gilman ddb73612bd
NIFI-3997-RC1prepare release nifi-1.3.0-RC1 2017-06-05 11:07:28 -04:00
Mark Payne 2c751a8e5b NIFI-3962: This closes #1846. Updated ConsumerLease to better handle batching of messages into a single FlowFile in the same way that it is handled for demarcated data 2017-05-23 23:20:13 -04:00
Mark Payne 6937a6cf64 NIFI-3953: This closes #1837. Allow multiple schemas on same kafka topic/partition for ConsumeKafkaRecord_0_10 
Also, updated record writers to ensure that they write the schema as appropriate if not using a RecordSet. Updated ConsumeKafkaRecord to allow for multiple schemas to be on same topic and partition

Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-22 14:37:53 -04:00
Mark Payne c49933f03d NIFI-3948: This closes #1834. Added flush() method to RecordWriter and call it when writing a single record to OutputStream for PublishKafkaRecord. Also removed no-longer-used class WriteAvroResult 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-19 23:05:04 -04:00
Tim Reardon 58ce52d5d6 NIFI-3854 This closes #1773. Expand expression language support for Kafka processors 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-19 15:51:23 -04:00
Bryan Bende d4f0c1d048 NIFI-3945 This closes #1829. Adding documentaion about security protocols to Kafka 0.10 processors 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-19 13:59:27 -04:00
Mark Payne 9bd0246a96 NIFI-3863: Initial implementation of Lookup Services. Implemented LookupRecord processors. This required some refactoring of RecordSetWriter interface, so refactored that interface and all implementations and references of it 2017-05-19 01:02:41 -04:00
Mark Payne fb94990e60 NIFI-3909: This closes #1806. If we have a FlowFile with 0 records, ensure that PublishKafkaRecord_0_10 handles the flowfile properly 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-16 14:42:42 -04:00
Mark Payne b1901d5fe0 NIFI-3838: Initial implementation of RecordPath and UpdateRecord processor 
NIFI-3838: Updated version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT; removed unneeded value from AttributeExpression.ResultType enum

NIFI-3838: Addressed PR Review feedback

NIFI-3838: Allow for schemas to be merged together for a record; refactored RecordSetWriterFactory so that there is a method to obtain the schema and then the writer is created with that schema. Added additional unit tests

NIFI-3838: Addressed problems with documentation based on PR Review

NIFI-3838: Fixed checkstyle violation

NIFI-3838: Addressed issue of comparing different types of Number objects

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1772
 2017-05-12 12:36:52 -04:00
Bryan Bende 3af53419af
NIFI-3770-RC2 prepare for next development iteration 2017-05-05 20:50:28 -04:00
Bryan Bende 3a605af8e0
NIFI-3770-RC2 prepare release nifi-1.2.0-RC2 2017-05-05 20:50:14 -04:00
Mark Payne 40de1b18d9 NIFI-3739: This closes #1728. Pass the proper InputStream to RecordSetWriterFactory in order to obtain RecordSetWriter; also fix error handling so that we don't kill kafka client if unable to create writer, since we roll back the offsets 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-02 00:21:15 -04:00
Mark Payne 07989b8460 NIFI-3739: This closes #1695. Added ConsumeKafkaRecord_0_10 and PublishKafkaRecord_0_10 processors 2017-05-01 18:47:51 -04:00
joewitt 8651d79778 NIFI-3757 grabs latest kafka client 0.10.2.1 
This closes #1714.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-04-28 11:16:35 +09:00
Jack Pickett 4bfb905f37 NIFI-2714 This closes #1549. Added regex support to ConsumeKafka_0_10 
Enabled the ability to specify wildcard topics as a regular expression
as supported in the Kafka client library.

Signed-off-by: joewitt <joewitt@apache.org>
 2017-04-07 21:52:33 -04:00
Pierre Villard 614fa6a6c4
NIFI-3528 Added support for keytab/principal to Kafka 0.10 processors 
This closes #1606.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-04-06 15:09:49 -04:00