Apache
/
nifi
mirror of https://github.com/apache/nifi.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
4,735 Commits 146 Branches 201 Tags 404 MiB
Commit Graph

154 Commits

Author SHA1 Message Date
Pierre Villard 854ee3021c NIFI-5599 - Bump Kafka versions 
This closes #3006

Signed-off-by: zenfenan <sivaprasanna246@gmail.com>
 2018-09-20 14:33:23 +05:30
Mark Payne 0da4f50ea0 NIFI-5518: Added processors for integrating with Apache Kafka 2.0 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2948.
 2018-09-19 15:50:08 +02:00
Mark Payne c8fc1327ee NIFI-5592: If an Exception is thrown by RecordReader.read() from ConsumeKafkaRecord, route Record to parse.failure relationship 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3001.
 2018-09-15 14:49:24 +02:00
Corey Fritz 89186fb96d NIFI-5388: enabled EL support for dynamic properties of Kafka 1.0 processors 
This closes #2915

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
 2018-08-30 07:08:00 -04:00
Jan Hentschel c2657ee043 NIFI-5423 Removed duplicated dependencies in sub-modules 
This closes #2887

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
 2018-08-15 11:01:51 -04:00
Corey Fritz 4baf35db5b NIFI-5083: enabled EL support for processor configuration of Kafka consumer `group.id` property 
Signed-off-by: zenfenan <zenfenan@apache.org>
 2018-07-26 22:04:49 +05:30
Andy LoPresto f60585a9b6
NIFI-5376 Removed deprecation warnings. 
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.

This closes #2841.

Signed-off-by: Joe Witt <joewitt@apache.org>
 2018-07-09 20:45:34 -07:00
Andy LoPresto d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto 99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Mark Payne 620d446860 NIFI-5194: Ensure that even if calling KafkaConsumer.resume() throws an Exception, that we still release the lock that we are holding 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2701.
 2018-05-15 11:52:55 +02:00
joewitt 59f625d3c2 Merge branch 'NIFI-4995-RC3' as part of NiFi 1.6.0 release process 2018-04-06 16:48:08 -07:00
Pierre Villard 4c787799ff NIFI-4149 - Indicate if EL is evaluated against FFs or not 
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
 2018-04-06 12:40:54 -04:00
joewitt d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
joewitt 7b5bf265a6 NIFI-4995 updating copyright year on all notices 2018-03-26 21:54:10 -04:00
Mark Payne 0b0aebe148
NIFI-4917: Externalize Keytab and Principal configuration from Processors to a Controller Service. This gives us the ability to allow users to interact with those Keytabs/Principals to which they've been given access without allowing them access to all Keytabs and Principals 
- Addressed review feedback; rebased against master

This closes #2552.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2018-03-21 14:23:41 -04:00
joewitt c71409fb5d
NIFI-4936 trying to quiet down the mvn output a bit so we dont exceed the travis-ci 4MB max 
NIFI-4936 updated dependency handling pushing down delcarations where they belong
This closes #2512
 2018-03-09 16:34:53 -05:00
Mark Payne 7c1ce17223
NIFI-4756: Updated PublishKafkaRecord processors to include attributes generated from schema write strategy into the message headers when appropriate 
This closes #2396.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2018-01-16 16:31:34 -05:00
joewitt 41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00
joewitt 36405e888c NIFI-4751-RC1 prepare for next development iteration 2018-01-08 23:39:49 -07:00
joewitt 46d30c7e92 NIFI-4751-RC1 prepare release nifi-1.5.0-RC1 2018-01-08 23:39:32 -07:00
Pierre Villard 10e3b14433 NIFI-4515 - This closes #2224. Enabled EL on Kerberos properties for Kafka 0.10 & 0.11 & 1.0 processors 
Signed-off-by: joewitt <joewitt@apache.org>
 2018-01-08 22:07:14 -07:00
Koji Kawamura e5ed62a98f NIFI-4724: Support 0 byte message with PublishKafka 
Before this fix, PublishKafka (0.9) and PublishKafka_0_10 fail with empty incoming FlowFiles due to 'transfer relationship not specified' error.
Because the internal 'publish' method is not called as StreamDemarcator does not emit any token regardless whether demarcator is set or not.

As for PublishKafka_0_11 and PublishKafka_1_0, empty FlowFiles are transferred to 'success' relationship, however no Kafka message is sent to Kafka.

Since Kafka allows 0 byte body empty messages, NiFi should be able to send it, too.

This commit changes above current situation to the followings, with all PublishKafka_* processors:

- If demarcator is not set, then publish incoming FlowFile content as it is. This enables sending an empty Kafka message.
- If demarcator is set, send each token as a separate message.
  Even if no token is found (empty incoming FlowFile), transfer the FlowFile to 'success'.

This closes #2362.

Signed-off-by: Mark Payne <markap14@hotmail.com>
 2018-01-05 10:42:58 -05:00
jknulst d543cfde25 NIFI-4675 Lifted restriction on demarcator and kafka.key usage together. This closes #2326. 2017-12-14 15:15:13 -05:00
Mark Payne c138987bb4 NIFI-4656, NIFI-4680: This closes #2330. Fix error handling in consume/publish kafka processors. Address issue with HortonworksSchemaRegistry throwing RuntimeException when it should be IOException. Fixed bug in ConsumeerLease/ConsumKafkaRecord that caused it to report too many records received 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-12-08 16:01:14 -05:00
Mark Payne 113ad5ecfa NIFI-4639: Updated PublishKafka 1.0 processor to use a fresh writer for each output record as well. This closes #2292. 2017-12-08 09:14:17 -05:00
matthew-silverman c9cc76b5c8 NIFI-4639: fresh writer for each output record 2017-12-08 08:39:22 -05:00
joewitt cdc1facf39
NIFI-4664, NIFI-4662, NIFI-4660, NIFI-4659 moved tests which are timing/threading/network dependent and brittle to integration tests and un-ignored tests that are IT. Updated travis to reduce impact on infra and appveyor now skips test runs so is just to prove build works on windows. This closes #2319 
squash
 2017-12-06 10:53:09 -05:00
Mark Payne 00b11e82b7 NIFI-4600: This closes #2312. Added nifi-kafka-1-0-nar and nifi-kafka-1-0-processors modules 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-12-04 16:51:59 -05:00
Janosch Woschitz e8b2387cb2 NIFI-4623: This closes #2281. Removed obsolete instability warning in documentation of newer (>= 0_10) Kafka processors 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-11-21 12:16:54 -05:00
Matt Gilman 6baea8ccff
NIFI-4444: 
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.

This closes #2206.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-10-12 10:27:02 -07:00
Mark Payne 7ad7520150 NIFI-4437: This closes #2183. When using ConsumeKafka_0_11 and no message demarcator, ensure that we add FlowFile Attributes for any Message Header that matches the 'Headers to Add as Attributes (Regex)' property 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-10-06 15:06:32 -04:00
Mark Payne 582df7f4e8 NIFI-4008: This closes #2189. Update ConsumeKafkaRecord 0.11 so that it can consume multiple records from a single Kafka message 
NIFI-4008: Ensure that we always check if a Kafka message's value is null before dereferencing it

Signed-off-by: joewitt <joewitt@apache.org>
 2017-10-06 15:03:37 -04:00
Koji Kawamura 58e4fb576e NIFI-4008: Allow 0 or more records within a message. This closes #1891. 2017-10-02 15:40:29 -04:00
Jeff Storck 2694adcca9 Merge branch 'NIFI-4412-RC2' 2017-10-02 13:58:54 -04:00
Mark Payne b3be2459e4 NIFI-4330: Fixed checkstyle violations (tabs instead of spaces). This closes #2185. 2017-09-29 10:33:56 -04:00
gardellajuanpablo 2d5b8c7267 NIFI-4330 ConsumeKafka* throw NullPointerException if Kafka message has a null value 
It is possible null values to be stored in Kafka topics. Fixed handle this scenario.
Notice without this fix, the consumer is unable to consume more messages (at least
without removing messages from the queue).
 2017-09-29 10:57:53 -03:00
Jeff Storck a57911d3db NIFI-4412-RC2 prepare for next development iteration 2017-09-28 13:45:36 -04:00
Jeff Storck e6508ba7d3 NIFI-4412-RC2 prepare release nifi-1.4.0-RC2 2017-09-28 13:45:21 -04:00
Mark Payne 3fb704c58f NIFI-4201: This closes #2024. Implementation of processors for interacting with Kafka 0.11 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-09-22 22:08:19 -04:00
Mark Payne d698f227df NIFI-4290: Ensure that the InFlightMessageTracker is instantiated when getTracker() is called 
This closes #2083.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-09-12 10:30:44 +09:00
Koji Kawamura 1f67cbf628 NIFI-4004: Use RecordReaderFactory without FlowFile. 
- Removed FlowFile from RecordReaderFactory, RecordSetWriterFactory and SchemaAccessStrategy.
- Renamed variable 'allowableValue' to 'strategy' to represent its meaning better.
- Removed creation of temporal FlowFile to resolve Record Schema from ConsumerLease.

- Removed unnecessary 'InputStream content' argument from
  RecordSetWriterFactory.getSchema method.

This closes #1877.
 2017-09-08 12:37:40 -04:00
Mark Payne 451f9cf124 NIFI-4142: This closes #2015. Refactored Record Reader/Writer to allow for reading/writing "raw records". Implemented ValidateRecord. Updated Record Reader to take two parameters for nextRecord: (boolean coerceTypes) and (boolean dropUnknownFields) 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-08-11 22:01:46 -07:00
Mark Payne cdc154f7c8 NIFI-4046: If we are unable to parse out any records from a Kafka Mesaage with ConsumeKafkaRecord, then we should route all of the bytes received to 'parse.failure' 
NIFI-4046: Addressed issue of Record Writer failing with ConsumeKafkaRecord

This closes #1906.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-06-30 17:56:14 +09:00
Matt Gilman 490e1da5db NIFI-4019: 
- Adding support for X-Forwarded-* headers.
- Unrelated code clean up.
- Addressing additional proxying issues.
- Addressing mistyped landing page.
- Handled trailing slashes in proxy headers for context path
 2017-06-15 14:17:35 -04:00
Mark Payne e7dcb6f6c5 NIFI-3921: Allow Record Writers to inherit schema from Record 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1902
 2017-06-09 16:13:25 -04:00
Matt Gilman cc741d2be6
NIFI-3997: 
- Bumping to next minor version.
 2017-06-08 15:22:51 -04:00
Matt Gilman 6ee12e9b47
NIFI-3997-RC1prepare for next development iteration 2017-06-05 11:07:43 -04:00
Matt Gilman ddb73612bd
NIFI-3997-RC1prepare release nifi-1.3.0-RC1 2017-06-05 11:07:28 -04:00
Mark Payne 2c751a8e5b NIFI-3962: This closes #1846. Updated ConsumerLease to better handle batching of messages into a single FlowFile in the same way that it is handled for demarcated data 2017-05-23 23:20:13 -04:00