Commit Graph

4099 Commits

Author SHA1 Message Date
Mark Payne b93cf7bbdb NIFI-3658: Created processor for converting records between data formats with like schemas
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3658: Incorporated PR review feedback; added counter; clarified documentation

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1668
2017-04-19 16:12:14 -04:00
Matt Gilman 7e7f9a5deb NIFI-3705:
- Using consistent logic when verifying connection creation. Removed some unecessary checks as verification has been changed to run in cluster and standalone mode.

This closes #1672.
2017-04-19 15:49:11 -04:00
ijokarumawak 220a870a1e NIFI-2697: Fix opening Remote Ports dialog.
If there is an unconnected remote port, right clicking a Remote Process Group
then select 'Remote Ports' causes a JS error saying 'Uncaught TypeError'.
Because there are few code paths those don't return the created
transmissionSwitch value from createPortOption function in
nf-remote-process-group-ports.js.

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #1679
2017-04-18 16:08:10 -04:00
joewitt 189b4ad63d NIFI-3694 removed the total cap size value as per review feedback 2017-04-18 14:07:38 -04:00
Andre F de Miranda cd260bab9f
NIFI-2747 - Introduce FuzzyHashContent processor
This closes #1619.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-04-18 10:27:15 -04:00
Matt Gilman 7e1ecad738
NIFI-3706: - Removing non-existent resource when converting a legacy authorized users file.
This closes #1673.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-18 09:46:21 -04:00
joewitt d08233b9fe NIFI-3694 updated latest logback,jetty,moved servlet-api and jetty-schemas to lib root as needed now
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1678
2017-04-18 07:41:21 -04:00
Matt Burgess edb9c644dd NIFI-3413: Standardize 'write' event to 'insert'
This closes #1675.

Signed-off-by: ijokarumawak <ijokarumawak@apache.org>
2017-04-18 00:55:17 +09:00
Yolanda M. Davis 5f73ba7f2c NIFI-3708 - removed typo from character escape regex
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1674
2017-04-14 22:13:09 -04:00
Scott Aslan deed25656f
[NIFI-3035] use URLSearchParams .toString() to update URL. This closes #1659 2017-04-13 14:32:30 -04:00
Oleg Zhurakousky 68c592ea43 NIFI-1280 added support for RecordSchema in SchemaRegistry
Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Updated SimpleKeyValueSchemaRegistry to make use of new CHOICE RecordFieldType - Update Record Readers to use SchemaRegistry controller service. Moved SchemaRegistry api into its own maven module and added to standard-services-api so that we can properly add dependencies on it. Code cleanup and bug fixes

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Fixed checkstyle violations and license exclusions for RAT plugin

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Addressed feedback from PR Review

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Additional changes/doc to support QueryFlowFile and Record Readers/Writers

This closes #1652
2017-04-11 19:29:35 -04:00
Mark Payne a88d3bfa3c NIFI-1280: Refactoring to make more generic so that other data types can be supported; created InputStreams to content on-demand so that multiple passes can be made over FlowFile content if required. Created new Controller Services for reading and writing specific data types
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2017-04-11 19:29:04 -04:00
Toivo Adams 4d5872a385 NIFI-1280 Create FilterCSVColumns Processor.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2017-04-11 19:29:04 -04:00
joewitt 52cf9a7953 NIFI-3690 Updated nifi-gcp-nar L&N, cat-x deps
Updated L&N in nifi-gcp-nar and nifi-assembly and fixed nifi-gcp-processors
to avoid jsr305 and json.org cat-x dependencies

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1665.
2017-04-11 15:56:25 -07:00
Joey Frazee 4a8398bb31 NIFI-3692 This closes #1663. Use hbc-core instead of hbc-twitter4j in social-media bundle to resolve org.json cat-x
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-11 14:25:04 -04:00
Koji Kawamura 7c19796eb2 NIFI-3561: Fix WS port forwarding
Fixed service instance lookup to use local port so that it works when
it is accessed through port forwarding.

This fixes: #1568

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-04-11 21:27:53 +10:00
Pierre Villard 776a00b0be NIFI-2860 Use different properties for JVM Heap Max and Min values
This closes: #1382

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-04-11 21:18:49 +10:00
Joe Percivall 5419891fe7 NIFI-3633 This closes #1610. Adding HttpNotificationService.
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-11 00:37:12 -04:00
Mark Payne 8a5398eba7 NIFI-3679: This closes #1655. Added Avro Content Viewer
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-11 00:00:09 -04:00
Koji Kawamura 9583ca99c1 NIFI-3414: Added EnforceOrder processor
Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.

nifi-mock is modified to support FlowFile assertion using Prioritizer.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3414: Added EnforceOrder processor

Incorporated review comments, added displayNames.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3414: Added EnforceOrder processor

Incorporate review comments:

- Moved nifi-standard-prioritizers dependency to top level nifi/pom.xml.
- Changed default initial order from 1 to 0.
- Fixed typos.
- Use session.get(batchCount).

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3414: Added EnforceOrder processor

When a FlowFile is transferred to success, remove attributes previously set when it was transferred to wait or failure.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1496
2017-04-10 14:03:22 -04:00
Andre F de Miranda c614a74437 Blank commit to close stalled github PRs
This closes #254
This closes #293
This closes #626
This closes #969
2017-04-11 02:08:14 +10:00
Mike Moser 2589df80ab NIFI-3304 removed unneeded Java 8 codecache info from Admin Guide
This closes #1621

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-04-11 02:05:20 +10:00
Pierre Villard 02d38b55bf NIFI-2952 Fixed search value validation in ReplaceText
This closes: #1378

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-04-11 00:00:07 +10:00
Francois Prunier 61c799d88b NIFI-3204 This closes #1561. fix handling deleting a path with a wildcard when the processor is invoqued via an incoming flowfile
applied Joseph Witts patch from NIFI-3204 JIRA

Signed-off-by: joewitt <joewitt@apache.org>
2017-04-10 08:51:10 -04:00
Jack Pickett 4bfb905f37 NIFI-2714 This closes #1549. Added regex support to ConsumeKafka_0_10
Enabled the ability to specify wildcard topics as a regular expression
as supported in the Kafka client library.

Signed-off-by: joewitt <joewitt@apache.org>
2017-04-07 21:52:33 -04:00
Mark Payne 778ba3957e NIFI-3631: This closes #1613. Do not change Active Directory in IndexDirectoryManager when it becomes full. Instead, wait until it is committed and is removed by the onIndexCommitted method. This resolved a bug where the index can exceed the configured limit but not yet be committed and as a result would no longer be the active index. As a result, this bug causes the IndexWriter never to get closed/removed from the IndexManager, and so a memory leak is created
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-07 14:42:18 -04:00
Mark Payne a5d630672a NIFI-3635: This closes #1631. Avoid using a static member variable for the 'Grok' object. Code cleanup
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-07 13:22:46 -04:00
Matt Gilman 35f4f48f37 NIFI-3683:
- Ensuring we use the appropriate action based on the previously displayed policy.

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #1657
2017-04-07 13:03:01 -04:00
Matt Gilman e8771e59a7
NIFI-2940: - Fixing broken integration test following changes introduced in NIFI-2940.
This closes #1653.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-07 10:42:11 -04:00
Mark Payne 292dd1d66b
NIFI-3678: Ensure that we catch EOFException when reading header information from WAL Partition files; previously, we caught EOFExceptions when reading a 'record' from the WAL but not when reading header info
NIFI-3678: If we have a transaction ID but then have no more data written to Partition file, we end up with a NPE. Added logic to avoid this and instead return null for the next record when this happens

This closes #1656.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-07 10:28:27 -04:00
Bryan Bende 6a75ab1740
Fixing checkstyle error in FlowSnippetDTO 2017-04-07 09:58:15 -04:00
Oleg Zhurakousky c7ecaba234
NIFI-3509 Fixed ID duplication
Fixed the possibility of duplicated MSB part of the component during template creation. This closes #1651
2017-04-06 16:04:27 -04:00
Pierre Villard 614fa6a6c4
NIFI-3528 Added support for keytab/principal to Kafka 0.10 processors
This closes #1606.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-06 15:09:49 -04:00
Bryan Bende 556f309df0
NIFI-3520 Refactoring instance class loading
- Fixing FlowController to use appropriate class loader when instantiating processor
- Updating ExtensionManager to leverage new flag in MANIFEST from NAR plugin
- Adding ReloadComponent interface and refactoring instance class loading to use it
- Fixing FetchHDFS issue with TDE by using ugi.doAs
- Refactoring nifi-nar-utils so that ExtensionManager only lives in nifi-framework
- Caching temp components found during service loader in ExtensionManager
- Updating authorizables, docs, and fingerprinting to use the cached components
- Introducing a flag on @RequiresInstanceClassLoading to indicate if ancestor resources should be cloned
- Updating developer guide regarding cloneAncestorResources flag
- This closes #1635
2017-04-06 13:51:08 -04:00
Matt Burgess 8f37ad4512 NIFI-3413: Add GetChangeDataCaptureMySQL processor
NIFI-3413: Incorporated review comments
NIFI-3413: Changed GetChangeDataCaptureMySQL to CaptureChangeMySQL, fixed some bugs
NIFI-3413: Refactored setup() for better error handling, more review comments incorporated
NIFI-3413: Refactored CDC into its own module(s), updated assembly and top-level POMs
NIFI-3413: Added RECEIVE prov event and Server ID property

Signed-off-by: ijokarumawak <ijokarumawak@apache.org>
2017-04-07 00:44:42 +09:00
Puspendu Banerjee 3386839ebc NIFI-3586: Fix for retrieving ProcessID for NiFi under windows.
This closes #1586

Signed-off-by: Tony Kurc <tkurc@apache.org>
2017-04-05 23:09:48 -04:00
Matt Gilman f5f54c468c
NIFI-3664: - Updating the unmarshalling to consider the current date when parsing the time.
This closes #1647.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-05 13:22:35 -04:00
Matt Gilman a565484ddd
NIFI-2940: - Allowing access to configuration actions through the Controller when the underlying component has been removed.
This closes #1648.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-04 17:40:09 -04:00
Koji Kawamura 5e62b4ae72 NIFI-3668: Fix purging expired replicate requests.
This closes #1646.

Newly created async response is added before checking map size nor
purging expired ones. If there are already 100 remaining requests,
the added request will not be executed nor removed.
2017-04-03 09:50:47 -04:00
Andre F de Miranda 84f1fb3959 NIFI-3222 - Allow ExecuteScript,InvokeScriptedProcessor and ScriptedReporting to use ExpressionLanguage to define the scripting engine modules path
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1640
2017-03-31 23:41:02 -04:00
joewitt 6a64b3cd9c NIFI-3440 This closes #1638. fixing tests not written for windows to not run on windows 2017-03-31 01:59:35 -04:00
Mark Payne 091359b450 NIFI-3630 This closes #1632. Use a BufferedOutputStream when checkpointing FlowFile Repository
Signed-off-by: joewitt <joewitt@apache.org>
2017-03-30 16:46:53 -04:00
Koji Kawamura fd92999daf
NIFI-3189: ConsumeKafka 0.9 and 0.10 with downstream backpressure
Currently, NiFi Kafka consumer processors have following issue.

While downstream connections are full, ConsumeKafka is not scheduled to run onTrigger.
It stopps executing poll to tell Kafka server that this client is alive.
Thus, after a while in that situation, Kafka server rebalances the client.
When downstream connections back to normal, although ConsumeKafka is scheduled again,
the client is no longer a part of a consumer group.

If this happens, Kafka client succeeds polling messages when ConsumeKafka processor resumes, but fails to commit offset.
Received messages are already committed into NiFi flow, but since consumer offset is not updated, those will be consumed again, duplicated.

In order to address above issue:

- For ConsumeKafka_0_10, use latest client library

    Above issue has been addressed by KIP-62.
    The latest Kafka consumer poll checks if the client instance is still valid, and rejoin the group if not, before consuming messages.

- For ConsumeKafka (0.9), added manual retention logic using pause/resume

    Kafka client 0.9 doesn't have background thread heartbeat, so similar machanism is added manually.
    Use Kafka pause/resume consumer API to tell Kafka server that the client stops consuming messages but is still alive.
    Another internal thread is used to perform paused poll periodically based on the time passed since the last onTrigger(poll) is executed.

This closes #1527.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-03-30 16:29:40 -04:00
Matt Gilman 282e1a7b1a NIFI-3421: This closes #1620.
- On contextDestroyed, referencing beans created during contextInitialized to prevent successive attempts to create a bean if that bean failed to be created initially.

Signed-off-by: joewitt <joewitt@apache.org>
2017-03-30 15:14:41 -04:00
Matt Gilman e203535163 NIFI-3272 This closes #1616.
- Updating the garbage collection column to use a icon with tooltip to relay all garbage collections.
2017-03-30 14:55:17 -04:00
Mark Payne 3aa1db6ee5 NIFI-3636: Lazily copy FlowFile Attributes Hash Map instead of doing so eagerly.
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3257: Added additional logging regarding timing information when replicating requests across cluster in order to glean insight as to what is taking so long when replicating some requests

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3649: Buffer node responses when replicating HTTP Requests up to a maximum buffer size

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3636: Added unit test to ensure that flowfile attribute maps are copied when appropriate

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3636: Removed patch file that should not have been in commit

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1612
2017-03-30 09:53:39 -04:00
Andre F de Miranda a7bf683a0d
NIFI-79 - Introduces the ability to set Bulletin Level to NONE 2017-03-30 08:54:24 -04:00
kedarchitale 457df93dd6 NIFI-3147 Build processor to parse CCDA into attributes
This closes #1312

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2017-03-29 20:30:57 +00:00
Mark Payne 83fa24b68f NIFI-3613: Fixed threading bug in JoltTransformJSON and adding caching of compiled Transforms
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1630
2017-03-29 10:45:34 -04:00
Gahan Rakholia 168fc72bba NIFI-3637 Fixed ListHDFS Documentation
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1633
2017-03-29 10:31:56 -04:00