- Added detailed description about how the URL property works with
GetHTMLElement
- Added Expression support with URL
- Made URL property dynamic with ModifyHTMLElement and PutHTMLElement,
since it won't be used to alter HTML element and need not to be
specified. Making it a dynamic property let existing processor configuration stays valid
* add exec to RUN_NIFI_CMD
* remove subshell for else
* tested compatible with runit with these changes
This closes#966.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
* Corrected handling of corrupt journal file records that prevents instance startup and loss of records from corrupt files. Specifically, exception handling was expanded to cover failures on records after the first the same as failures on the first record.
* Adjusted log messages to reflect that the remainder or all of the journal will be skipped, not just the current record.
This closes#1485.
H2 and Kafka broker uses the same default port 9092.
If an user is running Kafka broker on the same machine, or run the unit
tests in parallel, DBCPServiceTest can fail since some of its test
methods connects to port 9092.
This closes#1504.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
* Remove function based on JDK source.
* Add new function to find bytes based on RFC3629.
* Add field name to log entry when field is truncated.
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#1475
* Credentials service with tests
* Abstract processor definitions
* GCS-themed processors and their corresponding tests
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#1482.
use the FileNameFilter when not passing down explit jar paths
Filter out ^. files when reading lists of files from directories
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Added configure audits for Transport Protocol, HTTP Proxy Server Host,
Port, User and Password in RemoteProcessGroup configuration
- Added configure audits for enabling/disabling individual remote port
- Added configure audits for Concurrent Tasks and Compressed in Remote
Port configuration
- This closes#1476
* Updated StandardRecordWriter, even though it is now deprecated to consider the encoding behavior of java.io.DataOutputStream.writeUTF() and truncate string values such that the UTF representation will not be longer than that DataOutputStream's 64K UTF format limit.
* Updated the new SchemaRecordWriter class to similarly truncate long Strings that will be written as UTF.
* Add tests to confirm handling of large UTF strings and various edge conditions of UTF string handling.
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#1469.
- Marked PutKafka Partition Strategy property as deprecated, as Kafka 0.8 client doesn't use 'partitioner.class' as producer property, we don't have to specify it.
- Changed Partition Strategy property from a required one to a dynamic property, so that existing processor config can stay in valid state.
- Fixed partition property to work.
- Route a flow file if it failed to be published due to invalid partition.
This closes#1425
NIFI-2615 Addressing changes from P/R. Specifically, removing .gitignore as it should not be there for a nar. Removed non-used class. Changed name in notice
- Requiring WRITE permissions to the parent resource when attempting to remove a component.
- Updating expired certificates in the REST API integration tests.
This closes#1399.
Signed-off-by: James Wing <jvwing@gmail.com>
- Support counters at Wait/Notify processors so that NiFi flow can be
configured to wait for N signals
- Extract Wait/Notify logics into WaitNotifyProtocol
- Added FragmentAttributes to manage commonly used fragment attributes
- Changed existing split processors to set 'fragment.identifier' and
'fragment.count', so that Wait can use those to wait for all splits
get processed
This closes#1420.
Signed-off-by: Bryan Bende <bbende@apache.org>
Add support in SelectHiveQL to get script content from the Flow File to bring consistency with patterns used for PutHiveQL and support extra query management.
Changed behavior of using Flowfile to match ExecuteSQL. Handle query delimiter when embedded. Added test case for embedded delimiter
Formatting and License Header
PutHiveQL and SelectHiveQL Processor enhancements. Added support for multiple statements in a script. Options for delimiters, quotes, escaping, include header and alternate header.
Add support in SelectHiveQL to get script content from the Flow File to bring consistency with patterns used for PutHiveQL and support extra query management.
Changed behavior of using Flowfile to match ExecuteSQL. Handle query delimiter when embedded. Added test case for embedded delimiter
Removing dead code.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Comments to Clarify test case.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Final whitespace/formatting/typo changes
This closes#1316
- Adding additional parameters to be able to limit the size of the provenance response. Specifically, whether the events should be summarized and whether events should be returned incrementally before the query has completed.
- Ensuring the cluster node address is included in provenance events returned.
- Ensuring there is a cluster coordinator before attempting to get the cluster node address.
- Removing exponential back off between provenance requests.
- Ensuring the content viewer url is retrieve before initializing the provenance table.
This closes#1413.
- Using fetch and replace together can provide optimistic locking for
concurrency control.
- Added fetch to get cache entry with its meta data such as revision
number.
- Added replace to update cache only if it has not been updated.
- Added Map Cache protocol version 2 for those new operations.
- Existing operations such as get or put can work with protocol version
1.
This closes#1410.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-3004 Added logic to expire StandardSSLContextService customValidate cache after 5 invocations.
Updated unit test to demonstrate this logic.
This closes#1375.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Removing unnecessary authorization check during second phase of connection creation.
- Ensuring that the remote group port returns the correct resource type though not super critical since it is not possible to create policies for remote ports.
This closes#1353.
- Fixing the importUri when replicating a Template upload request. Previously was leveraging getResourceUri which considers proxy headers meant for client responses.
This closes#1334.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Removed host and port field from Peer since the same information is
available in PeerDescription
- Refactored variable names in SocketRemoteSiteListener to improve readability
- Changed how SocketRemoteSiteListener constructs PeerDescription
instance. It used to use hard-coded 'localhost' as hostname, and
getPort() which returns server's port. Since the peer is a remote peer,
i.e the client, it should be client hostname and port.
- Added hostname resolution at DataTransferResource to make s2s.host
value consistent with RAW transport. Without this, RAW uses hostname
while HTTP uses IP address. It will be hard to be used from downstream flows.
- Replaced heavy use of mockito which was difficult to maintain, with
nifi-mock
- Added SiteToSiteAttributes and more assertions in unit tests
This closes#1342.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Ensure that controller services are added to child process groups when creating snippet
- Addressed issue related to modifying higher-level process groups' controller services in snippet after having already visited the process group
This closes#1318
Signed-off-by: jpercivall <JPercivall@apache.org>
NIFI-1582 removing the option to use cluster state
NIFI-1582 addressing Oleg's comments
NIFI-1582 No longer forcing numbers as the init value and adding getStateValue() to EL instead of using 'ATTRIBUTE_state'
NIFI-1582 Removing init state value
NIFI-1582 Adding documentation for the changes to Init State value
This closes#319
NIFI-3133:
- Using getSourceAuthorizable() when accessing flow files and content.
NIFI-3133:
- Decouple local and remote connectable's to avoid ambiguity with self referencing RPGs.
NIFI-3133:
- Addressing comments from the PR.
NIFI-3133:
- Fixed check verifying source/destination when creating a connection.
NIFI-3133:
- Only showing the go to link when the source component is not a remote port.
NIFI-3133:
- Removing unnecessary checking of remote group port authorization since it's handled by the parent RPG.
NIFI-3133:
- Fixing issue showing the connection details dialog when the source component is a RPG.
NIFI-3133:
- Ensuring the local connectable was found.
This closes#1297
Signed-off-by: jpercivall <JPercivall@apache.org>
- Added urls in addition to the existing url, to support multiple target
URLs
- Backward compatibility is provided by returning the first url if
multipe urls are specified, but component accessing the url doesn't
support multiple urls
- UI is not fully updated yet. Following UI components are planned to be updated
by different commits
- Search component: only the first URL is searchable and shown
- Component status: RPG status shows only the first URL
- Component action history: only the first URL is searchable and shown
- Updated Search component to use URLs.
This closes#1208.
- Updating REST API docs accordingly.
- Adding some integration tests to exercise the additional restricted component checks.
This closes#1287.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Added unit test cases to simulate NiFi version update which fails without this fix.
- Added state object migration code, add file.0. prefix to state keys,
and add length from stored position.
This closes#1289
Fixed merging logic for root group status. Only consider a port transmitting when there is an active request, not when there is a 'queued' request
This closes#1279
- Updating the SnippetManager to automatically expire unused snippets.
- Making the checking of referenced services consistent across component removal.
- Adding checking of referenced services for all nested components in a snippet.
- Updating the REST API docs to clarify when a referenced service is authorized.
- Conditionally authorizing transitive service references.
This closes#1277.
Signed-off-by: Bryan Bende <bbende@apache.org>
- validate processors only when they are in STOPPED state
- report validation errors via REST API on processors/services/tasks/ports only when they are in the STOPPED state
- This closes#1192
- Addressing comments from PR.
- Adding restricted tags to relevant components.
- Showing a restricted icon overlay on the processor node on the canvas. (+1 squashed commit)
Squashed commits:
[f487682] NIFI-3050:
- Introducing a Restricted annotation for components that require elevated privileges to use.
- Updating the new Processor, Controller Service, and Reporting Task dialogs to include these details and prevent unauthorized selection.
- Including the Restricted description in the generated component documentation.
- Updating processor access control integration test to verify restricted component creation.
- Updating the developer, user, and admin guide to include the restricted component policy.
This closes#1247.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2750
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2750 Tweaking Property Retrieval
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1001
- Introducing a strategy for identifying users.
- Fixing issue with the referral strategy error message.
- Adding code to shutdown the application when the authorizer or login identity provider are not initialized successfully.
NIFI-3020:
- Updating the admin guide to document the identity strategy.
NIFI-3020:
- Ensuring the request replicator attempts to shutdown regardless of whether the flow service properly terminates.
This closes#1236
NIFI-2854: Incorporated PR review feedback
NIFI-2854: Implemented feedback from PR Review
NIFI-2854: Ensure that all resources are closed on CompressableRecordReader.close() even if an IOException is thrown when closing one of them
This closes#1202
- Returning the appropriate authorizable when accessing provenance events for a manual DROP event by emptying a queue.
- Populating the component details of a provenance event when the source is a connection.
Pierre Villard
Andy LoPresto
Koji Kawamura
JJ
patricker
Bryan Rosander
Andre F de Miranda
Hunter Morgan
Simon Elliston Ball
Matt Burgess
Mike Moser
Joe Skora
Selim Namsi
joewitt
Yolanda M. Davis
Scott Aslan
Gene Peters
Otto Fowler
Matt Gilman
Ryan Persaud
Nick Carenza
Oleg Zhurakousky
Giovanni Lanzani
Joe Gresock
James Wing
Ben Schofield
Jeff Storck
Andrew Psaltis
David W. Streever
Andrew Lim
afucs
Joey Frazee
jpercivall
Davy De Waele
Bryan Bende
Randy Gelhausen
Wesley-Lawrence
Mark Payne
Byunghwa Yun
jpercivall
Joey Frazee
Kathik Narayanan
Peter Wicks