- Added configure audits for Transport Protocol, HTTP Proxy Server Host,
Port, User and Password in RemoteProcessGroup configuration
- Added configure audits for enabling/disabling individual remote port
- Added configure audits for Concurrent Tasks and Compressed in Remote
Port configuration
- This closes#1476
* Updated StandardRecordWriter, even though it is now deprecated to consider the encoding behavior of java.io.DataOutputStream.writeUTF() and truncate string values such that the UTF representation will not be longer than that DataOutputStream's 64K UTF format limit.
* Updated the new SchemaRecordWriter class to similarly truncate long Strings that will be written as UTF.
* Add tests to confirm handling of large UTF strings and various edge conditions of UTF string handling.
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#1469.
- Marked PutKafka Partition Strategy property as deprecated, as Kafka 0.8 client doesn't use 'partitioner.class' as producer property, we don't have to specify it.
- Changed Partition Strategy property from a required one to a dynamic property, so that existing processor config can stay in valid state.
- Fixed partition property to work.
- Route a flow file if it failed to be published due to invalid partition.
This closes#1425
NIFI-2615 Addressing changes from P/R. Specifically, removing .gitignore as it should not be there for a nar. Removed non-used class. Changed name in notice
- Requiring WRITE permissions to the parent resource when attempting to remove a component.
- Updating expired certificates in the REST API integration tests.
This closes#1399.
Signed-off-by: James Wing <jvwing@gmail.com>
- Support counters at Wait/Notify processors so that NiFi flow can be
configured to wait for N signals
- Extract Wait/Notify logics into WaitNotifyProtocol
- Added FragmentAttributes to manage commonly used fragment attributes
- Changed existing split processors to set 'fragment.identifier' and
'fragment.count', so that Wait can use those to wait for all splits
get processed
This closes#1420.
Signed-off-by: Bryan Bende <bbende@apache.org>
Add support in SelectHiveQL to get script content from the Flow File to bring consistency with patterns used for PutHiveQL and support extra query management.
Changed behavior of using Flowfile to match ExecuteSQL. Handle query delimiter when embedded. Added test case for embedded delimiter
Formatting and License Header
PutHiveQL and SelectHiveQL Processor enhancements. Added support for multiple statements in a script. Options for delimiters, quotes, escaping, include header and alternate header.
Add support in SelectHiveQL to get script content from the Flow File to bring consistency with patterns used for PutHiveQL and support extra query management.
Changed behavior of using Flowfile to match ExecuteSQL. Handle query delimiter when embedded. Added test case for embedded delimiter
Removing dead code.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Comments to Clarify test case.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Final whitespace/formatting/typo changes
This closes#1316
- Adding additional parameters to be able to limit the size of the provenance response. Specifically, whether the events should be summarized and whether events should be returned incrementally before the query has completed.
- Ensuring the cluster node address is included in provenance events returned.
- Ensuring there is a cluster coordinator before attempting to get the cluster node address.
- Removing exponential back off between provenance requests.
- Ensuring the content viewer url is retrieve before initializing the provenance table.
This closes#1413.
- Using fetch and replace together can provide optimistic locking for
concurrency control.
- Added fetch to get cache entry with its meta data such as revision
number.
- Added replace to update cache only if it has not been updated.
- Added Map Cache protocol version 2 for those new operations.
- Existing operations such as get or put can work with protocol version
1.
This closes#1410.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-3004 Added logic to expire StandardSSLContextService customValidate cache after 5 invocations.
Updated unit test to demonstrate this logic.
This closes#1375.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Removing unnecessary authorization check during second phase of connection creation.
- Ensuring that the remote group port returns the correct resource type though not super critical since it is not possible to create policies for remote ports.
This closes#1353.
- Fixing the importUri when replicating a Template upload request. Previously was leveraging getResourceUri which considers proxy headers meant for client responses.
This closes#1334.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Removed host and port field from Peer since the same information is
available in PeerDescription
- Refactored variable names in SocketRemoteSiteListener to improve readability
- Changed how SocketRemoteSiteListener constructs PeerDescription
instance. It used to use hard-coded 'localhost' as hostname, and
getPort() which returns server's port. Since the peer is a remote peer,
i.e the client, it should be client hostname and port.
- Added hostname resolution at DataTransferResource to make s2s.host
value consistent with RAW transport. Without this, RAW uses hostname
while HTTP uses IP address. It will be hard to be used from downstream flows.
- Replaced heavy use of mockito which was difficult to maintain, with
nifi-mock
- Added SiteToSiteAttributes and more assertions in unit tests
This closes#1342.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Ensure that controller services are added to child process groups when creating snippet
- Addressed issue related to modifying higher-level process groups' controller services in snippet after having already visited the process group
This closes#1318
Signed-off-by: jpercivall <JPercivall@apache.org>
NIFI-1582 removing the option to use cluster state
NIFI-1582 addressing Oleg's comments
NIFI-1582 No longer forcing numbers as the init value and adding getStateValue() to EL instead of using 'ATTRIBUTE_state'
NIFI-1582 Removing init state value
NIFI-1582 Adding documentation for the changes to Init State value
This closes#319
NIFI-3133:
- Using getSourceAuthorizable() when accessing flow files and content.
NIFI-3133:
- Decouple local and remote connectable's to avoid ambiguity with self referencing RPGs.
NIFI-3133:
- Addressing comments from the PR.
NIFI-3133:
- Fixed check verifying source/destination when creating a connection.
NIFI-3133:
- Only showing the go to link when the source component is not a remote port.
NIFI-3133:
- Removing unnecessary checking of remote group port authorization since it's handled by the parent RPG.
NIFI-3133:
- Fixing issue showing the connection details dialog when the source component is a RPG.
NIFI-3133:
- Ensuring the local connectable was found.
This closes#1297
Signed-off-by: jpercivall <JPercivall@apache.org>
- Added urls in addition to the existing url, to support multiple target
URLs
- Backward compatibility is provided by returning the first url if
multipe urls are specified, but component accessing the url doesn't
support multiple urls
- UI is not fully updated yet. Following UI components are planned to be updated
by different commits
- Search component: only the first URL is searchable and shown
- Component status: RPG status shows only the first URL
- Component action history: only the first URL is searchable and shown
- Updated Search component to use URLs.
This closes#1208.
- Updating REST API docs accordingly.
- Adding some integration tests to exercise the additional restricted component checks.
This closes#1287.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Added unit test cases to simulate NiFi version update which fails without this fix.
- Added state object migration code, add file.0. prefix to state keys,
and add length from stored position.
This closes#1289
Fixed merging logic for root group status. Only consider a port transmitting when there is an active request, not when there is a 'queued' request
This closes#1279
- Updating the SnippetManager to automatically expire unused snippets.
- Making the checking of referenced services consistent across component removal.
- Adding checking of referenced services for all nested components in a snippet.
- Updating the REST API docs to clarify when a referenced service is authorized.
- Conditionally authorizing transitive service references.
This closes#1277.
Signed-off-by: Bryan Bende <bbende@apache.org>
- validate processors only when they are in STOPPED state
- report validation errors via REST API on processors/services/tasks/ports only when they are in the STOPPED state
- This closes#1192
- Addressing comments from PR.
- Adding restricted tags to relevant components.
- Showing a restricted icon overlay on the processor node on the canvas. (+1 squashed commit)
Squashed commits:
[f487682] NIFI-3050:
- Introducing a Restricted annotation for components that require elevated privileges to use.
- Updating the new Processor, Controller Service, and Reporting Task dialogs to include these details and prevent unauthorized selection.
- Including the Restricted description in the generated component documentation.
- Updating processor access control integration test to verify restricted component creation.
- Updating the developer, user, and admin guide to include the restricted component policy.
This closes#1247.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2750
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2750 Tweaking Property Retrieval
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1001
- Introducing a strategy for identifying users.
- Fixing issue with the referral strategy error message.
- Adding code to shutdown the application when the authorizer or login identity provider are not initialized successfully.
NIFI-3020:
- Updating the admin guide to document the identity strategy.
NIFI-3020:
- Ensuring the request replicator attempts to shutdown regardless of whether the flow service properly terminates.
This closes#1236
NIFI-2854: Incorporated PR review feedback
NIFI-2854: Implemented feedback from PR Review
NIFI-2854: Ensure that all resources are closed on CompressableRecordReader.close() even if an IOException is thrown when closing one of them
This closes#1202
- Returning the appropriate authorizable when accessing provenance events for a manual DROP event by emptying a queue.
- Populating the component details of a provenance event when the source is a connection.
NIFI-1002: Added WebSocket support.
- Reflecting review comments
- Added displayName to peroperty descriptors
NIFI-1002: Added WebSocket support.
This closes#1184
- Reflecting review comments:
- Removed unnecessary use of NarCloseable.withComponentNarLoader.
- Removed intermediate on memory queue to make it simpler and more
robust. Received messages in WebSocket layer now will be transferred
to downstream relationships directly.
Squashed commits:
[5dd22a9] NIFI-2654 Updated administration guide with login-identity-providers.xml flags.
Exposed master key retrieval code in NiFiPropertiesLoader.
Added logic to decrypt login identity providers XML configuration.
Updated login-identity-providers.xsd to include encryption scheme attribute.
Added unit tests. (+18 squashed commits)
Squashed commits:
[57c815f] NIFI-2654 Resolved issue where empty LIP property elements could not be encrypted.
Added unit test and resource.
[27d7309] NIFI-2654 Wired in serialization logic to write logic for LIP.
Added comprehensive unit test for LIP & NFP in same test.
[b450eb2] NIFI-2654 Finalized logic for preserving comments in LIP parsing.
[5aa6c9c] NIFI-2654 Added logic for maintaining XML formatting (comments and whitespace) for LIP.
Added unit tests (w/o encryption works; w/ does not).
[b53461f] NIFI-2654 Added unit test for full tool invocation migrating a login-identity-providers.xml file and updating file and bootstrap.conf with key.
[2d9686c] NIFI-2654 Updated tool description and various logging statements.
Added unit test for full tool invocation encrypting a login-identity-providers.xml file and updating file and bootstrap.conf with key.
[8c67cb2] NIFI-2654 Added logic to encrypt LIP XML content.
Added unit tests.
[8682d19] NIFI-2654 Added logic to handle "empty" (commented) LIP files.
Added unit tests.
[077230e] NIFI-2654 Fixed logic to decrypt multiline and multiple-per-line XML elements.
Added unit tests and resources.
[d5bb8da] NIFI-2654 Ignored unit test for unreadable conf directory because directory was causing Maven build issues.
Removed test resources.
[7e50506] NIFI-2654 Fixed AESSensitivePropertyProvider bug handling cipher text with whitespace.
Added unit test.
[b69a661] NIFI-2654 Fixed AESSensitivePropertyProviderFactoryTest to reflect absence of key causes errors.
[6f821b9] NIFI-2654 Added standard password to arbitrary encryption test for use in test resources.
[d289ffa] NIFI-2654 Added LIP XML decryption.
Added unit tests.
[a482245] NIFI-2654 Added LIP test resources.
[7204df4] NIFI-2654 Changed logic to only perform properties encryption when file path is provided.
[729e1df] NIFI-2654 Removed population of default file locations for bootstrap.conf, nifi.properties, and login-identity-providers.xml as not all files may be desired.
Added/updated unit tests.
[7dba5ef] NIFI-2654 Started LIP work (arguments & parsing).
Added unit tests.
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>
This closes#1216
Added unit tests and resources.
NIFI-3032 Fixed bug in AESSensitivePropertyProvider#getIdentifierKey where the result was always the max available key size, not the size of the current key.
Added unit test.
This closes#1220
Signed-off-by: Bryan Rosander <brosander@apache.org>
DefaultSettings annotation should be use on Custom Processor to set the default penalty period, the yield duration or the bulletin log level for each instance of the processor
- introduced org.apache.nifi.stream.io.util.TextLineDemarcator
- refactored SplitText to use org.apache.nifi.stream.io.util.TextLineDemarcator
- updated SplitText's capability discription to provide more clarity around splits with headers.
1 - Replace a r/w test over $NIFI_HOME/lib that is performed
but never utilised by a RO test
2 - Rename ensureDirectoryExistAndCanAccess to
ensureDirectoryExistAndCanReadAndWrite and deprecate the
former
- Removing legacy code when retrieving site to site details when the target instance is this local NiFi.
- Ensures the UI accurately reflects available remote ports.
NIFI-2603:
- Fixing color of remote output port status in connection label.
This closes#1189.
NIFI-1712 Applying per-instance class loading to HBaseClientService to allow specifying Phoenix Client JAR
-Refactoring the ClassLoading so that every processor, controller service, and reporting task gets an InstanceClassLoader with a parent of the NAR ClassLoader, and only components with @RequiresInstanceClassLoading will make a copy of the NAR ClassLoader resources, and addressing some review feedback
This closes#1156
- Updating the content of the delete policy dialog to make it a little more clear that the permissions for the component will revert back to the inherit policy.
This closes#1183.
- Ensuring the selection is cleared when going to the Process Group where the selected component policy is defined.
- Ensure that the selection context is updated when going to a component or group.
This closes#1174.
* Java version and vendor
* OS name and version
* Release Tag
* Build revision (commit SHA), branch, and timestamp
* Handles formal releases, ad-hoc builds, and non-release source builds
* Standalone UI presence in About dialog, Summary -> System Diagnostics
* Cluster UI as Versions tab in Cluster dialog
* Reduce About Dialog Content
* Fix Missing Property Display Bugs
* Marking the build time as type string.
* This closes#583
NIFI-2533: - Only including a user/group in the search results if they are not currently selected.
NIFI-2286: - Providing a tooltip for the add user and remove policy button.
This closes#1155.
Signed-off-by: Bryan Bende <bbende@apache.org>
This closes#1147.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Fixed typos in error messages, renamed variables in test, and cleaned up unnecessary imports. (+1 squashed commit)
Squashed commits:
[e755cbd] NIFI-2919 improved GetFile to fail if target directory is inaccessible
ExecuteProcess ignores error stream when Redirect Error Stream is
false, this commit let it to be logged instead so that user can see it on
bulletin.
This closes#1143.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Improved connection UI display when backpressure is enabled
- Updating the connection label to include backpressure indicators for object count and data size thresholds.
- Coloring the connection path and drop shadow once backpressure is engaged.
- Fixing bug with expiration icon tooltip.
- Including columns in the summary table for backpressure.
- Updating empty queue action to reload the connection status upon completion to ensure an updated count.
This closes#1080.
- Consolidated the target cluster URL resolving logic into
SiteToSiteRestApiClient's as a common method
- Changed to more descriptive error message
- Added more unit test cases
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Added NOTICE entries for test data and fixed checkstyle violations
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1123
- Showing process group name when possible.
- Providing a link to jump to the process group defined in the effective policy.
- Preventing editing an inherited policy.
- When overriding a policy, allowing the user to indicate if the policy should be empty or should copy the user/groups of the inherited policy.
This closes#1090.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Added following properties:
- nifi.web.http.port.forwarding
- nifi.web.https.port.forwarding
This closes#1100.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Ensure that we wait until a request is completed before unlocking the lock for request replication
- Ensure that failures do not trigger request completion logic unless the failure is the last node to report its status
- This closes#1109
NIFI-2856:
- Only performing response merging when the node is the cluster cooridinator even if there is a single response.
- Fixing PropertyDescriptor merging to ensure the 'choosen' descriptor is included in map of all responses.
This closes#1095.
Correction as suggested by users in JIRA.
+ adding a unit test to check attachments.
This closes: #1081
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
- getSystemLoadAverage of OperatingSystemMXBean sometimes returns a negative value.
- ProcessorLoadAverage sets -1.0. It added a defensive logic.
- When load average is zero(load average: 0.00, 0.02, 0.04), systemLoad is zero.
- This closes#1091
- Added ClusterWorkload message to retrieve workload information from a
cluster coordinator
- Use cluster workload to return queued flow file count to site-to-site
client so that it can calculate distribution of data transfer
This closes#1084.
Signed-off-by: Bryan Bende <bbende@apache.org>
Scott Aslan
Pierre Villard
Matt Gilman
Koji Kawamura
Ryan Persaud
Nick Carenza
Oleg Zhurakousky
Giovanni Lanzani
Joe Gresock
James Wing
Joe Skora
Matt Burgess
Andre F de Miranda
Ben Schofield
Jeff Storck
Andrew Psaltis
Andy LoPresto
Mike Moser
David W. Streever
Andrew Lim
afucs
Joey Frazee
Bryan Rosander
jpercivall
Davy De Waele
Bryan Bende
Randy Gelhausen
Wesley-Lawrence
Mark Payne
Byunghwa Yun
jpercivall
joewitt
Joey Frazee
Kathik Narayanan
Peter Wicks
Mathias Tiberghien
João Henrique Ferreira de Freitas
Brian Eugley
patricker
Joe N
simplesteph
Michal Klempa
d810146
Edgardo
Stephane Maarek
Keren Tseytlin
Peter Mitchell
mans2singh