- Processing properties and property descriptors in Controller Service referencing components unconditionally.
This closes#2602.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Ensuring appropriate response in checkAuthorization when user is null.
- Ensuring the user reference is passed down when applying variable changes.
This closes#2598.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Fixing required permission for PutParquet.
NIFI-5008:
- Ensuring all restricted components are tagged as such.
This closes#2583.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Fixing RPG port merging.
- Adding unit tests.
- Removing unecessary sorting that wasn't maintained while clustered.
This closes#2551.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Implemented review feedback. Refactored data model to make the API cleaner and delineate more along the lines of what permissions are required in order to see which details
Implementing review feedback
Removed sensitive information from the diagnostics reports
Fixed bug in merging logic for GCDiagnosticsSnapshots
This closes#2468
- Addressing memory leak from lingering authorization results that did not represent actual access attempts. This closes#2511.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Initial set of components marked with the HighResourceUsageScenario annotation.
- Added customized descriptions to SystemResourceConsideration annotations for MergeContent, SplitContent, SplitJson, SplitText, and SplitXml.
This closes#2475.
Signed-off-by: Mark Payne <markap14@hotmail.com>
...Search results
* Separated the search functionality.
* Added a unit test.
* Added the PG info to UI (a mere draft).
* Introduce the nearest versioned group
* Removed the top level group results in favour of the nearest versioned group.
* This closes#2364
NIFI-4772: Introduced a new thread-pool with 2 threads that will be used for monitoring lifecycle task. This means that if all threads in the timer-driven thead pool are blocked by processors that don't complete their @OnScheduled methods, we have a separate thread pool that at least gives us a chance of interrupting those threads
NIFI-4772: Remove unused import
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2403
- Everywhere that we ignore adding remote ports we should ignore removing remote ports as well in flow diffs
This closes#2462.
Signed-off-by: Bryan Bende <bbende@apache.org>
Adds stricter checks in AuthorizerFactoryBean for unique ids within
a given type of provider and requires unique providers in composite
and composite-configurable user group providers. Failed validation
checks cause startup to fail. Adds test cases for these new rules.
This closes#2419.
Signed-off-by: Bryan Bende <bbende@apache.org>
Added HostHeaderHandler regression unit tests before adding new functionality.
Added logic for parsing nifi.web.proxy.host property.
Added default property in nifi.properties and pom.xml.
Added logic for IPv6 and custom default hostnames.
Improved error messaging.
Added HostHeaderHandler unit tests.
Disabled HostHeaderSanitizationCustomizer in HTTP mode.
Fixed HTML escaping in error message.
Improved error message.
Added failing unit test for parsing custom hostnames.
Fixed custom hostname parsing.
Fixed unit tests.
Added TODO for IPv6 custom hostname parsing and unit test.
Added IPv6 custom hostname parsing and unit tests.
Fixed checkstyle issues.
Removed empty element in host list when no value defined.
Improved error message formatting.
Added unit tests.
Removed HostHeaderSanitizationCustomizer.
Removed InvalidPropertiesFormatException from NiFiProperties.
Removed InvalidPropertiesFormatException from HostHeaderHandler.
This closes#2415
- Only showing the import link in the new Process Group dialog when initiating from the toolbar.
This closes#2395.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Adds support for NiFI Registry config files to the encrypt-config tool
in NiFi Toolkit.
Also adds decryption capability to encrypt-config tool.
This closes#2376.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-4436: Fixed a bug that caused a flow not to successfully change version if a connection is added to an existing component and that component is running at time of version change
NIFI-4436: Fixed bug with ordering of controller services being enabled and disabled
NIFI-4436: Fixed bug that prevented local input and output ports from being stopped and started as needed
NIFI-4436: Fixed bugs around referencing controller services that are at a higher level than the versioned flow
NIFI-4436: Ensure that we clear components from FlowController's cache when removed and that they are added to cache when created.
NIFI-4436: Fixed error message coming back if component is invalid when trying to be restarted/re-enabled
NIFI-4436: Addressed issue with children of a removed process group not being considered 'affected components' and as a result not being stopped/disabled/restarted/re-enabled
This closes#2219.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
- Minor tweak to when the max height for the options list is calculated per PR comments.
- Resolving logic issue in two phase commit when updating variable registry.
- Fixing variable visibility.
- Updating buckets permissions based on new model.
- Adding check to ensure that flow name is non null before checking the length.
- Adding versioned flow state to the Process Group tab in the Summary table.
- Fixing issue with navigating to Controller Services from the local changes dialog.
- Code clean up.
- Improved error handling.
- Minor UX improvements.
- Always showing Process Group state to complement the aggregation counts.
- Adding the Process Group state to the top status bar.
- Code clean up.
- Improved error handling.
- Minor UX improvements.
- Adding message to indicate that variables do not support sensitive values.
- Preventing a user from changing the flow version to the current version.
- Only presenting buckets a user has appropriate permissions to.
- Adding basic auditing to the version control actions.
- Updating front end to use version control state/status.
- Fixing copy/paste issue during revert local changes.
- Code clean up in the breadcrumbs.
- Update VersionsResource authorization and two phase commit object usage.
- Addressing miscellaneous minor UX issues.
- Updating comments UX for all components.
- Updating the styling of PG and RPG to be more consistent.
- Adding the icons for nested versioned process groups.
- Calculating the number/states of nested versioned process groups.
- Fixing default border radius.
- Code clean up.
- Ensuring component visibility is updated after updating/reverting.
- Fixing sort on component name in local changes dialog
NIFI-4526:
- Added front end controls for updating RPG target URL.
- Clearing bucket/flow/versions when changing the selected registry/bucket.
- Using the versioned flow to get the group name when importing.
- Adding menu items for viewing local changes.
- Showing local changes during revert request.
- Adding the version number to the start version control, commit, and change version dialog.
- Showing a loading item in the combo's while querying for the registries, buckets, and flows.
- Adding tooltips to display version control information on the canvas.
- Adding progress bar dialogs for changing version and reverting local changes.
- Updating canvas and breadcrumb according to the version control state.
- Updating to use registry name, bucket name, and flow name where appropriate.
- Added the import dialog for importing a versioned flow into a new process group.
- Added the change version dialog for upgrading/downgrading a versioned flow.
- Adding support to save a version of a flow based on a selected Process Group.
- Adding support for revert changes back to the most recent version.
- Adding support to disconnect from version control.
- Moving the version control information out of the entity objects and into the dto's.
- Fixing checkstyle issues.
NIFI-4502:
- Updating the UI to allow for the user to register registry clients.
- Updating the version control menu item names.
- Initial checkpoint: able ot start version control and detect changes, in standalone mode, still 'crude' implementation
- Checkpoint: Can place flow under version control and can determine if modified
- Checkpoint: Change version working in some cases. Does not work if processor removed because COMPONENT_REMOVED type has ComponentA whose ID is the VersionedComponentID but we are trying to call ProcessorDAO.get() with this ID
- Checkpoint: Able to change flow from Version 1 to Version 2 and back. Not yet tested with controller services. Have not tried changing/removing connections. Not cluster-friendly yet. All inline, not in background. Have not taken into account ports, funnels, remote ports, etc. Have not tested with Labels yet
- Checkpoint after implementing ClusterReplicationComponentLifecycle instead of JerseyClientComponentLifecycle
- Checkpoint: Updated to allow starting version control and updating version in clustered mode
- Checkpoint: Updated versioning endpoint so that when version of a flow is updated, the bundle information is populated and the snapshot is replicated to the cluster.
- Checkpoint: Implemented endpoint for reverting to previously sync'ed version of a flow and updated version control endpoint so that Process Group can be pushed as a new version to existing flow instead of only creating a new flow
- Checkpoint: Updated so that if a Process Group is under Version Control and it has a child Process Group, which is also under Version Control, we can handle that gracefully. Not yet tested because it depends on updates to the nifi-registry module, which can't be compiled due to maven dependency conflicts
Removes user existence check from FileUserGroupProvider when
group is created or updated. Replaces it with check in the
Authorizer Decorator class created by Authorizer Factory, so
that all providers are used.
Also fixes bug when searching for group membership by user
that returns results across all providers.
NIFI-3472 NIFI-4350 Removed explicit relogin code from HDFS/Hive/HBase components and updated SecurityUtils.loginKerberos to use UGI.loginUserFromKeytab. This brings those components in line with daemon-process-style usage, made possible by NiFi's InstanceClassloader isolation. Relogin (on ticket expiry/connection failure) can now be properly handled by hadoop-client code implicitly.
NIFI-3472 Added default value (true) for javax.security.auth.useSubjectCredsOnly to bootstrap.conf
NIFI-3472 Added javadoc explaining the removal of explicit relogin threads and usage of UGI.loginUserFromKeytab
Readded Relogin Period property to AbstractHadoopProcessor, and updated its documentation to indicate that it is now a deprecated property
Additional cleanup of code that referenced relogin periods
Marked KerberosTicketRenewer is deprecated
NIFI-3472 Cleaned up imports in TestPutHiveStreaming
- Ensure the /nifi-api/controller redirection filter executes before matching.
This closes # 2358.
Signed-off-by: Andy LoPresto <alopresto.apache@gmail.com>
Adds authorizers.xml to the files understood by the encrypt-config
tool in the NiFi Toolkit. If enabled, then the sensitive properties
for LdapUserGroupProvider in authorizers.xml will be encrypted.
Also fixes a bug wherein encrypt-config replaces multiple XML nodes
in login-indentity-providers.xml when LdapProvider is not the first
provider listed in the file.
Enable properties in authorizers.xml to be encrypted by the master key.
This closes#2350.
Signed-off-by: Andy LoPresto <alopresto.apache@gmail.com>
Corrects time unit conversion for the Sync Interval config property
for LdapUserGroupProvider in authorizers.xml.
Also enforces a minimum value of 10 secs for the Sync Interval to help
catch unintentional misconfigurations, for example users upgrading
from previous versions, where tiny Sync Interval values could be set
as a workaround for NIFI-4667.
This closes#2341
Ensure that when ProcessSession.clone(FlowFile) is called, we obtain the most recent version of the FlowFile before attempting to obtain FlowFile size.
Signed-off-by: joewitt <joewitt@apache.org>
- Adding new properties to allow the referenced attribute of a user/group to be configurable when detecting group membership.
- Expanding on documentation regarding the new properties.
This closes#2274.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Ensuring a charset is specified when convert a string into bytes.
- Ensuring the aggregate snapshot is non null before attempting to promote the runStatus.
This closes#2278.
- Ensuring View Details button is visible if the user has read access and the service is defined in the current Process Group.
This closes#2262
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This commit includes changes to DeleteHDFS to report REMOTE_INVOCATION
event. In order to do so, the processor had to be changed to create
output FlowFile because a provenance event needs a FlowFile it
associates with.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2234.
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.
This closes#2206.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Upgraded immediately actionable dependency versions from Meterian report.
- Upgraded jackson-core test dependencies for HBase and Elasticsearch modules.
- Only 3 instances of jackson-core < 2.8.6 (Google Cloud Platform and Spark Receiver modules).
- Upgraded version of poi dependency in nifi-email-processors to 3.16.
- Resolving dependency issues after rebasing against 1.5.0-SNAPSHOT.
- Removed jackson-databind from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.version} in all referenced artifacts.
- Removed jackson-mapper-asl from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.old.version} in all referenced artifacts.
- Removed Jasypt from <dependencyManagement> and added explicit version in test dependency for legacy compatibility.
- This closes#2084
...during selection in Add Processor dialog
* Added navigation logic to both Add Processor and Add CS dialogs.
* No extending to the SlickGrid library done.
* Ported the new functionality to the Add Reporting Task dialog.
* Mimicked SlickGrid's PageUp/Down behavior.
* This closes#2170
Refactored the createDocsWebApp method of the JettyServer.java class. Previously NiFi would fail to
start up and instead throw an IllegalStatException if the 'docs' directory did not exist in the
installation directory. With the update, if the 'docs' directory is missing, an attempt to create
the missing directory will be made and if successful will enable NIFI to startup successfully, barring
any other startup errors of course. The side effect of this change is that the help documentation
under the 'General' heading of the help page will be missing.
Three small helper methods were extracted from the original method. Each related to a section of the
original code that could throw an exception. In each case if an exception is now thrown a more helpful
log message will be output and the process will be closed via the startUpfailure method rather than
throwing an exception.
This closes#2164.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Adding support for KnoxSSO.
- Updated the docs for nifi.security.user.knox.audiences.
- The KnoxSSO cookie is removed prior to request replication.
This closes#2177
Added external compatibility regression test for StringEncryptor to ensure continued functionality during removal of Jasypt.
Documents custom salt lengths and iteration counts for each encryption method.
Added (ignored) failing tests for keyed encryption (Jasypt does not support keyed encryption).
Changed StringEncryptor to non-final class and added protected default constructor.
Added failing test for initialization status.
Added utility methods in CipherUtility.
Moved PBE cipher providers (and tests) from nifi-standard-processors to nifi-security-utils module.
Implemented PBE and keyed encryption/decryption logic.
Moved Scrypt unit test back into scrypt package.
Resolved test failures in limited strength cryptographic environment.
Implemented keyed encryption/decryption and enabled unit tests.
Removed Jasypt dependency from production scope (kept in test scope for backward compatibility tests).
Signed-off-by: joewitt <joewitt@apache.org>
- Added XmlUtils class.
- Added unit test.
- Added XXE test resource.
- Refactored JAXB unmarshalling globally to prevent XXE attacks.
- Refactored duplicated/legacy code.
- Cleaned up commented code.
- Switched from FileInputStream back to StreamSource in AuthorizerFactoryBean.
- This closes#2134
...connection dialog if no relationships selected
Disabled confirmation button for both Create and Configure Connection
dialogs when no relationship was selected.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#2152
* Changed the tab title since sharing the name makes things
less clear for newcomers.
* Suggested info sentence is omitted.
This closes#2124.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Ensuring that sub context menus are removed when hiding to ensure they are correctly (re)created during mouseenter events.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#2109
...FlowFile has no content, UI does not indicate this fact
For FlowFiles with no content:
* Added a simple title replacing the Content Claim's one.
* Defaulted File Size to 0 bytes using common formatting.
..'View Details' button feels confusing and inconsistent with Processors
* Merged 'View Details' and 'Edit' buttons to a single 'Configure'/'View Details' one.
* Fixed confusing 'View Details' modal's title.
- When determining which controller services to return for a component, ensure that we don't show services that belong to 'child groups'
- Fixed a logic bug that determined which process group to use for obtaining controller services
- This closes#2087
- Initial implementation of Process Group level Variable Registry
- Updated to incorporate PR Feedback
- Changed log message because slf4j-simple apparently has a memory leak; passing a String instead of passing in the Controller Service object as an argument addresses this.
- This closes#2051
...name when I create a new Controller Service within a Processor/Service configuration dialog
Changed CS name default's behavior according to PR feedback.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#2081
...name when I create a new Controller Service within a Processor/Service configuration dialog
I'm not sure how to integrate/style the text box. Suggestions are welcome.
NIFI-4237 Cleaned up unused alternate approaches.
NIFI-4237 Added failing unit test for better error message.
NIFI-4237 Added logic to capture unhelpful encryption exception and provide context in message. All tests pass.
This closes#2077
- Introducing support for OpenId Connect.
- Updating REST API and UI to support the authorization code flow.
- Adding/fixing documentation.
- Implementing time constant equality checks where appropriate.
- Corrected error handling during startup and throughout the OIDC login sequence.
- Redacting the token values from the user log.
- Defaulting to RS256 when not preferred algorithm is specified.
- Marking the OIDC endpoints as non-guaranteed in to allow for minor adjustments if/when additional SSO techniques are introduced.
This closes#2047.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
The JIRA issue asks for treating both node and its label as one unit.
Described cursor issues seems to be fixed already.
However, there is an annoying dead space between a node and label
preventing displaying context menu, etc. Due to SVG group's nature
there has been added an opaque joint to remove the dead space.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2059.
* Added double-click shortcut opening config/details dialog to processors,
connections, ports and labels.
* Created a base for further default action selection, disabling, etc.
* Omitted default action configuration UI - that might be a separate JIRA ticket.
NIFI-4032: - Generating the appropriate fingerprint for the ManagedRangerAuthorizer based on whether the UserGroupProvider is configurable. - Adding unit tests.
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>
This closes#2019
- Expose processors' counters in Stats History
- Only include counters in Processors' Status History if user has read access to corresponding Processor
- Addressed review feedback. Found and addressed bug where a counter is not present in all of the aggregate snaphot values for status history, resulting in the UI not rendering the chart properly
- This closes#1872
NIFI-4060: Addressed threading issue with RecordBin being updated after it is completed; fixed issue that caused mime.type attribute not to be written properly if all incoming flowfiles already have a different value for that attribute
NIFI-4060: Bug fixes; improved documentation; added a lot of debug information; updated StandardProcessSession to produce more accurate logs in case of a session being committed/rolled back with open input/output streams
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1958
- Introducing composite ConfigurableUserGroupProvider and UserGroupProvider.
- Adding appropriate unit tests.
- Updating object model to support per resource (user/group/policy) configuration.
- Updating UI to support per resource (user/group/policy) configuration.
- Adding necessary documentation.
- Updating documentation to clarify integrity checks.
- Providing an example of configuring a composite implementation.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1978.
Fixed threads shutdown so that NiFi can shutdown gracefully
NIFI-4111 - Review - Handling SocketRemoteSiteListener (RAW S2S)
This closes#1963.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Addressing issues causing the eventId to not be relayed when submitting a lineage request under certain conditions.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1903.
When NiFi is clustered, and autopurge.purgeInterval is greater than 1, the DatadirCleanupManager will be started in order to automatically purge transaction log and snapshot files based on the autopurge settings in zookeeper.properties
This closes#1928.
NIFI-4061 Initial version of RedisStateProvider
- Adding PropertyContext and updating existing contexts to extend it
- Added embedded Redis for unit testing
- Added wrapped StateProvider with NAR ClassLoader in StandardStateManagerProvider
- Updating state-management.xml with config for RedisStateProvider
- Renaming tests that use RedisServer to be IT tests so they don't run all the time
This closes#1918.
- Introducing the LdapUserGroupProvider.
- Updating documentation accordingly.
- Moving the IdentityMapping utilities so they were accessible.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1923.
- Introducing FileUserGroupProvider and FileAccessPolicyProvider.
- Refactoring FileAuthorizer to utilize the file based implementations.
- Introducing the StandardManagedAuthorizer.
- Decorating the configured ManagedAuthorizer to ensure integrity checks are still performed.
- Loading user groups if possible to use during access decisions.
- Merging responses for requests for AccessPolicies, Users, and UserGroups.
- Adding unit tests as appropriate.
- Adding methods to the User, Group, and AccessPolicy builder that more easily supports generating UUIDs.
- Fixing typo when seeding policies during startup.
- Fixing type in documentation and error messages.
This closes#1897.
Signed-off-by: Bryan Bende <bbende@apache.org>
Matt Gilman
Scott Aslan
Mark Payne
joewitt
zenfenan
Derek Straka
Mike Moser
Bryan Bende
Jorge Machado
Kevin Doran
Takanobu Asanuma
Jeff Storck
Matthew Burgess
yuri1969
Peter G. Horvath
sbouchex
Andy LoPresto
Senthilannaswamy
Marco Gaido
Kevin Doran
Aldrin Piri
Koji Kawamura
Joe Percivall
Joey Frazee
Pierre Villard
Mark Owens
m-hogue
combine
Yolanda M. Davis
Tony Kurc
Niels Zeilemaker
Wesley-Lawrence
Andre F de Miranda