HTTP Site-to-Site can't handle TRANSACTION_FINISHED_BUT_DESTINATION_FULL
scenario as expected.
That happens if the remote NiFi's input port destination relationship
becomes full during Site-to-Site client sends data. The data which has
already sent to the remote NiFi has to be committed successfully.
However, the remote NiFi returns 503 as a response of commit HTTP
request. Because it does check port availability.
The port availability check shouldn't be called at commit request, since
the session at source NiFi has already been committed. The remote NiFi
should commit its session as well, and return
TRANSACTION_FINISHED_BUT_DESTINATION_FULL response.
This fix makes a remote NiFi to keep the handshaken properties when it holds
transaction to be committed. Then if a transaction already has
handshaken properties, then use it, instead of doing a handshake process
again.
Fixed Site-to-Site Transit URI for HTTP to be consistent with RAW socket.
- Removed url from CommunicationsSession since it's redundant as we have
Peer.url, too. The value was not used from anywhere other than HTTP
Site-to-Site.
- Added createTransitUri method in Communicant interface, so that
implementation can customize transitUri while providing consistent
interface.
Removed permission check causing "Cannot read property 'canRead' of
undefined". A given user won't have record level permission
difference for component state. It's not required here.
- Returning the enabled state of play and stop buttons in the palette. Allowing the user to always press if they have permissions to modify the selection.
- Only attempting to schedule components that are not running, not disabled, and valid.
This closes#745
Signed-off-by: jpercivall <joepercivall@yahoo.com>
- added SnippetUtilsTest
- renamed TypeOneUUIDGenerator to ComponentIdGenerator
- changed lsb part of ComponentIdGenerator back to long
- Fixed 'isCopy' condition for clustered environments
This closes#718.
[NIFI-2025] update birdseye after dragging/dropping element on the canvas
[NIFI-2367] update width of content viewer combo
[NIFI-2355] update table sorting based on auth efforts
[NIFI-2027] update EL editors
[NIFI-2387] update bulletin alert backgroud color to actually change the color of the icon not the div background
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2400] close any open combos contained within a shell when closing the shell
[NIFI-2404] remove extra scrollbar from #node-events
[NIFI-2027] account for min widths of EL editors
[NIFI-2025] update birdseye after changing color
[NIFI-2027] Update EL editor checkbox text
[NIFI-2027] update checkbox text
[NIFI-2355] update table sorting
[NIFI-2141] Hide bulletin icon/background on processors unless a bulletin exists
[NIFI-2027] update EL editors
[NIFI-2367] Overlapping links repositioned on error/login pages
This closes#715
- Introducing data resource for authorizing provenance events and queue listing.
- Authorizing entire proxy chain for data resource and data transfer resource.
NIFI-2338:
- Ensuring that replay authorization only happens once.
- Allowing users with access to policies for a component to be able to access all policies for that component.
-- Includes the component, data, data transfers, and policies.
- Fixing drop request completion to update the correct queued field.
- Fixing access control check for listing and emptying queues.
- Reseting selected policy when re-opening the policy management page.
- Fixing button/link visibility for available actions in policy management page.
- Fixing policy issues with policy removal when the underlying component is deleted.
- Updating file authorizer seeding to grant data access to node's in the cluster.
This closes#720.
- Allowing users with read only access to the tenants page.
- Fixing current user integration test.
- Ensuring schedule methods are locked appropriately.
- Addressing comments from PR.
This closes#698
Signed-off-by: jpercivall <joepercivall@yahoo.com>
- Ensuring the appropriate visibilty of the action in the policy management page.
NIFI-2273:
- Ensuring we load the policy or inform the user of the appropriate permissions of the effective policy.
NIFI-2239:
- Providing help tooltips for the policies in the management page.
NIFI-2283:
- Adding auditing for access policies, users, and groups.
NIFI-2263:
- Not replicating history requests throughout the cluster.
NIFI-2096:
- Fixing upload template file input in Firefox.
NIFI-2301:
- Removing relevant policies after component deletion.
- Updated site-to-site client and server to support clustered nifi instances
NIFI-2274:
- Ensuring we use the correct URI when updating a connection.
This closes#530
- Added NodeTypeProvider to expose flowController's isClustered and
isPrimaryNode so that processor can know if it's running on a cluster
and if it's a primary node.
- Added mechanism to simulate clustered or not, and primary or not, for testing
This closes#657
Signed-off-by: jpercivall <joepercivall@yahoo.com>
- Addressed issue enabling/disabling controller services where the wrong URI was referenced.
- Addressed with the update revisions in the controller service references.
- Addressed issue with showing the disconnected from cluster dialog on page load.
- Addressed issue with URI when adding a dynamic property.
This closes#654.
- Added following properties:
- nifi.flow.configuration.archive.enabled
- nifi.flow.configuration.archive.max.time
- nifi.flow.configuration.archive.max.storage
- Removed manual archive operation:
- Removed 'Back-up flow' link from UI since it's not needed any longer
- Removed corresponding REST API controller/archive and its
implementations
- Added FlowConfigurationArchiveManager to enclose archive related code
- Updated related docs
- Move bulletins out of the controller status endpoint.
NIFI-2238:
- Ensuring the controller bulletins are rendered on screen.
NIFI-2246:
- Ensuring the correct number of bulletins are returned when clustered.
- Switched from absolute layout to responsible using flex-box.
- Added toggle switch to control component listing pane visibility.
Signed-off-by: joewitt <joewitt@apache.org>
- Ensuring the active thread count is shown.
NIFI-2019:
- Ensuring correct color of the run status in the From connection label.
NIFI-2183:
- Removing the DownloadSvg servlet and hidding the download icon until we're able to support save the svg entirely from the client side.
This closes#634.
[NIFI-2217] fix single node and cluster link
[NIFI-2219] Fix styles on provenance cluster search combo
[NIFI-2180] Fix settings shell table text alignment for run status
[NIFI-2140] Update preview image for Change Color dialog
[NIFI-2131] update progress bars/percent complete to use angular material progress linear directive
[NIFI-2099] add header text to all ok and yes/no dialogs
[NIFI-2233] fix invalid/warning icon shifts position as tasks are added
[NIFI-2131] update progress bars/percent complete
[NIFI-2140] Update preview image for label. This closes#627
- Adding a page for managing users and groups.
- Adding a page for managing access policies.
- Renaming accessPolicy in entity to permissions to avoid confusion with the accessPolicy model.
- Adding an Authorizable for access policies.
- Refactoring access policies endpoints.
NIFI-2022:
- Implementing site to site authorizations.
NIFI-2201 Add support for seeding cluster nodes in authorizations.xml
- Passing client address along in user context on authorization requests
- This closes#628
Added unit tests for DN extraction.
Corrected typo in Javadoc.
Switched server/client socket logic for certificate extraction -- when the local socket is in client/server mode, the peer is necessarily the inverse.
Fixed unit tests.
Moved lazy-loading authentication access out of isDebugEnabled() control branch.
This closes#622
[NIFI-2190] About Dialog fixed text overlayed on image
[NIFI-2187] Update iconResize.png
[NIFI-2166] When adding Processor to canvas, previously selected tags is no longer selected
This closes#615
[NIFI-2031] update global menu styles
[NIFI-2037] Increase header icons font sizes
[NIFI-2036] update logo
[NIFI-2144] consistent view details icons
This closes#604
- Making FileAuthorizer not update the resource or action when updating an AccessPolicy
- Adding corresponding READ policies during initial seeding and legacy conversions
- Adding checks to FileAuthorizer to ensure only one policy per resource-action
- Removing merging of policies on legacy conversion since we have one action per policy now
- This closes#608
- Updating UI according to permissions through out the application.
- Shuffling provenance events, template, and cluster search REST APIs according to resources being authorized.
- Moving template upload controls.
- Removing username where appropriate.
- Addressing issues when authorizing flow configuration actions.
- Code clean up.
Cleanse more values from templates that are not necessary. Additionally, updated javadocs in ProcessorConfigDTO to provide further explanation of the getAutoTerminatedRelationships() method, since this was confusing
Removed additional unused fields from templates
Populating snippet response using actual components rather than the snippet contents.
This closes#593
Adding user and group summary objects (TenantEntity)
Fixed ComponentEntity JSON mapping issues when the id field is null
Removing unecessary revision checking.
Fixing error message when checking user, group, and policy revision.
This closes#589
Added spec for StandardPolicyBasedAuthorizerDAO
Added exception mapper for AuthorizationAccessException, added mapper to nifi-web-api-context.xml
Added rest endpoints to get all users and user groups
Merged UsersResource and UserGroupsResource into TenantsResource
This closes#582
- Addressing access controls for the Controller resource.
- Addressing access controls for RAW site to site clients.
- Addressing access controls for downloading content (from provenance and queue).
- Addressing access controls for accessing queues.
- Addressing access controls for cluster endpoints.
- Addressing access controls for counter endpoints.
- Removing redundant authorization calls.
NIFI-2044:
- Requiring revision when creating components.
- Requiring component creation over POST requests.
NIFI-1901
- Continuing to restore access control tests.
- Converting access control tests to itegration tests.
- Restoring contrib check to travis build.
- This closes#567
by using script that delegates to installed nifi.sh at NIFI_HOME. Prohibit installation of service for Cygwin and Darwin environments that are guaranteed to not support it.
This closes#561
Signed-off-by: James Wing <jvwing@gmail.com>
created REST Resources for users, groups, and access policies
added Authorizables for users, groups, and access policies
added methods to DtoFactory and EntityFactory to create objects for users, groups, and access policies
extracted anonymous AuthorizableLookup impl in StandardNiFiServiceFacade.java to a protected class to make the lookup call mockable in tests
added methods to manage users/groups/access policies to StandardNiFiServiceFacade
added StandardNiFiServiceFacadeSpec to unit-test management of users/groups/access policies
added implementations for UserDAO, GroupDAO, AccessPolicyDAO.
added spring config for user/group/policy resources and daos
Updated user/group/policy creation via REST resources, no longer requires the use of the revision manager
updated StandardNiFiServiceFacadeSpec based on user/group/policy creation changes
condensed user/group/policy DAOs to a single DAO (StandardPolicyBasedAuthorizerDAO)
fixed spring config of user/group/policy REST resources
Updated to return ComponentEntity objects instead of just their IDs
mid-progress on updating tests
updated code and tests to return component entities from REST endpoints for users, groups, policies
This closes#526
- Fixing issue rendering event rows while visible in the shell.
- Fixing issue go to/from the event table and lineage graph.
- Fixing visibility of the event table header, search, and filter controls.
- This closes#559
- added a logger entry for LogAttribute to specify INFO as log level
- replaced some tab characters to whitespace (existing checkstyle error)
This closes#414.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Expose BackPressureObjectThreshold and BackPressureDataSizeThreshold to
ConnectionStatus
This closes#377.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- Removing Resources class from file authorizer and updating ResourceType enum
- Updating ResourceFactory to be in sync with ResourceType enum and adding additional required permissions to the auto-conversion
- Adding root process group to the seeding of the initial admin
- Improvement so that users that are already part of a read-write policy, won't end up in a read policy for the same resource
- Removing rootGroupId from authorization context and auto-detecting it from the flow provided through nifi.properties
- This closes#507
- Restoring bulletin functionality.
- Ensuring appropriate merging of bulletins in clustered responses.
- Updated the phrasing for cluster-related bulletins
This closes#531.
- Listening for window resize events more selectively.
- Fixing malformed request when configuring remote process group ports.
- Fixing malformed request when starting/stopping a selected process group.
- Fixing default value for authorizers.xml.
- This closes#524
- Enable HTTP(S) for Site-to-Site communication
- Support HTTP Proxy in the middle of local and remote NiFi
- Support BASIC and DIGEST auth with Proxy Server
- Provide 2-phase style commit same as existing socket version
- [WIP] Test with the latest cluster env (without NCM) hasn't tested yet
- Fixed Buffer handling issues at asyc http client POST
- Fixed JS error when applying Remote Process Group Port setting from UI
- Use compression setting from UI
- Removed already finished TODO comments
- Added additional buffer draining code after receiving EOF
- Added inspection and assert code to make sure Site-to-Site client has
written data fully to output
stream
- Changed default nifi.remote.input.secure from true to false
This closes#497.
- Implementing CRUD operations and unit tests for Users
- Implementing CRUD operations and unit tests for Groups
- Implementing CRUD operations and unit tests for AccessPolicies
- Adding support for seeding with an initial admin user
- Fixing delete for user and group so it removes references from policies
- Adding example to authorizations.xml
- Adding back the old users schema in preparation for auto-converting to the new format, and providing the AuthorizationConfigurationContext with access to the root process group id
- Refactoring some of the FileAuthorizer to ensure thread safety
- Adding /groups to policies created for initial admin
- This closes#473
- Removing deprecated NiFiWebContext and related classes.
- Adding authorization to Custom UIs.
- Fixing issue when creating ControllerService inline.
- Addressing contentType issue when attempting to clear component state.
- This closes#489
- Refactoring to allow requests to be replicated from a node to other nodes
- Renaming cluster node connection/read timeout properties.
- Renaming NCM DN to Cluster Coordinator DN.
- Fixing default values in properties.
- Starting to fix Spring context to load correctly in standalone mode.
- Using the cluster protocol to handle connection failures instead of heartbeats.
- Ensured replicate call is returned from ControllerResource.
- Ensure the appropriate classloader when serializing templates.
- Handling when the flow contents are null.
- This closes#488
added scaling of templates upon instantiation for placement on the canvas
added template-0.7.0.xml for live-testing the import of templates
Fixing issue with potentially uninitialized RemoteGroupPorts in copySnippet.
This closes#471
- UI style updates to make the components stand out better.
- Reusing controller service table in different contexts (controller, process group, etc).
- This closes#469
added utility class to scale positions of components on the canvas,
extracted get/setPosition methods from ProcesGroup, RemoteProcessGroup, Label, and Connectable into new interface Positionable
added interface method for finding all Positionables in a ProcessGroup to the ProcessGroup interface and added implementation to StandardProcessGroup
added test flow for position rescaling
added Spock config to POM and a spec for testing the scaling of Positionables
forced Surefire to use JUnit (TestNG was on classpath and Surefire seems to prioritize that over JUnit),
added check in StandardFlowSynchronizer to scale positions only when flow encoding version is less than 1.0
added spec for StandardFlowfileSynchronizer
updated FlowConfiguration.xsd to allow encoding-version attribute
added new test flow used in StandardFlowSynchronizerSpec
This closes#442.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Including access policies in the breadcrumb's trail.
- Updating toolbox according to group access policies.
- Updating actions in palette based on selection access policies.
NIFI-1554:
- Introducing authorization during two phase commit.
- Introducing snippet authorization according to the encapsulated components and the action performed.
- This closes#461
- Introducing ControllerService and ReportingTask authorization and revision.
- Removing the revision where appropriate.
NIFI-1883
- Restoring referencing components automatically.
- This closes#448
- Addressing issues when creating a new inline controller service.
- Ensuring controller service referencing components are updated.
- Including revisions and status with each component.
- Dynamically updating component and authorization states.
- This closes#435
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
- Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information
- Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
NIFI-1727:
- Refactored logic for merging HTTP Requests that are federated across cluster
NIFI-1745:
- Refactoring how HTTP Requests are replicated to nodes
- Bug fixes and continuing to work on replication refactoring. Still need to handle cluster locking and revisions
- Begin work on RevisionManager
- Resolved some issues that resulted from rebase
- Fixed URIs to align with new URI's that will be used in 1.0.0
- This closes#413
- Populating component entities in the REST API to decouple key fields from the configuration DTOs.
- Added initial support for components in UI when access isn't allowed. Formal styling to come later.
Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java
Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information
Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.
Code cleanup and incorporate comments from peer review
This closes#323
- Introducing new REST endpoints to align with the authorizable resources.
- Additionally changes to support the new endpoints.
- Addressing comments in PR.
- This closes#374.
- Removing the AuthorityProvider.
- Refactoring REST API in preparation for introduction of the Authorizer.
- Updating UI accordingly.
- Removing unneeded properties from nifi.properties.
- Addressing comments from PR.
- This closes#359.
Fixed nifi-utils pom.xml comment about additional dependencies. (+5 squashed commits)
Squashed commits:
[965b766] NIFI-1753 Removed temporary work-around of duplicate certificate conversion util method and added nifi-security-utils as dependency of nifi-utils.
[cd35f9b] NIFI-1753 Replaced legacy X.509 certificate declarations with new declarations in SSLSocketChannel and EndpointConnectionPool.
Temporary work-around of duplicate certificate conversion util method because nifi-utils cannot depend on nifi-security-utils.
[6420897] NIFI-1753 Replaced legacy X.509 certificate declarations with new declarations in PostHTTP.
[b9868ef] NIFI-1753 Added convenience method for extracting DN from peer certificate chain in SSL socket (canonical implementation to reduce code duplication and references to legacy certificate implementations).
Refactored logic retrieving legacy X.509 certificates with reference to convenience method in NodeProtocolSenderImpl.
Replaced logic retrieving legacy X.509 certificates with reference to convenience method in SocketProtocolListener.
Cleaned up exception handling in SocketProtocolListener.
Replaced legacy X.509 certificate declarations with new declarations in HandleHttpRequest (needs manual test).
[e2d1c35] NIFI-1753 Added convenience methods for converting legacy X.509 certificates and abstract certificates to correct X.509 format.
Added unit tests for certificate manipulation.
Replaced logic retrieving legacy X.509 certificates with new logic in NodeProtocolSenderImpl.
Added bcpkix (Bouncy Castle PKI implementation) dependency to nifi-standard-processors pom.
This closes#346.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-1274 Cleaned up TODO statements. (+3 squashed commits)
Squashed commits:
[fd101cd] Removed logic to check for presence of services to determine if token support is enabled when username/password authentication is enabled (Kerberos also requires tokens).
[c2ce29f] Reverted import changes to RulesResource.java.
[c269d72] Added Kerberos authentication mechanism.
Moved Kerberos service wiring from XML to Java to handle scenario where admin has not configured Kerberos (previously threw NullPointerException in FileSystemResource constructor). (+15 squashed commits)
Squashed commits:
[09fc694] Added Kerberos documentation to Admin Guide.
[ecfb864] Cleaned up unused logic.
[157efb3] Added logic to determine if client certificates are required for REST API (login, anonymous, and Kerberos service authentication all disabled).
Cleaned up KerberosService by moving logic to NiFiProperties.
[5438619] Added documentation for Kerberos login-identity-providers.xml.
[3332d9f] Added NiFi properties for Kerberos SSO.
[b14a557] Fixed canvas call to only attempt Kerberos login if JWT not present in local storage.
Added logic to handle ticket validation failure in AccessResource.
Changed wiring of Kerberos service beans to XML in nifi-web-security-context.xml for consistency.
[c31ae3d] Kerberos SPNEGO works without additional filter (new entry endpoint accepts Kerberos ticket in Authorization header and returns JWT so the rest of the application functions the same as LDAP).
[98460e7] Added check to only instantiate beans when Kerberos enabled to allow access control integration tests to pass.
[6ed0724] Renamed Kerberos discovery method to be explicit about service vs. credential login.
[ed67d2e] Removed temporary solution for Rules Resource access via Kerberos ticket.
[c8b2b01] Added temporary solution for Rules Resource access via Kerberos ticket.
[81ca80f] NIFI-1274 Added KerberosAuthenticationFilter to conduct SPNEGO authentication with local (client) Kerberos ticket.
Added properties and accessors for service principal and keytab location for NiFi app server.
Added KAF to NiFiWebApiSecurityConfiguration.
Added AlternateKerberosUserDetailsService to provide user lookup without dependency on extension bundle (nifi-kerberos-iaa-provider).
Added dependencies on spring-security-kerberos-core and -web modules to pom.xml.
[0605ba8] Added working configuration files to test/resources in kerberos module to document necessary config. This version requires the user to enter their Kerberos username (without realm) and password into the NiFi login screen and will authenticate them against the running KDC.
Also includes a sample keystore and root CA public key for configuring a secure instance.
[49236c8] Added kerberos module dependencies to nifi/pom.xml and nifi-assembly/pom.xml.
Added default properties to login-identity-providers.xml.
[928c52b] Added nifi-kerberos-iaa-providers-bundle module to nifi/pom.xml.
Added skeleton of Kerberos authenticator using Spring Security Kerberos plugin.
This closes#284
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
Matt Gilman
Bryan Bende
Oleg Zhurakousky
joewitt
Koji Kawamura
Mark Payne
Scott Aslan
Yolanda M. Davis
Koji Kawamura
Aldrin Piri
Bryan Rosander
Matt Burgess
Andy LoPresto
Jeff Storck
jpercivall
Jeremy Dyer
James Wing
Pierre Villard
Joe Skora
rkarthik29
Mike Moser
Andy LoPresto
trkurc
puspendu.banerjee@gmail.com