Commit Graph

103 Commits

Author SHA1 Message Date
Mark Payne c955ec1689 NIFI-2033: Allow Controller Services to be scoped at Controller level instead of just group level. This closes #540 2016-06-17 13:26:30 -04:00
Matt Gilman 0b437e09a7 NIFI-2007:
- Restoring bulletin functionality.
- Ensuring appropriate merging of bulletins in clustered responses.
2016-06-17 13:20:20 -04:00
Bryan Bende 8d8a9cba79 NIFI-1916 Updating FileAuthorizer to extend AbstractPolicyBasedAuthorizer and adding intial loading of data users, groups, and policies
- Implementing CRUD operations and unit tests for Users
- Implementing CRUD operations and unit tests for Groups
- Implementing CRUD operations and unit tests for AccessPolicies
- Adding support for seeding with an initial admin user
- Fixing delete for user and group so it removes references from policies
- Adding example to authorizations.xml
- Adding back the old users schema in preparation for auto-converting to the new format, and providing the AuthorizationConfigurationContext with access to the root process group id
- Refactoring some of the FileAuthorizer to ensure thread safety
- Adding /groups to policies created for initial admin
- This closes #473
2016-06-03 17:26:22 -04:00
Matt Gilman 806f4d549d NIFI-1951:
- Removing deprecated NiFiWebContext and related classes.
- Adding authorization to Custom UIs.
- Fixing issue when creating ControllerService inline.
- Addressing contentType issue when attempting to clear component state.
- This closes #489
2016-06-03 16:52:02 -04:00
Mark Payne df0e4e7960 NIFI-1897:
- Refactoring to allow requests to be replicated from a node to other nodes
- Renaming cluster node connection/read timeout properties.
- Renaming NCM DN to Cluster Coordinator DN.
- Fixing default values in properties.
- Starting to fix Spring context to load correctly in standalone mode.
- Using the cluster protocol to handle connection failures instead of heartbeats.
- Ensured replicate call is returned from ControllerResource.
- Ensure the appropriate classloader when serializing templates.
- Handling when the flow contents are null.
- This closes #488
2016-06-03 15:29:16 -04:00
Bryan Bende 714925358e NIFI-1884
- Defining API for Users, Groups, and Policies
- Updating hashCode and equals methods
- Documenting that returned sets are unmodifiable
- Adding builders and unit tests
- Refactoring update methods in MutableAuthorizer to not take a string id
- Refactoring builders to use constructors for seeding
- Fixing toString() methods and cleaning up constructors that don't need to worry about null sets
- Changing MutableAuthorizer to an abstract class and providing implementation of authorize
- This closes #452
2016-05-24 09:24:11 -04:00
Matt Gilman 4dd50c80a4 NIFI-1781:
- Including access policies in the breadcrumb's trail.
- Updating toolbox according to group access policies.
- Updating actions in palette based on selection access policies.
NIFI-1554:
- Introducing authorization during two phase commit.
- Introducing snippet authorization according to the encapsulated components and the action performed.
- This closes #461
2016-05-23 14:15:32 -04:00
Pierre Villard 372ffb8aa0 NIFI-1811 Removed ProcessorLog and updated dependent interfaces
This closes #403.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-05-19 13:08:09 -04:00
Mark Payne 270944ec69 NIFI-1801: Scope Templates to Process Groups. This closes #446. 2016-05-16 16:12:43 -04:00
Mark Payne 25e7f314b1 NIFI-1800: Tie Controller Services to Process Groups. This closes #431 2016-05-11 14:56:05 -04:00
Mark Payne 04c41c0654 NIFI-1678:
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
- Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information
- Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator

NIFI-1727:
- Refactored logic for merging HTTP Requests that are federated across cluster

NIFI-1745:
- Refactoring how HTTP Requests are replicated to nodes
- Bug fixes and continuing to work on replication refactoring. Still need to handle cluster locking and revisions
- Begin work on RevisionManager
- Resolved some issues that resulted from rebase
- Fixed URIs to align with new URI's that will be used in 1.0.0
- This closes #413
2016-05-06 15:23:12 -04:00
ijokarumawak 3db14f58fe NIFI-1773 Removed unused import statement. 2016-05-06 14:35:16 +09:00
Pierre Villard 46f8693bc6 NIFI-1773 NIFI-1774 fixed typo/annotation
NIFI-1774 undid the change from OnConfigured to OnEnabled
This closes #406
2016-05-04 11:44:09 -04:00
Matt Gilman ff98d823e2 NIFI-1554:
- Populating component entities in the REST API to decouple key fields from the configuration DTOs.
- Added initial support for components in UI when access isn't allowed. Formal styling to come later.
2016-04-29 14:49:14 -04:00
Pierre Villard 5e55a543eb NIFI-1724 Added properties to configure log level when file not found and permission denied on FetchFile processor
NIFI-1724 Added unit test for logging with level
This closes #348
2016-04-18 08:15:27 -04:00
Matt Gilman 153f63ef43 NIFI-1551:
- Removing the AuthorityProvider.
- Refactoring REST API in preparation for introduction of the Authorizer.
- Updating UI accordingly.
- Removing unneeded properties from nifi.properties.
- Addressing comments from PR.
- This closes #359.
2016-04-15 16:03:00 -04:00
Matt Gilman 5de40ccec3 NIFI-1553:
- Implementing a file based authorizer.
- Providing an example authorizations files.
- Address comments from PR.
- This closes #330
2016-04-07 16:28:42 -04:00
Matt Gilman 3f4ac3156c Revert "NIFI-1551:"
This reverts commit c4d06f203d. Accidently merged wrong branch.
2016-04-07 16:22:35 -04:00
Matt Gilman c4d06f203d NIFI-1551:
- Starting to remove the AuthorityProvider.
- This closes #330
2016-04-07 16:18:36 -04:00
Matt Gilman 9f7dba491f NIFI-1552:
- Addressing unused imports.
2016-04-04 11:49:25 -04:00
Matt Gilman 9aa69b242e NIFI-1552: - Introducing the Authorizer API and additional components necessary for discovery and creation of configured instances. - Minor refactoring of existing Authority Provider API code/configuration to avoid some xsd naming conflicts. These components will be removed in NIFI-1551. - Introducing a number of the resource definitions that the Authorizer will make access decisions on. This list is likely not finalized may see some changes in NIFI-1554. - Address comments from PR. - This closes #318.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2016-04-04 11:47:43 -04:00
Mark Payne 0d3bd2c401 NIFI-1563: - Federate requests and merge responses from nodes instead of storing bulletins and stats at NCM - Updating UI to support restructured status history DTO. - Return 'Insufficient History' message if aggregate stats don't have enough data points, even if all nodes do (which can be the case if the node performing the aggregation has a different value for the 'nifi.components.status.snapshot.frequency' property than the other nodes) - Bug fixes; code cleanup; replicate requests to bulletin board endpoint - Refactored the <Component>StatusDTO objects into <Component>StatusDTO, <Component>StatusSnapshotDTO, Node<Component>StatusSnapshotDTO objects - Introducing endpoints for accessing individual component status. - Wiring up new endpoints and updated core. - Code clean up. - Starting to handling status merging of individual components. - Nodewise breakdown has been added to Processors but the remaining components still need to be updated. - Refactor so that System Diagnostics requests are replicated to nodes instead of the information being pulled from Heartbeats - Replicate request for counters instead of pulling them from heartbeats - Removed the getCounters / setCounters method from HeartbeatPayload - Implementing component specific endpoints. - Removing unused endpoints. - Supporting nodewise breakdown for system diagnostics and counters. - Updating DTOs to use more consistent naming. - Code clean up. - Addressing contrib issues. - Removed ProcessGroupStatus from HeartbeatPayload - Removing nodewise from the system diagnostics endpoint. Had included it for testing that option but did not intend for it to be committed. - Addressing comments in PR #294. - This closes #294
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2016-04-04 11:42:39 -04:00
Matt Gilman 2de7f3f884 Updating versions to 1.0.0-SNAPSHOT. 2016-04-04 11:36:20 -04:00
Aldrin Piri e977729b56 NIFI-1634-rc2 prepare for next development iteration 2016-03-23 18:56:34 -04:00
Aldrin Piri 0b9bd20d31 NIFI-1634-rc2 prepare release nifi-0.6.0-RC2 2016-03-23 18:56:22 -04:00
Mark Payne a7b97419e5 NIFI-1626: Throw an Exception proactively if too much state is attempting to be stored via ZooKeeperStateProvider
NIFI-1626: Updated State Management section of Developer Guide

Signed-off-by: jpercivall <joepercivall@yahoo.com>
2016-03-16 16:12:00 -04:00
Oleg Zhurakousky 0c5b1c27f2 NIFI-1464, Refactored Processor's life-cycle operation sequence
* Simplified and cleaned StandardProcessScheduler.start/stopProcessor methods
* Added stop/start operations to ProcessorNode.
* Removed unnecessary synchronization blocks related to ScheduledState in favor of enforcing order and idempotency via CAS operations. Those synchronization blocks were causing intermittent deadlocks whenever @OnScheduled blocks indefinitely.
* Added support for stopping the service when @OnScheduled operation hangs.
* Fixed the order of life-cycle operation invocation ensuring that each operation can *only* be invoked at the appropriate time
* Removed unnecessary locks from StandardProcessNode since Atomic variables are used.
* Removed calls to @OnStopped from ContinuallyRunningProcessTask while ensuring that procesor's full shut down in implementation of StandardProcessorNode.stop() method.
* Removed dead code
* Added comprehensive tests suite that covers 95% of Processor's life-cycle operations within the scope of FlowController, StandardProcesssScheduler and StandardProcessNode
* Improved and added javadocs on covered operations with detailed explanations.
2016-03-11 12:54:50 -05:00
Pasqualino Ferrentino 73e168e954 NIFI-1003 A relationship can be auto-terminable. In this case the processor will auto-terminate the relationship and allow the user to run it even if he does not connect those relationships and does not terminate them.
This closes #217

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-03-10 14:56:46 -05:00
Mark Payne 2839a2f215 NIFI-15: Address issue where incomplete swap files can result in continually attempting to swap in data without ever being successful
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-03-09 23:46:53 -05:00
Richard Miskin 738236a41d NIFI-1573 Allow programmatic access to a Processor's name 2016-02-27 15:18:22 +00:00
Tony Kurc c7e24c7569 NIFI-1513: fixed some easy to fix errors
Addressing checkstyle issue.

This closes #221

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2016-02-25 15:21:40 -05:00
Sönke Liebau fc92441981 NIFI-1539 - Add normalization of content type for content viewing
Add code to ContentViewerController to strip content type of any trailing parameters and lowercase the type and subtype.

Added function to ViewableContent to enable retrieving the original value of the content type if needed.

This closes #242

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2016-02-25 10:12:45 -05:00
Mark Payne 3bb18b9653 NIFI-1527: Ensure that we increment Claimant Counts for content claims that are referenced by Swapped-Out FlowFiles on restart of nifi 2016-02-18 16:25:57 -05:00
Tony Kurc ad73b5c9d9 NIFI-1379-RC3 prepare for next development iteration 2016-02-12 17:28:10 -05:00
Tony Kurc 8309dba80b NIFI-1379-RC3 prepare release nifi-0.5.0-RC3 2016-02-12 17:27:59 -05:00
Tony Kurc 303f8eabf1 NIFI-1379: Move to 0.5.0-SNAPSHOT, add tkurc code signing key to keys 2016-02-06 08:49:48 -05:00
Aldrin Piri 4df6512126 Merge branch 'NIFI-259' 2016-02-05 14:09:29 -05:00
Aldrin Piri 447e401912 NIFI-259: Extending the StateProvider interface to provide a getSupportedScopes method and implemented this based on the capabilities of each of its implementations. Used supported scope to evaluated configurations at startup and prevent issues when trying to make use of state 2016-02-01 16:29:12 -05:00
Aldrin Piri 6902812678 NIFI-259: Adjusting property description of the connect string to be more specific concerning socket tuple format and the associated default port if one is not specified. Providing minor cleanup and removing unused import. 2016-02-01 10:38:45 -05:00
Aldrin Piri fdda8f3e0f NIFI-45 Providing all validation results when more than one property is invalid for a referenced ControllerService 2016-01-25 09:26:33 -05:00
Matt Gilman 5c36358bc2 NIFI-45: - Include entire toString of validation result when referenced Controller Services are invalid.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-01-23 21:28:46 -05:00
Mark Payne f6ec437bc7 Merge branch 'master' into NIFI-259 2016-01-21 09:32:02 -05:00
Matt Gilman 0d7edcb3ac NIFI-108:
- Removing sort from UI.
- Addressing issues with listing and flowfile retrieval when clustered.
- Making the context menu item available when source and destination are still running.
- Adding a refresh button to the queue listing table.
- Fixing the flowfile summary sorting in the cluster manager.
- Adding a message when the source or destination of a connection is actively running.
- Updating the documentation regarding queue interaction.
- Updating the error message when a flowfile is no longer in the active queue.
- Updated queue listing to allow listing to be done while source and destination are running but not sort or have ability to search
- Added heartbeat when we finish clearing queue
- Addressing comments from review.
2016-01-21 08:13:32 -05:00
Mark Payne eba25ecaca NIFI-259, NIFI-1339: Added OnConfigurationRestored annotation, always invoke onPropertyModified even on restart when properties are changed from defaults, as was done previously 2016-01-17 14:40:27 -05:00
Mark Payne e943c41a47 NIFI-259: Minor bug fixes 2016-01-14 09:26:26 -05:00
Mark Payne 0151b1eed1 NIFI-259: Fixed checkstyle violations 2016-01-13 13:57:47 -05:00
Mark Payne 774c29a4da NIFI-259: Added Stateful annotation as described on ticket 2016-01-12 15:28:35 -05:00
Matt Gilman bbd35a0258 NIFI-259:
- Adding endpoints for getting and clearing component state.
2016-01-12 10:43:30 -05:00
Mark Payne d2a969e3d6 NIFI-259: Initial implementation of State Management feature 2016-01-11 16:38:52 -05:00
Mark Payne b330fd1601 NIFI-108: Built out 'skeleton' of the requests so that web tier can be written against it
NIFI-108: Implementing ability to list FlowFiles in a queue

NIFI-108:
- Starting to add support for endpoints that will listing flowfiles in a queue.

NIFI-108: Added merging of response for listing of flowfiles in cluster manager

NIFI-108:
- Starting to add support for endpoints that will listing flowfiles in a queue.

NIFI-108:
- Starting to add support for endpoints that will listing flowfiles in a queue.

NIFI-108:
- Adding checkstyle issues.

NIFI-108: Add clusterNodeId to FlowFileSummaryDTO

NIFI-108: Added unit tests; added verifyCanList method to queue; fixed bugs

NIFI-108:
- Adding compilation error for IOException from getFlowFile().
- Code clean up.
- Javadocs.

NIFI-108:
- Verifying two phase commit for queue listing.
- Fixing checkstyle.
- Ensuring drop and listing requests are merged when created when clustered.

NIFI-108:
- Adding initial listing capabilities.
- Passing through the sort column and direction.

NIFI-108:
- Removing Delete FlowFile button.
- Ensuring sort flags are being passed correctly.
- Setting column widths.
- Also including the cluster node address in the flowfile summaries.

NIFI-108:
- Including queue size statistics in listing request.
- Showing connection name.

NIFI-108:
- Including queue size statistics in listing request.
- Ensuring verifyCanList runs when appropriate.

NIFI-108:
- Adding initial support for viewing flowfile details dialog.
- Adding initial support for click to content.

NIFI-108:
- Allowing the flowfile details dialog to be draggable.

NIFI-108:
- Only showing the flowfile listing table when the listing is successful and the listing is not empty.

NIFI-108:
- Reseting the queue stats when closing the listing table.

NIFI-108: Implemented sorting when performing listing of FlowFiles

NIFI-108: Fixed bug that caused the listFlowFiles operation to wait on a readLock before returning and performing work asynchronously; fixed bug in Write-Ahead FlowFile Repository that caused ContentClaims to be queued up for destruction instead of ResourceClaims - this caused millions of ContentClaims to be queued up instead of a single ResourceClaim in some tests

NIFI-108:
- Ensured the column sort indicator is reset when a new listing is opened.
- Removing unused import.

NIFI-108:
- Addressed issues found during the review.
2016-01-06 16:20:23 -05:00