- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
- Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information
- Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
NIFI-1727:
- Refactored logic for merging HTTP Requests that are federated across cluster
NIFI-1745:
- Refactoring how HTTP Requests are replicated to nodes
- Bug fixes and continuing to work on replication refactoring. Still need to handle cluster locking and revisions
- Begin work on RevisionManager
- Resolved some issues that resulted from rebase
- Fixed URIs to align with new URI's that will be used in 1.0.0
- This closes#413
- Populating component entities in the REST API to decouple key fields from the configuration DTOs.
- Added initial support for components in UI when access isn't allowed. Formal styling to come later.
Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java
Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information
Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.
Code cleanup and incorporate comments from peer review
This closes#323
- Introducing new REST endpoints to align with the authorizable resources.
- Additionally changes to support the new endpoints.
- Addressing comments in PR.
- This closes#374.
- Removing the AuthorityProvider.
- Refactoring REST API in preparation for introduction of the Authorizer.
- Updating UI accordingly.
- Removing unneeded properties from nifi.properties.
- Addressing comments from PR.
- This closes#359.
Fixed nifi-utils pom.xml comment about additional dependencies. (+5 squashed commits)
Squashed commits:
[965b766] NIFI-1753 Removed temporary work-around of duplicate certificate conversion util method and added nifi-security-utils as dependency of nifi-utils.
[cd35f9b] NIFI-1753 Replaced legacy X.509 certificate declarations with new declarations in SSLSocketChannel and EndpointConnectionPool.
Temporary work-around of duplicate certificate conversion util method because nifi-utils cannot depend on nifi-security-utils.
[6420897] NIFI-1753 Replaced legacy X.509 certificate declarations with new declarations in PostHTTP.
[b9868ef] NIFI-1753 Added convenience method for extracting DN from peer certificate chain in SSL socket (canonical implementation to reduce code duplication and references to legacy certificate implementations).
Refactored logic retrieving legacy X.509 certificates with reference to convenience method in NodeProtocolSenderImpl.
Replaced logic retrieving legacy X.509 certificates with reference to convenience method in SocketProtocolListener.
Cleaned up exception handling in SocketProtocolListener.
Replaced legacy X.509 certificate declarations with new declarations in HandleHttpRequest (needs manual test).
[e2d1c35] NIFI-1753 Added convenience methods for converting legacy X.509 certificates and abstract certificates to correct X.509 format.
Added unit tests for certificate manipulation.
Replaced logic retrieving legacy X.509 certificates with new logic in NodeProtocolSenderImpl.
Added bcpkix (Bouncy Castle PKI implementation) dependency to nifi-standard-processors pom.
This closes#346.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-1274 Cleaned up TODO statements. (+3 squashed commits)
Squashed commits:
[fd101cd] Removed logic to check for presence of services to determine if token support is enabled when username/password authentication is enabled (Kerberos also requires tokens).
[c2ce29f] Reverted import changes to RulesResource.java.
[c269d72] Added Kerberos authentication mechanism.
Moved Kerberos service wiring from XML to Java to handle scenario where admin has not configured Kerberos (previously threw NullPointerException in FileSystemResource constructor). (+15 squashed commits)
Squashed commits:
[09fc694] Added Kerberos documentation to Admin Guide.
[ecfb864] Cleaned up unused logic.
[157efb3] Added logic to determine if client certificates are required for REST API (login, anonymous, and Kerberos service authentication all disabled).
Cleaned up KerberosService by moving logic to NiFiProperties.
[5438619] Added documentation for Kerberos login-identity-providers.xml.
[3332d9f] Added NiFi properties for Kerberos SSO.
[b14a557] Fixed canvas call to only attempt Kerberos login if JWT not present in local storage.
Added logic to handle ticket validation failure in AccessResource.
Changed wiring of Kerberos service beans to XML in nifi-web-security-context.xml for consistency.
[c31ae3d] Kerberos SPNEGO works without additional filter (new entry endpoint accepts Kerberos ticket in Authorization header and returns JWT so the rest of the application functions the same as LDAP).
[98460e7] Added check to only instantiate beans when Kerberos enabled to allow access control integration tests to pass.
[6ed0724] Renamed Kerberos discovery method to be explicit about service vs. credential login.
[ed67d2e] Removed temporary solution for Rules Resource access via Kerberos ticket.
[c8b2b01] Added temporary solution for Rules Resource access via Kerberos ticket.
[81ca80f] NIFI-1274 Added KerberosAuthenticationFilter to conduct SPNEGO authentication with local (client) Kerberos ticket.
Added properties and accessors for service principal and keytab location for NiFi app server.
Added KAF to NiFiWebApiSecurityConfiguration.
Added AlternateKerberosUserDetailsService to provide user lookup without dependency on extension bundle (nifi-kerberos-iaa-provider).
Added dependencies on spring-security-kerberos-core and -web modules to pom.xml.
[0605ba8] Added working configuration files to test/resources in kerberos module to document necessary config. This version requires the user to enter their Kerberos username (without realm) and password into the NiFi login screen and will authenticate them against the running KDC.
Also includes a sample keystore and root CA public key for configuring a secure instance.
[49236c8] Added kerberos module dependencies to nifi/pom.xml and nifi-assembly/pom.xml.
Added default properties to login-identity-providers.xml.
[928c52b] Added nifi-kerberos-iaa-providers-bundle module to nifi/pom.xml.
Added skeleton of Kerberos authenticator using Spring Security Kerberos plugin.
This closes#284
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
* Simplified and cleaned StandardProcessScheduler.start/stopProcessor methods
* Added stop/start operations to ProcessorNode.
* Removed unnecessary synchronization blocks related to ScheduledState in favor of enforcing order and idempotency via CAS operations. Those synchronization blocks were causing intermittent deadlocks whenever @OnScheduled blocks indefinitely.
* Added support for stopping the service when @OnScheduled operation hangs.
* Fixed the order of life-cycle operation invocation ensuring that each operation can *only* be invoked at the appropriate time
* Removed unnecessary locks from StandardProcessNode since Atomic variables are used.
* Removed calls to @OnStopped from ContinuallyRunningProcessTask while ensuring that procesor's full shut down in implementation of StandardProcessorNode.stop() method.
* Removed dead code
* Added comprehensive tests suite that covers 95% of Processor's life-cycle operations within the scope of FlowController, StandardProcesssScheduler and StandardProcessNode
* Improved and added javadocs on covered operations with detailed explanations.
Refactored and simplified ReflectionUtils while at it
Added ReflectionUtilsTest
This closes#260.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
With adjustments to formatting and whitespace.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Add code to ContentViewerController to strip content type of any trailing parameters and lowercase the type and subtype.
Added function to ViewableContent to enable retrieving the original value of the content type if needed.
This closes#242
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
Added utility method to return the maximum acceptable password length for PBE ciphers on JVM with limited strength crypto because BC implementation is undocumented (based on empirical evidence).
Updated EncryptionMethod definitions to accurately reflect need for unlimited strength crypto according to algorithm key length.
Added processor logic to invoke keyed cipher.
Added EncryptContent processor property for raw hex key (always visible until NIFI-1121).
Added validations for KDF (keyed and PBE) and hex key.
Added utility method to return list of valid key lengths for algorithm.
Added description to allowable values for KDF and encryption method in EncryptContent processor.
Added IV read/write to KeyedCipherProvider and changed from interface to abstract class.
Added salt read/write logic to NifiLegacy and OpenSSL cipher providers.
Changed RandomIVPBECipherProvider from interface to abstract class.
Updated strong KDF implementations.
Renamed CipherFactory to CipherProviderFactory.
Added unit test for registered KDF resolution from factory.
Updated default iteration count for PBKDF2 cipher provider.
Implemented Scrypt cipher provider.
Added salt translator from mcrypt format to Java format.
Added unit tests for salt formatting and validation.
Added surefire block to groovy unit test profile to enforce 3072 MB heap for Scrypt test.
Added local Java implementation of Scrypt KDF (and underlying PBKDF2 KDF) from Will Glozer.
Defined interface for KeyedCipherProvider.
Implemented AES implementation for KeyedCipherProvider.
Added Ruby script to test/resources for external compatibility check.
Added key length check to PBKDF2 cipher provider.
Changed default PRF to SHA-512.
Added salt and key length check to PBKDF2 cipher provider.
Added utility method to check key length validity for cipher families.
Added Bcrypt implementation.
Implemented PBKDF2 cipher provider.
Added default constructor with strong choices for PBKDF2 cipher provider.
Implemented NiFiLegacyCipherProvider and added unit tests.
Added key length parameter to PBKDF2 cipher provider.
Added PRF resolution to PBKDF2 cipher provider.
Added RandomIVPBECipherProvider to allow for non-deterministic IVs.
Added new keyed encryption methods and added boolean field for compatibility with new KDFs.
Added CipherFactory.
Improved Javadoc in NiFi legacy cipher provider and OpenSSL cipher provider.
Added KeyedCipherProvider interface.
Added OpenSSL PKCS#5 v1.5 EVP_BytesToKey cipher provider and unit test.
This closes#201.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- Incorporated comments for UUID format conformance & optimized.
- polished to satisfy this test-case. we don't need incrementAndGet
Reviewed by Tony Kurc (tkurc@apache.org). This closes#202
- Addressing typo in documentation.
- Minor tweaks to admin guide.
- Adding support to stand up a ZooKeeperServer when a quorum peer is not distributed (ie supporting both embedded standalone and cluster).
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Added Groovy support for unit tests to pom with skeleton test.
Added Groovy unit tests for OCSPCertificateValidator.
Implemented positive & negative unit tests with cache injection for valid/revoked OCSP certificate.
Modified pom.xml to support Groovy unit tests with custom variable.
mvn clean test -Dgroovy=test
Added local cache injection into Groovy tests for OCSP certificate validation (see NIFI-1324 and NIFI-1364).
Set Java version to 1.7 for Groovy test src/target.
Moved Groovy unit test profile from nifi-web-security to root pom.
Added null check for algorithm argument in PGPUtil.
Changed buffer length check from ">= 0" to "> -1" because it was confusing other developers.
Resolved contrib-check line length issues.
Fixed contrib-check issues in OpenPGPKeyBasedEncryptorTest.
This closes#163
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
- Removing sort from UI.
- Addressing issues with listing and flowfile retrieval when clustered.
- Making the context menu item available when source and destination are still running.
- Adding a refresh button to the queue listing table.
- Fixing the flowfile summary sorting in the cluster manager.
- Adding a message when the source or destination of a connection is actively running.
- Updating the documentation regarding queue interaction.
- Updating the error message when a flowfile is no longer in the active queue.
- Updated queue listing to allow listing to be done while source and destination are running but not sort or have ability to search
- Added heartbeat when we finish clearing queue
- Addressing comments from review.
- Merging responses when clustered to populate node details.
- Fixed bug when clearing processor state when clustered.
- Cleared the table after successfully clearing state.
Changed Maven dependencies for BouncyCastle bcprov and bcpg from jdk16:1.46 to jdk15on:1.53 (kept nifi-web-security on jdk16:1.46 because jdk15on:1.53 splits OCSP logic into new module bcpkix).
Added individual unit tests for PGP public keyring validation.
Passes all legacy unit tests.
Added TODOs for customizable brick encryption and refactoring shared code.
Cleaned up magic numbers to constants.
Added unit tests for OpenPGPPasswordBasedEncryptor (internal consistency and legacy file decrypt).
Began refactoring shared encrypt code from OpenPGP* implementations.
Extracted encrypt utility method from OpenPGPPasswordBasedEncryptor to PGPUtil class.
Added test resources (signed and unsigned key-encrypted files).
Added unit tests for OpenPGPKeyBasedEncryptor (internal consistency and external file decrypt).
Changed BC dependency for nifi-web-security to bcprov-jdk15on:1.53 and bcpkix-jdk15on:1.53.
Updated OCSPValidator to use new BC logic for OCSP validation. This code compiles but should be fully audited, as the legacy OCSP validation was not completely implemented.
Added skeleton of OCSP validator unit tests with successful keypair and certificate generation and signing code.
Added further unit tests for issued certificates.
Annotated unimplemented unit tests with note about Groovy integration.
Refactored Jersey call in OCSPCertificateValidator to internal method.
Added toString() to NiFi local OcspRequest.
Implemented positive & negative unit tests with cache injection for valid/revoked OCSP certificate.
Resolved contrib-check issues.
Removed ignored code in unit test.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
Increasing timeout values for TestStandardProcessScheduler#validateEnabledDisableMultiThread
Increasing timeout for testConcurrencyWithEnablingReferencingServicesGraph as 10s is not sufficient for overly taxed environments.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
NIFI-108: Implementing ability to list FlowFiles in a queue
NIFI-108:
- Starting to add support for endpoints that will listing flowfiles in a queue.
NIFI-108: Added merging of response for listing of flowfiles in cluster manager
NIFI-108:
- Starting to add support for endpoints that will listing flowfiles in a queue.
NIFI-108:
- Starting to add support for endpoints that will listing flowfiles in a queue.
NIFI-108:
- Adding checkstyle issues.
NIFI-108: Add clusterNodeId to FlowFileSummaryDTO
NIFI-108: Added unit tests; added verifyCanList method to queue; fixed bugs
NIFI-108:
- Adding compilation error for IOException from getFlowFile().
- Code clean up.
- Javadocs.
NIFI-108:
- Verifying two phase commit for queue listing.
- Fixing checkstyle.
- Ensuring drop and listing requests are merged when created when clustered.
NIFI-108:
- Adding initial listing capabilities.
- Passing through the sort column and direction.
NIFI-108:
- Removing Delete FlowFile button.
- Ensuring sort flags are being passed correctly.
- Setting column widths.
- Also including the cluster node address in the flowfile summaries.
NIFI-108:
- Including queue size statistics in listing request.
- Showing connection name.
NIFI-108:
- Including queue size statistics in listing request.
- Ensuring verifyCanList runs when appropriate.
NIFI-108:
- Adding initial support for viewing flowfile details dialog.
- Adding initial support for click to content.
NIFI-108:
- Allowing the flowfile details dialog to be draggable.
NIFI-108:
- Only showing the flowfile listing table when the listing is successful and the listing is not empty.
NIFI-108:
- Reseting the queue stats when closing the listing table.
NIFI-108: Implemented sorting when performing listing of FlowFiles
NIFI-108: Fixed bug that caused the listFlowFiles operation to wait on a readLock before returning and performing work asynchronously; fixed bug in Write-Ahead FlowFile Repository that caused ContentClaims to be queued up for destruction instead of ResourceClaims - this caused millions of ContentClaims to be queued up instead of a single ResourceClaim in some tests
NIFI-108:
- Ensured the column sort indicator is reset when a new listing is opened.
- Removing unused import.
NIFI-108:
- Addressed issues found during the review.
Added isActive check to the StandardControllerServiceNode:280 to ensure that
the IF statement can only have a chance to succeed if service is active. The service
will be indiscriminately deactivated as soon as disable(..) operation is invoked. This itself will
eliminate the race condition discovered by Mark
NIFI-1164 addressed PR comments
fixed the race condition described by Mark during disable call
NIFI-1164 polished javadoc
Changed ControllerServiceNode by adding enable(..), disable(..) and isActive() operations. See javadocs for more details in both ControllerServiceNode and StandardControllerServiceNode
Refactored service enable/disable logic in StandardProcessScheduler and StandardControllerServiceNode . Below are some of the notes:
- No need for resetting class loader since its going to derive from the class loader of the service. In other words any classes that aren’t loaded and will be loaded within the scope of the already loaded service will be loaded by the class lower of that service
- No need to control 'scheduleState.isScheduled()’ since the logic has changed to use CAS operation on state update and the service state change is now atomic.
- Removed Thread.sleep(..) and while(true) loop in favor of rescheduling re-tries achieving better thread utilization since the thread that would normally block in Thread.sleep(..) is now reused.
- Added tests and validated that the race condition no longer happening
Added additional logic that allows the initiation of the service disabling while it is in ENABLING state. See javadoc of StandardProcessScheduler.enable/disable for more details.
NIFI-1164 polishing
- Addressing issues around remote process groups automatically issuing new account requests.
- Ensuring authorization issues are updated with status refresh.
- Only showing the run duration setting when applicable.
- Showing the user a warning that a source processor with a non 0 run duration could lose data when NiFi is restarted.
Mark Payne
Scott Aslan
jpercivall
Matt Gilman
Aldrin Piri
Andy LoPresto
Pierre Villard
Mike Moser
Bryan Bende
Oleg Zhurakousky
Andy LoPresto
trkurc
puspendu.banerjee@gmail.com
Pasqualino Ferrentino
Richard Miskin
Tony Kurc
Sönke Liebau
James Wing
joewitt
ianwww