NIFI-1274 Cleaned up TODO statements. (+3 squashed commits)
Squashed commits:
[fd101cd] Removed logic to check for presence of services to determine if token support is enabled when username/password authentication is enabled (Kerberos also requires tokens).
[c2ce29f] Reverted import changes to RulesResource.java.
[c269d72] Added Kerberos authentication mechanism.
Moved Kerberos service wiring from XML to Java to handle scenario where admin has not configured Kerberos (previously threw NullPointerException in FileSystemResource constructor). (+15 squashed commits)
Squashed commits:
[09fc694] Added Kerberos documentation to Admin Guide.
[ecfb864] Cleaned up unused logic.
[157efb3] Added logic to determine if client certificates are required for REST API (login, anonymous, and Kerberos service authentication all disabled).
Cleaned up KerberosService by moving logic to NiFiProperties.
[5438619] Added documentation for Kerberos login-identity-providers.xml.
[3332d9f] Added NiFi properties for Kerberos SSO.
[b14a557] Fixed canvas call to only attempt Kerberos login if JWT not present in local storage.
Added logic to handle ticket validation failure in AccessResource.
Changed wiring of Kerberos service beans to XML in nifi-web-security-context.xml for consistency.
[c31ae3d] Kerberos SPNEGO works without additional filter (new entry endpoint accepts Kerberos ticket in Authorization header and returns JWT so the rest of the application functions the same as LDAP).
[98460e7] Added check to only instantiate beans when Kerberos enabled to allow access control integration tests to pass.
[6ed0724] Renamed Kerberos discovery method to be explicit about service vs. credential login.
[ed67d2e] Removed temporary solution for Rules Resource access via Kerberos ticket.
[c8b2b01] Added temporary solution for Rules Resource access via Kerberos ticket.
[81ca80f] NIFI-1274 Added KerberosAuthenticationFilter to conduct SPNEGO authentication with local (client) Kerberos ticket.
Added properties and accessors for service principal and keytab location for NiFi app server.
Added KAF to NiFiWebApiSecurityConfiguration.
Added AlternateKerberosUserDetailsService to provide user lookup without dependency on extension bundle (nifi-kerberos-iaa-provider).
Added dependencies on spring-security-kerberos-core and -web modules to pom.xml.
[0605ba8] Added working configuration files to test/resources in kerberos module to document necessary config. This version requires the user to enter their Kerberos username (without realm) and password into the NiFi login screen and will authenticate them against the running KDC.
Also includes a sample keystore and root CA public key for configuring a secure instance.
[49236c8] Added kerberos module dependencies to nifi/pom.xml and nifi-assembly/pom.xml.
Added default properties to login-identity-providers.xml.
[928c52b] Added nifi-kerberos-iaa-providers-bundle module to nifi/pom.xml.
Added skeleton of Kerberos authenticator using Spring Security Kerberos plugin.
This closes#284
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
* Simplified and cleaned StandardProcessScheduler.start/stopProcessor methods
* Added stop/start operations to ProcessorNode.
* Removed unnecessary synchronization blocks related to ScheduledState in favor of enforcing order and idempotency via CAS operations. Those synchronization blocks were causing intermittent deadlocks whenever @OnScheduled blocks indefinitely.
* Added support for stopping the service when @OnScheduled operation hangs.
* Fixed the order of life-cycle operation invocation ensuring that each operation can *only* be invoked at the appropriate time
* Removed unnecessary locks from StandardProcessNode since Atomic variables are used.
* Removed calls to @OnStopped from ContinuallyRunningProcessTask while ensuring that procesor's full shut down in implementation of StandardProcessorNode.stop() method.
* Removed dead code
* Added comprehensive tests suite that covers 95% of Processor's life-cycle operations within the scope of FlowController, StandardProcesssScheduler and StandardProcessNode
* Improved and added javadocs on covered operations with detailed explanations.
Refactored and simplified ReflectionUtils while at it
Added ReflectionUtilsTest
This closes#260.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
With adjustments to formatting and whitespace.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Add code to ContentViewerController to strip content type of any trailing parameters and lowercase the type and subtype.
Added function to ViewableContent to enable retrieving the original value of the content type if needed.
This closes#242
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
Added utility method to return the maximum acceptable password length for PBE ciphers on JVM with limited strength crypto because BC implementation is undocumented (based on empirical evidence).
Updated EncryptionMethod definitions to accurately reflect need for unlimited strength crypto according to algorithm key length.
Added processor logic to invoke keyed cipher.
Added EncryptContent processor property for raw hex key (always visible until NIFI-1121).
Added validations for KDF (keyed and PBE) and hex key.
Added utility method to return list of valid key lengths for algorithm.
Added description to allowable values for KDF and encryption method in EncryptContent processor.
Added IV read/write to KeyedCipherProvider and changed from interface to abstract class.
Added salt read/write logic to NifiLegacy and OpenSSL cipher providers.
Changed RandomIVPBECipherProvider from interface to abstract class.
Updated strong KDF implementations.
Renamed CipherFactory to CipherProviderFactory.
Added unit test for registered KDF resolution from factory.
Updated default iteration count for PBKDF2 cipher provider.
Implemented Scrypt cipher provider.
Added salt translator from mcrypt format to Java format.
Added unit tests for salt formatting and validation.
Added surefire block to groovy unit test profile to enforce 3072 MB heap for Scrypt test.
Added local Java implementation of Scrypt KDF (and underlying PBKDF2 KDF) from Will Glozer.
Defined interface for KeyedCipherProvider.
Implemented AES implementation for KeyedCipherProvider.
Added Ruby script to test/resources for external compatibility check.
Added key length check to PBKDF2 cipher provider.
Changed default PRF to SHA-512.
Added salt and key length check to PBKDF2 cipher provider.
Added utility method to check key length validity for cipher families.
Added Bcrypt implementation.
Implemented PBKDF2 cipher provider.
Added default constructor with strong choices for PBKDF2 cipher provider.
Implemented NiFiLegacyCipherProvider and added unit tests.
Added key length parameter to PBKDF2 cipher provider.
Added PRF resolution to PBKDF2 cipher provider.
Added RandomIVPBECipherProvider to allow for non-deterministic IVs.
Added new keyed encryption methods and added boolean field for compatibility with new KDFs.
Added CipherFactory.
Improved Javadoc in NiFi legacy cipher provider and OpenSSL cipher provider.
Added KeyedCipherProvider interface.
Added OpenSSL PKCS#5 v1.5 EVP_BytesToKey cipher provider and unit test.
This closes#201.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- Incorporated comments for UUID format conformance & optimized.
- polished to satisfy this test-case. we don't need incrementAndGet
Reviewed by Tony Kurc (tkurc@apache.org). This closes#202
- Addressing typo in documentation.
- Minor tweaks to admin guide.
- Adding support to stand up a ZooKeeperServer when a quorum peer is not distributed (ie supporting both embedded standalone and cluster).
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Added Groovy support for unit tests to pom with skeleton test.
Added Groovy unit tests for OCSPCertificateValidator.
Implemented positive & negative unit tests with cache injection for valid/revoked OCSP certificate.
Modified pom.xml to support Groovy unit tests with custom variable.
mvn clean test -Dgroovy=test
Added local cache injection into Groovy tests for OCSP certificate validation (see NIFI-1324 and NIFI-1364).
Set Java version to 1.7 for Groovy test src/target.
Moved Groovy unit test profile from nifi-web-security to root pom.
Added null check for algorithm argument in PGPUtil.
Changed buffer length check from ">= 0" to "> -1" because it was confusing other developers.
Resolved contrib-check line length issues.
Fixed contrib-check issues in OpenPGPKeyBasedEncryptorTest.
This closes#163
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
- Removing sort from UI.
- Addressing issues with listing and flowfile retrieval when clustered.
- Making the context menu item available when source and destination are still running.
- Adding a refresh button to the queue listing table.
- Fixing the flowfile summary sorting in the cluster manager.
- Adding a message when the source or destination of a connection is actively running.
- Updating the documentation regarding queue interaction.
- Updating the error message when a flowfile is no longer in the active queue.
- Updated queue listing to allow listing to be done while source and destination are running but not sort or have ability to search
- Added heartbeat when we finish clearing queue
- Addressing comments from review.
- Merging responses when clustered to populate node details.
- Fixed bug when clearing processor state when clustered.
- Cleared the table after successfully clearing state.
Changed Maven dependencies for BouncyCastle bcprov and bcpg from jdk16:1.46 to jdk15on:1.53 (kept nifi-web-security on jdk16:1.46 because jdk15on:1.53 splits OCSP logic into new module bcpkix).
Added individual unit tests for PGP public keyring validation.
Passes all legacy unit tests.
Added TODOs for customizable brick encryption and refactoring shared code.
Cleaned up magic numbers to constants.
Added unit tests for OpenPGPPasswordBasedEncryptor (internal consistency and legacy file decrypt).
Began refactoring shared encrypt code from OpenPGP* implementations.
Extracted encrypt utility method from OpenPGPPasswordBasedEncryptor to PGPUtil class.
Added test resources (signed and unsigned key-encrypted files).
Added unit tests for OpenPGPKeyBasedEncryptor (internal consistency and external file decrypt).
Changed BC dependency for nifi-web-security to bcprov-jdk15on:1.53 and bcpkix-jdk15on:1.53.
Updated OCSPValidator to use new BC logic for OCSP validation. This code compiles but should be fully audited, as the legacy OCSP validation was not completely implemented.
Added skeleton of OCSP validator unit tests with successful keypair and certificate generation and signing code.
Added further unit tests for issued certificates.
Annotated unimplemented unit tests with note about Groovy integration.
Refactored Jersey call in OCSPCertificateValidator to internal method.
Added toString() to NiFi local OcspRequest.
Implemented positive & negative unit tests with cache injection for valid/revoked OCSP certificate.
Resolved contrib-check issues.
Removed ignored code in unit test.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
Increasing timeout values for TestStandardProcessScheduler#validateEnabledDisableMultiThread
Increasing timeout for testConcurrencyWithEnablingReferencingServicesGraph as 10s is not sufficient for overly taxed environments.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
NIFI-108: Implementing ability to list FlowFiles in a queue
NIFI-108:
- Starting to add support for endpoints that will listing flowfiles in a queue.
NIFI-108: Added merging of response for listing of flowfiles in cluster manager
NIFI-108:
- Starting to add support for endpoints that will listing flowfiles in a queue.
NIFI-108:
- Starting to add support for endpoints that will listing flowfiles in a queue.
NIFI-108:
- Adding checkstyle issues.
NIFI-108: Add clusterNodeId to FlowFileSummaryDTO
NIFI-108: Added unit tests; added verifyCanList method to queue; fixed bugs
NIFI-108:
- Adding compilation error for IOException from getFlowFile().
- Code clean up.
- Javadocs.
NIFI-108:
- Verifying two phase commit for queue listing.
- Fixing checkstyle.
- Ensuring drop and listing requests are merged when created when clustered.
NIFI-108:
- Adding initial listing capabilities.
- Passing through the sort column and direction.
NIFI-108:
- Removing Delete FlowFile button.
- Ensuring sort flags are being passed correctly.
- Setting column widths.
- Also including the cluster node address in the flowfile summaries.
NIFI-108:
- Including queue size statistics in listing request.
- Showing connection name.
NIFI-108:
- Including queue size statistics in listing request.
- Ensuring verifyCanList runs when appropriate.
NIFI-108:
- Adding initial support for viewing flowfile details dialog.
- Adding initial support for click to content.
NIFI-108:
- Allowing the flowfile details dialog to be draggable.
NIFI-108:
- Only showing the flowfile listing table when the listing is successful and the listing is not empty.
NIFI-108:
- Reseting the queue stats when closing the listing table.
NIFI-108: Implemented sorting when performing listing of FlowFiles
NIFI-108: Fixed bug that caused the listFlowFiles operation to wait on a readLock before returning and performing work asynchronously; fixed bug in Write-Ahead FlowFile Repository that caused ContentClaims to be queued up for destruction instead of ResourceClaims - this caused millions of ContentClaims to be queued up instead of a single ResourceClaim in some tests
NIFI-108:
- Ensured the column sort indicator is reset when a new listing is opened.
- Removing unused import.
NIFI-108:
- Addressed issues found during the review.
Added isActive check to the StandardControllerServiceNode:280 to ensure that
the IF statement can only have a chance to succeed if service is active. The service
will be indiscriminately deactivated as soon as disable(..) operation is invoked. This itself will
eliminate the race condition discovered by Mark
NIFI-1164 addressed PR comments
fixed the race condition described by Mark during disable call
NIFI-1164 polished javadoc
Changed ControllerServiceNode by adding enable(..), disable(..) and isActive() operations. See javadocs for more details in both ControllerServiceNode and StandardControllerServiceNode
Refactored service enable/disable logic in StandardProcessScheduler and StandardControllerServiceNode . Below are some of the notes:
- No need for resetting class loader since its going to derive from the class loader of the service. In other words any classes that aren’t loaded and will be loaded within the scope of the already loaded service will be loaded by the class lower of that service
- No need to control 'scheduleState.isScheduled()’ since the logic has changed to use CAS operation on state update and the service state change is now atomic.
- Removed Thread.sleep(..) and while(true) loop in favor of rescheduling re-tries achieving better thread utilization since the thread that would normally block in Thread.sleep(..) is now reused.
- Added tests and validated that the race condition no longer happening
Added additional logic that allows the initiation of the service disabling while it is in ENABLING state. See javadoc of StandardProcessScheduler.enable/disable for more details.
NIFI-1164 polishing
- Addressing issues around remote process groups automatically issuing new account requests.
- Ensuring authorization issues are updated with status refresh.
- Only showing the run duration setting when applicable.
- Showing the user a warning that a source processor with a non 0 run duration could lose data when NiFi is restarted.
- Ensuring anonymous user label and login links are rendered when appropriate.
- Ensuring responses are accurate when making requests with a token when user log in is not supported.
- Update admin guide with documentation for username/password authentication.
- Setting default anonymous roles to none.
- Making account status messages to users more clear.
- Deleting user keys when an admin revokes/deletes an account.
- Updating authentication filter to error back whenever authentication fails.
- Refactoring web security to use Spring Security Java Configuration.
- Introducing security in Web UI in order to get JWT.
NIFI-655:
- Setting up the resources (js/css) for the login page.
NIFI-655:
- Adding support for configuring anonymous roles.
- Addressing checkstyle violations.
NIFI-655:
- Moving to token api to web-api.
- Creating an LoginProvider API for user/pass based authentication.
- Creating a module for funneling access to the authorized useres.
NIFI-655:
- Moving away from usage of DN to identity throughout the application (from the user db to the authorization provider).
- Updating the authorized users schema to support login users.
- Creating an extension point for authentication of users based on username/password.
NIFI-655:
- Creating an endpoint for returning the identity of the current user.
- Updating the LoginAuthenticationFilter.
NIFI-655:
- Moving NiFi registration to the login page.
- Running the authentication filters in a different order to ensure we can disambiguate each case.
- Starting to layout each case... Forbidden, Login, Create User, Create NiFi Account.
NIFI-655:
- Addressing checkstyle issues.
NIFI-655:
- Making nf-storage available in the login page.
- Requiring use of local storage.
- Ignoring security for GET requests when obtaining the login configuration.
NIFI-655:
- Adding a new endpoint to obtain the status of a user registration.
- Updated the login page loading to ensure all possible states work.
NIFI-655:
- Ensuring we know the necessary state before we attempt to render the login page.
- Building the proxy chain in the JWT authentication filter.
- Only rendering the login when appropriate.
NIFI-655:
- Starting to style the login page.
- Added simple 'login' support by identifying username/password. Issuing JWT token coming...
- Added logout support
- Rendering the username when appropriate.
NIFI-655:
- Extracting certificate validation into a utility class.
- Fixing checkstyle issues.
- Cleaning up the web security context.
- Removing proxy chain checking where possible.
NIFI-655:
- Starting to add support for registration.
- Creating registration form.
NIFI-655:
- Starting to implement the JWT service.
- Parsing JWT on client side in order to render who the user currently is when logged in.
NIFI-655:
- Allowing the user to link back to the log in page from the new account page.
- Renaming DN to identity where possible.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Adding more/better support for logging out.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Adding a few new exceptions for the login identity provider.
NIFI-655:
- Disabling log in by default initially.
- Restoring authorization service unit test.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Updating packages for log in filters.
- Handling new registration exceptions.
- Code clean up.
NIFI-655:
- Removing registration support.
- Removing file based implementation.
NIFI-655:
- Removing file based implementation.
NIFI-655:
- Removing unused spring configuration files.
NIFI-655:
- Making the auto wiring more explicit.
NIFI-655:
- Removing unused dependencies.
NIFI-655:
- Removing unused filter.
NIFI-655:
- Updating the login API authenticate method to use a richer set of exceptions.
- UI code clean.
NIFI-655:
- Ensuring the login identity provider is able to switch context classloaders via the standard NAR mechanisms.
NIFI-655:
- Initial commit of the LDAP based identity providers.
- Fixed issue when attempting to log into a NiFi that does not support new account requests.
NIFI-655:
- Allowing the ldap provider to specify if client authentication is required/desired.
NIFI-655:
- Persisting keys to sign user tokens.
- Allowing the identity provider to specify the token expiration.
- Code clean up.
NIFI-655:
- Ensuring identities are unique in the key table.
NIFI-655:
- Adding support for specifying the user search base and user search filter in the active directory provider.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Adding automatic client side token renewal.
NIFI-655:
- Ensuring the logout link is rendered when appropriate.
NIFI-655:
- Adding configuration options for referrals and connect/read timeouts
NIFI-655:
- Added an endpoint for access details including configuration, creating tokens, and checking status.
- Updated DTOs and client side to utilize new endpoints.
NIFI-655:
- Refactoring certificate extraction and validation.
- Refactoring how expiration is specified in the login identity providers.
- Adding unit tests for the access endpoints.
- Code clean up.
NIFI-655:
- Keeping token expiration between 1 minute and 12 hours.
NIFI-655:
- Using the user identity provided by the login identity provider.
NIFI-655: - Fixed typo in error message for unrecognized authentication strategy.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655. - Added logback-test.xml configuration resource for nifi-web-security.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655. - Added issuer field to LoginAuthenticationToken. - Updated AccessResource to pass identity provider class name when creating LoginAuthenticationTokens. - Began refactoring JWT logic from request parsing logic in JwtService. - Added unit tests for JWT logic.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655. - Changed issuer field to use FQ class name because some classes return an empty string for getSimpleName(). - Finished refactoring JWT logic from request parsing logic in JwtService. - Updated AccessResource and JwtAuthenticationFilter to call new JwtService methods decoupled from request header parsing. - Added extensive unit tests for JWT logic.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655:
- Refactoring key service to expose the key id.
- Handling client side expiration better.
- Removing specialized active directory provider and abstract ldap provider.
NIFI-655. - Updated JwtService and JwtServiceTest to use Key POJO instead of raw String key from KeyService.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655:
- Fixing typo when loading the ldap connect timeout.
- Providing a better experience for session expiration.
- Using ellipsis for lengthly user name.
- Adding an issuer to the authentication response so the LIP can specify the appropriate value.
NIFI-655:
- Showing a logging in notification during the log in process.
NIFI-655:
- Removing unnecessary class.
NIFI-655:
- Fixing checkstyle issues.
- Showing the progress spinner while submitting account justification.
NIFI-655:
- Removing deprecated authentication strategy.
- Renaming TLS to START_TLS.
- Allowing the protocol to be configured.
NIFI-655:
- Fixing issue detecting the presence of DN column
NIFI-655:
- Pre-populating the login-identity-providers.xml file with necessary properties and documentation.
- Renaming the Authentication Duration property name.
NIFI-655:
- Updating documentation for the failure response codes.
NIFI-655:
- Ensuring the user identity is not too long.
NIFI-655:
- Updating default authentication expiration to 12 hours.
NIFI-655:
- Remaining on the login form when there is any unsuccessful login attempt.
- Fixing checkstyle issues.
connections are scheduled to run, even if the connections have no FlowFiles;
expose these details to processor developers; update documentation
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Fixed the order of service state check in PropertyDescriptor
Encapsulated the check into private method for readability
Modified and documented test to validate correct behavior.
For more details please see comment in https://issues.apache.org/jira/browse/NIFI-1143
Current implementation of DBCPConnectionPool was attempting to test if connection could be obtained via dataSource.getConnection().
Such call is naturally a blocking call and the duration of the block is dependent on driver implementation. Some drivers (e.g., Phoenix - https://phoenix.apache.org/installation.html)
attempts numerous retries before failing creating a deadlock when attempt was made to disable DBCPConnectionPool which was still being enabled.
This fix removes the connection test from DBCPConnectionPool.onConfigured() operation returning successfully upon creation of DataSource.
For more details see comments in https://issues.apache.org/jira/browse/NIFI-1061
The exception was caused due to basic file permissions. This fix overrides
'visitFileFailed' method of SimpleFileVisitor to log WARN message and allow
FileSystemRepository to continue.
- Fixing empty java docs and adding sort by id asc to the history query
- Changing userDn to userIdentity in Action and FlowChangeAction
- Modifying NiFiAuditor to always save events locally, and implementing getFlowChanges for ClusteredEventAccess
- Starting to add support for deleting flow files from a queue by creating endpoints and starting to wire everything together.
- Adding context menu item for initiating the request to drop flow files.
NIFI-828:
- Always selecting the first item in the new component table.
- Enabling adding the selected component by typing Enter.
- Removing the 'filter by' in the new component dialogs and instead just searching every field.
- Ensuring that nodes are not disconnected when the user attempts to reset a counter that does not exist on that node. This can happen when/if counters are adjusted conditionally.
- Add krb5.conf to nifi.properties
nifi.kerberos.krb5.file | path to krb5.conf
- Connections to secure Hadoop clusters will be determined by their config,
that is, hadoop.security.authentication should be set to kerberos.
- Added two optional arguments to AbstractHadoopProcessor (principal and keytab),
these are only required if the cluster you're connecting to is secured. Both of
these options require the krb5.conf to be present in nifi.properties.
Signed-off-by: Bryan Bende <bbende@apache.org>
Before this change, the host given out to clients to connect to a Remote
Process Group Input Port is the host where the NiFi instance runs.
However, sometimes the binding host is different from the host that
clients connect to. For example, when a NiFi instance runs inside a
Docker container, a client on a separate machine must connect to the
Docker host which forwards the connection to the container.
Add a configuration property to specify the host name to give out to
clients to connect to a Remote Process Group Input Port. If the property
is not configured, then give out the name of host running the NiFi
instance.
Mark Payne
Aldrin Piri
Oleg Zhurakousky
Andy LoPresto
trkurc
puspendu.banerjee@gmail.com
Pierre Villard
Pasqualino Ferrentino
Richard Miskin
Tony Kurc
Matt Gilman
Sönke Liebau
James Wing
joewitt
ianwww
Mike Moser
Bryan Bende
Andrew Grande
Joseph Percivall
Alex Moundalexis
Venkatesh Sellappa
Mike Moser
ricky
Chin Huang