- Refactoring to allow requests to be replicated from a node to other nodes
- Renaming cluster node connection/read timeout properties.
- Renaming NCM DN to Cluster Coordinator DN.
- Fixing default values in properties.
- Starting to fix Spring context to load correctly in standalone mode.
- Using the cluster protocol to handle connection failures instead of heartbeats.
- Ensured replicate call is returned from ControllerResource.
- Ensure the appropriate classloader when serializing templates.
- Handling when the flow contents are null.
- This closes#488
Reverted whitespace changes. (+8 squashed commits)
Squashed commits:
[329755c] NIFI-1919 Reverted import re-organization from IDE.
[cf73c2f] NIFI-1919 Updated expression language guide.
[d9a1455] NIFI-1919 Reverted changes to ReplaceEvaluator.
Added ReplaceFirstEvaluator.
Added replace first logic to Query buildFunctionEvaluator.
Added unit tests.
[e2eb880] NIFI-1919 Added replaceFirst to AttributeExpression lexer and parser grammar definitions.
[11fe913] NIFI-1919 Ignored demonstrative test for replaceAll as it behaves as expected.
[af97be1] NIFI-1919 Changed ReplaceEvaluator to use String#replaceFirst which interprets regex instead of compiling as literal.
Demonstrative unit test now passes but two existing unit tests fail. I am not sure these tests are correct.
[f24f17b] NIFI-1919 Added working unit test to illustrate fix.
[8a0d43b] NIFI-1919 Added Groovy unit test to demonstrate issue.
Added DelegatingMetaClass code to record it (test not complete).
This closes#474.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Improved StreamScanner for better performance
- Renamed StreamScanner to StreamDemarcator as suggested by Joe
- Added failure handling logic to ensure both processors can be reset to their initial state (as if they were just started)
- Provided comprehensive test suite to validate various aspects of both Publish and Consume from Kafka
- Added relevant javadocs
- Added initial additionalDetails docs
- Addressed NPE reported by NIFI-1764
- Life-cycle refactoring for the existing PutKafka to ensure producer restart after errors
- Incorporated code changes contributed by Ralph Perko (see NIFI-1837)
- Addressed partition issue in RoundRobinPartitioner discussed in NIFI-1827
- Updated PropertyDescriptor descriptions to reflect their purpose
NIFI-1296 added @Ignore on some Kafka tests to improve test time
NIFI-1296 reworked tests to avoid dependency on embedded Kafka
NIFI-1296 fixed spelling error
NIFI-1296 fixed trailing whitespaces in non-java files
This closes#366
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
- Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information
- Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
NIFI-1727:
- Refactored logic for merging HTTP Requests that are federated across cluster
NIFI-1745:
- Refactoring how HTTP Requests are replicated to nodes
- Bug fixes and continuing to work on replication refactoring. Still need to handle cluster locking and revisions
- Begin work on RevisionManager
- Resolved some issues that resulted from rebase
- Fixed URIs to align with new URI's that will be used in 1.0.0
- This closes#413
Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java
Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information
Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.
Code cleanup and incorporate comments from peer review
This closes#323
- Removing the AuthorityProvider.
- Refactoring REST API in preparation for introduction of the Authorizer.
- Updating UI accordingly.
- Removing unneeded properties from nifi.properties.
- Addressing comments from PR.
- This closes#359.
Fixed nifi-utils pom.xml comment about additional dependencies. (+5 squashed commits)
Squashed commits:
[965b766] NIFI-1753 Removed temporary work-around of duplicate certificate conversion util method and added nifi-security-utils as dependency of nifi-utils.
[cd35f9b] NIFI-1753 Replaced legacy X.509 certificate declarations with new declarations in SSLSocketChannel and EndpointConnectionPool.
Temporary work-around of duplicate certificate conversion util method because nifi-utils cannot depend on nifi-security-utils.
[6420897] NIFI-1753 Replaced legacy X.509 certificate declarations with new declarations in PostHTTP.
[b9868ef] NIFI-1753 Added convenience method for extracting DN from peer certificate chain in SSL socket (canonical implementation to reduce code duplication and references to legacy certificate implementations).
Refactored logic retrieving legacy X.509 certificates with reference to convenience method in NodeProtocolSenderImpl.
Replaced logic retrieving legacy X.509 certificates with reference to convenience method in SocketProtocolListener.
Cleaned up exception handling in SocketProtocolListener.
Replaced legacy X.509 certificate declarations with new declarations in HandleHttpRequest (needs manual test).
[e2d1c35] NIFI-1753 Added convenience methods for converting legacy X.509 certificates and abstract certificates to correct X.509 format.
Added unit tests for certificate manipulation.
Replaced logic retrieving legacy X.509 certificates with new logic in NodeProtocolSenderImpl.
Added bcpkix (Bouncy Castle PKI implementation) dependency to nifi-standard-processors pom.
This closes#346.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Storing UGI so we can support multiple HBaseClientServices with different configs
- Creating nifi-hadoop-utils to hold utility code shared between HDFS and HBase processors
- Incorporating KerberosProperties into existing hadoop processors
This closes#281
NIFI-1274 Cleaned up TODO statements. (+3 squashed commits)
Squashed commits:
[fd101cd] Removed logic to check for presence of services to determine if token support is enabled when username/password authentication is enabled (Kerberos also requires tokens).
[c2ce29f] Reverted import changes to RulesResource.java.
[c269d72] Added Kerberos authentication mechanism.
Moved Kerberos service wiring from XML to Java to handle scenario where admin has not configured Kerberos (previously threw NullPointerException in FileSystemResource constructor). (+15 squashed commits)
Squashed commits:
[09fc694] Added Kerberos documentation to Admin Guide.
[ecfb864] Cleaned up unused logic.
[157efb3] Added logic to determine if client certificates are required for REST API (login, anonymous, and Kerberos service authentication all disabled).
Cleaned up KerberosService by moving logic to NiFiProperties.
[5438619] Added documentation for Kerberos login-identity-providers.xml.
[3332d9f] Added NiFi properties for Kerberos SSO.
[b14a557] Fixed canvas call to only attempt Kerberos login if JWT not present in local storage.
Added logic to handle ticket validation failure in AccessResource.
Changed wiring of Kerberos service beans to XML in nifi-web-security-context.xml for consistency.
[c31ae3d] Kerberos SPNEGO works without additional filter (new entry endpoint accepts Kerberos ticket in Authorization header and returns JWT so the rest of the application functions the same as LDAP).
[98460e7] Added check to only instantiate beans when Kerberos enabled to allow access control integration tests to pass.
[6ed0724] Renamed Kerberos discovery method to be explicit about service vs. credential login.
[ed67d2e] Removed temporary solution for Rules Resource access via Kerberos ticket.
[c8b2b01] Added temporary solution for Rules Resource access via Kerberos ticket.
[81ca80f] NIFI-1274 Added KerberosAuthenticationFilter to conduct SPNEGO authentication with local (client) Kerberos ticket.
Added properties and accessors for service principal and keytab location for NiFi app server.
Added KAF to NiFiWebApiSecurityConfiguration.
Added AlternateKerberosUserDetailsService to provide user lookup without dependency on extension bundle (nifi-kerberos-iaa-provider).
Added dependencies on spring-security-kerberos-core and -web modules to pom.xml.
[0605ba8] Added working configuration files to test/resources in kerberos module to document necessary config. This version requires the user to enter their Kerberos username (without realm) and password into the NiFi login screen and will authenticate them against the running KDC.
Also includes a sample keystore and root CA public key for configuring a secure instance.
[49236c8] Added kerberos module dependencies to nifi/pom.xml and nifi-assembly/pom.xml.
Added default properties to login-identity-providers.xml.
[928c52b] Added nifi-kerberos-iaa-providers-bundle module to nifi/pom.xml.
Added skeleton of Kerberos authenticator using Spring Security Kerberos plugin.
This closes#284
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
* Simplified and cleaned StandardProcessScheduler.start/stopProcessor methods
* Added stop/start operations to ProcessorNode.
* Removed unnecessary synchronization blocks related to ScheduledState in favor of enforcing order and idempotency via CAS operations. Those synchronization blocks were causing intermittent deadlocks whenever @OnScheduled blocks indefinitely.
* Added support for stopping the service when @OnScheduled operation hangs.
* Fixed the order of life-cycle operation invocation ensuring that each operation can *only* be invoked at the appropriate time
* Removed unnecessary locks from StandardProcessNode since Atomic variables are used.
* Removed calls to @OnStopped from ContinuallyRunningProcessTask while ensuring that procesor's full shut down in implementation of StandardProcessorNode.stop() method.
* Removed dead code
* Added comprehensive tests suite that covers 95% of Processor's life-cycle operations within the scope of FlowController, StandardProcesssScheduler and StandardProcessNode
* Improved and added javadocs on covered operations with detailed explanations.
- Adding Network Interface property to AbstractListenEventProcessor and ListenSyslog
- Adding sending host and sending port to DatagramChannelDispatcher
- Creation of common base class AbstractListenEventBatchingProcessor
- Refactor of ListenUDP, ListenTCP, and ListenRELP to all extend from AbstractListenEventBatchingProcessor
- Changing DatagramChannelDispatcher, socket handlers, and RELP handler to use offer() when queueing instead of put(), and log an error if the offer failed
This closes#266
Added utility method to return the maximum acceptable password length for PBE ciphers on JVM with limited strength crypto because BC implementation is undocumented (based on empirical evidence).
Updated EncryptionMethod definitions to accurately reflect need for unlimited strength crypto according to algorithm key length.
Added processor logic to invoke keyed cipher.
Added EncryptContent processor property for raw hex key (always visible until NIFI-1121).
Added validations for KDF (keyed and PBE) and hex key.
Added utility method to return list of valid key lengths for algorithm.
Added description to allowable values for KDF and encryption method in EncryptContent processor.
Added IV read/write to KeyedCipherProvider and changed from interface to abstract class.
Added salt read/write logic to NifiLegacy and OpenSSL cipher providers.
Changed RandomIVPBECipherProvider from interface to abstract class.
Updated strong KDF implementations.
Renamed CipherFactory to CipherProviderFactory.
Added unit test for registered KDF resolution from factory.
Updated default iteration count for PBKDF2 cipher provider.
Implemented Scrypt cipher provider.
Added salt translator from mcrypt format to Java format.
Added unit tests for salt formatting and validation.
Added surefire block to groovy unit test profile to enforce 3072 MB heap for Scrypt test.
Added local Java implementation of Scrypt KDF (and underlying PBKDF2 KDF) from Will Glozer.
Defined interface for KeyedCipherProvider.
Implemented AES implementation for KeyedCipherProvider.
Added Ruby script to test/resources for external compatibility check.
Added key length check to PBKDF2 cipher provider.
Changed default PRF to SHA-512.
Added salt and key length check to PBKDF2 cipher provider.
Added utility method to check key length validity for cipher families.
Added Bcrypt implementation.
Implemented PBKDF2 cipher provider.
Added default constructor with strong choices for PBKDF2 cipher provider.
Implemented NiFiLegacyCipherProvider and added unit tests.
Added key length parameter to PBKDF2 cipher provider.
Added PRF resolution to PBKDF2 cipher provider.
Added RandomIVPBECipherProvider to allow for non-deterministic IVs.
Added new keyed encryption methods and added boolean field for compatibility with new KDFs.
Added CipherFactory.
Improved Javadoc in NiFi legacy cipher provider and OpenSSL cipher provider.
Added KeyedCipherProvider interface.
Added OpenSSL PKCS#5 v1.5 EVP_BytesToKey cipher provider and unit test.
This closes#201.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- Update admin guide with documentation for username/password authentication.
- Setting default anonymous roles to none.
- Making account status messages to users more clear.
- Deleting user keys when an admin revokes/deletes an account.
- Updating authentication filter to error back whenever authentication fails.
- Refactoring web security to use Spring Security Java Configuration.
- Introducing security in Web UI in order to get JWT.
NIFI-655:
- Setting up the resources (js/css) for the login page.
NIFI-655:
- Adding support for configuring anonymous roles.
- Addressing checkstyle violations.
NIFI-655:
- Moving to token api to web-api.
- Creating an LoginProvider API for user/pass based authentication.
- Creating a module for funneling access to the authorized useres.
NIFI-655:
- Moving away from usage of DN to identity throughout the application (from the user db to the authorization provider).
- Updating the authorized users schema to support login users.
- Creating an extension point for authentication of users based on username/password.
NIFI-655:
- Creating an endpoint for returning the identity of the current user.
- Updating the LoginAuthenticationFilter.
NIFI-655:
- Moving NiFi registration to the login page.
- Running the authentication filters in a different order to ensure we can disambiguate each case.
- Starting to layout each case... Forbidden, Login, Create User, Create NiFi Account.
NIFI-655:
- Addressing checkstyle issues.
NIFI-655:
- Making nf-storage available in the login page.
- Requiring use of local storage.
- Ignoring security for GET requests when obtaining the login configuration.
NIFI-655:
- Adding a new endpoint to obtain the status of a user registration.
- Updated the login page loading to ensure all possible states work.
NIFI-655:
- Ensuring we know the necessary state before we attempt to render the login page.
- Building the proxy chain in the JWT authentication filter.
- Only rendering the login when appropriate.
NIFI-655:
- Starting to style the login page.
- Added simple 'login' support by identifying username/password. Issuing JWT token coming...
- Added logout support
- Rendering the username when appropriate.
NIFI-655:
- Extracting certificate validation into a utility class.
- Fixing checkstyle issues.
- Cleaning up the web security context.
- Removing proxy chain checking where possible.
NIFI-655:
- Starting to add support for registration.
- Creating registration form.
NIFI-655:
- Starting to implement the JWT service.
- Parsing JWT on client side in order to render who the user currently is when logged in.
NIFI-655:
- Allowing the user to link back to the log in page from the new account page.
- Renaming DN to identity where possible.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Adding more/better support for logging out.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Adding a few new exceptions for the login identity provider.
NIFI-655:
- Disabling log in by default initially.
- Restoring authorization service unit test.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Updating packages for log in filters.
- Handling new registration exceptions.
- Code clean up.
NIFI-655:
- Removing registration support.
- Removing file based implementation.
NIFI-655:
- Removing file based implementation.
NIFI-655:
- Removing unused spring configuration files.
NIFI-655:
- Making the auto wiring more explicit.
NIFI-655:
- Removing unused dependencies.
NIFI-655:
- Removing unused filter.
NIFI-655:
- Updating the login API authenticate method to use a richer set of exceptions.
- UI code clean.
NIFI-655:
- Ensuring the login identity provider is able to switch context classloaders via the standard NAR mechanisms.
NIFI-655:
- Initial commit of the LDAP based identity providers.
- Fixed issue when attempting to log into a NiFi that does not support new account requests.
NIFI-655:
- Allowing the ldap provider to specify if client authentication is required/desired.
NIFI-655:
- Persisting keys to sign user tokens.
- Allowing the identity provider to specify the token expiration.
- Code clean up.
NIFI-655:
- Ensuring identities are unique in the key table.
NIFI-655:
- Adding support for specifying the user search base and user search filter in the active directory provider.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Adding automatic client side token renewal.
NIFI-655:
- Ensuring the logout link is rendered when appropriate.
NIFI-655:
- Adding configuration options for referrals and connect/read timeouts
NIFI-655:
- Added an endpoint for access details including configuration, creating tokens, and checking status.
- Updated DTOs and client side to utilize new endpoints.
NIFI-655:
- Refactoring certificate extraction and validation.
- Refactoring how expiration is specified in the login identity providers.
- Adding unit tests for the access endpoints.
- Code clean up.
NIFI-655:
- Keeping token expiration between 1 minute and 12 hours.
NIFI-655:
- Using the user identity provided by the login identity provider.
NIFI-655: - Fixed typo in error message for unrecognized authentication strategy.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655. - Added logback-test.xml configuration resource for nifi-web-security.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655. - Added issuer field to LoginAuthenticationToken. - Updated AccessResource to pass identity provider class name when creating LoginAuthenticationTokens. - Began refactoring JWT logic from request parsing logic in JwtService. - Added unit tests for JWT logic.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655. - Changed issuer field to use FQ class name because some classes return an empty string for getSimpleName(). - Finished refactoring JWT logic from request parsing logic in JwtService. - Updated AccessResource and JwtAuthenticationFilter to call new JwtService methods decoupled from request header parsing. - Added extensive unit tests for JWT logic.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655:
- Refactoring key service to expose the key id.
- Handling client side expiration better.
- Removing specialized active directory provider and abstract ldap provider.
NIFI-655. - Updated JwtService and JwtServiceTest to use Key POJO instead of raw String key from KeyService.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655:
- Fixing typo when loading the ldap connect timeout.
- Providing a better experience for session expiration.
- Using ellipsis for lengthly user name.
- Adding an issuer to the authentication response so the LIP can specify the appropriate value.
NIFI-655:
- Showing a logging in notification during the log in process.
NIFI-655:
- Removing unnecessary class.
NIFI-655:
- Fixing checkstyle issues.
- Showing the progress spinner while submitting account justification.
NIFI-655:
- Removing deprecated authentication strategy.
- Renaming TLS to START_TLS.
- Allowing the protocol to be configured.
NIFI-655:
- Fixing issue detecting the presence of DN column
NIFI-655:
- Pre-populating the login-identity-providers.xml file with necessary properties and documentation.
- Renaming the Authentication Duration property name.
NIFI-655:
- Updating documentation for the failure response codes.
NIFI-655:
- Ensuring the user identity is not too long.
NIFI-655:
- Updating default authentication expiration to 12 hours.
NIFI-655:
- Remaining on the login form when there is any unsuccessful login attempt.
- Fixing checkstyle issues.
- Unified the way ExecuteStreamCommand and ExecuteProcess handle arguments
- Argument delimiters can now be specified. Their default being what they were using before (; and space)
- Add krb5.conf to nifi.properties
nifi.kerberos.krb5.file | path to krb5.conf
- Connections to secure Hadoop clusters will be determined by their config,
that is, hadoop.security.authentication should be set to kerberos.
- Added two optional arguments to AbstractHadoopProcessor (principal and keytab),
these are only required if the cluster you're connecting to is secured. Both of
these options require the krb5.conf to be present in nifi.properties.
Signed-off-by: Bryan Bende <bbende@apache.org>
Before this change, the host given out to clients to connect to a Remote
Process Group Input Port is the host where the NiFi instance runs.
However, sometimes the binding host is different from the host that
clients connect to. For example, when a NiFi instance runs inside a
Docker container, a client on a separate machine must connect to the
Docker host which forwards the connection to the container.
Add a configuration property to specify the host name to give out to
clients to connect to a Remote Process Group Input Port. If the property
is not configured, then give out the name of host running the NiFi
instance.
Bryan Bende
Mark Payne
Chris McDermott
Andy LoPresto
Andrew Grande
Pierre Villard
Oleg Zhurakousky
Matt Gilman
ijokarumawak
Matt Brown
Aldrin Piri
Andy LoPresto
Jeff Lord
Tony Kurc
gresockj
joewitt
Joseph Percivall
ricky
Chin Huang