NIFI-2854: Incorporated PR review feedback
NIFI-2854: Implemented feedback from PR Review
NIFI-2854: Ensure that all resources are closed on CompressableRecordReader.close() even if an IOException is thrown when closing one of them
This closes#1202
NIFI-1002: Added WebSocket support.
- Reflecting review comments
- Added displayName to peroperty descriptors
NIFI-1002: Added WebSocket support.
This closes#1184
- Reflecting review comments:
- Removed unnecessary use of NarCloseable.withComponentNarLoader.
- Removed intermediate on memory queue to make it simpler and more
robust. Received messages in WebSocket layer now will be transferred
to downstream relationships directly.
Reads from and writes to a Zookeeper that is open or secured via user/password digest or SASL
Supports persisting of data obtained from Zookeeper to a file or standard out as JSON
Supports sending of Zookeeper data to Zookeeper from a file or standard out as JSON
Does not allow data obtained from a source Zookeeper to be written back to the same Zookeeper connect string and path
This closes#1193
Signed-off-by: Bryan Rosander <brosander@apache.org>
* Java version and vendor
* OS name and version
* Release Tag
* Build revision (commit SHA), branch, and timestamp
* Handles formal releases, ad-hoc builds, and non-release source builds
* Standalone UI presence in About dialog, Summary -> System Diagnostics
* Cluster UI as Versions tab in Cluster dialog
* Reduce About Dialog Content
* Fix Missing Property Display Bugs
* Marking the build time as type string.
* This closes#583
This closes#695.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Defaulting to same keyStore, key password (+18 squashed commits)
Squashed commits:
[9d01ba0] NIFI-2193 - Fixing typo
[55440bc] NIFI-2193 - Standalone can run as long as there are no conflicting files/folders
[0ca34ed] NIFI-2193 - Fixing some filename, absolute path issues
[9d4f65b] NIFI-2193 - Incorporating feedback
[f7550b4] NIFI-2193 - Cleaning up imports
[59a7637] NIFI-2193 - Updating umask to allow owner to execute
[cf824e7] NIFI-2193 - Moving DN arg to CA service specific parent class
[921ee13] NIFI-2193 - Making keystore getInstance more consistent
[a283c4b] NIFI-2193 - Updating sample config files in assembly to reflect new structure
[8d3a21d] NIFI-2193 - Making TlsHelper static, adding option to use same password for Key, KeyStore
[b13d247] NIFI-2193 - Addressing PR feedback
[46ef8ed] NIFI-2193 - Removing commons-logging, log4j from notice
[d4cf41a] NIFI-2193 - Adding option to specify output file for CA certificate when using cli client
[b74bf25] NIFI-2193 - Removing Bouncy Castle from notice
[6e34f9a] NIFI-2193 - Adding CLI client for easier generation of client certificates
[2924fca] NIFI-2193 - nifi-toolkit-ssl -> nifi-toolkit-tls, removing unused constants
[886167e] NIFI-2193 - Adding slf4j to avoid runtime issue
[082de46] NIFI-2193 - Command line SSL config utility as well as certificate authority client/server
- Adding Authorizer implementation for Ranger
- Adding build profile and assembly that controls the inclusion of Ranger in the final assembly
- Add properties to specify ranger admin identity and a flag to indicate if ranger is using kerberos, plugin is updated to perform a UGI login if ranger is using kerberos
- Adding LICENSE and NOTICE for Ranger NAR, and some other licensing clean up
- Adding tests for kerberos properties, adding test for RangerBasePluginWithPolicies, cleaning up code to use Java 8 features
This closes#574
* Updated the AWS SDK dependency version to 1.11.8
* Removed GetDynamoDBTest references to JsonObject class dropped from AWS SDK
* Fixed integration test ITPutSNS.testPublish() - exception from not evaluating expressions
* Ignored integration test ITPutS3Object.testS3MultipartAgeoff() - did not work before or after SDK update
This closes#562.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
* Create new ExtractMediaMetadata processor using Apache Tika Detector and Parser.
* Refactored nifi-image-bundle, nifi-image-nar, and nifi-image-processors to nifi-media-bundle, nifi-media-nar, and nifi-media-processors to reflect broader media related purpose.
* Preserved existing ExtractImageMetadata and ResizeImage processors as well as existing ImageViewerController components to prevent impact on existing uses.
* Resolved collision between ExtractImage and ExtractMedia processors due to common dependency on Noakes' Metadata Extractor project.
- Updated bundle's Tika dependency from 1.7 to 1.8 and Drew Noakes' Metadata Extractor from 2.7.2 to 2.8.0.
- Adjusted ExtractImageMetadata tests for enhanced attribute names in new Noakes' Metadata Extractor version.
* Fix assembly POM to remove duplicate reference to site-to-site nar and change nifi-image-nar reference to nifi-media-nar.
* Note the potential attribute changes on upgrade due to underlying libraries.
This closes#556.
added utility class to scale positions of components on the canvas,
extracted get/setPosition methods from ProcesGroup, RemoteProcessGroup, Label, and Connectable into new interface Positionable
added interface method for finding all Positionables in a ProcessGroup to the ProcessGroup interface and added implementation to StandardProcessGroup
added test flow for position rescaling
added Spock config to POM and a spec for testing the scaling of Positionables
forced Surefire to use JUnit (TestNG was on classpath and Surefire seems to prioritize that over JUnit),
added check in StandardFlowSynchronizer to scale positions only when flow encoding version is less than 1.0
added spec for StandardFlowfileSynchronizer
updated FlowConfiguration.xsd to allow encoding-version attribute
added new test flow used in StandardFlowSynchronizerSpec
This closes#442.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Improved StreamScanner for better performance
- Renamed StreamScanner to StreamDemarcator as suggested by Joe
- Added failure handling logic to ensure both processors can be reset to their initial state (as if they were just started)
- Provided comprehensive test suite to validate various aspects of both Publish and Consume from Kafka
- Added relevant javadocs
- Added initial additionalDetails docs
- Addressed NPE reported by NIFI-1764
- Life-cycle refactoring for the existing PutKafka to ensure producer restart after errors
- Incorporated code changes contributed by Ralph Perko (see NIFI-1837)
- Addressed partition issue in RoundRobinPartitioner discussed in NIFI-1827
- Updated PropertyDescriptor descriptions to reflect their purpose
NIFI-1296 added @Ignore on some Kafka tests to improve test time
NIFI-1296 reworked tests to avoid dependency on embedded Kafka
NIFI-1296 fixed spelling error
NIFI-1296 fixed trailing whitespaces in non-java files
This closes#366
Squashed commits:
[b4e9b5f] NIFI-1197 fixed name/displayName on properties
[d39f82b] NIFI-1197 Added SSL support for MongoDB processors
This closes#360.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Storing UGI so we can support multiple HBaseClientServices with different configs
- Creating nifi-hadoop-utils to hold utility code shared between HDFS and HBase processors
- Incorporating KerberosProperties into existing hadoop processors
This closes#281
NIFI-1274 Cleaned up TODO statements. (+3 squashed commits)
Squashed commits:
[fd101cd] Removed logic to check for presence of services to determine if token support is enabled when username/password authentication is enabled (Kerberos also requires tokens).
[c2ce29f] Reverted import changes to RulesResource.java.
[c269d72] Added Kerberos authentication mechanism.
Moved Kerberos service wiring from XML to Java to handle scenario where admin has not configured Kerberos (previously threw NullPointerException in FileSystemResource constructor). (+15 squashed commits)
Squashed commits:
[09fc694] Added Kerberos documentation to Admin Guide.
[ecfb864] Cleaned up unused logic.
[157efb3] Added logic to determine if client certificates are required for REST API (login, anonymous, and Kerberos service authentication all disabled).
Cleaned up KerberosService by moving logic to NiFiProperties.
[5438619] Added documentation for Kerberos login-identity-providers.xml.
[3332d9f] Added NiFi properties for Kerberos SSO.
[b14a557] Fixed canvas call to only attempt Kerberos login if JWT not present in local storage.
Added logic to handle ticket validation failure in AccessResource.
Changed wiring of Kerberos service beans to XML in nifi-web-security-context.xml for consistency.
[c31ae3d] Kerberos SPNEGO works without additional filter (new entry endpoint accepts Kerberos ticket in Authorization header and returns JWT so the rest of the application functions the same as LDAP).
[98460e7] Added check to only instantiate beans when Kerberos enabled to allow access control integration tests to pass.
[6ed0724] Renamed Kerberos discovery method to be explicit about service vs. credential login.
[ed67d2e] Removed temporary solution for Rules Resource access via Kerberos ticket.
[c8b2b01] Added temporary solution for Rules Resource access via Kerberos ticket.
[81ca80f] NIFI-1274 Added KerberosAuthenticationFilter to conduct SPNEGO authentication with local (client) Kerberos ticket.
Added properties and accessors for service principal and keytab location for NiFi app server.
Added KAF to NiFiWebApiSecurityConfiguration.
Added AlternateKerberosUserDetailsService to provide user lookup without dependency on extension bundle (nifi-kerberos-iaa-provider).
Added dependencies on spring-security-kerberos-core and -web modules to pom.xml.
[0605ba8] Added working configuration files to test/resources in kerberos module to document necessary config. This version requires the user to enter their Kerberos username (without realm) and password into the NiFi login screen and will authenticate them against the running KDC.
Also includes a sample keystore and root CA public key for configuring a secure instance.
[49236c8] Added kerberos module dependencies to nifi/pom.xml and nifi-assembly/pom.xml.
Added default properties to login-identity-providers.xml.
[928c52b] Added nifi-kerberos-iaa-providers-bundle module to nifi/pom.xml.
Added skeleton of Kerberos authenticator using Spring Security Kerberos plugin.
This closes#284
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
Added Groovy support for unit tests to pom with skeleton test.
Added Groovy unit tests for OCSPCertificateValidator.
Implemented positive & negative unit tests with cache injection for valid/revoked OCSP certificate.
Modified pom.xml to support Groovy unit tests with custom variable.
mvn clean test -Dgroovy=test
Added local cache injection into Groovy tests for OCSP certificate validation (see NIFI-1324 and NIFI-1364).
Set Java version to 1.7 for Groovy test src/target.
Moved Groovy unit test profile from nifi-web-security to root pom.
Added null check for algorithm argument in PGPUtil.
Changed buffer length check from ">= 0" to "> -1" because it was confusing other developers.
Resolved contrib-check line length issues.
Fixed contrib-check issues in OpenPGPKeyBasedEncryptorTest.
This closes#163
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
Changed Maven dependencies for BouncyCastle bcprov and bcpg from jdk16:1.46 to jdk15on:1.53 (kept nifi-web-security on jdk16:1.46 because jdk15on:1.53 splits OCSP logic into new module bcpkix).
Added individual unit tests for PGP public keyring validation.
Passes all legacy unit tests.
Added TODOs for customizable brick encryption and refactoring shared code.
Cleaned up magic numbers to constants.
Added unit tests for OpenPGPPasswordBasedEncryptor (internal consistency and legacy file decrypt).
Began refactoring shared encrypt code from OpenPGP* implementations.
Extracted encrypt utility method from OpenPGPPasswordBasedEncryptor to PGPUtil class.
Added test resources (signed and unsigned key-encrypted files).
Added unit tests for OpenPGPKeyBasedEncryptor (internal consistency and external file decrypt).
Changed BC dependency for nifi-web-security to bcprov-jdk15on:1.53 and bcpkix-jdk15on:1.53.
Updated OCSPValidator to use new BC logic for OCSP validation. This code compiles but should be fully audited, as the legacy OCSP validation was not completely implemented.
Added skeleton of OCSP validator unit tests with successful keypair and certificate generation and signing code.
Added further unit tests for issued certificates.
Annotated unimplemented unit tests with note about Groovy integration.
Refactored Jersey call in OCSPCertificateValidator to internal method.
Added toString() to NiFi local OcspRequest.
Implemented positive & negative unit tests with cache injection for valid/revoked OCSP certificate.
Resolved contrib-check issues.
Removed ignored code in unit test.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
- Introduced nifi-riemann-bundle for future Riemann backed monitoring
- Added initial PutRiemann processor for writing events to Riemann
using the Riemann batch client.
- Values for events are provided using the NiFi expression language
e.g. Metric -> ${latency.milliseconds:divide(1000)}
- Refactoring web security to use Spring Security Java Configuration.
- Introducing security in Web UI in order to get JWT.
NIFI-655:
- Setting up the resources (js/css) for the login page.
NIFI-655:
- Adding support for configuring anonymous roles.
- Addressing checkstyle violations.
NIFI-655:
- Moving to token api to web-api.
- Creating an LoginProvider API for user/pass based authentication.
- Creating a module for funneling access to the authorized useres.
NIFI-655:
- Moving away from usage of DN to identity throughout the application (from the user db to the authorization provider).
- Updating the authorized users schema to support login users.
- Creating an extension point for authentication of users based on username/password.
NIFI-655:
- Creating an endpoint for returning the identity of the current user.
- Updating the LoginAuthenticationFilter.
NIFI-655:
- Moving NiFi registration to the login page.
- Running the authentication filters in a different order to ensure we can disambiguate each case.
- Starting to layout each case... Forbidden, Login, Create User, Create NiFi Account.
NIFI-655:
- Addressing checkstyle issues.
NIFI-655:
- Making nf-storage available in the login page.
- Requiring use of local storage.
- Ignoring security for GET requests when obtaining the login configuration.
NIFI-655:
- Adding a new endpoint to obtain the status of a user registration.
- Updated the login page loading to ensure all possible states work.
NIFI-655:
- Ensuring we know the necessary state before we attempt to render the login page.
- Building the proxy chain in the JWT authentication filter.
- Only rendering the login when appropriate.
NIFI-655:
- Starting to style the login page.
- Added simple 'login' support by identifying username/password. Issuing JWT token coming...
- Added logout support
- Rendering the username when appropriate.
NIFI-655:
- Extracting certificate validation into a utility class.
- Fixing checkstyle issues.
- Cleaning up the web security context.
- Removing proxy chain checking where possible.
NIFI-655:
- Starting to add support for registration.
- Creating registration form.
NIFI-655:
- Starting to implement the JWT service.
- Parsing JWT on client side in order to render who the user currently is when logged in.
NIFI-655:
- Allowing the user to link back to the log in page from the new account page.
- Renaming DN to identity where possible.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Adding more/better support for logging out.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Adding a few new exceptions for the login identity provider.
NIFI-655:
- Disabling log in by default initially.
- Restoring authorization service unit test.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Updating packages for log in filters.
- Handling new registration exceptions.
- Code clean up.
NIFI-655:
- Removing registration support.
- Removing file based implementation.
NIFI-655:
- Removing file based implementation.
NIFI-655:
- Removing unused spring configuration files.
NIFI-655:
- Making the auto wiring more explicit.
NIFI-655:
- Removing unused dependencies.
NIFI-655:
- Removing unused filter.
NIFI-655:
- Updating the login API authenticate method to use a richer set of exceptions.
- UI code clean.
NIFI-655:
- Ensuring the login identity provider is able to switch context classloaders via the standard NAR mechanisms.
NIFI-655:
- Initial commit of the LDAP based identity providers.
- Fixed issue when attempting to log into a NiFi that does not support new account requests.
NIFI-655:
- Allowing the ldap provider to specify if client authentication is required/desired.
NIFI-655:
- Persisting keys to sign user tokens.
- Allowing the identity provider to specify the token expiration.
- Code clean up.
NIFI-655:
- Ensuring identities are unique in the key table.
NIFI-655:
- Adding support for specifying the user search base and user search filter in the active directory provider.
NIFI-655:
- Fixing checkstyle issues.
NIFI-655:
- Adding automatic client side token renewal.
NIFI-655:
- Ensuring the logout link is rendered when appropriate.
NIFI-655:
- Adding configuration options for referrals and connect/read timeouts
NIFI-655:
- Added an endpoint for access details including configuration, creating tokens, and checking status.
- Updated DTOs and client side to utilize new endpoints.
NIFI-655:
- Refactoring certificate extraction and validation.
- Refactoring how expiration is specified in the login identity providers.
- Adding unit tests for the access endpoints.
- Code clean up.
NIFI-655:
- Keeping token expiration between 1 minute and 12 hours.
NIFI-655:
- Using the user identity provided by the login identity provider.
NIFI-655: - Fixed typo in error message for unrecognized authentication strategy.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655. - Added logback-test.xml configuration resource for nifi-web-security.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655. - Added issuer field to LoginAuthenticationToken. - Updated AccessResource to pass identity provider class name when creating LoginAuthenticationTokens. - Began refactoring JWT logic from request parsing logic in JwtService. - Added unit tests for JWT logic.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655. - Changed issuer field to use FQ class name because some classes return an empty string for getSimpleName(). - Finished refactoring JWT logic from request parsing logic in JwtService. - Updated AccessResource and JwtAuthenticationFilter to call new JwtService methods decoupled from request header parsing. - Added extensive unit tests for JWT logic.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655:
- Refactoring key service to expose the key id.
- Handling client side expiration better.
- Removing specialized active directory provider and abstract ldap provider.
NIFI-655. - Updated JwtService and JwtServiceTest to use Key POJO instead of raw String key from KeyService.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
NIFI-655:
- Fixing typo when loading the ldap connect timeout.
- Providing a better experience for session expiration.
- Using ellipsis for lengthly user name.
- Adding an issuer to the authentication response so the LIP can specify the appropriate value.
NIFI-655:
- Showing a logging in notification during the log in process.
NIFI-655:
- Removing unnecessary class.
NIFI-655:
- Fixing checkstyle issues.
- Showing the progress spinner while submitting account justification.
NIFI-655:
- Removing deprecated authentication strategy.
- Renaming TLS to START_TLS.
- Allowing the protocol to be configured.
NIFI-655:
- Fixing issue detecting the presence of DN column
NIFI-655:
- Pre-populating the login-identity-providers.xml file with necessary properties and documentation.
- Renaming the Authentication Duration property name.
NIFI-655:
- Updating documentation for the failure response codes.
NIFI-655:
- Ensuring the user identity is not too long.
NIFI-655:
- Updating default authentication expiration to 12 hours.
NIFI-655:
- Remaining on the login form when there is any unsuccessful login attempt.
- Fixing checkstyle issues.
NIFI-980 Add support for HTTP Digest authentication to InvokeHttp
NIFI-1080 Provide additional InvokeHttp unit tests
NIFI-1133 InvokeHTTP Processor does not save Location header for 3xx responses
NIFI-1009 InvokeHTTP should be able to be scheduled without any incoming connection for GET operations
NIFI-61 Multiple improvements for InvokeHTTP inclusive of providing unique tx.id across clusters, dynamic HTTP header properties
Signed-off-by: Aldrin Piri <aldrin@apache.org>
James Wing
joewitt
Matt Burgess
Mark Payne
Koji Kawamura
Bryan Bende
Jeff Storck
Andre F de Miranda
Aldrin Piri
Bryan Rosander
vijayk
Joseph Percivall
Andy LoPresto
mans2singh
Joe Skora
Pierre Villard
Adam Lamar
Oleg Zhurakousky
Bryan Rosander
Matt Gilman
info@richards-tech.com
Chris McDermott
Andy LoPresto
Tony Kurc
Matt Burgess
ricky
Jeremy Dyer