Commit Graph

256 Commits

Author SHA1 Message Date
Andy LoPresto 4b9df7d1e2 NIFI-2186 Refactored CertificateUtils to separate logic for DN extraction from server/client sockets. Added logic to detect server/client mode encapsulated in exposed method.
Added unit tests for DN extraction.
Corrected typo in Javadoc.
Switched server/client socket logic for certificate extraction -- when the local socket is in client/server mode, the peer is necessarily the inverse.
Fixed unit tests.
Moved lazy-loading authentication access out of isDebugEnabled() control branch.
This closes #622
2016-07-11 23:15:28 -04:00
Matt Gilman ce5330330a NIFI-1781:
- Updating UI according to permissions through out the application.
- Shuffling provenance events, template, and cluster search REST APIs according to resources being authorized.
- Moving template upload controls.
- Removing username where appropriate.
- Addressing issues when authorizing flow configuration actions.
- Code clean up.
2016-07-01 15:10:27 -04:00
Mark Payne ae9e2fdf0b NIFI-2123: Add authorization of provenance events; refactor core classes so that Authorizable is located within nifi-api. This closes #592 2016-06-30 07:57:17 -04:00
jpercivall cd846c8d62 NIFI-1834 adding fixes to PutTCP
This closes #543
2016-06-21 12:05:39 -04:00
Matt Brown a7c912ac18 PutTCP Processor created.
Signed-off-by: jpercivall <joepercivall@yahoo.com>
2016-06-21 12:05:13 -04:00
Will Song 9546bef86e
NIFI-1958 Added "wks" as an acceptable parameter and added unit test (+6 squashed commits)
Squashed commits:
[16dd4ba] NIFI-1958 fixed logic on incoming time units. Removed feature tests to convert to weeks because it will not be implemented.
[1b22e58] NIFI-1958 added logic to getTimeDuration to handle weeks as a string value to parse but did not add TimeUnit yet. 3/5 feature test pass.
[7136544] NIFI-1958 Moved tests to correct module.
[7d95653] NIFI-1958 Added feature tests for negative values.
[ffc3941] NIFI-1958 Added second feature test.
[7d16bbe] NIFI-1958 Added new feature test and regression test for week conversion.

This closes #544.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-06-17 17:24:19 -07:00
Joshi 548561ed4c NIFI-1955: Deprecate IntegerHolder, LongHolder, BooleanHolder
This closes #537
2016-06-17 12:32:53 -04:00
Matt Gilman fbd299e885 NIFI-1901: - Building component based access control tests for Connections, Funnels, Labels, Input Ports, Output Ports, Processors, and Process Groups. - Restoring Access Token Endpoint tests.
This closes #518

Signed-off-by: Bryan Bende <bbende@apache.org>
2016-06-17 10:16:09 -04:00
Andy LoPresto 8b0c5a4911 NIFI-1981 Resolved issue where cluster communication without client certificates would fail even if needClientAuth set to false.
Fixed IDE setting for import wildcarding on Groovy files. (+4 squashed commits)
Squashed commits:
[4c3b174] NIFI-1981 Lowered logging level of client auth setting on cluster connection receive.
[b50f473] NIFI-1981 Finished logic to suppress exception on missing client certificates when clientAuth is set to WANT.
Added unit tests for CertificateUtil methods.
[ace35a2] NIFI-1981 Added test scope dependency on BouncyCastle and BC PKIX modules for CertificateUtils tests.
[2c463d1] NIFI-1981 Added ClientAuth enum and CertificateUtil methods to extract this setting from an SSLSocket.
Added logic to compare X509Certificate DNs regardless of RDN element order.
Added logic to suppress peer certificate exceptions when client authentication is not required.
Removed duplicate dependency in pom.xml.
2016-06-14 11:18:31 -04:00
Matt Gilman 75bb4bfaa2 NIFI-1781:
- Listening for window resize events more selectively.
- Fixing malformed request when configuring remote process group ports.
- Fixing malformed request when starting/stopping a selected process group.
- Fixing default value for authorizers.xml.
- This closes #524
2016-06-13 16:55:01 -04:00
Koji Kawamura c120c4982d NIFI-1857: HTTPS Site-to-Site
- Enable HTTP(S) for Site-to-Site communication
- Support HTTP Proxy in the middle of local and remote NiFi
- Support BASIC and DIGEST auth with Proxy Server
- Provide 2-phase style commit same as existing socket version
- [WIP] Test with the latest cluster env (without NCM) hasn't tested yet

- Fixed Buffer handling issues at asyc http client POST
- Fixed JS error when applying Remote Process Group Port setting from UI
- Use compression setting from UI
- Removed already finished TODO comments

- Added additional buffer draining code after receiving EOF
- Added inspection and assert code to make sure Site-to-Site client has
  written data fully to output
stream
- Changed default nifi.remote.input.secure from true to false

This closes #497.
2016-06-09 15:09:57 -04:00
Bryan Bende 37488d2789 Adding RAT exclusion for test resource in nifi-commons/nifi-expression-language 2016-06-07 17:55:07 -04:00
Mark Payne 6de738fd04 NIFI-1660: Fix formatting; update to NOTICE files of nars that bring in EL. Closed Reader that was opened in TestQuery. 2016-06-07 15:34:37 -04:00
Chris McDermott abad7d805e NIFI-1660 - Enhance the expression language with jsonPath function 2016-06-07 15:34:36 -04:00
Mark Payne df0e4e7960 NIFI-1897:
- Refactoring to allow requests to be replicated from a node to other nodes
- Renaming cluster node connection/read timeout properties.
- Renaming NCM DN to Cluster Coordinator DN.
- Fixing default values in properties.
- Starting to fix Spring context to load correctly in standalone mode.
- Using the cluster protocol to handle connection failures instead of heartbeats.
- Ensured replicate call is returned from ControllerResource.
- Ensure the appropriate classloader when serializing templates.
- Handling when the flow contents are null.
- This closes #488
2016-06-03 15:29:16 -04:00
Andy LoPresto 8127314975
NIFI-1919 Added replaceFirst() expression language method which accepts literal or pattern for replacement.
Reverted whitespace changes. (+8 squashed commits)
Squashed commits:
[329755c] NIFI-1919 Reverted import re-organization from IDE.
[cf73c2f] NIFI-1919 Updated expression language guide.
[d9a1455] NIFI-1919 Reverted changes to ReplaceEvaluator.
Added ReplaceFirstEvaluator.
Added replace first logic to Query buildFunctionEvaluator.
Added unit tests.
[e2eb880] NIFI-1919 Added replaceFirst to AttributeExpression lexer and parser grammar definitions.
[11fe913] NIFI-1919 Ignored demonstrative test for replaceAll as it behaves as expected.
[af97be1] NIFI-1919 Changed ReplaceEvaluator to use String#replaceFirst which interprets regex instead of compiling as literal.
Demonstrative unit test now passes but two existing unit tests fail. I am not sure these tests are correct.
[f24f17b] NIFI-1919 Added working unit test to illustrate fix.
[8a0d43b] NIFI-1919 Added Groovy unit test to demonstrate issue.
Added DelegatingMetaClass code to record it (test not complete).

This closes #474.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-06-01 16:39:58 -07:00
Bryan Bende 5df67c5dc2 NIFI-1907 Moving lazy init of SSLContext to StandardSiteToSiteClientConfig rather than the builder
This closes #457.
2016-05-24 09:51:18 -04:00
Andrew Grande d42ea467ca Reviewed, typos.
This closes #460
2016-05-22 10:58:48 -04:00
Mark Payne 4b74e4de74 NIFI-1745: Refactor how revisions are handled at NCM/Distributed to Node. This closes #454 2016-05-20 14:04:24 -04:00
Pierre Villard 372ffb8aa0 NIFI-1811 Removed ProcessorLog and updated dependent interfaces
This closes #403.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-05-19 13:08:09 -04:00
Oleg Zhurakousky 2d03489ec5 NIFI-1296, NIFI-1680, NIFI-1764, NIFI-1837, NIFI-1827, NIFI-1699 implemented new Kafka processors that leverage Kafka 0.9 API
- Improved StreamScanner for better performance
- Renamed StreamScanner to StreamDemarcator as suggested by Joe
- Added failure handling logic to ensure both processors can be reset to their initial state (as if they were just started)
- Provided comprehensive test suite to validate various aspects of both Publish and Consume from Kafka
- Added relevant javadocs
- Added initial additionalDetails docs
- Addressed NPE reported by NIFI-1764
- Life-cycle refactoring for the existing PutKafka to ensure producer restart after errors
- Incorporated code changes contributed by Ralph Perko (see NIFI-1837)
- Addressed partition issue in RoundRobinPartitioner discussed in NIFI-1827
- Updated PropertyDescriptor descriptions to reflect their purpose

NIFI-1296 added @Ignore on some Kafka tests to improve test time

NIFI-1296 reworked tests to avoid dependency on embedded Kafka

NIFI-1296 fixed spelling error

NIFI-1296 fixed trailing whitespaces in non-java files

This closes #366
2016-05-19 07:33:03 -04:00
Pierre Villard 7910e98ba1 NIFI-1826 Expression Language: add function to check enumerator
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2016-05-11 10:01:15 -04:00
Mark Payne 04c41c0654 NIFI-1678:
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
- Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information
- Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator

NIFI-1727:
- Refactored logic for merging HTTP Requests that are federated across cluster

NIFI-1745:
- Refactoring how HTTP Requests are replicated to nodes
- Bug fixes and continuing to work on replication refactoring. Still need to handle cluster locking and revisions
- Begin work on RevisionManager
- Resolved some issues that resulted from rebase
- Fixed URIs to align with new URI's that will be used in 1.0.0
- This closes #413
2016-05-06 15:23:12 -04:00
Mark Payne fb7b3fe4b8 NIFI-1678: Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart

Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java

Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator

Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information

Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.

Code cleanup and incorporate comments from peer review

This closes #323
2016-04-22 15:01:04 -04:00
Chris McDermott 32398693d5 NIFI-1661 add random() function to expression language.
Signed-off-by: Joe Skora <jskora@apache.org>
2016-04-21 14:05:00 -04:00
Oleg Zhurakousky 0fed158d14 NIFI-1785 added NON_BLANK_VALIDATOR
NIFI-1785 addressed PR comment

NIFI-1785 fixed white spaces

This closes #365.

Signed-off-by: Bryan Bende <bbende@apache.org>
2016-04-19 16:19:17 -04:00
Oleg Zhurakousky 13488fd9b3
NIFI-1771 deprecated ObjectHolder
This closes #351.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-04-18 11:00:33 -07:00
Matt Gilman 153f63ef43 NIFI-1551:
- Removing the AuthorityProvider.
- Refactoring REST API in preparation for introduction of the Authorizer.
- Updating UI accordingly.
- Removing unneeded properties from nifi.properties.
- Addressing comments from PR.
- This closes #359.
2016-04-15 16:03:00 -04:00
Andy LoPresto 378ccf53c2
NIFI-1753 Replaced usage of javax.security.cert.X509Certificate with java.security.cert.X509Certificate and resolved user-reported ClassCastException when handling client certificates during TLS mutual authentication.
Fixed nifi-utils pom.xml comment about additional dependencies. (+5 squashed commits)
Squashed commits:
[965b766] NIFI-1753 Removed temporary work-around of duplicate certificate conversion util method and added nifi-security-utils as dependency of nifi-utils.
[cd35f9b] NIFI-1753 Replaced legacy X.509 certificate declarations with new declarations in SSLSocketChannel and EndpointConnectionPool.
Temporary work-around of duplicate certificate conversion util method because nifi-utils cannot depend on nifi-security-utils.
[6420897] NIFI-1753 Replaced legacy X.509 certificate declarations with new declarations in PostHTTP.
[b9868ef] NIFI-1753 Added convenience method for extracting DN from peer certificate chain in SSL socket (canonical implementation to reduce code duplication and references to legacy certificate implementations).
Refactored logic retrieving legacy X.509 certificates with reference to convenience method in NodeProtocolSenderImpl.
Replaced logic retrieving legacy X.509 certificates with reference to convenience method in SocketProtocolListener.
Cleaned up exception handling in SocketProtocolListener.
Replaced legacy X.509 certificate declarations with new declarations in HandleHttpRequest (needs manual test).
[e2d1c35] NIFI-1753 Added convenience methods for converting legacy X.509 certificates and abstract certificates to correct X.509 format.
Added unit tests for certificate manipulation.
Replaced logic retrieving legacy X.509 certificates with new logic in NodeProtocolSenderImpl.
Added bcpkix (Bouncy Castle PKI implementation) dependency to nifi-standard-processors pom.

This closes #346.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-04-13 18:30:21 -07:00
Matt Gilman 3f4ac3156c Revert "NIFI-1551:"
This reverts commit c4d06f203d. Accidently merged wrong branch.
2016-04-07 16:22:35 -04:00
Matt Gilman c4d06f203d NIFI-1551:
- Starting to remove the AuthorityProvider.
- This closes #330
2016-04-07 16:18:36 -04:00
ijokarumawak 9235a28f82 NIFI-1736 Move kafka.StreamScanner to nifi-utils. This closes #333 2016-04-07 07:37:28 -04:00
Matt Brown b0cc6ae7e8 NIFI-1630 PutUDP processor created.
Signed-off-by: Bryan Bende <bbende@apache.org>
2016-04-06 11:04:01 -04:00
Mark Payne 1ac05266a5 NIFI-483: Use ZooKeeper's Leader Election to determine Primary Node. This closes #301
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2016-04-04 11:47:08 -04:00
Matt Gilman 2de7f3f884 Updating versions to 1.0.0-SNAPSHOT. 2016-04-04 11:36:20 -04:00
Bryan Bende 04c6830937 NIFI-1698 Improving customValidate in AbstractHadoopProcessor and HBaseClient service to not reload Configuration unless it changed. This closes #313 2016-04-04 10:14:05 -04:00
Aldrin Piri e977729b56 NIFI-1634-rc2 prepare for next development iteration 2016-03-23 18:56:34 -04:00
Aldrin Piri 0b9bd20d31 NIFI-1634-rc2 prepare release nifi-0.6.0-RC2 2016-03-23 18:56:22 -04:00
Mark Payne f120952ab7 NIFI-1669: Ensure that we take into account the number of 'ready bins' when determining whether or not we have reached out Max Bin threshold for BinFiles
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-03-22 20:49:39 -04:00
Aldrin Piri b44b177039 NIFI-1605 Adjust documentation and resources to reflect nifi.provenance.repository.rollover.time default
This closes #263

Signed-off-by: Matt Burgess <mattyb149@apache.org>
2016-03-17 22:14:24 -04:00
Bryan Bende 8773ec3d3e NIFI-1488 Refactoring HBase Kerberos support
- Storing UGI so we can support multiple HBaseClientServices with different configs
- Creating nifi-hadoop-utils to hold utility code shared between HDFS and HBase processors
- Incorporating KerberosProperties into existing hadoop processors

This closes #281
2016-03-17 17:12:57 -04:00
Andy LoPresto 76f2d5702f NIFI-1274 Added Kerberos authentication mechanism.
NIFI-1274 Cleaned up TODO statements. (+3 squashed commits)
Squashed commits:
[fd101cd] Removed logic to check for presence of services to determine if token support is enabled when username/password authentication is enabled (Kerberos also requires tokens).
[c2ce29f] Reverted import changes to RulesResource.java.
[c269d72] Added Kerberos authentication mechanism.

Moved Kerberos service wiring from XML to Java to handle scenario where admin has not configured Kerberos (previously threw NullPointerException in FileSystemResource constructor). (+15 squashed commits)
Squashed commits:
[09fc694] Added Kerberos documentation to Admin Guide.
[ecfb864] Cleaned up unused logic.
[157efb3] Added logic to determine if client certificates are required for REST API (login, anonymous, and Kerberos service authentication all disabled).
Cleaned up KerberosService by moving logic to NiFiProperties.
[5438619] Added documentation for Kerberos login-identity-providers.xml.
[3332d9f] Added NiFi properties for Kerberos SSO.
[b14a557] Fixed canvas call to only attempt Kerberos login if JWT not present in local storage.
Added logic to handle ticket validation failure in AccessResource.
Changed wiring of Kerberos service beans to XML in nifi-web-security-context.xml for consistency.
[c31ae3d] Kerberos SPNEGO works without additional filter (new entry endpoint accepts Kerberos ticket in Authorization header and returns JWT so the rest of the application functions the same as LDAP).
[98460e7] Added check to only instantiate beans when Kerberos enabled to allow access control integration tests to pass.
[6ed0724] Renamed Kerberos discovery method to be explicit about service vs. credential login.
[ed67d2e] Removed temporary solution for Rules Resource access via Kerberos ticket.
[c8b2b01] Added temporary solution for Rules Resource access via Kerberos ticket.
[81ca80f] NIFI-1274 Added KerberosAuthenticationFilter to conduct SPNEGO authentication with local (client) Kerberos ticket.
Added properties and accessors for service principal and keytab location for NiFi app server.
Added KAF to NiFiWebApiSecurityConfiguration.
Added AlternateKerberosUserDetailsService to provide user lookup without dependency on extension bundle (nifi-kerberos-iaa-provider).
Added dependencies on spring-security-kerberos-core and -web modules to pom.xml.
[0605ba8] Added working configuration files to test/resources in kerberos module to document necessary config. This version requires the user to enter their Kerberos username (without realm) and password into the NiFi login screen and will authenticate them against the running KDC.
Also includes a sample keystore and root CA public key for configuring a secure instance.
[49236c8] Added kerberos module dependencies to nifi/pom.xml and nifi-assembly/pom.xml.
Added default properties to login-identity-providers.xml.
[928c52b] Added nifi-kerberos-iaa-providers-bundle module to nifi/pom.xml.
Added skeleton of Kerberos authenticator using Spring Security Kerberos plugin.
This closes #284

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2016-03-17 13:10:09 -04:00
Bryan Bende b188b0abd6 NIFI-1420 Fixing bug where a FlowFile should route to failure when PutSplunk can't createa connection, defaulting PutSplunk to TCP
Signed-off-by: joewitt <joewitt@apache.org>
2016-03-16 17:43:28 -04:00
Mark Payne 07d4d7005b NIFI-1612: Found a spot within the site-to-site client where the timeout was not utilized properly
Signed-off-by: joewitt <joewitt@apache.org>
2016-03-13 13:40:56 -04:00
Oleg Zhurakousky 59fac58c96 NIFI-1464 ensured that OnUnscheduled is treated the same as OnScheduled
NIFI-1464 polished javadocs, error messages and docs
2016-03-11 12:54:50 -05:00
Oleg Zhurakousky 0c5b1c27f2 NIFI-1464, Refactored Processor's life-cycle operation sequence
* Simplified and cleaned StandardProcessScheduler.start/stopProcessor methods
* Added stop/start operations to ProcessorNode.
* Removed unnecessary synchronization blocks related to ScheduledState in favor of enforcing order and idempotency via CAS operations. Those synchronization blocks were causing intermittent deadlocks whenever @OnScheduled blocks indefinitely.
* Added support for stopping the service when @OnScheduled operation hangs.
* Fixed the order of life-cycle operation invocation ensuring that each operation can *only* be invoked at the appropriate time
* Removed unnecessary locks from StandardProcessNode since Atomic variables are used.
* Removed calls to @OnStopped from ContinuallyRunningProcessTask while ensuring that procesor's full shut down in implementation of StandardProcessorNode.stop() method.
* Removed dead code
* Added comprehensive tests suite that covers 95% of Processor's life-cycle operations within the scope of FlowController, StandardProcesssScheduler and StandardProcessNode
* Improved and added javadocs on covered operations with detailed explanations.
2016-03-11 12:54:50 -05:00
Bryan Bende 1373bf6725 NIFI-899 Rewrite of ListenUDP to use new listener framework, includes the following changes:
- Adding Network Interface property to AbstractListenEventProcessor and ListenSyslog
- Adding sending host and sending port to DatagramChannelDispatcher
- Creation of common base class AbstractListenEventBatchingProcessor
- Refactor of ListenUDP, ListenTCP, and ListenRELP to all extend from AbstractListenEventBatchingProcessor
- Changing DatagramChannelDispatcher, socket handlers, and RELP handler to use offer() when queueing instead of put(), and log an error if the offer failed

This closes #266
2016-03-11 09:07:05 -05:00
Jeff Lord 7f15626af5 NIFI-1488: Add HBase Kerberos Support with UGI
Added `kerberos-principal` and `kerberos-keytab` properties
to the HBase service.
2016-03-09 15:51:30 -05:00
Bryan Bende 6f5fb59479 NIFI-1420 Adding Splunk bundle containing PutSplunk, and GetSplunk, and adding a ListenTCP processor to standard processors. Refactored internal code from PutSyslog to create a generic AbstractPutEventProcessor which PutSplunk extends from.
This closes #233
2016-03-07 18:21:17 -05:00
Bryan Bende 19e53962ca NIFI-1579 Performance improvements for ListenSyslog which include removing an unnecessary yield and exposing a configurable size for the internal queue used by the processor, changing ListenSyslog to use a 20ms poll and use a long poll when batching, also including same improvements for ListenRELP 2016-03-04 09:17:45 -05:00
Mark Payne 62333c9e0a NIFI-1574: Ensure that we never flush a BufferedOutputStream's buffer on close of the write-ahead log 2016-02-29 15:42:54 -05:00
Mark Payne 1149bc61cb NIFI-1572: Ensure that if an Exception is thrown when processing a bin, all sessions involved are rolled back or otherwise accounted for 2016-02-29 15:42:33 -05:00
Mark Payne 28c2a3e5a6 NIFI-1256: Added a 'Satisfies Expression' option for Matching Strategy of RouteText
Signed-off-by: jpercivall <joepercivall@yahoo.com>
2016-02-25 15:59:57 -05:00
Tony Kurc c7e24c7569 NIFI-1513: fixed some easy to fix errors
Addressing checkstyle issue.

This closes #221

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2016-02-25 15:21:40 -05:00
Tony Kurc ad73b5c9d9 NIFI-1379-RC3 prepare for next development iteration 2016-02-12 17:28:10 -05:00
Tony Kurc 8309dba80b NIFI-1379-RC3 prepare release nifi-0.5.0-RC3 2016-02-12 17:27:59 -05:00
Tony Kurc 303f8eabf1 NIFI-1379: Move to 0.5.0-SNAPSHOT, add tkurc code signing key to keys 2016-02-06 08:49:48 -05:00
Aldrin Piri 4df6512126 Merge branch 'NIFI-259' 2016-02-05 14:09:29 -05:00
Andy LoPresto 498b5023ce NIFI-1257 NIFI-1259
Added utility method to return the maximum acceptable password length for PBE ciphers on JVM with limited strength crypto because BC implementation is undocumented (based on empirical evidence).
Updated EncryptionMethod definitions to accurately reflect need for unlimited strength crypto according to algorithm key length.
Added processor logic to invoke keyed cipher.
Added EncryptContent processor property for raw hex key (always visible until NIFI-1121).
Added validations for KDF (keyed and PBE) and hex key.
Added utility method to return list of valid key lengths for algorithm.
Added description to allowable values for KDF and encryption method in EncryptContent processor.
Added IV read/write to KeyedCipherProvider and changed from interface to abstract class.
Added salt read/write logic to NifiLegacy and OpenSSL cipher providers.
Changed RandomIVPBECipherProvider from interface to abstract class.
Updated strong KDF implementations.
Renamed CipherFactory to CipherProviderFactory.
Added unit test for registered KDF resolution from factory.
Updated default iteration count for PBKDF2 cipher provider.
Implemented Scrypt cipher provider.
Added salt translator from mcrypt format to Java format.
Added unit tests for salt formatting and validation.
Added surefire block to groovy unit test profile to enforce 3072 MB heap for Scrypt test.
Added local Java implementation of Scrypt KDF (and underlying PBKDF2 KDF) from Will Glozer.
Defined interface for KeyedCipherProvider.
Implemented AES implementation for KeyedCipherProvider.
Added Ruby script to test/resources for external compatibility check.
Added key length check to PBKDF2 cipher provider.
Changed default PRF to SHA-512.
Added salt and key length check to PBKDF2 cipher provider.
Added utility method to check key length validity for cipher families.
Added Bcrypt implementation.
Implemented PBKDF2 cipher provider.
Added default constructor with strong choices for PBKDF2 cipher provider.
Implemented NiFiLegacyCipherProvider and added unit tests.
Added key length parameter to PBKDF2 cipher provider.
Added PRF resolution to PBKDF2 cipher provider.
Added RandomIVPBECipherProvider to allow for non-deterministic IVs.
Added new keyed encryption methods and added boolean field for compatibility with new KDFs.
Added CipherFactory.
Improved Javadoc in NiFi legacy cipher provider and OpenSSL cipher provider.
Added KeyedCipherProvider interface.
Added OpenSSL PKCS#5 v1.5 EVP_BytesToKey cipher provider and unit test.

This closes #201.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-02-04 19:40:55 -05:00
Aldrin Piri 6902812678 NIFI-259: Adjusting property description of the connect string to be more specific concerning socket tuple format and the associated default port if one is not specified. Providing minor cleanup and removing unused import. 2016-02-01 10:38:45 -05:00
Bryan Bende b7f7e6ed80 NIFI-1436 This closes #189. Combining stop() and close() into a single method to simplify, and adding checks on stopped flag in the run method of SocketChannelDispatcher and DatagramChannelDispatcher to ensure the run() method exits as soon as possible upon close() being called
NIFI-1436 Adding synchronization on keys set in close() method  based on Selector JavaDoc

Signed-off-by: joewitt <joewitt@apache.org>
2016-01-25 22:50:25 -05:00
Bryan Bende 1089f0a956 NIFI-1273 Adding ListenRELP processor which includes refactoring code that was previously part of ListenSyslog into a reusable framework for implementing listener processors, as well as back-end code for handling the RELP protocol. Addressing some feedback from review and fixing logging statements Addressing review feedback, added AsyncChannelDispatcher and made ChannelHandlerFactory use generics for the dispatcher This closes #179
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2016-01-22 16:28:58 -05:00
Mark Payne 4d88aaedc5 NIFI-1258: Added a new function named getDelimitedField to the Expression Language and put together a guide that walks through how to add a new function
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-01-21 22:09:25 -08:00
Mark Payne 09357297e2 NIFI-259: Ensured that thread pools were being shutdown properly 2016-01-21 12:00:40 -05:00
Mark Payne d2a969e3d6 NIFI-259: Initial implementation of State Management feature 2016-01-11 16:38:52 -05:00
gresockj 34bd2061f7 NIFI-1291: Adding BinFiles in nifi-processor-utilities, deprecating version in nifi-standard-processors. This closes #155.
Signed-off-by: joewitt <joewitt@apache.org>
2016-01-03 23:05:28 -05:00
Oleg Zhurakousky ebcefaac23 NIFI-1289 reverted new method of NiFiProperties in favor of the localized reflection call in test to refresh properties.
Reviewed and Amended (added comments) by Tony Kurc (tkurc@apache.org). This closes #150
2015-12-26 09:58:08 -05:00
Oleg Zhurakousky 2845e93812 NIFI-1289 added support for refreshing properties - Added _getNewInstance()_ operation to NiFiProperties to ensure there is a way to refresh/reload NiFi properties - Fixed javadocs
Signed-off-by: Bryan Bende <bbende@apache.org>

This closes #142
2015-12-23 14:30:08 -05:00
joewitt f4ac8d75c5 NIFI-1312-RC1 prepare for next development iteration 2015-12-19 00:41:04 -05:00
joewitt d624ea4866 NIFI-1312-RC1 prepare release nifi-0.4.1-RC1 2015-12-19 00:40:53 -05:00
Mark Payne bef3fc8b40 NIFI-1301: Ensure that when creating site-to-site connection, if remote instance is applying backpressure that we do not block indefinitely waiting for the connection to be made
Signed-off-by: joewitt <joewitt@apache.org>
2015-12-18 16:08:41 -05:00
Bryan Bende 4249fc943a NIFI-1284 Creating inner class for SiteToSiteClientConfig to fix serialization issue 2015-12-18 13:08:51 -05:00
joewitt d755e43ec8 NIFI-1122_nifi-0.4.0-RC2prepare for next development iteration 2015-12-08 13:00:10 -05:00
joewitt b66c029090 NIFI-1122_nifi-0.4.0-RC2prepare release nifi-0.4.0-RC2 2015-12-08 12:59:59 -05:00
Andy LoPresto f83e6d33c5 NIFI-1242:
Added logic and test resources to debug JCE unlimited strength cryptography policy issues and incorporated into processor property validation.
Excluded test resources from RAT check
Added KeyDerivationFunction enum.
Added kdf property in EncryptContent processor and provided to PasswordBasedEncryptor.
Added logic in PasswordBasedEncryptor to handle variable KDF.
Added unit tests for EncryptContent processor.
Added test resources and excluded from RAT check.

plain.txt: This is a plaintext message.

0s @ 12:20:32 $ openssl enc -aes-256-cbc -e -in plain.txt -out salted_raw.enc -k thisIsABadPassword -p
salt=31DC301A6C7B8A0B
key=CB878A6E167A5B530B8F2BD175E6359E3092AFF7C83274A22A5B421D79E599AC
iv =0C614A72FC06B454B84E035B3FA8F877
0s @ 12:20:44 $ xxd salted_raw.enc
0000000: 5361 6c74 6564 5f5f 31dc 301a 6c7b 8a0b  Salted__1.0.l{..
0000010: 616b c65d f767 504d c085 ba7a c517 d0cb  ak.].gPM...z....
0000020: 7832 211e f573 b6f1 ded2 8f59 88e8 088f  x2!..s.....Y....

0s @ 20:14:00 $ openssl enc -aes-256-cbc -e -in plain.txt -out unsalted_raw.enc -k thisIsABadPassword -p -nosalt
key=711E85689CE7AFF6F410AEA43ABC5446842F685B84879B2E00F977C22B9E9A7D
iv =0C90ABF8ECE84B92BAA2CD448EC760F0
0s @ 20:14:17 $ xxd unsalted_raw.enc
0000000: 70cd 2984 fdbb 0e7c c01b 7206 88b1 6b50  p.)....|..r...kP
0000010: 5eeb e4f3 4036 773b 00ce dd8e 85d8 f90a  ^...@6w;........

This closes #140

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2015-12-04 19:45:10 -05:00
Mark Payne f378ee9021 NIFI-1249: Allow Processors to add their own variables to those referencable by Expression Language. Make ReplaceText allow users to reference back-references of regex matches 2015-12-04 13:17:37 -05:00
Bryan Bende 4653908b22 NIFI-1227 Adding SSL/TLS support to Syslog processors
-Removing connection creation from PutSyslog onScheduled, and reusing constructor in SSLSocketChannel
-Reducing visiblity of inner classes
2015-12-03 11:42:33 -05:00
Tony Kurc 3a7ddc6a35 NIFI-1054: Fixed DOS line endings in xml, java and js source files
Signed-off-by: joewitt <joewitt@apache.org>
2015-12-01 22:49:51 -05:00
Matt Gilman c073253366 NIFI-655:
- Update admin guide with documentation for username/password authentication.
- Setting default anonymous roles to none.
- Making account status messages to users more clear.
- Deleting user keys when an admin revokes/deletes an account.
- Updating authentication filter to error back whenever authentication fails.
2015-11-25 14:17:23 -05:00
Matt Gilman aaf14c45c9 NIFI-655:
- Refactoring web security to use Spring Security Java Configuration.
- Introducing security in Web UI in order to get JWT.

NIFI-655:
- Setting up the resources (js/css) for the login page.

NIFI-655:
- Adding support for configuring anonymous roles.
- Addressing checkstyle violations.

NIFI-655:
- Moving to token api to web-api.
- Creating an LoginProvider API for user/pass based authentication.
- Creating a module for funneling access to the authorized useres.

NIFI-655:
- Moving away from usage of DN to identity throughout the application (from the user db to the authorization provider).
- Updating the authorized users schema to support login users.
- Creating an extension point for authentication of users based on username/password.

NIFI-655:
- Creating an endpoint for returning the identity of the current user.
- Updating the LoginAuthenticationFilter.

NIFI-655:
- Moving NiFi registration to the login page.
- Running the authentication filters in a different order to ensure we can disambiguate each case.
- Starting to layout each case... Forbidden, Login, Create User, Create NiFi Account.

NIFI-655:
- Addressing checkstyle issues.

NIFI-655:
- Making nf-storage available in the login page.
- Requiring use of local storage.
- Ignoring security for GET requests when obtaining the login configuration.

NIFI-655:
- Adding a new endpoint to obtain the status of a user registration.
- Updated the login page loading to ensure all possible states work.

NIFI-655:
- Ensuring we know the necessary state before we attempt to render the login page.
- Building the proxy chain in the JWT authentication filter.
- Only rendering the login when appropriate.

NIFI-655:
- Starting to style the login page.
- Added simple 'login' support by identifying username/password. Issuing JWT token coming...
- Added logout support
- Rendering the username when appropriate.

NIFI-655:
- Extracting certificate validation into a utility class.
- Fixing checkstyle issues.
- Cleaning up the web security context.
- Removing proxy chain checking where possible.

NIFI-655:
- Starting to add support for registration.
- Creating registration form.

NIFI-655:
- Starting to implement the JWT service.
- Parsing JWT on client side in order to render who the user currently is when logged in.

NIFI-655:
- Allowing the user to link back to the log in page from the new account page.
- Renaming DN to identity where possible.

NIFI-655:
- Fixing checkstyle issues.

NIFI-655:
- Adding more/better support for logging out.

NIFI-655:
- Fixing checkstyle issues.

NIFI-655:
- Adding a few new exceptions for the login identity provider.

NIFI-655:
- Disabling log in by default initially.
- Restoring authorization service unit test.

NIFI-655:
- Fixing checkstyle issues.

NIFI-655:
- Updating packages for log in filters.
- Handling new registration exceptions.
- Code clean up.

NIFI-655:
- Removing registration support.
- Removing file based implementation.

NIFI-655:
- Removing file based implementation.

NIFI-655:
- Removing unused spring configuration files.

NIFI-655:
- Making the auto wiring more explicit.

NIFI-655:
- Removing unused dependencies.

NIFI-655:
- Removing unused filter.

NIFI-655:
- Updating the login API authenticate method to use a richer set of exceptions.
- UI code clean.

NIFI-655:
- Ensuring the login identity provider is able to switch context classloaders via the standard NAR mechanisms.

NIFI-655:
- Initial commit of the LDAP based identity providers.
- Fixed issue when attempting to log into a NiFi that does not support new account requests.

NIFI-655:
- Allowing the ldap provider to specify if client authentication is required/desired.

NIFI-655:
- Persisting keys to sign user tokens.
- Allowing the identity provider to specify the token expiration.
- Code clean up.

NIFI-655:
- Ensuring identities are unique in the key table.

NIFI-655:
- Adding support for specifying the user search base and user search filter in the active directory provider.

NIFI-655:
- Fixing checkstyle issues.

NIFI-655:
- Adding automatic client side token renewal.

NIFI-655:
- Ensuring the logout link is rendered when appropriate.

NIFI-655:
- Adding configuration options for referrals and connect/read timeouts

NIFI-655:
- Added an endpoint for access details including configuration, creating tokens, and checking status.
- Updated DTOs and client side to utilize new endpoints.

NIFI-655:
- Refactoring certificate extraction and validation.
- Refactoring how expiration is specified in the login identity providers.
- Adding unit tests for the access endpoints.
- Code clean up.

NIFI-655:
- Keeping token expiration between 1 minute and 12 hours.

NIFI-655:
- Using the user identity provided by the login identity provider.

NIFI-655: - Fixed typo in error message for unrecognized authentication strategy.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655. - Added logback-test.xml configuration resource for nifi-web-security.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655. - Added issuer field to LoginAuthenticationToken. - Updated AccessResource to pass identity provider class name when creating LoginAuthenticationTokens. - Began refactoring JWT logic from request parsing logic in JwtService. - Added unit tests for JWT logic.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655. - Changed issuer field to use FQ class name because some classes return an empty string for getSimpleName(). - Finished refactoring JWT logic from request parsing logic in JwtService. - Updated AccessResource and JwtAuthenticationFilter to call new JwtService methods decoupled from request header parsing. - Added extensive unit tests for JWT logic.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655:
- Refactoring key service to expose the key id.
- Handling client side expiration better.
- Removing specialized active directory provider and abstract ldap provider.

NIFI-655. - Updated JwtService and JwtServiceTest to use Key POJO instead of raw String key from KeyService.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655:
- Fixing typo when loading the ldap connect timeout.
- Providing a better experience for session expiration.
- Using ellipsis for lengthly user name.
- Adding an issuer to the authentication response so the LIP can specify the appropriate value.

NIFI-655:
- Showing a logging in notification during the log in process.

NIFI-655:
- Removing unnecessary class.

NIFI-655:
- Fixing checkstyle issues.
- Showing the progress spinner while submitting account justification.

NIFI-655:
- Removing deprecated authentication strategy.
- Renaming TLS to START_TLS.
- Allowing the protocol to be configured.

NIFI-655:
- Fixing issue detecting the presence of DN column

NIFI-655:
- Pre-populating the login-identity-providers.xml file with necessary properties and documentation.
- Renaming the Authentication Duration property name.

NIFI-655:
- Updating documentation for the failure response codes.

NIFI-655:
- Ensuring the user identity is not too long.

NIFI-655:
- Updating default authentication expiration to 12 hours.

NIFI-655:
- Remaining on the login form when there is any unsuccessful login attempt.
- Fixing checkstyle issues.
2015-11-23 14:50:13 -05:00
Aldrin Piri 1580edb558 NIFI-1196 Correcting treatment of FETCH events as a continuation of a lineage trail and not the start of a new one.
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-21 20:21:36 -05:00
Aldrin Piri 08d59e4374 NIFI-1196 Providing handling of FETCH provenance events for their "unique" property, transit URI, within the framework and UI.
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-19 17:42:15 -05:00
Tony Kurc 528dab78d6 NIFI-1073: Fixing coverity discovered errors. Resource leaks, and statics
Reviewed by Bryan Bende (bbende@apache.org)
2015-11-14 18:56:43 -05:00
Mark Payne 358cabafa3 NIFI-1142: Added typo that referenced the wrong function name when indicating that isEmpty has the wrong number of arguments 2015-11-10 10:55:08 -05:00
Mark Payne 73c1671975 NIFI-1132: Limited number of Lineage Identifiers held to 100 and marked the getLineageIdentifiers() method as deprecated 2015-11-09 12:09:56 -05:00
Mark Payne 5f8fdae909 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi 2015-11-09 12:07:45 -05:00
Mark Payne 59a49aea12 Revert "NIFI-1082: Ensure that events returned from the provenance repository are ordered such that newest events are provided first"
This reverts commit cf8ca3dc2c.
2015-11-08 21:46:06 -05:00
joewitt 99629646fe NIFI-1122 moved to 0.4.0-SNAPSHOT 2015-11-06 23:41:15 -05:00
Joseph Percivall a549621267 NIFI-1077 enabling expression language on ConvertCharacterSet input and output
Signed-off-by: Mark Payne <markap14@hotmail.com>
2015-11-02 11:46:17 -05:00
Mark Payne 814e8b212c Merge branch 'NIFI-1082' 2015-11-02 11:29:52 -05:00
Joseph Percivall 37e2f178f8 NIFI-1068 Fix EndpointConnectionPool to properly remove connections from activeConnections when terminating connections
Signed-off-by: Mark Payne <markap14@hotmail.com>
2015-11-01 14:47:23 -05:00
Mark Payne cf8ca3dc2c NIFI-1082: Ensure that events returned from the provenance repository are ordered such that newest events are provided first 2015-10-28 17:32:51 -04:00
Mark Payne 193dbe6533 Merge branch 'NIFI-604' of https://github.com/rickysaltzer/nifi into NIFI-604 2015-09-23 16:03:46 -04:00
Matt Gilman ded74ec94c NIFI-876 prepare for next development iteration 2015-09-14 21:48:11 -04:00
Matt Gilman 2ec735e350 NIFI-876 prepare release nifi-0.3.0-RC1 2015-09-14 21:48:00 -04:00
Mark Payne e4e263c292 NIFI-914: If no krb5 file set in nifi properties, treat it the same as an empty string set 2015-08-31 15:18:33 -04:00
Mark Payne a8b063d61b NIFI-902: Ensure that we close the underlying file stream when we roll over a partition instead of the bufferedoutputstream, which could cause corruption of there was a failure to flush the entire buffer previously. 2015-08-30 19:48:19 -04:00
Mark Payne 5de37f63d9 NIFI-902: Ensure that if we get an IOException during rollover of WAL, we are able to recover 2015-08-28 10:04:58 -04:00
ricky 9cefc4a5a5 NIFI-604: Custom Argument Delimiters ExecuteStreamCommand / ExecuteProcess
- Unified the way ExecuteStreamCommand and ExecuteProcess handle arguments
- Argument delimiters can now be specified. Their default being what they were using before (; and space)
2015-08-27 15:26:56 -04:00
Mark Payne 4baffacc42 NIFI-892: If nifi.flowfile.repository.partitions property is changed, but repository already exists, just previous value 2015-08-25 09:58:37 -04:00
ricky 7fb6e884a7 NIFI-866: Add Kerberos Support for Hadoop
- Add krb5.conf to nifi.properties

	nifi.kerberos.krb5.file | path to krb5.conf

- Connections to secure Hadoop clusters will be determined by their config,
  that is, hadoop.security.authentication should be set to kerberos.

- Added two optional arguments to AbstractHadoopProcessor (principal and keytab),
  these are only required if the cluster you're connecting to is secured. Both of
  these options require the krb5.conf to be present in nifi.properties.

Signed-off-by: Bryan Bende <bbende@apache.org>
2015-08-24 15:36:22 -04:00
Mark Payne 68d94cc01b NIFI-744: Refactored ContentClaim into ContentClaim and ResourceClaim and allowed resource claim to be reused across sessions 2015-08-21 11:08:34 -04:00
Mark Payne 8ebf1f03c2 Merge branch 'nifi-868' of https://github.com/pukkaone/nifi into NIFI-868 2015-08-19 12:56:26 -04:00
Mark Payne 19f7db6986 NIFI-793: Added multi-threading to the indexing in the Persistent Provenance Repository 2015-08-19 12:24:25 -04:00
Chin Huang 8f98f80938 NIFI-868 Configure advertised host name of Remote Process Group Input Port
Before this change, the host given out to clients to connect to a Remote
Process Group Input Port is the host where the NiFi instance runs.
However, sometimes the binding host is different from the host that
clients connect to. For example, when a NiFi instance runs inside a
Docker container, a client on a separate machine must connect to the
Docker host which forwards the connection to the container.

Add a configuration property to specify the host name to give out to
clients to connect to a Remote Process Group Input Port. If the property
is not configured, then give out the name of host running the NiFi
instance.
2015-08-18 21:35:28 -07:00
joewitt aa99884782 NIFI-850 removed nifi parent, updated nifi pom, moved all nifi subdirs up one level, fixed readme. 2015-08-15 13:12:22 -04:00