Commit Graph

153 Commits

Author SHA1 Message Date
Koji Kawamura ae251c1a6f NIFI-2718: Show HTTP S2S Auth error on bulletin
This commit fixes following two issues, that happens when a Root Group Port
policy for S2S data transfer is removed at a remote NiFi, after a client NiFi has
connected to that port:

1. At client side, Remote Process Group should show that authorization
is failing on its bulletin, but the Exception is caught and
ignored. Nothing is shown on the UI with HTTP transport protocol.
RAW S2S shows error on RPG bulletin. This commit fixes HTTP S2S to
behave the same.

2. At server side, corresponding input-port or output-port should show
that it is accessed by an unauthorized client on its bulletin, but it's
not shown with HTTP transport protocol.
RAW S2S shows warning messages for this. This commit fixes HTTP S2S to
behave the same.

In order to fix the 2nd issue above, request authorization at
DataTransferResource is changed from using DataTransferAuthorizable
directly, to call RootGroupPort.checkUserAuthorization().

Because the blettin is tied to the Port instance and it's
difficult to produce blettin message from this resource.

Since RootGroupPort.checkUserAuthorization uses
DataTransferAuthorizable inside, the check logic stays the same as
before.

Adding a RootGroupPortAuthorizable to provide access to necessary components for performing the authorization.

This closes #996
2016-09-08 13:43:38 -04:00
Mark Payne a7e76cc00a NIFI-1966: When cluster is started up, do not assume that Cluster Coordinator has the golden copy of the flow but instead wait for some period of time or until the required number of nodes have connected, and then choose which flow is correct. This closes #977 2016-09-06 16:31:37 -04:00
Joseph Percivall 1fe18a1567 NIFI-2676-rc1 prepare for next development iteration 2016-08-26 11:40:58 -04:00
Joseph Percivall 74d5224783 NIFI-2676-rc1 prepare release nifi-1.0.0-RC1 2016-08-26 11:40:44 -04:00
Matt Burgess 0745990c2d NIFI-2604: Added validators and logic for multiple URLs/files/folders for DB driver location
This closes #912

Signed-off-by: jpercivall <joepercivall@yahoo.com>
2016-08-25 23:35:41 -04:00
Bryan Bende 957c120343 NIFI-2664 Moving System.setProperty for krb5.conf to NiFi startup, and removing conflicting property from KerberosProvider config
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #946
2016-08-25 17:26:37 -04:00
Mark Payne f908ae3c3b NIFI-2669: This closes #949. Ensure that if Exception is thrown during Transaction initialization that the underlying client is closed/cleaned up. Also ensure that we generate bulletins when logging error/warn level log messages 2016-08-25 16:39:00 -04:00
Matt Gilman a6133d4ce3 NIFI-2635:
- Fixing contrib check issues.
- Clean up pom.
- Addressing issue where reporting task property descriptor using wrong scope.

NIFI-2635:
- Fixing issue with revisions when creating users and user groups.
- Forwarding requests to the coordinator instead of replicating.
- Tweaking verbage in dialog for removing users and groups.

This closes #943
2016-08-25 13:08:35 -04:00
Mark Payne 8536ad65f4 NIFI-2651: Ensure that when we disable transmission on an RPG that we interrupt any transactions in progress for http-based site-to-site
This closes #937.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2016-08-25 15:33:10 +09:00
Matt Gilman c2bfc4ef24 NIFI-2635: - Re-using the original request during the second phase of the two phase commit. - Forwarding requests to the coordinator when received by a node.
This closes #933

Signed-off-by: jpercivall <joepercivall@yahoo.com>
2016-08-24 22:42:10 -04:00
Koji Kawamura 671301193b NIFI-2525: Fix Proxy auth issue with async send.
Without this fix, NiFi fails to send data via HTTP Site-to-Site through
Proxy which requires authentication due to AsynchronousCloseException.
It happens when async client replays producing contents in order to re-send the
request with auth credential for the proxy server, however the
connection is already closed.
This fix makes NiFi to send actual data only at the second round of requests, so that flow-file
contents can be sent without reading it twice.

Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #915
2016-08-24 20:39:17 -04:00
Koji Kawamura a3586e04d9 NIFI-2459: Site-to-Site bootstrap node failure
Refresh remote peer statuses even if the bootstrap node goes down.

Migrate existing code which handles the situation from
EndpointConnectionPool to PeerSelector, so that both RAW and HTTP
transport protocol has the same capability.

This closes #927.
2016-08-24 17:28:10 -04:00
Andy LoPresto c638191a47
NIFI-1831 Added internal logic and command-line tool to allow AES-encrypted sensitive configuration values in nifi.properties.
This closes #834.
2016-08-23 20:34:31 -07:00
Bryan Rosander 23350543ff
NIFI-2621 - Generating unique serial numbers for certificates
This closes #909.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-08-23 01:37:25 -07:00
Matt Burgess 7123a1a276 NIFI-2619: Added unit test showing bugs, Added logic to ClassLoaderUtils to trim module paths and accept URLs
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #907
2016-08-22 15:29:03 -04:00
Koji Kawamura a919844461 NIFI-2567: Site-to-Site to send large data via HTTPS
- It couldn't send data larger than about 7KB due to the mis-use of
  httpasyncclient library
- Updated httpasyncclient from 4.1.1 to 4.1.2
- Let httpasyncclient framework to call produceContent multiple times as
  it gets ready to send more data via SSL session
- Added HTTPS test cases to TestHttpClient, which failed without this
  fix
2016-08-19 14:24:53 -04:00
joewitt a5261914fb NIFI-2574 merging latest kerb changes to adjust for NiFiProperties 2016-08-17 09:06:18 -07:00
Matt Gilman 3f7216ab84 NIFI-2561: - Decoupling kerberos service and spnego principles and keytabs.
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #873
2016-08-17 08:14:05 -04:00
joewitt 7d7401add4 NIFI-2574 Changed NiFiProperties to avoid static initializer and updated all references to it. 2016-08-17 00:10:07 -07:00
Bryan Rosander fa5da543e6
NIFI-2526 - DN order, multiple standalone runs, client certificates
- Logic for sorting DN, reversing X500Names before using them to generate certificate
- Logging reordered dn
- Accounting for limited crypto pkcs12, allowing password specification for client certificates
- Updating tests to work with or without jce unlimited
- Loading keystore for test in try-with

This closes #824.

Signed-off-by: Bryan Bende <bbende@apache.org>
2016-08-15 10:21:34 -04:00
Bryan Bende fd0dd51ff5 NIFI-2553 Fixing handling of Paths in HDFS processors
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #843
2016-08-15 08:59:42 -04:00
joewitt d3b96dcac1 NIFI-2519 This closes #856. aligned threading model with subethastmp 2016-08-14 13:32:00 -04:00
Oleg Zhurakousky 48fa76ecff NIFI-2519 Fixed and refactored ListenSMTP processor
- Removed message queueing which could result in data loss
- Fixed life-cycle issues that coudl put processor in an unstable state
- Fixed PropertyDescriptor translation for Time units and Byte sizes
- Fixed broken tests
- Added additional tests

NIFI-2519 added default for SMTP_MAXIMUM_CONNECTIONS

NIFI-2519 addressed PR comments, polishing
- fixed intermittent deadlock on processor stop and added test for it
- the attributes that can not be extracted from the message but available via MessageContext are written into the outgoing FlowFile
- other minor fixes

NIFI-2519 addressed lates PR comments

NIFI-2519 added better messaging when server closes the connection

NIFI-2519 some polishing and additional tests to validate deadlocks

NIFI-2519 address latest PR comments
fixed deadlock condition for when the consumer is stopped while server is distributing messages
fixed MAX message size issue ensuring it is validated
set max connections to SMTPServer
polished pom
added L&N

NIFI-2519 PR comments
- fixed LICENSE
- Added usage of LimitingInputStream
- simplified SmtpConsumer by removing hasMessage operation
2016-08-14 13:31:46 -04:00
Devin Fisher 098a35c915 Fixed CharSequenceTranslatorEvaluator to meet style standard and added license text block
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #833
2016-08-12 11:31:50 -04:00
devin fisher 219e0e96fb Added more test for escape and unescape functions
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2016-08-12 11:30:06 -04:00
devin fisher 8f74241a10 added functions for escaping text to the expression language
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2016-08-12 11:30:06 -04:00
Devin Fisher 6f85440ebd fix a copy past error
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2016-08-12 11:30:06 -04:00
Devin Fisher ebd11b1d8f EscapeJson function added to expression-language
Made use of org.apache.commons.lang3.StringEscapeUtils to do that
actual processing

Signed-off-by: Matt Burgess <mattyb149@apache.org>
2016-08-12 11:30:06 -04:00
Mark Payne 76a4a2c48b
NIFI-2544: Created integration tests for clustering and addressed a few minor bugs that were found in doing so
This closes #832.

Signed-off-by: Bryan Bende <bbende@apache.org>
2016-08-11 10:43:38 -04:00
Mark Payne 42df02f014 NIFI-2406 This closes #820. Addressed regression introduced in NIFI-2406 where the cluster does not recognize a new Cluster Coordinator when the coordinator is shutdown 2016-08-09 15:19:49 -04:00
Bryan Rosander fa4c6ab03c
NIFI-2193 - Added functionality to automate certificate generation, keystore and truststore generation, and nifi.properties keystore and truststore password population. Follow-on changes will be made under NIFI-2476.
This closes #695.

Signed-off-by: Andy LoPresto <alopresto@apache.org>

Defaulting to same keyStore, key password (+18 squashed commits)
Squashed commits:
[9d01ba0] NIFI-2193 - Fixing typo
[55440bc] NIFI-2193 - Standalone can run as long as there are no conflicting files/folders
[0ca34ed] NIFI-2193 - Fixing some filename, absolute path issues
[9d4f65b] NIFI-2193 - Incorporating feedback
[f7550b4] NIFI-2193 - Cleaning up imports
[59a7637] NIFI-2193 - Updating umask to allow owner to execute
[cf824e7] NIFI-2193 - Moving DN arg to CA service specific parent class
[921ee13] NIFI-2193 - Making keystore getInstance more consistent
[a283c4b] NIFI-2193 - Updating sample config files in assembly to reflect new structure
[8d3a21d] NIFI-2193 - Making TlsHelper static, adding option to use same password for Key, KeyStore
[b13d247] NIFI-2193 - Addressing PR feedback
[46ef8ed] NIFI-2193 - Removing commons-logging, log4j from notice
[d4cf41a] NIFI-2193 - Adding option to specify output file for CA certificate when using cli client
[b74bf25] NIFI-2193 - Removing Bouncy Castle from notice
[6e34f9a] NIFI-2193 - Adding CLI client for easier generation of client certificates
[2924fca] NIFI-2193 - nifi-toolkit-ssl -> nifi-toolkit-tls, removing unused constants
[886167e] NIFI-2193 - Adding slf4j to avoid runtime issue
[082de46] NIFI-2193 - Command line SSL config utility as well as certificate authority client/server
2016-08-03 21:34:11 -07:00
Mark Payne 83a23f90d4
NIFI-2466: Added option to provide separate key password to StandardSSLContextService.
Fixed NPE (+2 squashed commits)
Squashed commits:
[c5d521a] NIFI-2466: Added unit test to verify changes; fixed validation
[aa4d418] NIFI-2446: Add option to specify key password when different than keystore password

This closes #776.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-08-03 15:56:12 -07:00
Koji Kawamura 809f042353 NIFI-2028: Fixed Site-to-Site Transit URI
Fixed Site-to-Site Transit URI for HTTP to be consistent with RAW socket.

- Removed url from CommunicationsSession since it's redundant as we have
  Peer.url, too. The value was not used from anywhere other than HTTP
Site-to-Site.
- Added createTransitUri method in Communicant interface, so that
  implementation can customize transitUri while providing consistent
interface.
2016-08-02 09:08:00 -04:00
joewitt 05a99a93cb NIFI-2208 This closes #754. refactored as per comments on JIRA. Reduced API expsosure and tightened lifecycle management. 2016-08-01 14:17:26 -04:00
Oleg Zhurakousky 1bf10944ea NIFI-2366 - Fixed ID generation semantics in clustered environment
- added SnippetUtilsTest
- renamed TypeOneUUIDGenerator to ComponentIdGenerator

- changed lsb part of ComponentIdGenerator back to long
- Fixed 'isCopy' condition for clustered environments

This closes #718.
2016-07-31 15:24:02 -04:00
Yolanda M. Davis 8412d2662a NIFI-2208 - initial commit Custom Property Expression Language support with Variable Registry, includes bug fix for NIFI-2057
This closes #529

Signed-off-by: jpercivall <joepercivall@yahoo.com>
2016-07-29 17:10:20 -04:00
Koji Kawamura b396867847 NIFI-2386 This closes #716. Site-to-Site fails without port no
It fails if a given URL doesn't have port in it.
This fixes its behavior with default http 80 and https 443 port.
2016-07-28 22:23:39 -04:00
Oleg Zhurakousky 6c7c3c0822 NIFI-826 (part 3)
- fixed ID generation routine that was causing miss-identification of the components
2016-07-19 15:52:35 -04:00
Oleg Zhurakousky 52a961873b NIFI-826 This closes #617. Added deterministic template support 2016-07-15 20:41:18 -04:00
Mark Payne c81dc1959a NIFI-1992:
- Updated site-to-site client and server to support clustered nifi instances
NIFI-2274:
- Ensuring we use the correct URI when updating a connection.

This closes #530
2016-07-15 16:13:59 -04:00
Aldrin Piri d1129706e2 NIFI-1896 This closes #650. Refactored nifi-api into nifi-framework-api and other locations. The nifi-api is specific to that which is needed for intended extension points. 2016-07-14 18:24:48 -04:00
Yolanda M. Davis 048ba5366c NIFI-2020 - initial commit for custom transformation support
NIFI-2020 - updates to use lambdas/stream wherever possible and fix potential nullpointer issue.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #564
2016-07-14 18:04:14 -04:00
Koji Kawamura 30889995cb NIFI-2145: Auto flow.xml archive
- Added following properties:
  - nifi.flow.configuration.archive.enabled
  - nifi.flow.configuration.archive.max.time
  - nifi.flow.configuration.archive.max.storage
- Removed manual archive operation:
  - Removed 'Back-up flow' link from UI since it's not needed any longer
  - Removed corresponding REST API controller/archive and its
    implementations
- Added FlowConfigurationArchiveManager to enclose archive related code
- Updated related docs
2016-07-14 10:35:16 -04:00
joewitt f987b21609 NIFI-1157 searched for and resolved all remaining references to deprecated items that were clearly addressable. 2016-07-14 09:32:35 -04:00
joewitt 961be21a38 NIFI-1157 resolved deprecated nifi-api items and ripple effects 2016-07-14 09:32:34 -04:00
Matt Gilman e0c96794fa NIFI-2095:
- Adding a page for managing users and groups.
- Adding a page for managing access policies.
- Renaming accessPolicy in entity to permissions to avoid confusion with the accessPolicy model.
- Adding an Authorizable for access policies.
- Refactoring access policies endpoints.
NIFI-2022:
- Implementing site to site authorizations.
2016-07-12 15:45:13 -04:00
Bryan Bende ba763b95e8 NIFI-2003 Creating abstract authentication provider and incorporating into existing providers
NIFI-2201 Add support for seeding cluster nodes in authorizations.xml
- Passing client address along in user context on authorization requests
- This closes #628
2016-07-12 11:20:29 -04:00
Andy LoPresto 4b9df7d1e2 NIFI-2186 Refactored CertificateUtils to separate logic for DN extraction from server/client sockets. Added logic to detect server/client mode encapsulated in exposed method.
Added unit tests for DN extraction.
Corrected typo in Javadoc.
Switched server/client socket logic for certificate extraction -- when the local socket is in client/server mode, the peer is necessarily the inverse.
Fixed unit tests.
Moved lazy-loading authentication access out of isDebugEnabled() control branch.
This closes #622
2016-07-11 23:15:28 -04:00
Matt Gilman ce5330330a NIFI-1781:
- Updating UI according to permissions through out the application.
- Shuffling provenance events, template, and cluster search REST APIs according to resources being authorized.
- Moving template upload controls.
- Removing username where appropriate.
- Addressing issues when authorizing flow configuration actions.
- Code clean up.
2016-07-01 15:10:27 -04:00
Mark Payne ae9e2fdf0b NIFI-2123: Add authorization of provenance events; refactor core classes so that Authorizable is located within nifi-api. This closes #592 2016-06-30 07:57:17 -04:00