Commit Graph

4462 Commits

Author SHA1 Message Date
Peter Toth a1794b101e NIFI-5041: Adds SPNEGO authentication to LivySessionController
NIFI-5041: fixes http client version issue

Change-Id: I1b87ec4752ff6e1603025883a72113919aba5dd4

NIFI-5041: fixes Kerberos configuration

Change-Id: I868fdf3ea7cfd28cf415164e420f23bf3f6eefeb

NIFI-5041: adds new NOTICE entries

NIFI-5041: yields processor if no session is available, fixes error handling in session manager thread, fixes error returned in KerberosKeytabSPNegoScheme on authentication failure

Change-Id: I443e063ae21c446980087e5464a4b70373d730f6

NIFI-5041: makes the session manager thread exceptions visible to the users

Change-Id: I33fde5df6933cec2a87a4d82e681d4464f21b459

NIFI-5041: adds special SessionManagerException to identify error occurred on session manager thread

Change-Id: I25a52c025376a0cd238f14bda533d6f5f3e5fb4a

This closes #2630

Signed-off-by: Matthew Burgess <mattyb149@apache.org>
2018-05-31 11:07:11 -04:00
Mike Thomsen 83fccc4beb NIFI-5024 Fixed a checkstyle issue that was missed in the merge of 5024. 2018-05-31 08:35:25 -04:00
Nicolas Sanglard 498fd8f22d NIFI-5024 Resolves deadlock in ExecuteStreamCommand processor
This closes #2594

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-31 06:35:39 -04:00
Bryan Bende f69b720464
NIFI-5148 Refactoring Kerberos auth for Solr processors
- Created resuable KeytabUser and KeytabConfiguration in nifi-security-utils
- Refactored Solr processors to use a KeytabControllerService and no longer rely on JAAS system property
- Wrapped all calls in SolrProcessor onTrigger in a doAs when kerberos is enabled
- Added IT tests against MiniKDC
- This closes #2674
2018-05-30 14:22:28 -04:00
Matt Gilman 9093d280f2 NIFI-5208:
- Removing test for unsupported functionality.

This closes #2740

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-30 13:41:20 -04:00
Matt Gilman 05a1d63090 NIFI-5208:
- Fixing missing message when a node is disconnected from a cluster.
- Updating endpoints to accept a flag to allow for changes to be made to a disconnected node.
- Updating custom UIs to acknowledge disconnected nodes prior to performing modifications.
2018-05-30 13:41:03 -04:00
Seiji Sogabe 3a248e96de NIFI-5245: SimpleCSVFileLookupService should take account of charset
This closes #2745

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-30 07:10:00 -04:00
Pierre Villard 1d21e3baf8 NIFI-5122 - Add Record Writer for S2S RTs
This closes #2663

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-30 06:58:01 -04:00
zenfenan 1663a6c094 NIFI-5133: Implemented Google Cloud PubSub Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2724.
2018-05-30 09:47:18 +02:00
Scott Aslan 9aa7e65f70
[NIFI-5215] Upgrade AngularJS to v1.7.0. Also, adding package-lock.json to source control
[NIFI-5215] add package-lock.json to rat excludes
This closes #2721
2018-05-29 13:20:49 -04:00
Mike Thomsen 688f906a73
NIFI-5244 Fixed a bug in MockSchemaRegistry that prevented it from loading using the schema name strategy.
This closes #2742.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-05-29 10:31:14 -04:00
Matthew Burgess 65beec39bb NIFI-5242: Exclude JTS dependency from Elasticsearch client
This closes #2741

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-27 19:00:34 -04:00
Mark Payne eb0b4283e8 NIFI-5222: Prevent validating components multiple times for each update
- Avoid triggering async validation for each update to component when instantiating a template (such as copy/paste or templates). Added debug logging to indicate when and why we are triggering validation; removed unit test that made poor assumptions about the inner workings of the FlowSynchronizer that resulted in failures when we make calls into processors that the unit test doesn't know about"

This closes #2731.
2018-05-25 13:44:15 -04:00
zenfenan 0973c2d8d1 NIFI-543 Added annotation to restrict processor to run only on the primary node
- PR Fix - 'Execution' dropdown will now be shown in all cases
- Annotated ListGCSBucket with PrimaryNodeOnly

This closes #2509.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-25 11:50:29 -04:00
Matt Gilman dbef69040f NIFI-5066:
- Fixing predicate for finding all processors/ports that can be disabled.

This closes #2739.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-25 11:22:21 -04:00
Matt Gilman 18ec386150 NIFI-5066:
- Ensuring we verify we can enable/disable when appropriate.
2018-05-25 11:22:21 -04:00
Pierre Villard 64cd34016b NIFI-5233 - Add EL support with Variable Registry scope for HBase client service
This closes #2738

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-25 10:37:45 -04:00
Bryan Bende cf6089196f NIFI-5229 Adding a DBCPService implementation that can lookup other DBCPServices dynamically at runtime
This closes #2735

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-25 08:54:03 -04:00
Mark Payne f15e6c7ab3 This closes #2037. 2018-05-24 16:03:36 -04:00
Mark Payne 397e88c858 NIFI-4227: Fixed typo 2018-05-24 15:57:28 -04:00
Pierre Villard be0ed70423 NIFI-4227 - add a ForkRecord processor
Added split/extract modes, unit tests, and additional details

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-24 15:55:17 -04:00
veteranbv 3b3d6d4eb2 NIFI-5171: fixed Yandex Jersey issues by adding dependency to POM and modified API call to now detect for languages
Added license to YandexTranslate NOTICE file
Updated to use StringUtils.isBlank for detecting sourceLanguage field being blank and languages to file to align with new logic

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-24 14:35:33 -04:00
Mark Payne 1597492fed NIFI-5073: Removed unneeded code for changing ClassLoader since it is now managed by framework
This closes #2653.
2018-05-24 11:30:39 -04:00
zenfenan db259628c7 NIFI-5073: JMSConnectionFactoryProvider now resolves EL Expression from VariableRegistry
- CLIENT_LIB_PATH is updated to include 'dynamicallyModifiesClasspath(true)'
2018-05-24 11:29:47 -04:00
Mike Thomsen 716587d09f NIFI-5169 This closes #2705. Upgrade to JSONPath 2.4
NIFI-5169 Added explicit checking for empty strings in JsonPath expression property.

Signed-off-by: joewitt <joewitt@apache.org>
2018-05-23 16:16:25 -05:00
zenfenan 06d45c3a6e
NIFI-4987: Added TTL to RedisDistributedMapCacheClientService
NIFI-4987: PR Review Fixes - Reverted getAndPutIfAbsent and added TTL setting with a different approach

NIFI-4987: PR Review Fixes - Added TTL to putIfAbsent()

This closes #2726.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-05-23 15:28:39 -04:00
Frederik Petersen d75ba167cd NIFI-5225: Purge event data from event repository when Connectable is removed
This closes #2732.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-23 13:39:20 -04:00
Mark Payne cca520aab7
NIFI-5228: Make the storing of file attributes as FlowFile attributes optional for ListFile
NIFI-5228: Addressed review feedback
This closes #2733
2018-05-23 10:50:01 -04:00
Mark Payne 167014441b
NIFI-5191: Revert "NIFI-4631: Use java.nio.file.Files in ListFile to improve performance"
This reverts commit 758e44682f.
2018-05-23 10:50:01 -04:00
Matthew Burgess 23feb8f429 NIFI-5230: Fixed NPE in InvokeScriptedProcessor on customValidate
This closes #2734

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-23 10:00:25 -04:00
Mike Thomsen 3a29c1e4ca NIFI-5052 Added DeleteByQuery ElasticSearch processor.
NIFI-5052 fixed checkstyle issue.

NIFI-5052 Added changes requested in a code review.

NIFI-5052 Fixed a typo.

NIFI-5052 Added changes requested in a code review.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2616
2018-05-23 09:14:04 -04:00
Koji Kawamura f7035f0497 NIFI-4637: Fix ExtensionManager warning on VisibilityLabelService
This closes #2729

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-23 07:34:49 -04:00
Max Viazovskyi dd8b25ab48 NIFI-5109 Reset justElectedPrimaryNode flag right after reelection happen
Read full cluster state and return if no update required

This closes #2657.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-05-23 08:52:49 +09:00
Andrew Lim 470d85042d NIFI-4585 Add nifi.cluster.node.max.concurrent.requests to Cluster Node Properties section of Admin Guide
This closes #2719

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-22 12:03:37 -04:00
Matt Gilman 0305ca0fb8 NIFI-5186:
- Addressing sort issue discovered during PR.

This closes #2722

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-22 11:05:35 -04:00
Matt Gilman 568288dbcd NIFI-5186:
- Updating UI to support showing when a component is validating.
2018-05-22 11:05:28 -04:00
Andrew Psaltis 6356d7b9ee NIFI-4988 This closes #2725. Changed the exception handling so that an invliad ZIP is handled. When an invalid zip is processed, the exception is an IllegalArgumentException which was not being handled and thus the session was being rollbacked.
Signed-off-by: joewitt <joewitt@apache.org>
2018-05-22 10:24:57 -04:00
Otto Fowler 6bf6a92abd NIFI-5220: add validation for host and password, and tests
This closes #2727.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-05-22 09:28:30 +09:00
Koji Kawamura 2834fa4ce4 NIFI-4199: Added ProxyConfigurationService
- Added ProxyConfigurationService to manage centralized proxy
configurations
- Adopt ProxyConfigurationService at FTP and HTTP processors

NIFI-4175 - Add HTTP proxy support to *SFTP processors

This closes #2018.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>

NIFI-4199: Add ProxyConfigurationService to SFTP processors

- Fixed check style issue
- Use the same proxy related PropertyDescriptors from FTPTransfer and
SFTPTransfer
- Dropped FlowFile EL evaluation support to make it align with other
processors spec, Now it supports VARIABLE_REGISTRY
- Added ProxyConfigurationService to SFTP processors
- Added SOCKS proxy support to SFTP processors

NIFI-4199: Added ProxyConfigurationService to ElasticsearchHttp processors

- ElasticsearchHttp processors now support SOCKS proxy, too
- Added proxy support to PutElasticsearchHttpRecord
- Moved more common property descriptors to
AbstractElasticsearchHttpProcessor and just return static unmodifiable
property descriptor list at each implementation processors

NIFI-4196 - Expose AWS proxy authentication settings

NIFI-4196 - Fix jUnit errors

This closes #2016.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>

NIFI-4199: Add ProxyConfigService to AWS processors

- Applied ProxyConfigService to S3 processors
- Added proxy support to following processors:
  - PutKinesisFirehose, PutKinesisStream
  - PutDynamoDB, DeleteDynamoDB, GetDynamoDB
  - PutKinesisStream
- All AWS processors support HTTP proxy now

NIFI-4199: Add ProxyConfigService to Azure processors

NIFI-4199: More explicit validation and docs for Proxy spec

- Each processor has different supporting Proxy specs
- Show supported spec to ProxyConfigurationService property doc
- Validate not only Proxy type, but also with Authentication

NIFI-4199: Incorporated review comments

- Fixed TestListS3 property descriptor check
- Separate name and displayName

This closes #2016
This closes #2018
This closes #2704

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-20 17:57:47 -04:00
Matthew Burgess d79216d6b0 NIFI-5217: Allow non-nullable fields to be added to MockRecordParser
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2720
2018-05-18 15:59:12 -04:00
Matthew Burgess 29e96ed0e0 NIFI-5145: Fixed MockPropertyValue to call the correct evaluateAttributeExpressions when FF is null
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2717
2018-05-17 21:33:46 -04:00
Mike Thomsen 6bb43bd471 NIFI-5145 Changed the override in MockPropertyValue to use a better way of passing over null flowfiles in evaluateExpressionLanguage.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2714
2018-05-17 20:00:24 -04:00
Andrew Lim 8aaa51af31 NIFI-5205 Updated default value for nifi.content.claim.max.appendable.size in Admin Guide
This closes #2716

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-17 15:09:37 -04:00
Mark Payne 2afbf96381
NIFI-5204: Ensure that verifyCanStop throws ISE if component is disabled
NIFI-5204: If processor joins cluster and inherits 'disabled' state but is still stopping, ensure that the state becomes disabled when the processor finishes stopping
This closes #2713
2018-05-17 13:44:52 -04:00
Mike Thomsen f5108ea839 NIFI-5197 Fixed a bunch of EL scope bugs.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2712
2018-05-17 09:57:35 -04:00
Matthew Burgess 5613bf4012 NIFI-5202: Fixed timing issue in TestListDatabaseTables
This closes #2709

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-17 07:10:35 -04:00
Jeff Storck 807e1e5c7a NIFI-5175 Updated NiFi compiled on Java 1.8 to run on Java 9
The bootstrap process (RunNiFi) detects Java 9 and adds "--add-modules=java.xml.bind" to the command to start NiFi
Updated OSUtils to detect Java 9 and reflectively invoke the Process.pid() method to get the PID of the NiFi process
Added java debug variable to nifi.sh to allow debugging of the bootstrap process (RunNiFi)

This closes #2708

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-17 07:00:46 -04:00
Andrew Lim 775cf42560 NIFI-2966 Add policy resources to Access Policies tables in Admin Guide
This closes #2710

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-16 16:18:52 -04:00
Mark Payne 604656fe88
NIFI-950: Make component validation asynchronous
NIFI-950: Still seeing some slow response times when instantiating a large template in cluster mode so making some minor tweaks based on the results of CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired intermediate data model that can be serialized, separate from serializing. This allows us to hold the FlowController's Read Lock only while creating the data model, not while actually serializing the data. Configured Jersey Client in ThreadPoolRequestReplicator not to look for features using the Service Loader for every request. Updated Template object to hold a DOM Node that represents the template contents instead of having to serialize the DTO, then parse the serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so that it will clean up any resource related to component lifecycle.
NIFI-950: Avoid gathering the Status objects for entire flow when we don't need them; removed unnecessary code
NIFI-950: Bug fixes
NIFI-950: Bug fix; added validation status to ProcessorDTO, ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause time to be set in the customValidate method for testing functionality
NIFI-950: Addressing test failures
NIFI-950: Bug fixes
NIFI-950: Addressing review feedback
NIFI-950: Fixed validation logic in mock framework
This closes #2693
2018-05-16 14:39:23 -04:00
Mike Thomsen ea41b41ede NIFI-5145 Made MockPropertyValue.evaluateExpressionLanguage(FlowFile able to accommodate null flowfiles the way live NiFi does.
NIFI-5145 Fixed a mistake with evaluateAttributeExpressions found in a code review.

NIFI-5145 Removed a property descriptor from GetHBase that was a duplication of the one in VisibilityFetchSupport.

NIFI-5145 Added comments in the code that were requested in the review.

NIFI-5145 Fixed check style issue.

NIFI-5145 Fixed a few improperly scoped items in ListFileTransfer that impacted the FTP processor(s).

NIFI-5145 Changed which override gets called based on code review feedback.

NIFI-5145: Reverted changes to ListFileTransfer re ExpressionLanguageScope

NIFI-5145: Removed TODO from MockPropertyValue javadoc

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2672
2018-05-16 13:50:27 -04:00