Commit Graph

4633 Commits

Author SHA1 Message Date
Matthew Burgess 23feb8f429 NIFI-5230: Fixed NPE in InvokeScriptedProcessor on customValidate
This closes #2734

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-23 10:00:25 -04:00
Mike Thomsen 3a29c1e4ca NIFI-5052 Added DeleteByQuery ElasticSearch processor.
NIFI-5052 fixed checkstyle issue.

NIFI-5052 Added changes requested in a code review.

NIFI-5052 Fixed a typo.

NIFI-5052 Added changes requested in a code review.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2616
2018-05-23 09:14:04 -04:00
Koji Kawamura f7035f0497 NIFI-4637: Fix ExtensionManager warning on VisibilityLabelService
This closes #2729

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-23 07:34:49 -04:00
Max Viazovskyi dd8b25ab48 NIFI-5109 Reset justElectedPrimaryNode flag right after reelection happen
Read full cluster state and return if no update required

This closes #2657.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-05-23 08:52:49 +09:00
Andrew Lim 470d85042d NIFI-4585 Add nifi.cluster.node.max.concurrent.requests to Cluster Node Properties section of Admin Guide
This closes #2719

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-22 12:03:37 -04:00
Matt Gilman 0305ca0fb8 NIFI-5186:
- Addressing sort issue discovered during PR.

This closes #2722

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-22 11:05:35 -04:00
Matt Gilman 568288dbcd NIFI-5186:
- Updating UI to support showing when a component is validating.
2018-05-22 11:05:28 -04:00
Andrew Psaltis 6356d7b9ee NIFI-4988 This closes #2725. Changed the exception handling so that an invliad ZIP is handled. When an invalid zip is processed, the exception is an IllegalArgumentException which was not being handled and thus the session was being rollbacked.
Signed-off-by: joewitt <joewitt@apache.org>
2018-05-22 10:24:57 -04:00
Otto Fowler 6bf6a92abd NIFI-5220: add validation for host and password, and tests
This closes #2727.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-05-22 09:28:30 +09:00
Koji Kawamura 2834fa4ce4 NIFI-4199: Added ProxyConfigurationService
- Added ProxyConfigurationService to manage centralized proxy
configurations
- Adopt ProxyConfigurationService at FTP and HTTP processors

NIFI-4175 - Add HTTP proxy support to *SFTP processors

This closes #2018.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>

NIFI-4199: Add ProxyConfigurationService to SFTP processors

- Fixed check style issue
- Use the same proxy related PropertyDescriptors from FTPTransfer and
SFTPTransfer
- Dropped FlowFile EL evaluation support to make it align with other
processors spec, Now it supports VARIABLE_REGISTRY
- Added ProxyConfigurationService to SFTP processors
- Added SOCKS proxy support to SFTP processors

NIFI-4199: Added ProxyConfigurationService to ElasticsearchHttp processors

- ElasticsearchHttp processors now support SOCKS proxy, too
- Added proxy support to PutElasticsearchHttpRecord
- Moved more common property descriptors to
AbstractElasticsearchHttpProcessor and just return static unmodifiable
property descriptor list at each implementation processors

NIFI-4196 - Expose AWS proxy authentication settings

NIFI-4196 - Fix jUnit errors

This closes #2016.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>

NIFI-4199: Add ProxyConfigService to AWS processors

- Applied ProxyConfigService to S3 processors
- Added proxy support to following processors:
  - PutKinesisFirehose, PutKinesisStream
  - PutDynamoDB, DeleteDynamoDB, GetDynamoDB
  - PutKinesisStream
- All AWS processors support HTTP proxy now

NIFI-4199: Add ProxyConfigService to Azure processors

NIFI-4199: More explicit validation and docs for Proxy spec

- Each processor has different supporting Proxy specs
- Show supported spec to ProxyConfigurationService property doc
- Validate not only Proxy type, but also with Authentication

NIFI-4199: Incorporated review comments

- Fixed TestListS3 property descriptor check
- Separate name and displayName

This closes #2016
This closes #2018
This closes #2704

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-20 17:57:47 -04:00
Matthew Burgess d79216d6b0 NIFI-5217: Allow non-nullable fields to be added to MockRecordParser
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2720
2018-05-18 15:59:12 -04:00
Matthew Burgess 29e96ed0e0 NIFI-5145: Fixed MockPropertyValue to call the correct evaluateAttributeExpressions when FF is null
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2717
2018-05-17 21:33:46 -04:00
Mike Thomsen 6bb43bd471 NIFI-5145 Changed the override in MockPropertyValue to use a better way of passing over null flowfiles in evaluateExpressionLanguage.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2714
2018-05-17 20:00:24 -04:00
Andrew Lim 8aaa51af31 NIFI-5205 Updated default value for nifi.content.claim.max.appendable.size in Admin Guide
This closes #2716

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-17 15:09:37 -04:00
Mark Payne 2afbf96381
NIFI-5204: Ensure that verifyCanStop throws ISE if component is disabled
NIFI-5204: If processor joins cluster and inherits 'disabled' state but is still stopping, ensure that the state becomes disabled when the processor finishes stopping
This closes #2713
2018-05-17 13:44:52 -04:00
Mike Thomsen f5108ea839 NIFI-5197 Fixed a bunch of EL scope bugs.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2712
2018-05-17 09:57:35 -04:00
Matthew Burgess 5613bf4012 NIFI-5202: Fixed timing issue in TestListDatabaseTables
This closes #2709

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-17 07:10:35 -04:00
Jeff Storck 807e1e5c7a NIFI-5175 Updated NiFi compiled on Java 1.8 to run on Java 9
The bootstrap process (RunNiFi) detects Java 9 and adds "--add-modules=java.xml.bind" to the command to start NiFi
Updated OSUtils to detect Java 9 and reflectively invoke the Process.pid() method to get the PID of the NiFi process
Added java debug variable to nifi.sh to allow debugging of the bootstrap process (RunNiFi)

This closes #2708

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-17 07:00:46 -04:00
Andrew Lim 775cf42560 NIFI-2966 Add policy resources to Access Policies tables in Admin Guide
This closes #2710

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-16 16:18:52 -04:00
Mark Payne 604656fe88
NIFI-950: Make component validation asynchronous
NIFI-950: Still seeing some slow response times when instantiating a large template in cluster mode so making some minor tweaks based on the results of CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired intermediate data model that can be serialized, separate from serializing. This allows us to hold the FlowController's Read Lock only while creating the data model, not while actually serializing the data. Configured Jersey Client in ThreadPoolRequestReplicator not to look for features using the Service Loader for every request. Updated Template object to hold a DOM Node that represents the template contents instead of having to serialize the DTO, then parse the serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so that it will clean up any resource related to component lifecycle.
NIFI-950: Avoid gathering the Status objects for entire flow when we don't need them; removed unnecessary code
NIFI-950: Bug fixes
NIFI-950: Bug fix; added validation status to ProcessorDTO, ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause time to be set in the customValidate method for testing functionality
NIFI-950: Addressing test failures
NIFI-950: Bug fixes
NIFI-950: Addressing review feedback
NIFI-950: Fixed validation logic in mock framework
This closes #2693
2018-05-16 14:39:23 -04:00
Mike Thomsen ea41b41ede NIFI-5145 Made MockPropertyValue.evaluateExpressionLanguage(FlowFile able to accommodate null flowfiles the way live NiFi does.
NIFI-5145 Fixed a mistake with evaluateAttributeExpressions found in a code review.

NIFI-5145 Removed a property descriptor from GetHBase that was a duplication of the one in VisibilityFetchSupport.

NIFI-5145 Added comments in the code that were requested in the review.

NIFI-5145 Fixed check style issue.

NIFI-5145 Fixed a few improperly scoped items in ListFileTransfer that impacted the FTP processor(s).

NIFI-5145 Changed which override gets called based on code review feedback.

NIFI-5145: Reverted changes to ListFileTransfer re ExpressionLanguageScope

NIFI-5145: Removed TODO from MockPropertyValue javadoc

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2672
2018-05-16 13:50:27 -04:00
Mark Payne 0688363d87 NIFI-5200: Fixed bug that caused the wrong InputStream to be closed by StandardProcessSession if calling Session.read() from the callback of another Session.read(); also changed default of the 'allowSessionStreamManagement' flag from 'false' to 'true' as it will provide performance benefits in some cases and NIFI-516 outlined that it should be 'true' initially but ended up being false when the PR was merged.
This closes #2707

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-16 12:27:19 -04:00
James Wing 10254a03c2 NIFI-4015 NIFI-3999 Fix DeleteSQS Issues
* Avoid exception by providing id in DeleteMessageBatchRequestEntry
* Include receipt handle property descriptor for user configuration

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1888.
2018-05-16 18:15:20 +02:00
Matt Gilman c3059939e8 NIFI-5173:
- Removing unnecessary logic in the zoom handler since the zoom event is no longer triggered during onClick.

This closes #2692

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-16 11:12:16 -04:00
Koji Kawamura de1ad3eb6f NIFI-4198: ElasticsearchHttp processors Proxy settings
- Added Expression Language support with variable registry at proxy user
and password
2018-05-16 14:02:28 +09:00
Arun Manivannan 01a01a3dfc NIFI-4198 *ElasticsearchHttp processors do not expose Proxy settings
This closes #2094.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-05-16 14:01:19 +09:00
Otto Fowler ccedc71c8b NIFI-5077 ExtractGrok support for `keep empty captures`
Support for the new option to keep empty captures.  I did not add to the GrokReader because I am not sure of the effect on
the schema extraction.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2698.
2018-05-15 19:24:10 +02:00
JohannesDaniel 2c8c9374af NIFI-5113: Add XMLRecordSetWriter
This closes #2675.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-15 11:07:19 -04:00
Mark Payne 620d446860 NIFI-5194: Ensure that even if calling KafkaConsumer.resume() throws an Exception, that we still release the lock that we are holding
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2701.
2018-05-15 11:52:55 +02:00
Koji Kawamura 79a7014a95 NIFI-4971: ReportLineageToAtlas complete path can miss one-time lineages
- Separated Hook message de-duplication logic from NiFiAtlasHook to
NotificationSender
- NiFiAtlasHook used to send individual CREATE_ENTITY messages for each entities,
this commit changed it to bundle all new entities into a single
CREATE_ENTITY to preserve entity creation order, so that new DataSet
entities can be referred from new nifi_flow_path entities
- Added more unit tests

This closes #2542

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-15 05:50:26 -04:00
Fokko Driesprong ed9263811b [NIFI-5190] Bump Apache Parquet to 1.10.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2697.
2018-05-15 11:40:11 +02:00
JohannesDaniel 04745a556a NIFI-5189 Schema name is not available for RecordSchema
This closes #2700

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-14 14:35:12 -04:00
Koji Kawamura 03adaeca22 NIFI-5114: Added Basic auth support to WebSocket components
This closes #2652

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-14 14:26:47 -04:00
Otto Fowler 61fe493786 NIFI-5170 Upgrad Grok to version 0.1.9
This closes #2691

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-13 08:40:20 -04:00
Gardella Juan Pablo 64356e0014 NIFI-5049 Fix handling of Phonenix datetime columns
This closes #2625

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-11 09:51:24 -04:00
Matthew Burgess 099bfcdf3a NIFI-5121: Added DBCPService API method for passing in flow file attributes when available
This closes #2658

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-05-11 08:19:16 -04:00
Koji Kawamura 500a254e3f NIFI-4637: HBase visibility lables
Fixed additional docs directory structure.
2018-05-11 07:30:19 +00:00
Mike Thomsen 0b851910f3 NIFI-4637 Added support for visibility labels to the HBase processors.
NIFI-4637 Removed integration test and updated the hbase client from 1.1.2 to 1.1.13 which is the final version for 1.1.X

NIFI-4637 Fixed EL support issue w/ tests.

NIFI-4637 Added more documentation to DeleteHBaseCells.

NIFI-4637 changed PutHBaseCell/JSON to use dynamic properties instead of a 'default visibility string.'

NIFI-4637 Added changes requested in a code review.

NIFI-4637 Moved pickVisibilityString to a utility class to make testing easier.

NIFI-4637 Added additionalDetails.html for PutHBaseRecord.

NIFI-4637 Added additional documentation and testing.

NIFI-4637 Added documentation for DeleteHBaseCells.

NIFI-4637 Added pickVisibilityLabel support to PutHBaseRecord and updated documentation to reflect that.

NIFI-4637 Reverted version bump to hbase client.

This closes #2518.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-05-11 00:55:40 +00:00
himanshu 6e0be8e641 NIFI-4938 Upgraded org.eclipse.paho.client.mqttv3 dependency version to 1.2.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2514.
2018-05-10 11:04:32 +02:00
Mark Bean def9741a39 NIFI-5180: update JMS additional details to set Destination Type to Required, default 'QUEUE'
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2694.
2018-05-10 10:47:20 +02:00
Matt Gilman 22342a0e0c NIFI-5167:
- Updating how the nf-reporting-task module is injected to the nf-controller-service module.

This closes #2688

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2018-05-09 14:06:19 -04:00
Andy LoPresto 4f1444c0e0 NIFI-4942 This closes #2690. Resolved test failures in JCE limited mode.
Signed-off-by: joewitt <joewitt@apache.org>
2018-05-09 13:52:05 -04:00
Junegunn Choi 02ba4cf2c8 NIFI-5064 - Fixes and improvements to PutKudu processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2621.
2018-05-09 19:39:29 +02:00
patricker 0a44bad76e NIFI-5168 - ReplaceText Processor Should Use Single FlowFile Processing Instead of Batch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2687.
2018-05-09 18:46:35 +02:00
Mark Payne 4700b8653d NIFI-5138: Bug fix to ensure that when we have a CHOICE between two or more REOCRD types that we choose the appropriate RECORD type when creating the Record in the JSON Reader.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2670.
2018-05-09 18:38:04 +02:00
zenfenan f742a3a6ac NIFI-5156: Updated GCP SDK to latest version
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2680.
2018-05-09 18:19:17 +02:00
Andy LoPresto 7a4990e7fe
NIFI-5146 Only support HTTP or HTTPS operation for NiFi API/UI
- Added logic to check for simultaneous configuration of HTTP and HTTPS connectors in JettyServer.
- Added test logging resources. Added unit tests.
- Refactored shared functionality to generic method which accepts lambdas.
  Fixed unit test with logging side effects.
- Added note about exclusive HTTP/HTTPS behavior to Admin Guide. Fixed typos.

This closes #2683.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-05-09 12:14:11 -04:00
Bryan Bende 92b4a3208f NIFI-5136 Ensure processor references are removed from LogRepository and from ProcessScheduler
- Forcing FileSystem statistics thread to be interrupted when HDFS processors are stopped
- Stop creating temp components during import from registry, use bundle info instead

This closes #2668.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-05-08 15:29:37 -04:00
thenatog 2094786ec8
NIFI-5161 - Moved filename escaping method to TlsHelper.java to allow use by the different Tls modes.
Added another test for special characters in the DN/output key filename.
Added a method to escape special characters in the alias name for keys in the truststore. This fixes an error with the TlsToolkit which occurs when extracting keys and writing them to file.

This closes #2684.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-05-07 14:07:20 -07:00
Bryan Bende a9e85c358c
NIFI-5163 Clearing version control info when creating a template
This closes #2685.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-05-07 16:13:26 -04:00