2910 Commits

Author SHA1 Message Date
Mark Payne
2a964681ec NIFI-5514: Fixed bugs in MergeRecord around minimum thresholds not being honored and validation not being performed to ensure that minimum threshold is smaller than max threshold (would previously allow min record = 100, max records = 2 as a valid configuration)
NIFI-5514: Do not rely on ProcessSession.getQueueSize() to return a queue size of 0 objects because if the processor is holding onto data, the queue size won't be 0.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2954.
2018-09-24 22:36:02 +02:00
Andrew I. Christianson
be2ed060a2
NIFI-5542 Added support for node groups to FileAccessPolicyProvider
This closes #2970.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-09-20 10:27:28 -04:00
Matthew Burgess
2ee0af9663 NIFI-5614: Update commons-dbcp to commons-dbcp2 for DBCPConnectionPool
This closes #3014

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-20 08:17:52 -04:00
Mark Payne
73a7c8c6b4 NIFI-5613: Upgraded version of geoip2 library to version 2.12.0; fixed unit tests
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3013.
2018-09-20 13:55:13 +02:00
Kemix Koo
8cf0662e5b NIFI-5615: set the avro-binary MIME type for converting processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3016.
2018-09-20 12:07:43 +02:00
Pierre Villard
854ee3021c NIFI-5599 - Bump Kafka versions
This closes #3006

Signed-off-by: zenfenan <sivaprasanna246@gmail.com>
2018-09-20 14:33:23 +05:30
Mark Payne
3dd548e807 NIFI-5609: Fixed NullPointer when attempting to view status history for a component that has not yet run
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3012
2018-09-19 16:56:42 -04:00
Koji Kawamura
f570cb980d
NIFI-375: Added operation policy
The operation policy allows that a user to operate components even if they does not have direct READ/WRITE
permission of the component.

Following operations are controlled by the new operate policy:
- Start/stop/enable/disable Processors, ControllerServices,
ReportingTasks, Input/OuputPorts
- Enable/disable transmission of RemoteInput/OutputPorts and
RemoteProcessGroups
- Terminate Processor threads

Refactored what API exposes

The previous commit let API exposes few fields in DTO. But we should
avoid returning partial DTO as it complicates authorization logic.

Instead, this commit adds StatusDTO for ReportingTaskEntity and
ControllerServiceEntity, so that it can be returned regardless of having
READ permission. Component DTO can only be returned with a READ
permission.

Refactor RPG same as ControllerService.

WIP incorporating review comments.

Incorporated review comments

- Cleaned up merger classes
- Recreate DTO instance at each function during two phase commmit

Restrict enabling ControllerService without read permission

Revert the last commit.

Fix review comments.

- Renamed confusing static method names and its parameters
- Removed unnecessary permission checks from UI condition

Fixed delete action display condition.

Fixed NPE at Summary.

Apply operation policy to activateControllerServices.

Removed OperationPermissible from ComponentEntity.

This closes #2990
2018-09-19 15:28:41 -04:00
“PrashanthVenkatesan”
9161b1787a NIFI-5327 Adding Netflowv5 protocol parser
NIFI-5327 Formated Code to adhere NiFi checkstyle
NIFI-5327 Incorporated code review comments
NIFI-5327 Update additionalDetails docs
NIFI-5327 Format IPV4 fields & rename nar
NIFI-5327 Update dependency jar version
NIFI-5327 Updated Code review comments
NIFI-5327 Updated Code review comments
NIFI-5327 Patch for code checkstyle violations
NIFI-5327 commit for retriggering build

This closes #2820

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-19 13:15:41 -04:00
Mark Payne
ad80f5f064 NIFI-5598: Allow JMS Processors to lookup Connection Factory via JNDI
NIFI-5598: Expose JNDI Principal & Credentails as explicit properties
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3005
2018-09-19 12:56:58 -04:00
Mark Payne
0da4f50ea0 NIFI-5518: Added processors for integrating with Apache Kafka 2.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2948.
2018-09-19 15:50:08 +02:00
Mike Thomsen
1dea8faa06 NIFI-5239 Made a client service an optional source of connection pooling in Mongo processors.
This closes #2896

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-09-19 17:04:25 +05:30
zenfenan
c56a7e9ba5 NIFI-5510: Introducing PutCassandraRecord processor
NIFI-5510: Fixes for PR review comments

This closes #2992

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-18 08:10:14 -04:00
Otto Fowler
7e627f2fed
NIFI-5566 Added unit test for PR feedback.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-09-17 17:12:53 -07:00
Andy LoPresto
cd6877404b
NIFI-5147, NIFI-5566 Added CryptographicHashAttribute and CryptographicHashContent processors.
Deprecated HashContent processor.
Added documentation to HashAttribute processor.
Added shared HashService and HashAlgorithm enum.
Added unit tests.
Added #clearProperties() to TestRunner, StandardProcessorTestRunner, and MockProcessContext.
Updated processor manifest.
Updated Javadoc.
Added documentation about deprecated/renamed components.
Added logic handling for UTF-16 encoding where UTF-16BE is overridden so the BOM is not inserted prior to hashing.

This closes #2983.

Co-authored-by: Otto Fowler <ottobackwards@gmail.com>
Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-09-17 17:03:57 -07:00
Mohit Garg
07ad132a2b NIFI-5596 Upgraded splunk-sdk-java version
NIFI-5596 Updated L&N

L&N commit added by Mike Thomsen

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-17 12:52:17 -04:00
Mark Payne
c8fc1327ee NIFI-5592: If an Exception is thrown by RecordReader.read() from ConsumeKafkaRecord, route Record to parse.failure relationship
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3001.
2018-09-15 14:49:24 +02:00
uday
57ae9b65a0 NIFI-1184 Mock FileSystem for PutHDFSTest
This closes #2892

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-14 13:30:03 -04:00
Matthew Burgess
4a25402c1a NIFI-3425: Provide ability to cache CQL statements
This closes #2986.

Co-authored-by: Michael A Giroux <magirou@rd6ul-92373g.infosec.tycho.ncsc.mil>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-09-14 11:37:18 -04:00
Bryan Bende
14729be837 NIFI-5577 Fixing update method in NaiveRevisionManager so that failed updates don't change the revision
This closes #2995.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-09-14 11:33:53 -04:00
Mark Payne
647152ab16
NIFI-5597: Restored FreeFormTextRecordSetWriter to the META-INF services file
This closes #3004.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-09-14 09:52:39 -04:00
Vijeta
e0f7f244b8 Clarifying upsert query documentation for PutMongo processor.
This closes #2999

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-12 17:46:43 -04:00
Andy LoPresto
7e8b7752ca NIFI-5569 Added keywords to Route* and ScanAttribute processors to improve discoverability.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2984.
2018-09-11 18:09:48 +02:00
Mark Payne
0274bd4ff3 Revert "NIFI-4272 support multiple captures when EL is present in replacement value"
This reverts commit f7f809c3d3632eea5234b31740984b73de322464.

NIFI-5474, NIFI-4272: When using Regex Replace with ReplaceText, and there are capturing groups, ensure that we populate the 'additionalVariables' map for each match of the regex. This allows Expression Language to reference the back-references properly even when there are multiple matches

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2951
2018-09-11 11:07:33 -04:00
Mark Payne
87cf474e54
NIFI-5581: Disable connection pooling for OkHttpReplicationClient. We should revisit this in the future, but for now, it appears that Jetty is having problems with the connections if they are reused. By disabling the Connection Pooling, we address the concern, but for secure connections this means that every request results in a TLS handshake - and for a mutable request, both the verification and the 'performance' stages require the TLS handshake. But it's better than timing out, which is the currently observed behavior
This closes #2996
2018-09-10 09:43:53 -04:00
Mike Thomsen
57820d0d88 NIFI-5495 Made date format configurable.
This closes #2969

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-09-09 12:33:34 +05:30
thenatog
fc1461298a
NIFI-5366 - Added ContentSecurityPolicyFilter which stops framing of NiFi resources. It applies the Content-Security-Policy header. This protects against clickjacking.
NIFI-5366 - Added unit test. Added single quotes around 'self' for frame-ancestors CSP header.
NIFI-5366 - Fixed dependencies.

This closes #2989.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-09-05 17:38:53 -07:00
zenfenan
4f0e2f5562 NIFI-5456: AWS clients now work with private link endpoints (VPC)
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2968.
2018-09-05 15:04:29 +02:00
Lars Francke
1b73578c48 NIFI-5408 Add documentation for NIFI-4279
This closes #2878

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-08-31 19:09:58 -04:00
zenfenan
05e32cff28 NIFI-5544: GetMongo refactored
NIFI-5544: PR Review changes

This closes #2958

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-08-31 15:32:44 -04:00
thenatog
97e0f6a6a7
NIFI-4558 - Set JKS as the default keystore type and truststore type.
This closes #2982.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-08-31 10:35:21 -07:00
Koji Kawamura
5a58c9a171
NIFI-5282: Add ProxyConfigurationService to GCSProcessors
This closes #2981
2018-08-31 11:07:09 -04:00
jugi92
cdae2b14b3 NIFI-5282: GCPProcessor with HTTP Proxy with Auth
added http proxy support with authentication for GCP processors
added proxy support for Google Credential Service

This closes #2943.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-31 20:28:00 +09:00
Andy LoPresto
8f37b5ee10
NIFI-4426 Removed exclude from nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/pom.xml. 2018-08-30 13:05:09 -07:00
Pierre Villard
cbd942df10
NIFI-5561 - Add component name filtering to S2S Provenance Reporting Task.
Added regression test for ProvenanceEventConsumer#isFilteringEnabled().
Changed isFilteringEnabled implementation to be expandable as other attributes are added using Streams.
EL + indentation.

This closes #2973.

Co-authored-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-08-30 11:30:10 -07:00
Corey Fritz
89186fb96d NIFI-5388: enabled EL support for dynamic properties of Kafka 1.0 processors
This closes #2915

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-08-30 07:08:00 -04:00
Mike Thomsen
c755ed9322 NIFI-5413 Added additional documentation for PutMongoRecord and fixed a typo in RunMongoAggregation's documentation.
This closes #2881

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-08-30 09:43:44 +05:30
Andrew Lim
860663ae3e
NIFI-5526 Improve PutS3Object processor documentation
This closes #2975.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-08-29 15:07:22 -07:00
Andy LoPresto
e83ea1f9ae
NIFI-5558 Fixed unit test to avoid contamination from System property "nifi.properties.file.path" when creating test NiFiProperties instance.
This closes #2972.

Co-authored-by: thenatog <thenatog@gmail.com>
Signed-off-by: Marc Parisi <phrocker@apache.org>
2018-08-29 14:52:07 -07:00
Matthew Burgess
8a8a8cb072 NIFI-5553: Fixed issue with rollback on result set errors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2967.
2018-08-29 13:22:10 +02:00
Matthew Burgess
53969adcc8 NIFI-5353: Add JoltTransformRecord processor 2018-08-28 14:32:27 -04:00
Matt Gilman
f04cd8681d NIFI-5479: Using the SUN provider when the keystore type is JKS.
This closes #2961.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-28 15:28:30 +09:00
joewitt
a27ccd8a56 NIFI-5479 Upgraded Jetty. Moved where we unpack bundled deps to so we can avoid a new jetty bug with META-INF loading logic.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-28 15:28:11 +09:00
Mark Bean
58cb900b90
NIFI-5543: move bendPointDrag below mousedown.selection in nf-connection.js
This closes #2957
2018-08-27 11:04:07 -04:00
Mark Payne
6ceee25859 NIFI-5551: If session.get() returns null, then return fron onTrigger instead of assuming that the FlowFile is non-null
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2964.
2018-08-27 16:52:07 +02:00
Mike Moser
8c0705cb6b NIFI-5275 PostHTTP SocketConfig setup, fixed connection pool when using HTTPS, setup idle connection checker, setup request retry handler, improved some exception handling and logging, and NIFI-1336 fix as well
This closes #2796.
Signed-off-by: Brandon Devries <devriesb@apache.org>
2018-08-24 10:07:11 -04:00
Matthew Burgess
cfc858c901 NIFI-5517 - Add support for remaining Hive types to PutHive3Streaming
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2950.
2018-08-24 11:09:05 +02:00
Mark Payne
aac2c6a60e
NIFI-5482: Made WriteAheadProvenanceRepository the default implementation
This closes #2960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-08-23 16:49:55 -07:00
Andy LoPresto
744b15b4a7
NIFI-5540 Added unit test to demonstrate missing default sensitive properties key in flow election encryptor creation.
NIFI-5540 Added failing unit test and ignored regression test to demonstrate missing default sensitive properties key in flow election encryptor creation.

NIFI-5540 Added equality logic to StringEncryptor and utility equality methods to CryptoUtils.

NIFI-5540 Added default sensitive properties key population logic and log warning to StringEncryptor.

NIFI-5540 Cleaned up formatting.

NIFI-5540 Cleaned up boolean logic.

NIFI-5540 Added Javadoc to StringEncryptor.

NIFI-5540 Added unit test for StringEncryptor#equals().

NIFI-5540 Added performance benchmarking unit tests for constantTimeEquals methods for String, byte[], and char[].

NIFI-5540 Fixed checkstyle issue.

NIFI-5540 Fixed unit tests for default key population.

This closes #2959.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-08-22 13:56:16 -04:00
Mark Payne
5b57935bae NIFI-5527: Refactored unit test to make member variables for 'delay tolerances' so that they are easier to update in the future. Increased delay tolerances so that tests pass on heavily loaded system.
This closes #2955.
2018-08-22 10:51:47 -04:00