Commit Graph

341 Commits

Author SHA1 Message Date
Matthew Burgess 5a84d650c3 NIFI-5449: Added Base64 Encode/Decode functions to RecordPath
NIFI-5449: Incorporated review comments

This closes #2920

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-09-14 14:04:25 -04:00
thenatog c9267347ed
NIFI-4426 - Replaced Java7 jBCrypt implementation which was made for Java7 backwards compatibility. It now uses a normal maven import to provide jBCrypt.
This closes #2976.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-08-30 12:15:44 -07:00
Matt Gilman f04cd8681d NIFI-5479: Using the SUN provider when the keystore type is JKS.
This closes #2961.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-28 15:28:30 +09:00
Andy LoPresto 744b15b4a7
NIFI-5540 Added unit test to demonstrate missing default sensitive properties key in flow election encryptor creation.
NIFI-5540 Added failing unit test and ignored regression test to demonstrate missing default sensitive properties key in flow election encryptor creation.

NIFI-5540 Added equality logic to StringEncryptor and utility equality methods to CryptoUtils.

NIFI-5540 Added default sensitive properties key population logic and log warning to StringEncryptor.

NIFI-5540 Cleaned up formatting.

NIFI-5540 Cleaned up boolean logic.

NIFI-5540 Added Javadoc to StringEncryptor.

NIFI-5540 Added unit test for StringEncryptor#equals().

NIFI-5540 Added performance benchmarking unit tests for constantTimeEquals methods for String, byte[], and char[].

NIFI-5540 Fixed checkstyle issue.

NIFI-5540 Fixed unit tests for default key population.

This closes #2959.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-08-22 13:56:16 -04:00
Jan Hentschel c2657ee043 NIFI-5423 Removed duplicated dependencies in sub-modules
This closes #2887

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-08-15 11:01:51 -04:00
thenatog 8106af699c
NIFI-5400 - Changed the hostname verifier from the custom NiFi verifier to the Apache http-client DefaultHostnameVerifier
Removed NiFiHostnameVerifier. Removed NiFi WebUtils usage of NiFiHostnameVerifier.
Added unit tests for the DefaultHostnameVerifier to WebUtils.java
Added groovy-eclipse-compiler definition to nifi-web-utils/pom.xml to execute Groovy unit tests.

This closes #2919.

Co-authored-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-08-06 19:57:49 -07:00
Matthew Burgess 3d546b8d87
NIFI-5484: Fixed PutHive3Streaming to use the Hive Metastore URI property (to include multiple URIs)
NIFI-5484: Incorporated review comments, added unit test for new validator

This closes #2934
2018-08-03 10:36:56 -04:00
Andy LoPresto e62aa0252d
NIFI-5442 Get X-ProxyContextPath value from request attributes rather than directly from headers.
NIFI-5442 Populate request contextPath attribute during AccessResource before displaying on message-page.jsp.
Refactored shared code from CatchAllFilter to WebUtils.
NIFI-5442 Refactored filter and context path code to shared parent filter and subclass.
NIFI-5442 Removed unnecessary initParams from nifi-web-ui web.xml.
NIFI-5442 Added explicit dispatchers to nifi-web-ui web.xml and removed unnecessary code from AccessResource.

This closes #2908
2018-08-01 12:10:14 -04:00
Mark Payne 72a5e6bfba NIFI-4407: Updated Expression Language Guide to provide details about how escaping $ is accomplished, and when the $ character should and should not be escaped. Fixed bug in the Query compiler that mistakenly would blindly replace 484 with $ even when the 484 did not precede an Expression. Added additional test cases.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2898.
2018-07-18 10:52:13 +02:00
Koji Kawamura e94f0757db
NIFI-4654: Support reporting RAS S2S lineage to Atlas
- Added 's2s.port.id' FlowFile attribute to track target remote Port id
- Use 's2s.port.id' to analyze RAW S2S provenance events
- This closes #2863
2018-07-10 11:44:00 -04:00
Andy LoPresto f60585a9b6
NIFI-5376 Removed deprecation warnings.
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.

This closes #2841.

Signed-off-by: Joe Witt <joewitt@apache.org>
2018-07-09 20:45:34 -07:00
patricker 3be08b7e49 NIFI-5344 Fix Write Ahead Log Unit Test on Windows
This closes #2819

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-28 22:39:45 +05:30
Andy LoPresto d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto 99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
zenfenan cf3c666683 NIFI-5286: Updated FasterXML Jackson libraries to 2.9.5
This closes #2775

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-09 14:19:45 -04:00
Mike Thomsen d2fd7e5e7d NIFI-5271 Moved JSON_VALIDATOR to its own maven module.
This closes #2765

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-06-07 20:38:25 +05:30
zenfenan 504152eaa1 NIFI-5261: Added JSON_VALIDATOR to StandardValidators
This closes #2758

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-06-05 18:24:47 -04:00
Mark Payne 06d1276f09 NIFI-5141: Updated regex for doubles to allow for numbers that have no decimal
NIFI-5141: Loosened regex for floating-point numbers to account for decimal place followed by 0 digits, such as '13.' and also added unit tests

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2679
2018-06-04 12:23:57 -04:00
Bryan Bende f69b720464
NIFI-5148 Refactoring Kerberos auth for Solr processors
- Created resuable KeytabUser and KeytabConfiguration in nifi-security-utils
- Refactored Solr processors to use a KeytabControllerService and no longer rely on JAAS system property
- Wrapped all calls in SolrProcessor onTrigger in a doAs when kerberos is enabled
- Added IT tests against MiniKDC
- This closes #2674
2018-05-30 14:22:28 -04:00
Mike Thomsen 716587d09f NIFI-5169 This closes #2705. Upgrade to JSONPath 2.4
NIFI-5169 Added explicit checking for empty strings in JsonPath expression property.

Signed-off-by: joewitt <joewitt@apache.org>
2018-05-23 16:16:25 -05:00
Matthew Burgess d79216d6b0 NIFI-5217: Allow non-nullable fields to be added to MockRecordParser
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2720
2018-05-18 15:59:12 -04:00
Mark Payne 604656fe88
NIFI-950: Make component validation asynchronous
NIFI-950: Still seeing some slow response times when instantiating a large template in cluster mode so making some minor tweaks based on the results of CPU profiling
NIFI-5112: Refactored FlowSerializer so that it creates the desired intermediate data model that can be serialized, separate from serializing. This allows us to hold the FlowController's Read Lock only while creating the data model, not while actually serializing the data. Configured Jersey Client in ThreadPoolRequestReplicator not to look for features using the Service Loader for every request. Updated Template object to hold a DOM Node that represents the template contents instead of having to serialize the DTO, then parse the serialized form as a DOM object each time that it needs to be serialized.
NIFI-5112: Change ThreadPoolRequestReplicator to use OkHttp client instead of Jersey Client
NIFI-5111: Ensure that if a node is no longer cluster coordinator, that it clears any stale heartbeats.
NIFI-5110: Notify StandardProcessScheduler when a component is removed so that it will clean up any resource related to component lifecycle.
NIFI-950: Avoid gathering the Status objects for entire flow when we don't need them; removed unnecessary code
NIFI-950: Bug fixes
NIFI-950: Bug fix; added validation status to ProcessorDTO, ControllerServiceDTO, ReportingTaskDTO; updated DebugFlow to allow for pause time to be set in the customValidate method for testing functionality
NIFI-950: Addressing test failures
NIFI-950: Bug fixes
NIFI-950: Addressing review feedback
NIFI-950: Fixed validation logic in mock framework
This closes #2693
2018-05-16 14:39:23 -04:00
Mark Payne 4700b8653d NIFI-5138: Bug fix to ensure that when we have a CHOICE between two or more REOCRD types that we choose the appropriate RECORD type when creating the Record in the JSON Reader.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2670.
2018-05-09 18:38:04 +02:00
Matt Gilman 5cfa29e48f
NIFI-5135:
- Adding support for applying transforms to user identities and group names.

This closes #2673.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-05-07 11:17:26 -04:00
greencee fdea876ede NIFI-5060 Updated SubstringAfter record processing to support multi-character search trimming
This closes #2623.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-04-25 15:40:20 -04:00
joewitt ac9944ccee
NIFI-5108 Updated all explicit refs and media nar usage of commons-compress to latest version and updated spring redis client
This closes #2651.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-04-24 15:16:04 -04:00
Matthew Burgess ba32879ec8 NIFI-5082: Added support for custom Oracle timestamp types to Avro conversion
This closes #2638

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-04-22 19:59:48 -04:00
Andy LoPresto 6d06defa63 NIFI-4942 [WIP] Added skeleton for secure hash handling in encrypt-config toolkit. Added test resource for Python scrypt implementation/verifier. Added unit tests.
NIFI-4942 [WIP] More unit tests passing.

NIFI-4942 All unit tests pass and test artifacts are cleaned up.

NIFI-4942 Added RAT exclusions.

NIFI-4942 Added Scrypt hash format checker. Added unit tests.

NIFI-4942 Added NiFi hash format checker. Added unit tests.

NIFI-4942 Added check for simultaneous use of -z/-y. Added logic to check hashed password/key. Added logic to retrieve secure hash from file to compare. Added unit tests (125/125).

NIFI-4942 Added new ExitCode. Added logic to return current hash params in JSON for Ambari to consume. Fixed typos in error messages. Added unit tests (129/129).

NIFI-4942 Added Scrypt hash format verification for hash check. Added unit tests.

NIFI-4942 Fixed RAT checks.

Signed-off-by: Yolanda Davis <ymdavis@apache.org>

This closes #2628
2018-04-13 18:25:09 -04:00
Koji Kawamura b2fae5f56e NIFI-5065: HTTP Site-to-Site fails sending data more than 2GB at once
Int variables got overflown.

This closes #5065

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-04-10 16:08:11 -04:00
Matthew Burgess b29304df79 NIFI-4857: Support String<->byte[] conversion for records
This closes #2570.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-04-09 14:53:21 -04:00
joewitt 59f625d3c2 Merge branch 'NIFI-4995-RC3' as part of NiFi 1.6.0 release process 2018-04-06 16:48:08 -07:00
Mike Moser dc9b4cb516
NIFI-3599 Allow back pressure object count and data size to be configurable in nifi.properties. This closes #2497 2018-04-05 13:50:16 -04:00
joewitt d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
Mark Payne fd31c161a2 NIFI-5026: Refactored StandardPreparedQuery so that instead of a List of Strings that may or may not correspond to compiled expressions and a Map of String to Compiled Expression, the StandardPreparedQuery now just takes a List of Expression objects, and those Expressions can be evaluated to return the proper result
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2592.
2018-03-29 10:54:25 +02:00
Derek Straka 5bdb7cf6e7 NIFI-4912: This closes #2494. Update jackson version to latest stable version (2.9.4)
Signed-off-by: joewitt <joewitt@apache.org>
2018-03-19 10:22:50 -04:00
Bryan Bende de71a41bd0 NIFI-4935 Refactoring to support specifying schema branch or schema version when using schema by name strategy
This closes #2523.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-15 16:16:12 -04:00
Bryan Bende 9b098f3235
NIFI-4864 - Additional improvements to additonal resource loading
This closes #2470.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-13 13:23:28 -04:00
zenfenan fad152f389
NIFI-4864 Fixing Additional Resources property pointing at a directory won't find new JARs
NIFI-4864: Code refactor and improved additionalUrlsFingerprint implementation
Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-13 13:23:16 -04:00
joewitt c71409fb5d
NIFI-4936 trying to quiet down the mvn output a bit so we dont exceed the travis-ci 4MB max
NIFI-4936 updated dependency handling pushing down delcarations where they belong
This closes #2512
2018-03-09 16:34:53 -05:00
Mike Moser 42e6fa42a3 NIFI-2630 Allow PublishJMS to send TextMessages
- Added configurable character set encoding for JMS TextMessages
- Improved PublishJMS/ConsumeJMS documentation
- Validate character set in property validator instead of OnScheduled
2018-03-02 09:11:58 -05:00
Mark Payne 9f95a10df9 NIFI-4794: Updated event writers to avoid creating a lot of byte[] by reusing buffers. Also removed synchronization on EventWriter when rolling over the writer and just moved the writing of the header to happen before making the writer available to any other threads. This reduces thread contention during rollover.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2437
2018-02-19 09:31:11 -05:00
Mark Payne 0bcb241db3 NIFI-4774: Implementation of SequentialAccessWriteAheadLog and updates to WriteAheadFlowFileRepository to make use of the new implementation instead of MinimalLockingWriteAheadLog.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2416
2018-02-19 09:26:01 -05:00
Andy LoPresto 28067a29fd
NIFI-4869 Added SAX utility method for SplitXML processor.
Added unit tests.
Added test resources to RAT exclude list.
This closes #2466
2018-02-13 15:29:29 -05:00
Andy LoPresto 336d3cf1f2
NIFI-4856 Removed deprecated ByteArrayInputStream references in ByteCountingInputStreamTest.
Added failing unit test for #available() at various states (initial, during read, after read).
Implemented #available() delegation.
All tests pass.

This closes #2461.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-02-09 10:49:58 -05:00
Koji Kawamura dbbf78f22c
NIFI-4848: Update HttpComponents version
- httpclient 4.5.3 -> 4.5.5
- httpcore 4.4.4 -> 4.4.9
  - ThreadSafe annotation is removed since 4.4.5, HTTPCLIENT-1743.
    Removed the annotation from DebugFlow processor.
- httpasyncclient 4.1.2 -> 4.1.3
- This closes #2453
2018-02-07 08:55:43 -05:00
Andy LoPresto 8cb09c301d
NIFI-4761
Added HostHeaderHandler regression unit tests before adding new functionality.
Added logic for parsing nifi.web.proxy.host property.
Added default property in nifi.properties and pom.xml.
Added logic for IPv6 and custom default hostnames.
Improved error messaging.
Added HostHeaderHandler unit tests.
Disabled HostHeaderSanitizationCustomizer in HTTP mode.
Fixed HTML escaping in error message.
Improved error message.
Added failing unit test for parsing custom hostnames.
Fixed custom hostname parsing.
Fixed unit tests.
Added TODO for IPv6 custom hostname parsing and unit test.
Added IPv6 custom hostname parsing and unit tests.
Fixed checkstyle issues.
Removed empty element in host list when no value defined.
Improved error message formatting.
Added unit tests.
Removed HostHeaderSanitizationCustomizer.
Removed InvalidPropertiesFormatException from NiFiProperties.
Removed InvalidPropertiesFormatException from HostHeaderHandler.
This closes #2415
2018-01-19 10:53:13 -05:00
Matthew Burgess ecb8067883 NIFI-4428: Replaced JSON input with RecordReader
using per-record flowfiles
NIFI-4428: Added initial L&N, fixed dependency hierarchy

NIFI-4428: Exposed additional ZK properties

NIFI-4428: More work to reduce dependencies in Druid artifacts

NIFI-4428: Removed inaccessible attribute

NIFI-4428: Fixed typo in property description

NIFI-4428: Updates to Druid NAR L&N

NIFI-4428: Updated LICENSE to refer to ICU License for icu4j
2018-01-16 13:08:02 -05:00
joewitt 41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00
joewitt 42edfa75b7 Merge branch 'NIFI-4751-RC1' 2018-01-12 15:00:32 -05:00