Commit Graph

3792 Commits

Author SHA1 Message Date
Bence Simon 83400789f6
NIFI-7292 Preventing file listing from fail because of insufficient privileges
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4195.
2020-04-15 18:54:51 +02:00
Rob Fellows c263daf20b
[NIFI-7358] - Fix: Sorting on 'Estimated Time to Back Pressure' in the Connection summary table does not work properly
- fix style issues
- review feedback

This closes #4208
2020-04-15 10:06:38 -04:00
sjyang18 9df53e7204
NIFI-6977 - Change the reporting behavior of Azure Reporting task to report report the time when metrics are generated
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4211.
2020-04-15 12:53:18 +02:00
Mark Payne a61a4c2b58
NIFI-7347: Fixed NullPointerException that can happen if a bin is merged due to timeout and has no records
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4210.
2020-04-15 12:39:13 +02:00
Andy LoPresto 07a8311b4c
NIFI-7319 Add walkthrough document (#4193)
* NIFI-7319 Added first draft of walkthroughs doc.

* NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance.

* NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate.

* NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates.

* NIFI-7319 Added instructions and screenshots for building NiFi from source.

* NIFI-7319 [WIP] Converting secure cluster instructions to match format.
Fixed instructions regarding embedded ZooKeeper configuration.

* NIFI-7319 Completed secure cluster walkthrough.

* NIFI-7319 Added walkthroughs to documentation navigation list.

* NIFI-7319 Incorporated PR feedback on broken links.

* NIFI-7319 Removed line number helpers from update sections.

* NIFI-7319 Incorporated final PR review items.

Co-authored-by: Sandra Pius <spiusapache@gmail.com>
2020-04-10 22:25:40 -07:00
Mark Payne c99a187018 NIFI-7346: Ensure that the Provenance Repository doesn't delete the Active Event File 2020-04-09 13:34:37 -04:00
Mark Payne 728bdec0f4 NIFI-7339: Fixed bug that caused Write Ahead Provenance Repository not to rollover event files after specified time. Code cleanup. Updated some default properties. 2020-04-09 13:32:34 -04:00
Matthew Burgess 84968e70d2
NIFI-7087: Use FlowManager.findAllConnections() when available
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4026
2020-04-09 12:59:01 -04:00
Mark Payne 683b1d9952
NIFI-6849: Reworked how nodes inherit cluster information when joining a cluster. Now, if there are conflicts, a local copy is made of the flow/authorizations/etc. and the cluster's flow is inherited.
- Refactored Flow Synchronization to make code cleaner
 - Updated Authorizers to forcibly inherit Users, Groups, and Access Policies if the local flow is empty.
 - Updated FlowFileRepositories to use SerializedRepositoryRecord instead of RepositoryRecord, so that we have the ability to read records without already knowing the Queue objects. Updated StandardFlowSynchronizer so that if the flow is not inheritable but the controller has not yet been initialized, the flow is backed up and replaced instead of NiFi failing to start
- Added system tests. Updated FlowController so that if it fails to inherit flow due to flow uninheritability that it notifies the cluster of this instead of remaining in the 'CONNECTING' state.
- Added additional log statements to aid in debugging

NIFI-6849: Rebased against master. Updated Admin Guide to describe new cluster flow inheritance behavior

NIFI-6849: Addressed review feedback

NIFI-6849: Addressed review feedback: Relocated logic for bundle compatibility into the BundleCompatibilityCheck class. Fixed logic that prevented users/groups/policies from being forcibly inherited during startup

This closes #3891
2020-04-09 10:42:05 -04:00
Peter Turcsanyi 08dcd4af14
NIFI-7345: Fixed Hive database and table names case insensitivity in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4198.
2020-04-09 14:13:54 +02:00
muazmaz 63379c3520 NIFI-7259 Adding DeleteDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage.
Updated to remove unused variables
NIFI-7259 import and property description changes

This closes #4189.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-08 10:58:48 +02:00
Tamas Palfy 09cece8e99 NIFI-7314 HandleHttpRequest stops Jetty in OnUnscheduled instead of OnStopped. Also reject pending request and clean their queue when shutting down.
NIFI-7314 In HandleHttpRequest returning 503 when rejecting pending requests before shutdown.
NIFI-7314 In HandleHttpRequest add logs and better response message during cleanup.

This closes #4191.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-07 20:12:18 +02:00
Joe Witt 445efcfde6
NIFI-7326 updated URL to find splunk artifacts (#4188)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-06 20:36:56 -07:00
Dustin Rodrigues f0e8daf45d
NIFI-7317 - make .java files non-executable
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4181.
2020-04-06 12:19:46 +02:00
Matthew Burgess a093af2d42
NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping (#4156)
* NIFI-7273: Add flow metrics REST endpoint with for Prometheus scraping

* NIFI-7273: Changed method name, fix handling when analytics not enabled

* NIFI-7273: Removed attachment header from Prometheus metrics endpoint

* NIFI-7273: Removed unused variable
2020-04-03 16:12:53 -04:00
Bence Simon c0f5fcb484 NIFI-7188 Extending UI search with filters and refactoring existing solution
This closes #4123.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-02 21:49:40 +02:00
Mike Thomsen 4a2a91135e
NIFI-7293 Add in-memory janusgraph implementation of GraphClientService to help with live testing.
Added new in memory janus graph client for testing.
Added integration test to ExecuteGraphQuery.

NIFI-7293 Added missing getter.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4168
2020-04-01 10:00:41 -04:00
muazmaz daddf400a2 NIFI-7103 Adding PutDataLakeStorage Processor to provide native support for Azure Data Lake Storage Gen 2 Storage.
added data-lake dependency
NIFI-7103 fixed indentation
Update to add IllegalArgumentException
Fixed indentation and logging
nifi-7103 review changes
nifi-7103 root directory and exception change

This closes #4126.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-03-31 11:15:07 +02:00
Mubashir Kazia fee1b8b8e0
NIFI-7294 Address deprecation issues in solrj and httpclient
Some calls to deprecated methods in httpclient were resulting in
UnsupportedOperationException. Use the new API calls in both httpclient
and solrj. Add an integration test to include test coverage for
org.apache.nifi.processors.solr.SolrUtils.createClient

This closes #4171.
2020-03-30 16:49:24 -04:00
Joe Witt 992d990dd2
NIFI-7291 updated enforcement of dependency rules and build command
Reviewed by markap14
This closes #4166.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-27 16:34:59 -04:00
Marc Parisi 1343b00e0b NIFI-7290: omit transitive that is causing the build to fail and not needed for the test scope 2020-03-27 13:12:13 -04:00
Matthew Burgess be2929b8b0
NIFI-7287: Move services-api dependency from Prometheus reporting task to its NAR
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4162
2020-03-27 08:48:41 -04:00
Tamas Palfy c79473baf5 NIFI-7286 ListenTCPRecord cleanup changed from @OnStopped to @OnUnscheduled 2020-03-26 11:45:09 -04:00
Troy Melhase 483f23a8aa
NIFI-7153 Adds ContentLengthFilter to enforce configurable maximum length on incoming HTTP requests.
Adds DoSFilter to enforce configurable maximum on incoming HTTP requests per second.
Redirected log messages for ContentLengthFilter to nifi-app.log in logback.xml.

This closes #4125.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-25 17:23:22 -07:00
Bryan Bende 975f2bdc4f
NIFI-7279 This closes #4160. Protect against NPE in RedisDistributedMapCacheClientService when value is null
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-25 17:40:20 -04:00
Bryan Bende 5784888082
NIFI-7281 This closes #4159. Use BufferedInputStream in StandardSocketChannelRecordReader in order to support mark/reset
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-25 17:39:44 -04:00
Michael Hogue aeaf953e06
NIFI-5925: Added controller services to set of components that are searched
NIFI-5925: cleanup, add negative test

NIFI-5925: fixed checkstyle

This closes #4105

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-03-25 08:55:44 -04:00
Bryan Bende 5fd25d6235
NIFI-7278 Adding support for SCRAM-SHA-512 to Kafka 2.0 processors 2020-03-24 16:53:22 -04:00
Mubashir Kazia 9b4292024b
NIFI-7269 - Upgrade solrj version to 7 in nifi-solr-processors
Remove unused imports

Use the latest solrj version(8.4.1)

Setup default schemaFactory for tests
The default schemaFactory ManagedIndexSchemaFactory creates
additional files in test's resources directory. Change it to
ClassicIndexSchemaFactory for classic behavior.

This closes #4152.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-03-24 14:55:51 -04:00
Bryan Bende 6412097eb2
NIFI-7257 Added HadoopDBCPConnectionPool
- Updated InstanceClassLoader to resolve files that are in the instance urls or additional urls
- Updated nifi-mock to support KerberosContext and removeProperty for ControllerServices
- Added unit test for HadoopDBCPConnectionPool
- Addressing review feedback

This closes #4149.
2020-03-24 11:05:18 -04:00
Bryan Bende 7f32aa56db
NIFI-7271 Make command timeout configurable for ShellUserGroupProvider
- Changing ShellRunner to use a separate thread for reading the output of the process
- Removing unused member variable
- Addressing review feedback

This closes #4154.
2020-03-23 20:43:06 -04:00
Joe Witt 254b20fa5f
NIFI-7274 add time for test conditions to be met 2020-03-22 09:54:43 -04:00
Bryan Bende 2feeb57159
NIFI-7221 Support v2 and v3 protocol version for Hortonworks Schema Registry
- Update nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-avro-record-utils/src/main/java/org/apache/nifi/serialization/SchemaRegistryRecordSetWriter.java
- Addressing review feedback

This closes #4120.
2020-03-20 10:56:47 -04:00
Pierre Villard 1fe79021b5
NIFI-7221 Initial work 2020-03-20 10:56:32 -04:00
Joe Witt f694e6464f NIFI-7187 adding missing version strings from accumulo bundle pom
- Removed Cat X JSON.org dep inclusion which seems to not be necessary
- Updated a ton of easier/safer looking deps
- Updated tika due to CVE

This closes #4086

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-03-20 10:07:56 -04:00
Eduardo Fontes 943de310ad
NIFI-6293
Add support to Mongo Extended JSON v2
Add org.json lib
Replace evil json
Replace evil json for alternative
Include testExtendedJsonSupport
Style adjustment
Remove unecessary new JSON parser
Fix query in testExtendedJsonSupport
Parse with Jackson and BSON
Back to default MONGO_URI

This closes #4068

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-03-18 18:36:09 -04:00
M Tien f91d6c420d
NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.fa… (#4151)
* NIFI-7268 Removed org.mindrot.jBcrypt library and replaced with at.favre.lib.bcrypt library.
Updated LICENSE and NOTICE files to reflect changes.
Updated unit tests.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

* NIFI-7268 Fixed typo in Javadoc.

Co-authored-by: Andy LoPresto <alopresto@apache.org>
2020-03-17 19:49:15 -07:00
Pierre Villard 7105bf36a8
NIFI-7267 - Upgrade spring-data-redis in Redis bundle (#4150)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-03-17 15:51:28 -07:00
Pierre Villard 71bf12976f
NIFI-7258 - fix overflow in PutAzureEventHub when not configured correctly
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4146.
2020-03-16 21:28:12 +01:00
Andy LoPresto 3feb85a030
NIFI-7223 Resolved merge conflicts from additional test case for null key password.
This closes #4145.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-16 12:47:41 -04:00
Nathan Gough 69b6c231fd
NIFI-7223 - Added another test for when keyPasswd is not present. 2020-03-16 12:28:18 -04:00
Andy LoPresto f9d75056fa
NIFI-7223 [WIP] Resolved compilation issues in unit test on OpenJDK 11 by removing Sun security class references.
Added OkHttpReplicationClient#isTLSConfigured() method.
Added unit test.
NIFI-7223 Fixed remaining unit tests for TLS regression.
Renamed tests for clarity.
2020-03-16 12:28:01 -04:00
Nathan Gough 7374361b5c
NIFI-7223 - Fixed a minor issue where the OkHttpReplicationClient class loaded blank properties as empty string instead of an expected null value. Added a isNotBlank check. Added unit tests for replication client and HTTPNotificationService.
NIFI-7223 - Renamed some variables and methods.
NIFI-7223 - Removed unused dependency. Corrected security properties in administration-guide.
2020-03-16 12:26:50 -04:00
Matthew Burgess ace5754524 NIFI-7208: Restore default timezone in JdbcCommon 2020-03-13 15:00:24 -04:00
Mark Payne 0ad58200af
NIFI-7256: This closes #4142. Fixed thresholds in unit test. Instead of assuming that multiple runs of the processor will occur within 100 milliseconds, allowed the multiple runs to occur within 3 mins of one another.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-03-13 14:05:40 -04:00
Joe Witt 561be89a21
NIFI-7250 activate user.timezone appropriate to each region
NIFI-7250 fix a test which appears brittle at least on windows builds on slow environments
NIFI-7250 activated a timezone run for AU Australia/Melbourne which exposed a poor magic number and needless assertion but interesting results worth keeping

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4140.
2020-03-13 17:30:19 +01:00
Matthew Burgess 798a8eeb50
NIFI-7249: Force String keys in maps in DataTypeUtils.inferDataType()
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4139.
2020-03-13 17:20:45 +01:00
Adam Taft decb5d062e
NIFI-6742 Use JUnit TemporaryFolder when creating test databases
- Add @Rule for TemporaryFolder
- Replace use of previous target/db with TemporaryFolder/db
- Remove use of ~/test db (in home directory)
- Remove System.out lines

Signed-off-by: Marc Parisi <phrocker@apache.org>

This closes #4137.
2020-03-12 20:48:52 -04:00
Joe Witt 97e250cdaa
NIFI-7244 Updated all tests which dont run well on windows to either work or be ignored on windows
Also dealt with unreliable tests which depend on timing by ignoring them or converting to IT.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4132.
2020-03-12 19:13:59 +01:00
Peter Turcsanyi dfaef38805
NIFI-7248: Atlas reporting task handles PutHive3Streaming
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4138.
2020-03-12 10:44:48 +01:00