Commit Graph

212 Commits

Author SHA1 Message Date
vvaks 7fa0a34aba NIFI-4428: - Implement PutDruid Processor and Controller
update

added provenance report

added parameters for batch control

WIP
2018-01-16 13:07:56 -05:00
joewitt 41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00
joewitt 36405e888c NIFI-4751-RC1 prepare for next development iteration 2018-01-08 23:39:49 -07:00
joewitt 46d30c7e92 NIFI-4751-RC1 prepare release nifi-1.5.0-RC1 2018-01-08 23:39:32 -07:00
Matt Gilman 20b539aac3
NIFI-4436:
- Bumping registry version to 0.1.0.
- Addressing PR comments.
2018-01-08 12:44:56 -05:00
Mark Payne 6b00dff1a8
NIFI-4436: Integrate with actual Flow Registry via REST Client - Store Bucket Name, Flow Name, Flow Description for VersionControlInformation - Added endpoint for determining local modifications to a process group - Updated authorizations required for version control endpoints - Add state and percent complete fields ot VersionedFlowUpdateRequestDTO
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2018-01-08 12:44:53 -05:00
Mark Payne 6a58d780d7
NIFI-4436:
- Initial checkpoint: able ot start version control and detect changes, in standalone mode, still 'crude' implementation
- Checkpoint: Can place flow under version control and can determine if modified
- Checkpoint: Change version working in some cases. Does not work if processor removed because COMPONENT_REMOVED type has ComponentA whose ID is the VersionedComponentID but we are trying to call ProcessorDAO.get() with this ID
- Checkpoint: Able to change flow from Version 1 to Version 2 and back. Not yet tested with controller services. Have not tried changing/removing connections. Not cluster-friendly yet. All inline, not in background. Have not taken into account ports, funnels, remote ports, etc. Have not tested with Labels yet
- Checkpoint after implementing ClusterReplicationComponentLifecycle instead of JerseyClientComponentLifecycle
- Checkpoint: Updated to allow starting version control and updating version in clustered mode
- Checkpoint: Updated versioning endpoint so that when version of a flow is updated, the bundle information is populated and the snapshot is replicated to the cluster.
- Checkpoint: Implemented endpoint for reverting to previously sync'ed version of a flow and updated version control endpoint so that Process Group can be pushed as a new version to existing flow instead of only creating a new flow
- Checkpoint: Updated so that if a Process Group is under Version Control and it has a child Process Group, which is also under Version Control, we can handle that gracefully. Not yet tested because it depends on updates to the nifi-registry module, which can't be compiled due to maven dependency conflicts
2018-01-08 12:44:52 -05:00
joewitt 6ee6b5e57c NIFI-4734 refactored couchbase bundle, jms bundle, removed jms test-lib usage, fixed broken SeeAlso and service links causing docs warns. Corrected L&N in each bundle. This closes #2369.
Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-01-04 09:16:17 -05:00
joewitt 7467bb7b0f
NIFI-4730 restructure of nifi-gcp-bundle
This closes #2367.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-01-03 11:21:52 -05:00
Koji Kawamura fc73c60924 NIFI-3709: Export NiFi flow dataset lineage to Apache Atlas 2017-12-18 12:23:58 -05:00
Matthew Burgess 2192138b06 NIFI-4683: Add ability to execute Spark jobs via Livy 2017-12-14 16:23:41 -05:00
joewitt 7e61c6333c
NIFI-2375 updated to latest Tika across codebase. Fixed
tests and updated L&N for Tika parsers.  Turned unstable grpc test to IT
 test. Updated deps due to Tika testing findings. This closes #2320
2017-12-06 11:53:15 -05:00
joewitt cdc1facf39
NIFI-4664, NIFI-4662, NIFI-4660, NIFI-4659 moved tests which are timing/threading/network dependent and brittle to integration tests and un-ignored tests that are IT. Updated travis to reduce impact on infra and appveyor now skips test runs so is just to prove build works on windows. This closes #2319
squash
2017-12-06 10:53:09 -05:00
Mark Payne 00b11e82b7 NIFI-4600: This closes #2312. Added nifi-kafka-1-0-nar and nifi-kafka-1-0-processors modules
Signed-off-by: joewitt <joewitt@apache.org>
2017-12-04 16:51:59 -05:00
joewitt 2c68d0e098
NIFI-4652 updating to latest apache version and updating key maven plugins, fixing versions for grpc to avoid extranous build dir, correcting checkstyle issues, resolving build problem with incorrect usage of activeByDefault in maven. Key change is now for the 'dir-only' profile it is now activated via a system property -Ddir-only instead of profile. No other activeByDefault profiles exist except to minifi the javascript but that usage is acceptible as-is. This closes #2313 2017-12-04 09:59:23 -05:00
Joey Frazee a458a139b3 NIFI-4650 This closes #2308. Added independent hadoop.http.core.version property
This closes #2308

Signed-off-by: joewitt <joewitt@apache.org>
2017-12-02 12:58:29 -05:00
dlukyanov c7a5a09b8a groovyx initial version
NIFI-3688 license update to ASF
NIFI-3688 add groovyx dependency
NIFI-3688 comments from @alopresto
- reformat code https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-EclipseUsers
- enabled java-doc
NIFI-3688
- try fix mess with licenses
- remove commented import
- remove unused var `sql`
NIFI-3688 the properties/skip javadoc/src
NIFI-3688 remove not used sql var
NIFI-3688
- first test case
NIFI-3688
+ 2 more tests based on current groovy tests
NIFI-3688 comment from @mattyb149
- processor renamed to ExecuteGroovyScript
NIFI-3688 fix script path validation
NIFI-3688
- refactor to compile on validation
- prepare 4 test cases with database
NIFI-3688 new test cases with groovy and sql
NIFI-3688 documentation
NIFI-3688
- refactor groovy extended methods
- add more test cases
NIFI-3688
- codestyle
- javadoc
- refactor flowfile voids to self-reference
- fix test cases
NIFI-3688 minor comments changes
NIFI-3688 rename additional documentation according to processor classname
NIFI-3688 exclude json test file from rat check
NIFI-3688 codestyle
NIFI-3688 add nar bundle to root pom.xml and to nifi-assembly/pom.xml
NIFI-3688
- fix & extend additional documentation
- fix pom.xml to bundle groovy into nar
NIFI-3688 add examples into additional processor documentation
NIFI-3688 fix pom.xml to exclude unnecessary libs from nar bundle
NIFI-3688 add restricted annotation
NIFI-3688 change version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT after rebase
NIFI-3688 new method in ProcessSession: public OutputStream write(FlowFile source)

NIFI-3688 change version from 1.3.0-SNAPSHOT to 1.4.0-SNAPSHOT after rebase

NIFI-3688 fix for @mattyb149 comment: The bundles referred to here and below are not included with this NAR and should be removed.

NIFI-3688 fix for @mattyb149 comment: This unused line can be removed

NIFI-3688
- removed `require flowfile` property
- fixed test cases according to deprecated property

change version to 1.5.0-SNAPSHOT

[NIFI-3688] Commented by mistake

[NIFI-3688] remove unused class

[NIFI-3688] fix javadoc comments

[NIFI-3688] refactor CTL & SQL properties

b3eecec901

2916ce1ec8

8e15392e2f

NIFI-3688: Checkstyle and typo fixes

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #1662
2017-11-13 14:32:58 -05:00
Koji Kawamura d914ad2924 NIFI-4547: Add ProvenanceEventConsumer utility class
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2236.
2017-10-30 09:50:44 +01:00
Christopher Currie c894246e03 NIFI-3950 Refactor AWS bundle
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2140.
2017-10-21 12:58:40 -07:00
Matt Gilman 6baea8ccff
NIFI-4444:
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.

This closes #2206.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-10-12 10:27:02 -07:00
Omer Hadari 6201c06c99
NIFI-4392 Create a MetricReportingTask with GraphiteMetricService
This closes #2171.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-10-08 10:44:11 -04:00
Andy LoPresto d4168f5ff1
NIFI-4297
- Upgraded immediately actionable dependency versions from Meterian report.
- Upgraded jackson-core test dependencies for HBase and Elasticsearch modules.
- Only 3 instances of jackson-core < 2.8.6 (Google Cloud Platform and Spark Receiver modules).
- Upgraded version of poi dependency in nifi-email-processors to 3.16.
- Resolving dependency issues after rebasing against 1.5.0-SNAPSHOT.
- Removed jackson-databind from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.version} in all referenced artifacts.
- Removed jackson-mapper-asl from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.old.version} in all referenced artifacts.
- Removed Jasypt from <dependencyManagement> and added explicit version in test dependency for legacy compatibility.
- This closes #2084
2017-10-05 15:23:52 -04:00
Matt Gilman 697c382d9b
NIFI-4328:
- Upgrading swagger and the corresponding maven plugin to address issues discovered in recursive domain models.
2017-10-03 09:50:11 -04:00
Jeff Storck a57911d3db NIFI-4412-RC2 prepare for next development iteration 2017-09-28 13:45:36 -04:00
Jeff Storck e6508ba7d3 NIFI-4412-RC2 prepare release nifi-1.4.0-RC2 2017-09-28 13:45:21 -04:00
Jeff Storck 502adc6e8a NIFI-4412 Removed duplicate dependency management definition for nifi-mongodb-client-service-api. 2017-09-28 11:08:47 -04:00
Matthew Burgess 1ca4fa3a83
NIFI-4345: Removed lookup-services dependency from MongoDB Services NAR, cleaned up deps
- Added nifi-lookup-service-api to nifi-nar-bundles POM

This closes #2174.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-09-25 14:52:33 -04:00
Mark Payne 3fb704c58f NIFI-4201: This closes #2024. Implementation of processors for interacting with Kafka 0.11
Signed-off-by: joewitt <joewitt@apache.org>
2017-09-22 22:08:19 -04:00
Mike Thomsen 9a8e6b2eb1 NIFI-4345 Added a MongoDB controller service and a lookup service.
NIFI-4345: Changed Lookup Key to Lookup Value Field
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2123
2017-09-22 13:33:07 -04:00
cam e3da44fb62 [NiFi-3973] Add PutKudu processor to ingest data to Kudu 2017-08-25 10:53:31 -04:00
Bryan Bende 60d4672195 NIFI-4304 Extracting HWX Schema Registry client version to a property and bumping to latest 0.3.0 release
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2096
2017-08-17 16:07:03 -04:00
Matt Gilman 528b82634f
NIFI-4210:
- Introducing support for OpenId Connect.
- Updating REST API and UI to support the authorization code flow.
- Adding/fixing documentation.
- Implementing time constant equality checks where appropriate.
- Corrected error handling during startup and throughout the OIDC login sequence.
- Redacting the token values from the user log.
- Defaulting to RS256 when not preferred algorithm is specified.
- Marking the OIDC endpoints as non-guaranteed in to allow for minor adjustments if/when additional SSO techniques are introduced.

This closes #2047.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-08-10 11:15:35 -07:00
Matt Gilman 743c6b9c17 NIFI-4032: - Introducing the ManagedRangerAuthorizer. - Introducing the AuthorizationAuditor. - Updating authorization requests to utilize Authorizable where ever possible so allow for a singular place to audit resource not found as denied when the parent authorizable is null (no more inheritance). - Updating unit tests as appropriate. - Addressing issues with broken web-api integration tests.
NIFI-4032: - Generating the appropriate fingerprint for the ManagedRangerAuthorizer based on whether the UserGroupProvider is configurable. - Adding unit tests.

Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #2019
2017-07-24 15:51:27 -04:00
mans2singh affc88e599 NIFI-4118 RethinkDB bundle with put processor
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1942.
2017-07-09 11:10:23 -07:00
Mark Payne 3906d4e1d2 NIFI-1763: Initial implementation of ConfluentSchemaRegistry.
NIFI-1763: Fixed bug where the Confluent Schema Registry Schema Access Writer was not being created

Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1938
2017-07-06 15:52:07 -04:00
m-hogue 58a623dfa2 NIFI-4037 added InvokeGRPC processor, with proto service IDL
NIFI-4038 added ListenGRPC processor

This closes #1947

Signed-off-by: Tony Kurc <tkurc@apache.org>
2017-06-29 22:01:22 -04:00
Andy LoPresto 2b5aaec6d9
NIFI-3716 Updated flowanalyzer version to 1.4.0-SNAPSHOT in pom.xml. 2017-06-26 17:06:49 -07:00
Eric 7314066b6a
NIFI-3716 Created flow analyzer in NiFi Toolkit.
This closes #1747.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-06-26 15:24:30 -07:00
Bryan Bende aabd4a25d2 NIFI-4043 Initial commit of nifi-redis-bundle
NIFI-4061 Initial version of RedisStateProvider
- Adding PropertyContext and updating existing contexts to extend it
- Added embedded Redis for unit testing
- Added wrapped StateProvider with NAR ClassLoader in StandardStateManagerProvider
- Updating state-management.xml with config for RedisStateProvider
- Renaming tests that use RedisServer to be IT tests so they don't run all the time

This closes #1918.
2017-06-21 12:47:19 -04:00
Matt Gilman 4ed7511bee
NIFI-3653: - Introducing UserGroup and Policy provider interfaces.
- Introducing FileUserGroupProvider and FileAccessPolicyProvider.
- Refactoring FileAuthorizer to utilize the file based implementations.
- Introducing the StandardManagedAuthorizer.
- Decorating the configured ManagedAuthorizer to ensure integrity checks are still performed.
- Loading user groups if possible to use during access decisions.
- Merging responses for requests for AccessPolicies, Users, and UserGroups.
- Adding unit tests as appropriate.
- Adding methods to the User, Group, and AccessPolicy builder that more easily supports generating UUIDs.
- Fixing typo when seeding policies during startup.
- Fixing type in documentation and error messages.

This closes #1897.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-06-09 13:54:10 -04:00
Matt Gilman cc741d2be6
NIFI-3997:
- Bumping to next minor version.
2017-06-08 15:22:51 -04:00
Matt Gilman 6ee12e9b47
NIFI-3997-RC1prepare for next development iteration 2017-06-05 11:07:43 -04:00
Matt Gilman ddb73612bd
NIFI-3997-RC1prepare release nifi-1.3.0-RC1 2017-06-05 11:07:28 -04:00
Andy LoPresto 6d07ebf50c NIFI-3954 This closes #1841. Upgraded version of httpclient library to handle TLS issues.
Added explicit dependency on httpcore for @ThreadSafe annotation.

Signed-off-by: joewitt <joewitt@apache.org>
2017-05-23 23:42:28 -04:00
Mark Payne 9bd0246a96 NIFI-3863: Initial implementation of Lookup Services. Implemented LookupRecord processors. This required some refactoring of RecordSetWriter interface, so refactored that interface and all implementations and references of it 2017-05-19 01:02:41 -04:00
Mark Payne b1901d5fe0 NIFI-3838: Initial implementation of RecordPath and UpdateRecord processor
NIFI-3838: Updated version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT; removed unneeded value from AttributeExpression.ResultType enum

NIFI-3838: Addressed PR Review feedback

NIFI-3838: Allow for schemas to be merged together for a record; refactored RecordSetWriterFactory so that there is a method to obtain the schema and then the writer is created with that schema. Added additional unit tests

NIFI-3838: Addressed problems with documentation based on PR Review

NIFI-3838: Fixed checkstyle violation

NIFI-3838: Addressed issue of comparing different types of Number objects

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1772
2017-05-12 12:36:52 -04:00
Bryan Bende 3af53419af
NIFI-3770-RC2 prepare for next development iteration 2017-05-05 20:50:28 -04:00
Bryan Bende 3a605af8e0
NIFI-3770-RC2 prepare release nifi-1.2.0-RC2 2017-05-05 20:50:14 -04:00
Andre F de Miranda 54d47c7f74 NIFI-3726 - Introduces CompareFuzzyHash processor
- Abstract FuzzyhashContent to reduce a bit of code
            duplication

NIFI-3726 - Attempt to address peer review comments

NIFI-3726 - Addresses additional PR comments

NIFI-3726 - Fix checkstyle violations

NIFI-3726 - Fix issues with POMs

NIFI-3726 - Fixes the logging level to error in case of failure due to message too short

NIFI-3726 - Capture ArrayIndexOutOfBoundsException when consuming a hash source

            that matches but lacks a separator AND filename or matches but lacks
            a filename(i.e. ends with trailing separator)

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1692
2017-05-03 09:18:17 -04:00
Jeff Storck 26d90fbccf
NIFI-1833 - Addressed issues from PR review.
NIFI-1833 Moved AbstractListProcessor.java, EntityListing.java, and ListableEntity.java from nifi-standard-processors into nifi-processor-utils
Moved TestAbstractListProcessor.java into nifi-processor-utils
Set nifi-azure-nar's nar dependency back to nifi-standard-services-api-nar
Fixed failing integration tests (ITFetchAzureBlobStorage.java, ITListAzureBlobStorage.java, and ITPutAzureStorageBlob.java) and refactored them to be able to run in parallel

NIFI-1833 Moved security notice info in the additional details documentation into the descriptions of the specific attributes for which those notices are intended
Added displayName usage to properties
Updated exception handling in FetchAzureBlobStorage.java and PutAzureBlobStorage.java to cause flowfiles with Output/InputStreamCallback failures to be routed to the processor's failure relationship
Cleaned up dependencies in pom

NIFI-1833 Removed unnecessary calls to map on Optional in the onTrigger exception handling of FetchAzureBlobStorage.java and PutAzureBlobStorage.java

NIFI-1833 Updates due to nifi-processor-utils being moved under nifi-nar-bundles

This closes #1719.

Signed-off-by: Bryan Rosander <brosander@apache.org>
2017-05-02 14:39:46 -04:00