1
0
mirror of https://github.com/apache/nifi.git synced 2025-03-02 15:39:12 +00:00

51 Commits

Author SHA1 Message Date
Emilio Setiadarma
bee65b8447 NIFI-12825: implemented ListHBaseRegions processor
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes 
2024-03-06 18:56:40 -05:00
Matt Burgess
2f42b44efa
NIFI-12731: Ensure state is updated in GetHBase whenever the session is committed
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes .
2024-02-02 09:05:44 +04:00
Pierre Villard
68228921a4
NIFI-12664 Removed deprecated DMC in GetHBase
This closes 

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2024-01-25 17:48:43 -06:00
exceptionfactory
34aebc1f69
NIFI-12473 Deleted removeByPattern Methods from Cache Services
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes .
2023-12-07 12:42:08 +01:00
Pierre Villard
43cc5ad122
NIFI-12079 This closes . Remove Variables / Variable Registry
Signed-off-by: Joseph Witt <joewitt@apache.org>
2023-09-25 09:39:53 -07:00
Pierre Villard
5405a3b08b
NIFI-12102 Fixed deprecation warnings when instantiating primitive types
This closes 

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-09-21 13:41:30 -05:00
Peter Turcsanyi
ebd0d60e49
NIFI-12075 Deprecated ComponentLog methods with Object[] and Throwable
- Updated component references to remove use of Object[] wrapping for log methods

This closes 

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-09-19 08:53:00 -05:00
Pierre Villard
150bfbe990
NIFI-11813 Removed Event Driven Scheduling Strategy
This closes 

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2023-07-18 12:22:52 -05:00
exceptionfactory
512155ba2f
NIFI-10975 Add Kubernetes Leader Election and State Provider ()
* NIFI-10975 Added Kubernetes Leader Election and State Provider
- Added Kubernetes Leader Election Manager based on Kubernetes Leases
- Added Kubernetes State Provider based on Kubernetes ConfigMaps
- Added nifi-kubernetes-client for generalized access to Fabric8 Kubernetes Client
- Added nifi.cluster.leader.election.implementation Property defaulting to CuratorLeaderElectionManager
- Refactored LeaderElectionManager to nifi-framework-api for Extension Discovering Manager
- Refactored shared ZooKeeper configuration to nifi-framework-cluster-zookeeper

* NIFI-10975 Updated Kubernetes Client and StateMap
- Upgraded Kubernetes Client from 6.2.0 to 6.3.0
- Added getStateVersion to StateMap and deprecated getVersion
- Updated Docker start.sh with additional properties

* NIFI-10975 Corrected MockStateManager.assertStateSet()
* NIFI-10975 Upgraded Kubernetes Client from 6.3.0 to 6.3.1
* NIFI-10975 Corrected unregister leader and disabled release on cancel

* NIFI-10975 Corrected findLeader handling of Lease expiration
- Changed LeaderElectionManager.getLeader() return to Optional String

* NIFI-10975 Corrected StandardNiFiServiceFacade handling of Optional Leader
* NIFI-10975 Changed getLeader() to call findLeader() to avoid stale cached values
* NIFI-10975 Updated LeaderElectionCommand to run LeaderElector in loop
* NIFI-10975 Rebased on project version 2.0.0-SNAPSHOT

* NIFI-10975 Corrected Gson and AspectJ versions
- Updated versions to match current main branch and avoid reverting
2023-03-07 14:14:24 -05:00
dan-s1
e86786433a
NIFI-11108 Further changes to upgrade from JUnit 4 to JUnit 5.
Signed-off-by: Chris Sampson <chris.sampson82@gmail.com>

This closes 
2023-02-02 12:27:42 +00:00
sopan98
889254a9dc
NIFI-10744 Used LinkedHashMap in TestScanHBase methods
This closes 

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-11-08 14:51:10 -06:00
Lehel Boér
feb3148b68
NIFI-10628: Added Default Schedule Time and Run Duration to processors
This closes 

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-10-21 14:57:46 -05:00
Mike Thomsen
589a1c24d1
NIFI-9124 Refactored nifi-jms-bundle to use JUnit 5
- NIFI-9122 Refactored nifi-influxdb-bundle to use JUnit 5
- NIFI-9121 Refactored nifi-ignite-bundle to use JUnit 5
- NIFI-9120 Refactored nifi-html-bundle to use JUnit 5
- NIFI-9119 Refactored nifi-hl7-bundle to use JUnit 5
- NIFI-9118 Refactored nifi-hive-bundle to use JUnit 5
- NIFI-9117 Refactored nifi-hbase-bundle to use JUnit 5
- NIFI-9116 Refactored nifi-hazelcast-bundle to use JUnit 5
- NIFI-9114 Refactored nifi-hadoop-bundle to use JUnit 5
- Added DisabledOnOs(OS.WINDOWS) to Hive tests with @TempDir from JUnit 5 due to Windows-specific problems in JUnit 5

This closes 

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-02-28 10:39:47 -06:00
xinyuwu4
22b47ccc7b
NIFI-9672: fix flaky tests caused by the use of HashMap.
This closes 

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-02-22 08:06:52 -05:00
Bryan Bende
e436381c3a
NIFI-8975 Integrate KerberosUserService into HBase processors/services
This closes 

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-08-27 17:41:58 -05:00
Mark Payne
ecacfdaa4c NIFI-8469: Introduced ProcessSession.commitAsync and updated processors to use it. Deprecated ProcessSession.commit()
- Updated Mock Framework to now fail tests that use ProcessSession.commit() unless they first call TestRunner.setAllowSynchronousSessionCommits(true)
- Updated stateless nifi in order to make use of async session commits
- Fixed bug that caused stateless to not properly handle Additional Classpath URLs and bug that caused warnings about validation to get generated when a flow that used controller services was initialized. While this is not really in scope of NIFI-8469, it was found when testing and blocked further progress so addresssed here.
- If Processor fails to progress when run from stateless, trigger from start of flow until that is no longer the case
- Introduced notion of TransactionThresholds that can limit the amount of data that a flow will bring in for a given invocation of stateless dataflow
- Several new system-level tests
2021-05-19 08:51:16 -04:00
Mark Payne
097edf4f7c
NIFI-8136: Added getState/setState/replaceState/clearState methods to ProcessSession, updated processors to use these methods instead of StateManager version where appropriate
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes 
2021-01-15 10:02:09 -05:00
Pierre Villard
4b0d349a49
NIFI-7667 - add variable support to GetHBase properties () 2020-07-23 09:29:44 -04:00
Matthew Burgess
71226ce077
NIFI-7044: Fixed 'InputStream not closed' issue in PutElasticsearchRecord and DeleteHBaseCells
This closes 

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-01-18 10:39:47 -05:00
Tomer Ghelber
ba642485cc NIFI-5933 Stopped using sun's IOUtils. Using apahce's instead.
NIFI-5933 Removed unused file with sun.* import
NIFI-5933 Removed unsafe of sun.*.
NIFI-5933 Added openjdk8 to test

This closes 

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-07 21:58:16 -04:00
Dustin Rodrigues
161e4b5763 NIFI-6231 - fix source code permissions to be non-executable
This closes .

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-04-22 10:13:07 +09:00
Sandish Kumar
82e2c97782
NIFI-5956 Option BlockCache HBaseScanProcessor
This closes .

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-03-14 12:22:41 -04:00
joewitt
8e233ca2ef
NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin
This closes 
2018-10-04 09:32:04 -04:00
Mike Thomsen
ea41b41ede NIFI-5145 Made MockPropertyValue.evaluateExpressionLanguage(FlowFile able to accommodate null flowfiles the way live NiFi does.
NIFI-5145 Fixed a mistake with evaluateAttributeExpressions found in a code review.

NIFI-5145 Removed a property descriptor from GetHBase that was a duplication of the one in VisibilityFetchSupport.

NIFI-5145 Added comments in the code that were requested in the review.

NIFI-5145 Fixed check style issue.

NIFI-5145 Fixed a few improperly scoped items in ListFileTransfer that impacted the FTP processor(s).

NIFI-5145 Changed which override gets called based on code review feedback.

NIFI-5145: Reverted changes to ListFileTransfer re ExpressionLanguageScope

NIFI-5145: Removed TODO from MockPropertyValue javadoc

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes 
2018-05-16 13:50:27 -04:00
Koji Kawamura
500a254e3f NIFI-4637: HBase visibility lables
Fixed additional docs directory structure.
2018-05-11 07:30:19 +00:00
Mike Thomsen
0b851910f3 NIFI-4637 Added support for visibility labels to the HBase processors.
NIFI-4637 Removed integration test and updated the hbase client from 1.1.2 to 1.1.13 which is the final version for 1.1.X

NIFI-4637 Fixed EL support issue w/ tests.

NIFI-4637 Added more documentation to DeleteHBaseCells.

NIFI-4637 changed PutHBaseCell/JSON to use dynamic properties instead of a 'default visibility string.'

NIFI-4637 Added changes requested in a code review.

NIFI-4637 Moved pickVisibilityString to a utility class to make testing easier.

NIFI-4637 Added additionalDetails.html for PutHBaseRecord.

NIFI-4637 Added additional documentation and testing.

NIFI-4637 Added documentation for DeleteHBaseCells.

NIFI-4637 Added pickVisibilityLabel support to PutHBaseRecord and updated documentation to reflect that.

NIFI-4637 Reverted version bump to hbase client.

This closes .

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-05-11 00:55:40 +00:00
zenfenan
1dbfcb9445 NIFI-5062: Removed hbase-client dependecy from hbase bundle
This closes 

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-04-20 08:50:25 -04:00
Pierre Villard
4c787799ff NIFI-4149 - Indicate if EL is evaluated against FFs or not
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
2018-04-06 12:40:54 -04:00
Ed
c2616e6fe7
NIFI-4833 Add ScanHBase Processor
- New processor for scanning HBase records based on verious params like range of rowkeys, range of timestamps. Supports result limit and reverse scan.
- Adds Atlas Support for ScanHBase processor
- Fixed not recent version of FF
- Formatting and Style changes
- Single line to multiline if-then statements
- Removed HTML formatting that is not used for doc generation
- Fixed issue with limitRows
- Fixed issue with filter expression
- Refactored "processRows"
- Fixed possible NPE for bulkSize var
- Changed provenance to "receive" to indicate new data from external source.
- Updated min/max timestamp custom validation
- JSON array support
- Removed in-memory caching for records. Now records are being written directly to FF
- Removed unfinished flowfile from session, transfered original to "failure". Test cases update as well

This closes .

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-13 10:07:07 -04:00
Ed
373cf090a4 NIFI-4953 - FetchHBaseRow - update log level for "not found" to DEBUG instead of ERROR
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes .
2018-03-12 20:21:40 +01:00
Jeff Storck
e916594b69 NIFI-4872 Added annotation for specifying scenarios in which components can cause high usage of system resources.
- Initial set of components marked with the HighResourceUsageScenario annotation.
- Added customized descriptions to SystemResourceConsideration annotations for MergeContent, SplitContent, SplitJson, SplitText, and SplitXml.

This closes .

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-05 15:32:23 -05:00
Mike Thomsen
143d7e6829 NIFI-3538 Added DeleteHBaseRow
This closes .

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-02-14 10:04:25 +09:00
Mike Thomsen
ad2d12a20c NIFI-4578 Added strategy for dealing with nullable fields in PutHBaseRecord.
NIFI-4578 Added changes from code review.

This closes .

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-11-14 13:59:30 +09:00
Koji Kawamura
116c846342 NIFI-4543: Improve HBase processors provenance transit URL
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes .
2017-11-02 18:24:40 +01:00
Matt Gilman
6baea8ccff
NIFI-4444:
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.

This closes .

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-10-12 10:27:02 -07:00
Bryan Bende
f7da7e67f4 NIFI-4275 Adding support for specifying the timestamp on PutHBase processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes .
2017-08-11 18:04:13 +02:00
Bryan Bende
f8f1cc8d0d
NIFI-4024 Ensuring InputStream gets closed and cleaning up complex field handling
This closes .

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-08-04 14:54:05 -04:00
Mike Thomsen
496a32e12c
NIFI-4024 Added org.apache.nifi.hbase.PutHBaseRecord
Signed-off-by: Bryan Bende <bbende@apache.org>
2017-08-04 14:54:01 -04:00
baolsen
152f002abf
Completed initial development of HBase_1_1_2_ClientMapCacheService.java which is compatible with DetectDuplicate (and other processors)
Signed-off-by: Bryan Bende <bbende@apache.org>
2017-05-24 14:59:23 -04:00
Matt Burgess
d1ebddce98 NIFI-3627: Added removeByPattern() to DistributedMapCache interfaces
NIFI-3627: Updated unit tests that use MapCache interface(s)

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes .
2017-03-22 19:22:08 +01:00
Bryan Bende
b207397a11 NIFI-1784 This closes . Initial commit for FetchHBaseRow processor 2017-02-15 12:59:38 -05:00
Andre F de Miranda
3b408f5601 NIFI-2816 - Clean typos across the code - Part 2. This closes 2016-10-05 13:07:57 -04:00
Andrew Psaltis
0303805c01
NIFI-2620 Adding support for Binary Row Keys for both PutHBaseCell and PutHBaseJSON. This also involved making changes to PutFlowFile and PutColumn to carry around byte[] and not all strings. This closes .
Signed-off-by: Bryan Bende <bbende@apache.org>
2016-08-23 13:25:55 -04:00
joewitt
f987b21609 NIFI-1157 searched for and resolved all remaining references to deprecated items that were clearly addressable. 2016-07-14 09:32:35 -04:00
Bryan Bende
8593bd771f NIFI-1895 Adding a property to PutHBaseJSON to allow specifying how to store the values
This closes .

Signed-off-by: Bryan Bende <bbende@apache.org>
2016-06-20 13:19:20 -04:00
rtempleton
0d2a9dc7e5 NIFI-1895 PutHBaseJSON processor treats all values as Strings
The operator will now inspect the node value to determine type and convert as such.
Numeric integral - Long (assumes widest type)
Numeric not integral - Double (assumes widest type)
Logical - Boolean
everything else (including current Complex Type logic) - String

Values that represent the row key continue to be implictly treated as Strings by the processor

Removed depenency on HBase utility Bytes class from the PutHBaseJSON processor.
Convenience methods to encode to byte array are now wrapped by the appropriate HBaseClientService instance.

Signed-off-by: Bryan Bende <bbende@apache.org>
2016-06-20 13:19:13 -04:00
Bryan Bende
8773ec3d3e NIFI-1488 Refactoring HBase Kerberos support
- Storing UGI so we can support multiple HBaseClientServices with different configs
- Creating nifi-hadoop-utils to hold utility code shared between HDFS and HBase processors
- Incorporating KerberosProperties into existing hadoop processors

This closes 
2016-03-17 17:12:57 -04:00
jpercivall
29fb9c9393 NIFI-259 Corrected some logic in AbstractListProcessor regarding being elected primary node and improved readability in GetHBase a smidge 2016-02-04 16:07:18 -05:00
Mark Payne
bbce596d74 NIFI-259: Updated GetHBase to use new State Management; bug fixes; updated docs 2016-01-13 12:47:08 -05:00
Bryan Bende
40dd8a0a84 NIFI-1174 Refactoring the HBase client API and adding a PutHBaseJSON which can write a whole row from a single json document - Adding Complex Field Strategy to PutHBaseJSON to allow more control of complex fields - Improving error messages to indicate what the problem was with an invalid row
Signed-off-by: Bryan Bende <bbende@apache.org>
2015-11-19 13:49:02 -05:00