- Removing problematic timeout for SMTP Listen
- Converting anonymous method to lambda
- Adding debug and error logging
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1924.
To be used with cache engines that does not have revision number.
NIFI-4049: Refactor AtomicDistributedMapCacheClient
Removed old methods completely.
This closes#1904.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Introducing FileUserGroupProvider and FileAccessPolicyProvider.
- Refactoring FileAuthorizer to utilize the file based implementations.
- Introducing the StandardManagedAuthorizer.
- Decorating the configured ManagedAuthorizer to ensure integrity checks are still performed.
- Loading user groups if possible to use during access decisions.
- Merging responses for requests for AccessPolicies, Users, and UserGroups.
- Adding unit tests as appropriate.
- Adding methods to the User, Group, and AccessPolicy builder that more easily supports generating UUIDs.
- Fixing typo when seeding policies during startup.
- Fixing type in documentation and error messages.
This closes#1897.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Oracle NUMBER can return 0 precision and -127 or 0 scale with variable scale NUMBER such as ROWNUM or function result
- Added 'Default Decimal Precision' and 'Default Decimal Scale' property to ExecuteSQL and QueryDatabaseTable to apply default precision and scale if those are unknown
- Coerce BigDecimal scale to field schema logical type, so that BigDecimals having different scale can be written
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1851
- Added dependency notice.
- Added EL evaluation at SimpleKeyValueLookupService.
- Updated documentation.
- Updated CommonsConfigurationLookupService to throw LookupFailureException if it fails to get configuration so that error messages can be displayed at each processor bulletin.
- Added calling getConfiguration at OnEnabled of CommonsConfigurationLookupService, so that the service will stay in Enabling state if there is any issue.
Signed-off-by: Joey Frazee <jfrazee@apache.org>
* Added non-integration unit tests for Lambda, S3, SNS, and SQS processors
* Moved non-integration tests out of integration test files
* Moved SQS integration tests to IT* files for clarification
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1814.
- Filtering out the EventTime searchable field for all implementation of ProvenanceRepository. Prefer user query using the startDate/endDate fields.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1844.
- When monitoring heartbeats use the connected nodes as the basis for the check. This addresses the case when a node is terminated and no corresponding heartbeats exist.
This closes#1838.
- When monitoring heartbeats use the connected nodes as the basis for the check. This addresses the case when a node is terminated and no corresponding heartbeats exist.
Also, updated record writers to ensure that they write the schema as appropriate if not using a RecordSet. Updated ConsumeKafkaRecord to allow for multiple schemas to be on same topic and partition
Signed-off-by: joewitt <joewitt@apache.org>
Before this fix, 'NullPointerException: Inflater has been closed' can be thrown as the Inflater is closed before input stream is consumed.
Also, calling close from AbstractTransaction.receive is removed, because the DataPacket is exposed as its return value and this class will not be able to know when to close the stream.
Signed-off-by: joewitt <joewitt@apache.org>
Forces a node reconnecting to a cluster to serialize the updated flow to disk
Added most processor settings to the flow fingerprint (excluding name, style, comment, position, and schedule state)
Updated some test data for FingerprintFactoryTest to test for new fields added to the flow fingerprint
Updated StandardProcessorNode to allow processor comments and name to be settable while a processor is running
Updated StandardFlowSynchronizer to inherit non-fingerprinted processor settings (name, style, comment, and position) when flow is already synchronized
This closes#1812
- Removed remaining duplicate lines of code left by NIFI-3861 refactoring.
- Added test case that writes Avro record having union field.
This closes#1813.
NIFI-3900: Rebased against master and updated new unit test to use new method signature for FlowSerializer.serialize
This closes#1804.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Addressing response code of POST /provenance/lineage.
- Ensuring cluster node details are set accordingly.
This closes#1803.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
- Filtering out certain control characters and unpaired Unicode surrogate codepoints prior to saving the flow.xml.
This closes#1784.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Added Logical type support for DECIMAL/NUMBER, DATE, TIME and TIMESTAMP column types.
- Added Logical type 'decimal' to AvroReader so that Avro records with logical types written by ExecuteSQL and QueryDatabaseTable can be consumed by AvroReader.
- Added JdbcCommon.AvroConversionOptions to consolidate conversion options.
- Added 'Use Avro Logical Types' property to ExecuteSQL and QueryDatabaseTable to toggle whether to use Logical types.
- Added 'mime.type' FlowFile attribute as 'application/avro-binary' so that output FlowFiles can be displayed by content viewer.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1798
NIFI-3855 Refactor deprecated API
NIFI-3855 Log separate error messages in PutSlack
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1774
- adjust testFullyDocumentedProcessor to correctly eval to false when needed
- Introduce the ability to deprecate a component
- Allow documentation to inform user about deprecation of a component
- This closes#1718
NIFI-391 - Add set/getDeprecationReson to DocumentedDTO and use it within DtoFactory'
NIFI-3838: Updated version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT; removed unneeded value from AttributeExpression.ResultType enum
NIFI-3838: Addressed PR Review feedback
NIFI-3838: Allow for schemas to be merged together for a record; refactored RecordSetWriterFactory so that there is a method to obtain the schema and then the writer is created with that schema. Added additional unit tests
NIFI-3838: Addressed problems with documentation based on PR Review
NIFI-3838: Fixed checkstyle violation
NIFI-3838: Addressed issue of comparing different types of Number objects
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1772
Previous fix#1779 refactored the way to check Logical type to use string constants.
One of those refactoring used wrong constant mistakenly in normalizeValue method.
Fortunately, this defect is harmless since even though normalizeValue did not convert int to Time, DataTypeUtils.convertType does the same conversion.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1782
This closes#1787.
- When converting from a raw value to an Avro object, convert the values
of any Avro map types so that they can be complex types like other
records.
- AvroReader did not convert logical types if those are defined with union
- Consolidated createSchema method in AvroSchemaRegistry and AvroTypeUtil as both has identical implementation and mai
ntaining both would be error-prone
This closes#1779.
realized that the session should be cloned here because its inside a for loop and the original flow file would be transferred but not be the latest flow file if an error occurred in the for loop
@trixpan at a high level what do you think about this approach?
NIFI-3600: Added unit test
NIFI-3600: Removed the hdfs.error.code attribute
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1595
- Making the selector for the enable/disable controller service state more specific. Previously name was too general and was affect other parts of the application.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#1759
Changed from async append to sync as it breaks 'recursionSet' check in StandardProcessSession by updating it from multiple threads, resulting IllegalStateException to happen.
This closes#1761.
Signed-off-by: Bryan Bende <bbende@apache.org>
Removing buildnumber plugin and build-info-no-git profile from nifi-assembly
and adding build-info-no-git profile to nifi-nar-bundles pom
Signed-off-by: James Wing <jvwing@gmail.com>
I still want to add unit tests and integration tests, but I ran a flow which had previously caused the reproducible exception and this worked fine (flow showed objects were put in S3, no exceptions, and I verified through AWS Web Console that new objects were present).
This closes#1753.
Signed-off-by: Bryan Rosander <brosander@apache.org>
- Abstract FuzzyhashContent to reduce a bit of code
duplication
NIFI-3726 - Attempt to address peer review comments
NIFI-3726 - Addresses additional PR comments
NIFI-3726 - Fix checkstyle violations
NIFI-3726 - Fix issues with POMs
NIFI-3726 - Fixes the logging level to error in case of failure due to message too short
NIFI-3726 - Capture ArrayIndexOutOfBoundsException when consuming a hash source
that matches but lacks a separator AND filename or matches but lacks
a filename(i.e. ends with trailing separator)
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1692
NIFI-1833 Moved AbstractListProcessor.java, EntityListing.java, and ListableEntity.java from nifi-standard-processors into nifi-processor-utils
Moved TestAbstractListProcessor.java into nifi-processor-utils
Set nifi-azure-nar's nar dependency back to nifi-standard-services-api-nar
Fixed failing integration tests (ITFetchAzureBlobStorage.java, ITListAzureBlobStorage.java, and ITPutAzureStorageBlob.java) and refactored them to be able to run in parallel
NIFI-1833 Moved security notice info in the additional details documentation into the descriptions of the specific attributes for which those notices are intended
Added displayName usage to properties
Updated exception handling in FetchAzureBlobStorage.java and PutAzureBlobStorage.java to cause flowfiles with Output/InputStreamCallback failures to be routed to the processor's failure relationship
Cleaned up dependencies in pom
NIFI-1833 Removed unnecessary calls to map on Optional in the onTrigger exception handling of FetchAzureBlobStorage.java and PutAzureBlobStorage.java
NIFI-1833 Updates due to nifi-processor-utils being moved under nifi-nar-bundles
This closes#1719.
Signed-off-by: Bryan Rosander <brosander@apache.org>
Addressed dependency issues from the review.
Addressed a checkstyle issue.
Review: reworded the descriptions.
Review: implemented the reset condition logic.
Review: dropped static qualifier from method signatures, not required really
Review: removed sys.out, inlined a single method to get access to the ProcessContext.getName()
Switched to HTTPS as per MSFT recommendation. Some DRY. Dropped cruft.
Addressing review suggestions from 4/5
Review: documentation improvements
Review: documentation improvements
This closes#1636.
Signed-off-by: Bryan Rosander <brosander@apache.org>
Added src/test/resources/logback-test.xml files resetting log level from DEBUG (in nifi-data-provenance-utils) to WARN because later tests depend on MockComponentLog recording a certain number of messages and this number is different than expected if the log level is DEBUG.
This closes#1686.
Signed-off-by: Bryan Bende, Yolanda M. Davis, and Mark Payne
- Added description on what session maintenance does.
- Added calling deregister when initial connection attempt fails so that a processor can retry connecting at next onTrigger.
This closes#1597
Signed-off-by: Jeremy Dyer <jeremydyer@apache.org>
- Removed unused disconnect method from WebSocketService interface.
- Added session maintenance background thread at JettyWebSocketClient
which reconnects sessions those are still referred by ConnectWebSocket
processor but no longer active.
- Added Session Maintenance Interval property to JettyWebSocketClient.
- Allowed specifying existing session id so that it can be recovered
transparently.
- Moved test classes to appropriate package.
- Added test cases that verify the same session id can be used after
WebSocket server restarts.
- Addressing issue when enabling/disabling controller services when referencing services are not in scope.
- Fixing enabled/disabled icons for referencing services.
- Rendering when a reference cycle is detected.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#1725
- Creating nifi-records-utils to share utility code from record services
- Refactoring Parquet tests to use MockRecorderParser and MockRecordWriter
- Refactoring AbstractPutHDFSRecord to use schema access strategy
- Adding custom validate to AbstractPutHDFSRecord and adding handling of UNION types when writing Records as Avro
- Refactoring project structure to get CS API references out of nifi-commons, introducing nifi-extension-utils under nifi-nar-bundles
- Updating abstract put/fetch processors to obtain the WriteResult and update flow file attributes
This closes#1712.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
- Added org.apache.nifi.processor.util.pattern package in nifi-processor-utils containing reusable functions to mix-in 'Rollback on Failure' capability.
- Created a process pattern classes, Put and PutGroup. It will be helpful to standardize Processor implementations.
- Applied Rollback on Failure to PutSQL, PutHiveQL, PutHiveStreaming and PutDatabaseRecord.
- Stop using AbstractProcessor for these processors, as it penalizes FlowFiles being processed when it rollback a process session. If FlowFiles are penalized, it will not be fetched again until penalization expires.
- Yield processor when a failure occurs and RollbackOnFailure is enabled. If we do not penalize nor yield, a failed FlowFile retries too frequently.
- When Rollback on Failure is enabled but processor is not transactional, discontinue when an error occurred after successful processes.
- Fixed existing issues on PutHiveStreaming:
- Output FlowFile Avro format was corrupted by concatenating multiple Avro files.
- Output FlowFile records had incorrect values because of reusing GenericRecord instance.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1658
- added proxy dn flag to support providing authorized username for secured environments
- addressed pr comments including fix to ensure proxy info added when getting cluster info, showing cleaner error messaging and improving help text. Also fixed potential issue with versioning comparison (mismatched lengths)
- Printing response body when requests fails.
- This closes#1697
- Added batchCount, batchSize, batchDuration to limit flow files to be
included in a single Site-to-Site transaction.
- Added batch throttling logic when StandardRemoteGroupPort transfers
flow files to a remote input port using the batch limit configurations,
so that users can limit batch not only for pulling data, but also pushing data.
- Added destination list shuffle to provide better load distribution.
Previously, the load distribution algorithm produced the same host consecutively.
- Added new batch settings to FlowConfiguration.xsd.
- Added new batch settings to Flow Fingerprint.
- Added new batch settings to Audit.
- Sort ports by name at 'Remote Process Group Ports' dialog.
- Show 'No value set' when a batch configuration is not set
- Updated batch settings tooltip to clearly explain how it works the configuration works differently for input and output ports.
- Updated DTO by separating batch settings to BatchSettingsDTO to indicate count, size and duration are a set of configurations.
- This closes#1306
NIFI-3704: Incorporated review comments
NIFI-3704: Updated PutDatabaseRecord documentation, including limitations on Update records
NIFI-3704: Added retry paths for transient SQL errors
NIFI-3704: Updated to use renamed RecordReaderFactory and catch SchemaNotFoundException
NIFI-3704: Added better error handling and rollback
This closes#1677.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-3738 Fixed NPE when ParseSyslog UDP datagram has zero length.
Added unit test.
NIFI-3738 Added licenses to new unit tests.
This closes#1694.
Signed-off-by: Bryan Bende <bbende@apache.org>
This approaches a proper solution on how to hand over the key from
RunNiFi to NiFi. Insofar the password file is pruned as part of the
startup, NiFi processors can't read it.
See also: NIFI-3045.
This closes#1302.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
connections (Think chat to all clients instead of individual person).
The core change was a change in WebSocketMessageRouter.java where if a
sessionId is not present the message is sent to all connected clients.
So the key is leaving the sessionId to empty or null to send to all
clients. If the sessionId is specified the message will be sent just to
that session specified.
This closes#1649.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Fixed SiteInfoProvider and HttpClient to use specified local address with its SiteToSiteRestApiClient
- Removed setupRequestConfig method call from connection and read timeout setter methods at SiteToSiteRestApiClient, because it created config object before local address was set
- Null clear StandardRemoteProcessGroup localAddress when user clears Local Network Interface
Signed-off-by: joewitt <joewitt@apache.org>
- Refactored ServerProtocol.sendPeerList method signature to clarify the
meaning of arguments, and avoid null pointer exception when converting null Integer to int.
- Refactored SocketRemoteSiteListener handleRequest method to make it
more unit test friendly.
- Added more unit tests.
- Using consistent logic when verifying connection creation. Removed some unecessary checks as verification has been changed to run in cluster and standalone mode.
This closes#1672.
If there is an unconnected remote port, right clicking a Remote Process Group
then select 'Remote Ports' causes a JS error saying 'Uncaught TypeError'.
Because there are few code paths those don't return the created
transmissionSwitch value from createPortOption function in
nf-remote-process-group-ports.js.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
This closes#1679
Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Updated SimpleKeyValueSchemaRegistry to make use of new CHOICE RecordFieldType - Update Record Readers to use SchemaRegistry controller service. Moved SchemaRegistry api into its own maven module and added to standard-services-api so that we can properly add dependencies on it. Code cleanup and bug fixes
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Fixed checkstyle violations and license exclusions for RAT plugin
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Addressed feedback from PR Review
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Additional changes/doc to support QueryFlowFile and Record Readers/Writers
This closes#1652
Updated L&N in nifi-gcp-nar and nifi-assembly and fixed nifi-gcp-processors
to avoid jsr305 and json.org cat-x dependencies
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#1665.
Fixed service instance lookup to use local port so that it works when
it is accessed through port forwarding.
This fixes: #1568
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.
nifi-mock is modified to support FlowFile assertion using Prioritizer.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporated review comments, added displayNames.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporate review comments:
- Moved nifi-standard-prioritizers dependency to top level nifi/pom.xml.
- Changed default initial order from 1 to 0.
- Fixed typos.
- Use session.get(batchCount).
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
When a FlowFile is transferred to success, remove attributes previously set when it was transferred to wait or failure.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1496
Enabled the ability to specify wildcard topics as a regular expression
as supported in the Kafka client library.
Signed-off-by: joewitt <joewitt@apache.org>
NIFI-3678: If we have a transaction ID but then have no more data written to Partition file, we end up with a NPE. Added logic to avoid this and instead return null for the next record when this happens
This closes#1656.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Fixing FlowController to use appropriate class loader when instantiating processor
- Updating ExtensionManager to leverage new flag in MANIFEST from NAR plugin
- Adding ReloadComponent interface and refactoring instance class loading to use it
- Fixing FetchHDFS issue with TDE by using ugi.doAs
- Refactoring nifi-nar-utils so that ExtensionManager only lives in nifi-framework
- Caching temp components found during service loader in ExtensionManager
- Updating authorizables, docs, and fingerprinting to use the cached components
- Introducing a flag on @RequiresInstanceClassLoading to indicate if ancestor resources should be cloned
- Updating developer guide regarding cloneAncestorResources flag
- This closes#1635
NIFI-3413: Incorporated review comments
NIFI-3413: Changed GetChangeDataCaptureMySQL to CaptureChangeMySQL, fixed some bugs
NIFI-3413: Refactored setup() for better error handling, more review comments incorporated
NIFI-3413: Refactored CDC into its own module(s), updated assembly and top-level POMs
NIFI-3413: Added RECEIVE prov event and Server ID property
Signed-off-by: ijokarumawak <ijokarumawak@apache.org>
This closes#1646.
Newly created async response is added before checking map size nor
purging expired ones. If there are already 100 remaining requests,
the added request will not be executed nor removed.
Currently, NiFi Kafka consumer processors have following issue.
While downstream connections are full, ConsumeKafka is not scheduled to run onTrigger.
It stopps executing poll to tell Kafka server that this client is alive.
Thus, after a while in that situation, Kafka server rebalances the client.
When downstream connections back to normal, although ConsumeKafka is scheduled again,
the client is no longer a part of a consumer group.
If this happens, Kafka client succeeds polling messages when ConsumeKafka processor resumes, but fails to commit offset.
Received messages are already committed into NiFi flow, but since consumer offset is not updated, those will be consumed again, duplicated.
In order to address above issue:
- For ConsumeKafka_0_10, use latest client library
Above issue has been addressed by KIP-62.
The latest Kafka consumer poll checks if the client instance is still valid, and rejoin the group if not, before consuming messages.
- For ConsumeKafka (0.9), added manual retention logic using pause/resume
Kafka client 0.9 doesn't have background thread heartbeat, so similar machanism is added manually.
Use Kafka pause/resume consumer API to tell Kafka server that the client stops consuming messages but is still alive.
Another internal thread is used to perform paused poll periodically based on the time passed since the last onTrigger(poll) is executed.
This closes#1527.
Signed-off-by: Bryan Bende <bbende@apache.org>
- On contextDestroyed, referencing beans created during contextInitialized to prevent successive attempts to create a bean if that bean failed to be created initially.
Signed-off-by: joewitt <joewitt@apache.org>
Scott Aslan
Wesley-Lawrence
Chris Herrera
Matt Gilman
Mark Payne
Bryan Bende
Koji Kawamura
Steve Champagne
Joe Gresock
Andre F de Miranda
Matt Burgess
Yolanda M. Davis
Pierre Villard
Joseph Niemiec
Joey Frazee
baolsen
Mike Moser
Andy LoPresto
James Wing
Joe Skora
Tim Reardon
joewitt
Jeff Storck
Aldrin Piri
Peter G. Horvath
Jeremy Dyer
Andrew Grande
Simon Elliston Ball
Jon Anderson III
Bryan Rosander
Andrew Lim
Anders Breindahl
ShellyLC
Oleg Zhurakousky
Toivo Adams
Joe Percivall
Francois Prunier
Jack Pickett