diff --git a/nifi-assembly/NOTICE b/nifi-assembly/NOTICE index b62e10a256..ad8350e3c5 100644 --- a/nifi-assembly/NOTICE +++ b/nifi-assembly/NOTICE @@ -2257,6 +2257,11 @@ The following binary components are provided under the Apache Software License v atinject Copyright + (ASLv2) The Dropbox Project + The following NOTICE information applies: + The Dropbox Project + Copyright 2013-2021 Dropbox Inc. + ************************ Common Development and Distribution License 1.1 ************************ diff --git a/nifi-assembly/pom.xml b/nifi-assembly/pom.xml index a7a351cbcf..2a90a924ac 100644 --- a/nifi-assembly/pom.xml +++ b/nifi-assembly/pom.xml @@ -862,6 +862,24 @@ language governing permissions and limitations under the License. --> 1.18.0-SNAPSHOT nar + + org.apache.nifi + nifi-dropbox-processors-nar + 1.18.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-dropbox-services-nar + 1.18.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-dropbox-services-api-nar + 1.18.0-SNAPSHOT + nar + diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors-nar/pom.xml b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors-nar/pom.xml new file mode 100644 index 0000000000..25c6403cb2 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors-nar/pom.xml @@ -0,0 +1,45 @@ + + + + 4.0.0 + + + org.apache.nifi + nifi-dropbox-bundle + 1.18.0-SNAPSHOT + + + nifi-dropbox-processors-nar + nar + + true + true + + + + + org.apache.nifi + nifi-dropbox-services-api-nar + 1.18.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-dropbox-processors + 1.18.0-SNAPSHOT + + + \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors-nar/src/main/resources/META-INF/LICENSE b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors-nar/src/main/resources/META-INF/LICENSE new file mode 100644 index 0000000000..7a4a3ea242 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors-nar/src/main/resources/META-INF/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors-nar/src/main/resources/META-INF/NOTICE b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors-nar/src/main/resources/META-INF/NOTICE new file mode 100644 index 0000000000..e5f54b695d --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors-nar/src/main/resources/META-INF/NOTICE @@ -0,0 +1,38 @@ +nifi-dropbox-processors-nar +Copyright 2015-2022 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + +****************** +Apache Software License v2 +****************** + +The following binary components are provided under the Apache Software License v2 + (ASLv2) The Dropbox Project + The following NOTICE information applies: + The Dropbox Project + Copyright 2013-2021 Dropbox Inc. + + (ASLv2) Jackson JSON processor + The following NOTICE information applies: + # Jackson JSON processor + + Jackson is a high-performance, Free/Open Source JSON processing library. + It was originally written by Tatu Saloranta (tatu.saloranta@iki.fi), and has + been in development since 2007. + It is currently developed by a community of developers, as well as supported + commercially by FasterXML.com. + + ## Licensing + + Jackson core and extension components may licensed under different licenses. + To find the details that apply to this artifact see the accompanying LICENSE file. + For more information, including possible other licensing options, contact + FasterXML.com (http://fasterxml.com). + + ## Credits + + A list of contributors may be found from CREDITS file, which is included + in some artifacts (usually source distributions); but is always available + from the source code management (SCM) system project uses. \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/pom.xml b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/pom.xml new file mode 100644 index 0000000000..2391a78f99 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/pom.xml @@ -0,0 +1,91 @@ + + + + 4.0.0 + + + org.apache.nifi + nifi-dropbox-bundle + 1.18.0-SNAPSHOT + + + nifi-dropbox-processors + jar + + + + org.apache.nifi + nifi-api + + + org.apache.nifi + nifi-utils + 1.18.0-SNAPSHOT + + + org.apache.nifi + nifi-record + + + org.apache.nifi + nifi-listed-entity + 1.18.0-SNAPSHOT + + + org.apache.nifi + nifi-dropbox-services-api + 1.18.0-SNAPSHOT + provided + + + com.dropbox.core + dropbox-core-sdk + ${dropbox.client.version} + + + org.apache.nifi + nifi-mock + test + + + org.apache.nifi + nifi-record-serialization-services + 1.18.0-SNAPSHOT + test + + + org.apache.nifi + nifi-record-serialization-service-api + test + + + org.apache.nifi + nifi-distributed-cache-client-service-api + test + + + org.apache.nifi + nifi-schema-registry-service-api + test + + + org.apache.nifi + nifi-dropbox-services + 1.18.0-SNAPSHOT + test + + + \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/java/org/apache/nifi/processors/dropbox/DropboxFileInfo.java b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/java/org/apache/nifi/processors/dropbox/DropboxFileInfo.java new file mode 100644 index 0000000000..22eda85a51 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/java/org/apache/nifi/processors/dropbox/DropboxFileInfo.java @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.dropbox; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import org.apache.nifi.processor.util.list.ListableEntity; +import org.apache.nifi.serialization.SimpleRecordSchema; +import org.apache.nifi.serialization.record.MapRecord; +import org.apache.nifi.serialization.record.Record; +import org.apache.nifi.serialization.record.RecordField; +import org.apache.nifi.serialization.record.RecordFieldType; +import org.apache.nifi.serialization.record.RecordSchema; + +public class DropboxFileInfo implements ListableEntity { + + public static final String ID = "dropbox.id"; + public static final String PATH = "path"; + public static final String FILENAME = "filename"; + public static final String SIZE = "dropbox.size"; + public static final String TIMESTAMP = "dropbox.timestamp"; + public static final String REVISION = "dropbox.revision"; + + private static final RecordSchema SCHEMA; + + static { + final List recordFields = new ArrayList<>(); + recordFields.add(new RecordField(ID, RecordFieldType.STRING.getDataType(), false)); + recordFields.add(new RecordField(PATH, RecordFieldType.STRING.getDataType(), false)); + recordFields.add(new RecordField(FILENAME, RecordFieldType.STRING.getDataType(), false)); + recordFields.add(new RecordField(SIZE, RecordFieldType.LONG.getDataType(), false)); + recordFields.add(new RecordField(TIMESTAMP, RecordFieldType.LONG.getDataType(), false)); + recordFields.add(new RecordField(REVISION, RecordFieldType.STRING.getDataType(), false)); + + SCHEMA = new SimpleRecordSchema(recordFields); + } + + public static RecordSchema getRecordSchema() { + return SCHEMA; + } + + private final String id; + private final String path; + private final String filename; + private final long size; + private final long timestamp; + private final String revision; + + private DropboxFileInfo(final Builder builder) { + this.id = builder.id; + this.path = builder.path; + this.filename = builder.filename; + this.size = builder.size; + this.timestamp = builder.timestamp; + this.revision = builder.revision; + } + + public String getId() { + return id; + } + + public String getPath() { + return path; + } + + public String getRevision() { + return revision; + } + + public String getFileName() { + return filename; + } + + @Override + public Record toRecord() { + final Map values = new HashMap<>(); + + values.put(ID, getId()); + values.put(PATH, getPath()); + values.put(FILENAME, getFileName()); + values.put(SIZE, getSize()); + values.put(TIMESTAMP, getTimestamp()); + values.put(REVISION, getRevision()); + + return new MapRecord(SCHEMA, values); + } + + @Override + public String getName() { + return getFileName(); + } + + @Override + public String getIdentifier() { + return getId(); + } + + @Override + public long getTimestamp() { + return timestamp; + } + + @Override + public long getSize() { + return size; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + DropboxFileInfo that = (DropboxFileInfo) o; + return id.equals(that.id) && size == that.size && timestamp == that.timestamp && path.equals(that.path) && filename.equals(that.filename) + && revision.equals(that.revision); + } + + @Override + public int hashCode() { + return Objects.hash(id, path, filename, size, timestamp, revision); + } + + public static final class Builder { + + private String id; + private String path; + private String filename; + private long size; + private long timestamp; + private String revision; + + public Builder id(String id) { + this.id = id; + return this; + } + + public Builder path(String path) { + this.path = path; + return this; + } + + public Builder name(String filename) { + this.filename = filename; + return this; + } + + public Builder size(long size) { + this.size = size; + return this; + } + + public Builder timestamp(long createdTime) { + this.timestamp = createdTime; + return this; + } + + public Builder revision(String revision) { + this.revision = revision; + return this; + } + + public DropboxFileInfo build() { + return new DropboxFileInfo(this); + } + } +} diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/java/org/apache/nifi/processors/dropbox/DropboxFlowFileAttribute.java b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/java/org/apache/nifi/processors/dropbox/DropboxFlowFileAttribute.java new file mode 100644 index 0000000000..0a5898cb2a --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/java/org/apache/nifi/processors/dropbox/DropboxFlowFileAttribute.java @@ -0,0 +1,45 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.dropbox; + +import java.util.function.Function; + +public enum DropboxFlowFileAttribute { + ID(DropboxFileInfo.ID, DropboxFileInfo::getId), + PATH(DropboxFileInfo.PATH, DropboxFileInfo::getPath), + FILENAME(DropboxFileInfo.FILENAME, DropboxFileInfo::getName), + SIZE(DropboxFileInfo.SIZE, fileInfo -> String.valueOf(fileInfo.getSize())), + TIMESTAMP(DropboxFileInfo.TIMESTAMP, fileInfo -> String.valueOf(fileInfo.getTimestamp())), + REVISION(DropboxFileInfo.REVISION, DropboxFileInfo::getRevision); + + private final String name; + private final Function fromFileInfo; + + DropboxFlowFileAttribute(String attributeName, Function fromFileInfo) { + this.name = attributeName; + this.fromFileInfo = fromFileInfo; + } + + public String getName() { + return name; + } + + public String getValue(DropboxFileInfo fileInfo) { + return fromFileInfo.apply(fileInfo); + } + +} diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/java/org/apache/nifi/processors/dropbox/ListDropbox.java b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/java/org/apache/nifi/processors/dropbox/ListDropbox.java new file mode 100644 index 0000000000..2c1b19fe40 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/java/org/apache/nifi/processors/dropbox/ListDropbox.java @@ -0,0 +1,293 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.processors.dropbox; + +import static java.lang.String.format; +import static java.util.stream.Collectors.toList; + +import com.dropbox.core.DbxException; +import com.dropbox.core.DbxRequestConfig; +import com.dropbox.core.oauth.DbxCredential; +import com.dropbox.core.v2.DbxClientV2; +import com.dropbox.core.v2.files.FileMetadata; +import com.dropbox.core.v2.files.ListFolderBuilder; +import com.dropbox.core.v2.files.ListFolderResult; +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.TimeUnit; +import java.util.function.Predicate; +import java.util.regex.Pattern; +import org.apache.nifi.annotation.behavior.InputRequirement; +import org.apache.nifi.annotation.behavior.InputRequirement.Requirement; +import org.apache.nifi.annotation.behavior.PrimaryNodeOnly; +import org.apache.nifi.annotation.behavior.Stateful; +import org.apache.nifi.annotation.behavior.TriggerSerially; +import org.apache.nifi.annotation.behavior.WritesAttribute; +import org.apache.nifi.annotation.behavior.WritesAttributes; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnScheduled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.components.state.Scope; +import org.apache.nifi.context.PropertyContext; +import org.apache.nifi.dropbox.credentials.service.DropboxCredentialDetails; +import org.apache.nifi.dropbox.credentials.service.DropboxCredentialService; +import org.apache.nifi.expression.ExpressionLanguageScope; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.processor.util.StandardValidators; +import org.apache.nifi.processor.util.list.AbstractListProcessor; +import org.apache.nifi.processor.util.list.ListedEntityTracker; +import org.apache.nifi.serialization.record.RecordSchema; + +@PrimaryNodeOnly +@TriggerSerially +@Tags({"dropbox", "storage"}) +@CapabilityDescription("Retrieves a listing of files from Dropbox (shortcuts are ignored)." + + " Each listed file may result in one FlowFile, the metadata being written as Flowfile attributes." + + " When the 'Record Writer' property is set, the entire result is written as records to a single FlowFile." + + " This Processor is designed to run on Primary Node only in a cluster. If the primary node changes, the new Primary Node will pick up where the" + + " previous node left off without duplicating all of the data.") +@InputRequirement(Requirement.INPUT_FORBIDDEN) +@WritesAttributes({@WritesAttribute(attribute = DropboxFileInfo.ID, description = "The Dropbox identifier of the file"), + @WritesAttribute(attribute = DropboxFileInfo.PATH, description = "The folder path where the file is located"), + @WritesAttribute(attribute = DropboxFileInfo.FILENAME, description = "The name of the file"), + @WritesAttribute(attribute = DropboxFileInfo.SIZE, description = "The size of the file"), + @WritesAttribute(attribute = DropboxFileInfo.TIMESTAMP, description = "The server modified time, when the file was uploaded to Dropbox"), + @WritesAttribute(attribute = DropboxFileInfo.REVISION, description = "Revision of the file")}) +@Stateful(scopes = {Scope.CLUSTER}, description = "The processor stores necessary data to be able to keep track what files have been listed already. " + + "What exactly needs to be stored depends on the 'Listing Strategy'.") +public class ListDropbox extends AbstractListProcessor { + + public static final PropertyDescriptor FOLDER = new PropertyDescriptor.Builder() + .name("folder") + .displayName("Folder") + .description("The Dropbox identifier or path of the folder from which to pull list of files." + + " 'Folder' should match the following regular expression pattern: /.*|id:.* ." + + " Example for folder identifier: id:odTlUvbpIEAAAAAAAAAGGQ." + + " Example for folder path: /Team1/Task1.") + .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) + .required(true) + .addValidator(StandardValidators.createRegexMatchingValidator(Pattern.compile("/.*|id:.*"))) + .defaultValue("/") + .build(); + + public static final PropertyDescriptor RECURSIVE_SEARCH = new PropertyDescriptor.Builder() + .name("recursive-search") + .displayName("Search Recursively") + .description("Indicates whether to list files from subfolders of the Dropbox folder.") + .required(true) + .defaultValue("true") + .allowableValues("true", "false") + .build(); + + public static final PropertyDescriptor MIN_AGE = new PropertyDescriptor.Builder() + .name("min-age") + .displayName("Minimum File Age") + .description("The minimum age a file must be in order to be considered; any files newer than this will be ignored.") + .required(true) + .addValidator(StandardValidators.TIME_PERIOD_VALIDATOR) + .defaultValue("0 sec") + .build(); + + public static final PropertyDescriptor CREDENTIAL_SERVICE = new PropertyDescriptor.Builder() + .name("dropbox-credential-service") + .displayName("Dropbox Credential Service") + .description("Controller Service used to obtain Dropbox credentials (App Key, App Secret, Access Token, Refresh Token)." + + " See controller service's Additional Details for more information.") + .identifiesControllerService(DropboxCredentialService.class) + .required(true) + .build(); + + public static final PropertyDescriptor LISTING_STRATEGY = new PropertyDescriptor.Builder() + .fromPropertyDescriptor(AbstractListProcessor.LISTING_STRATEGY) + .allowableValues(BY_TIMESTAMPS, BY_ENTITIES, BY_TIME_WINDOW, NO_TRACKING) + .build(); + + public static final PropertyDescriptor TRACKING_STATE_CACHE = new PropertyDescriptor.Builder() + .fromPropertyDescriptor(ListedEntityTracker.TRACKING_STATE_CACHE) + .dependsOn(LISTING_STRATEGY, BY_ENTITIES) + .build(); + + public static final PropertyDescriptor TRACKING_TIME_WINDOW = new PropertyDescriptor.Builder() + .fromPropertyDescriptor(ListedEntityTracker.TRACKING_TIME_WINDOW) + .dependsOn(LISTING_STRATEGY, BY_ENTITIES) + .build(); + + public static final PropertyDescriptor INITIAL_LISTING_TARGET = new PropertyDescriptor.Builder() + .fromPropertyDescriptor(ListedEntityTracker.INITIAL_LISTING_TARGET) + .dependsOn(LISTING_STRATEGY, BY_ENTITIES) + .build(); + + private static final List PROPERTIES = Collections.unmodifiableList(Arrays.asList( + CREDENTIAL_SERVICE, + FOLDER, + RECURSIVE_SEARCH, + MIN_AGE, + LISTING_STRATEGY, + TRACKING_STATE_CACHE, + TRACKING_TIME_WINDOW, + INITIAL_LISTING_TARGET, + RECORD_WRITER + )); + + private DbxClientV2 dropboxApiClient; + + @OnScheduled + public void onScheduled(final ProcessContext context) { + dropboxApiClient = getDropboxApiClient(context); + } + + @Override + protected List getSupportedPropertyDescriptors() { + return PROPERTIES; + } + + @Override + protected Map createAttributes( + final DropboxFileInfo entity, + final ProcessContext context) { + final Map attributes = new HashMap<>(); + + for (DropboxFlowFileAttribute attribute : DropboxFlowFileAttribute.values()) { + Optional.ofNullable(attribute.getValue(entity)) + .ifPresent(value -> attributes.put(attribute.getName(), value)); + } + + return attributes; + } + + @Override + protected String getPath(final ProcessContext context) { + return context.getProperty(FOLDER).evaluateAttributeExpressions().getValue(); + } + + protected DbxClientV2 getDropboxApiClient(ProcessContext context) { + final DropboxCredentialService credentialService = context.getProperty(CREDENTIAL_SERVICE) + .asControllerService(DropboxCredentialService.class); + DbxRequestConfig config = new DbxRequestConfig(format("%s-%s", getClass().getSimpleName(), getIdentifier())); + DropboxCredentialDetails credential = credentialService.getDropboxCredential(); + return new DbxClientV2(config, new DbxCredential(credential.getAccessToken(), -1L, + credential.getRefreshToken(), credential.getAppKey(), credential.getAppSecret())); + } + + @Override + protected List performListing(ProcessContext context, Long minTimestamp, + ListingMode listingMode) throws IOException { + final List listing = new ArrayList<>(); + + final String folderName = getPath(context); + final Boolean recursive = context.getProperty(RECURSIVE_SEARCH).asBoolean(); + final Long minAge = context.getProperty(MIN_AGE).asTimePeriod(TimeUnit.MILLISECONDS); + + try { + Predicate metadataFilter = createMetadataFilter(minTimestamp, minAge); + + ListFolderBuilder listFolderBuilder = dropboxApiClient.files().listFolderBuilder(convertFolderName(folderName)); + ListFolderResult result = listFolderBuilder + .withRecursive(recursive) + .start(); + + List metadataList = new ArrayList<>(filterMetadata(result, metadataFilter)); + + while (result.getHasMore()) { + result = dropboxApiClient.files().listFolderContinue(result.getCursor()); + metadataList.addAll(filterMetadata(result, metadataFilter)); + } + + for (FileMetadata metadata : metadataList) { + DropboxFileInfo.Builder builder = new DropboxFileInfo.Builder() + .id(metadata.getId()) + .path(getParentPath(metadata.getPathDisplay())) + .name(metadata.getName()) + .size(metadata.getSize()) + .timestamp(metadata.getServerModified().getTime()) + .revision(metadata.getRev()); + + listing.add(builder.build()); + } + } catch (DbxException e) { + throw new IOException("Failed to list Dropbox folder [" + folderName + "]", e); + } + + return listing; + } + + @Override + protected boolean isListingResetNecessary(final PropertyDescriptor property) { + return LISTING_STRATEGY.equals(property) + || FOLDER.equals(property) + || RECURSIVE_SEARCH.equals(property); + } + + @Override + protected Scope getStateScope(final PropertyContext context) { + return Scope.CLUSTER; + } + + @Override + protected RecordSchema getRecordSchema() { + return DropboxFileInfo.getRecordSchema(); + } + + @Override + protected Integer countUnfilteredListing(final ProcessContext context) throws IOException { + return performListing(context, null, ListingMode.CONFIGURATION_VERIFICATION).size(); + } + + @Override + protected String getListingContainerName(final ProcessContext context) { + return format("Dropbox Folder [%s]", getPath(context)); + } + + private Predicate createMetadataFilter(Long minTimestamp, Long minAge) { + Predicate metadataFilter = FileMetadata::getIsDownloadable; + + if (minTimestamp != null && minTimestamp > 0) { + metadataFilter = metadataFilter.and(metadata -> metadata.getServerModified().getTime() >= minTimestamp); + } + + if (minAge != null && minAge > 0) { + long maxTimestamp = System.currentTimeMillis() - minAge; + metadataFilter = metadataFilter.and(metadata -> metadata.getServerModified().getTime() < maxTimestamp); + } + return metadataFilter; + } + + private List filterMetadata(ListFolderResult result, Predicate metadataFilter) { + return result.getEntries().stream() + .filter(metadata -> metadata instanceof FileMetadata) + .map(FileMetadata.class::cast) + .filter(metadataFilter) + .collect(toList()); + } + + private String getParentPath(String fullPath) { + int idx = fullPath.lastIndexOf("/"); + String parentPath = fullPath.substring(0, idx); + return "".equals(parentPath) ? "/" : parentPath; + } + + private String convertFolderName(String folderName) { + return "/".equals(folderName) ? "" : folderName; + } +} diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor new file mode 100644 index 0000000000..ab3ec13b23 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor @@ -0,0 +1,15 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +org.apache.nifi.processors.dropbox.ListDropbox diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/AbstractDropboxIT.java b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/AbstractDropboxIT.java new file mode 100644 index 0000000000..346116035a --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/AbstractDropboxIT.java @@ -0,0 +1,128 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.dropbox; + +import com.dropbox.core.DbxRequestConfig; +import com.dropbox.core.oauth.DbxCredential; +import com.dropbox.core.v2.DbxClientV2; +import com.dropbox.core.v2.files.FileMetadata; +import com.dropbox.core.v2.files.GetMetadataErrorException; +import java.io.ByteArrayInputStream; +import java.nio.charset.StandardCharsets; +import org.apache.nifi.processor.Processor; +import org.apache.nifi.services.dropbox.StandardDropboxCredentialService; +import org.apache.nifi.util.TestRunner; +import org.apache.nifi.util.TestRunners; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; + +/** + * Set the following constants before running:
+ *
APPLICATION_KEY - App Key of your app + *
APPLICATION_SECRET - Your App Secret of your app + *
ACCESS_TOKEN - Access Token generated for your app + *
REFRESH_TOKEN - Refresh Token generated for your app + *

+ * For App Key, App Secret, Access Token and Refresh Token generation please check controller service's Additional Details page.

+ * NOTE: Since the integration test creates the test files you need "files.content.write" permission besides the "files.content.read" + * permission mentioned in controller service's Additional Details page.
+ * + *
Created files and folders are cleaned up, but it's advisable to dedicate a folder for this test so that it can + * be cleaned up easily should the test fail to do so.
+ *
WARNING: The creation of a file is not a synchronized operation, may need to adjust tests accordingly! + */ +public abstract class AbstractDropboxIT { + + public static final String APP_KEY = ""; + public static final String APP_SECRET = ""; + public static final String ACCESS_TOKEN = ""; + public static final String REFRESH_TOKEN = ""; + + public static final String MAIN_FOLDER = "/testFolder"; + protected T testSubject; + protected TestRunner testRunner; + protected String mainFolderId; + + private DbxClientV2 client; + + @AfterEach + public void teardown() throws Exception { + deleteFolderIfExists(MAIN_FOLDER); + } + + protected abstract T createTestSubject(); + + @BeforeEach + protected void init() throws Exception { + testSubject = createTestSubject(); + testRunner = createTestRunner(); + DbxCredential credential = + new DbxCredential(ACCESS_TOKEN, -1L, REFRESH_TOKEN, APP_KEY, APP_SECRET); + DbxRequestConfig config = new DbxRequestConfig("nifi"); + client = new DbxClientV2(config, credential); + mainFolderId = createFolder(MAIN_FOLDER); + } + + protected TestRunner createTestRunner() throws Exception { + TestRunner testRunner = TestRunners.newTestRunner(testSubject); + + StandardDropboxCredentialService controllerService = new StandardDropboxCredentialService(); + testRunner.addControllerService("dropbox_credential_provider_service", controllerService); + testRunner.setProperty(controllerService, StandardDropboxCredentialService.APP_KEY, APP_KEY); + testRunner.setProperty(controllerService, StandardDropboxCredentialService.APP_SECRET, APP_SECRET); + testRunner.setProperty(controllerService, StandardDropboxCredentialService.ACCESS_TOKEN, ACCESS_TOKEN); + testRunner.setProperty(controllerService, StandardDropboxCredentialService.REFRESH_TOKEN, REFRESH_TOKEN); + testRunner.enableControllerService(controllerService); + testRunner.setProperty(ListDropbox.CREDENTIAL_SERVICE, "dropbox_credential_provider_service"); + + return testRunner; + } + + protected void deleteFolderIfExists(String path) throws Exception { + if (folderExists(path)) { + client.files().deleteV2(path); + } + } + + protected FileMetadata createFile(String name, String fileContent, String folder) throws Exception { + ByteArrayInputStream content = new ByteArrayInputStream(fileContent.getBytes(StandardCharsets.UTF_8)); + return client.files().upload(folder + "/" + name).uploadAndFinish(content); + } + + private String createFolder(String path) throws Exception { + if (folderExists(path)) { + deleteFolder(path); + } + return client.files().createFolderV2(path).getMetadata().getId(); + } + + private void deleteFolder(String path) throws Exception { + client.files().deleteV2(path); + } + + private boolean folderExists(String path) throws Exception { + try { + return client.files().getMetadata(path) != null; + } catch (GetMetadataErrorException e) { + if (e.errorValue.isPath() && e.errorValue.getPathValue().isNotFound()) { + return false; + } + throw e; + } + } +} \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxIT.java b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxIT.java new file mode 100644 index 0000000000..d027f23a5c --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxIT.java @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.dropbox; + +import static java.util.Collections.emptyList; +import static java.util.Collections.singletonList; +import static java.util.stream.Collectors.toList; +import static org.junit.jupiter.api.Assertions.assertEquals; + +import java.util.Arrays; +import java.util.List; +import org.apache.nifi.util.MockFlowFile; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +public class ListDropboxIT extends AbstractDropboxIT { + + private static final String NOT_MAIN_FOLDER = "/notMainFolder"; + private static final String YOUNG_FILE_NAME = "just_created"; + + @BeforeEach + public void init() throws Exception { + super.init(); + testRunner.setProperty(ListDropbox.FOLDER, MAIN_FOLDER); + } + + @AfterEach + public void teardown() throws Exception { + super.teardown(); + deleteFolderIfExists(NOT_MAIN_FOLDER); + } + + @Override + protected ListDropbox createTestSubject() { + return new ListDropbox(); + } + + @Test + void testEmbeddedDirectoriesAreListed() throws Exception { + createFile("test_file1", "test_file_content1", MAIN_FOLDER); + createFile("test_file2", "test_file_content2", MAIN_FOLDER); + createFile("test_file11", "test_file_content11", MAIN_FOLDER + "/testFolder1"); + createFile("test_file112", "test_file_content112", MAIN_FOLDER + "/testFolder2"); + + createFile("test_file_not_in_main_folder", "test_file_content31", NOT_MAIN_FOLDER); + + List expectedFileNames = Arrays.asList("test_file1", "test_file2", "test_file11", "test_file112"); + + waitForFileCreation(); + + testRunner.run(); + + List successFlowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS); + + List actualFileNames = getFilenames(successFlowFiles); + + assertEquals(expectedFileNames, actualFileNames); + } + + @Test + void testFolderIsListedById() throws Exception { + testRunner.setProperty(ListDropbox.FOLDER, mainFolderId); + + createFile("test_file1", "test_file_content1", MAIN_FOLDER); + createFile("test_file11", "test_file_content11", MAIN_FOLDER + "/testFolder1"); + + List expectedFileNames = Arrays.asList("test_file1", "test_file11"); + + waitForFileCreation(); + + testRunner.run(); + + List successFlowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS); + + List actualFileNames = getFilenames(successFlowFiles); + + assertEquals(expectedFileNames, actualFileNames); + } + + @Test + void testTooYoungFilesNotListedWhenMinAgeIsSet() throws Exception { + testRunner.setProperty(ListDropbox.MIN_AGE, "15 s"); + + createFile(YOUNG_FILE_NAME, "test_file_content1", MAIN_FOLDER); + + waitForFileCreation(); + + testRunner.run(); + + List successFlowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS); + + List actualFileNames = getFilenames(successFlowFiles); + + assertEquals(emptyList(), actualFileNames); + + // Next, wait for another 10+ seconds for MIN_AGE to expire then list again. + Thread.sleep(10000); + + List expectedFileNames = singletonList(YOUNG_FILE_NAME); + + testRunner.run(); + + successFlowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS); + + actualFileNames = getFilenames(successFlowFiles); + + assertEquals(expectedFileNames, actualFileNames); + } + + private void waitForFileCreation() throws InterruptedException { + // We need to wait since the creation of the files are not (completely) synchronized. + Thread.sleep(5000); + } + + private List getFilenames(List flowFiles) { + return flowFiles.stream() + .map(flowFile -> flowFile.getAttribute("filename")) + .collect(toList()); + } +} diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxTest.java b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxTest.java new file mode 100644 index 0000000000..33899936c4 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxTest.java @@ -0,0 +1,285 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.nifi.processors.dropbox; + +import static java.util.Collections.singletonList; +import static java.util.Spliterators.spliteratorUnknownSize; +import static java.util.stream.Collectors.toList; +import static org.apache.nifi.services.dropbox.StandardDropboxCredentialService.ACCESS_TOKEN; +import static org.apache.nifi.services.dropbox.StandardDropboxCredentialService.APP_KEY; +import static org.apache.nifi.services.dropbox.StandardDropboxCredentialService.APP_SECRET; +import static org.apache.nifi.services.dropbox.StandardDropboxCredentialService.REFRESH_TOKEN; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.mockito.Mockito.when; + +import com.dropbox.core.DbxException; +import com.dropbox.core.v2.DbxClientV2; +import com.dropbox.core.v2.files.DbxUserFilesRequests; +import com.dropbox.core.v2.files.FileMetadata; +import com.dropbox.core.v2.files.FolderMetadata; +import com.dropbox.core.v2.files.ListFolderBuilder; +import com.dropbox.core.v2.files.ListFolderResult; +import com.dropbox.core.v2.files.Metadata; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; +import java.util.Date; +import java.util.List; +import java.util.Spliterator; +import java.util.stream.StreamSupport; +import org.apache.nifi.json.JsonRecordSetWriter; +import org.apache.nifi.processor.ProcessContext; +import org.apache.nifi.reporting.InitializationException; +import org.apache.nifi.serialization.RecordSetWriterFactory; +import org.apache.nifi.services.dropbox.StandardDropboxCredentialService; +import org.apache.nifi.util.MockFlowFile; +import org.apache.nifi.util.TestRunner; +import org.apache.nifi.util.TestRunners; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +@ExtendWith(MockitoExtension.class) +public class ListDropboxTest { + + public static final String ID_1 = "id:11111"; + public static final String ID_2 = "id:22222"; + public static final String TEST_FOLDER = "/testFolder"; + public static final String FILENAME_1 = "file_name_1"; + public static final String FILENAME_2 = "file_name_2"; + public static final long SIZE = 125; + public static final long CREATED_TIME = 1659707000; + public static final String REVISION = "5e4ddb1320676a5c29261"; + public static final boolean IS_RECURSIVE = true; + public static final long MIN_TIMESTAMP = 1659707000; + public static final long OLD_CREATED_TIME = 1657375066; + private TestRunner testRunner; + + @Mock + private DbxClientV2 mockDropboxClient; + + @Mock + private StandardDropboxCredentialService credentialService; + + @Mock + private DbxUserFilesRequests mockDbxUserFilesRequest; + + @Mock + private ListFolderResult mockListFolderResult; + + @Mock + private ListFolderBuilder mockListFolderBuilder; + + @BeforeEach + void setUp() throws Exception { + ListDropbox testSubject = new ListDropbox() { + @Override + public DbxClientV2 getDropboxApiClient(ProcessContext context) { + return mockDropboxClient; + } + + @Override + protected List performListing( + ProcessContext context, Long minTimestamp, ListingMode ignoredListingMode) throws IOException { + return super.performListing(context, MIN_TIMESTAMP, ListingMode.EXECUTION); + } + }; + + testRunner = TestRunners.newTestRunner(testSubject); + + mockStandardDropboxCredentialService(); + + testRunner.setProperty(ListDropbox.RECURSIVE_SEARCH, Boolean.toString(IS_RECURSIVE)); + testRunner.setProperty(ListDropbox.MIN_AGE, "0 sec"); + } + + @Test + void testFolderValidity() { + testRunner.setProperty(ListDropbox.FOLDER, "id:odTlUvbpIEAAAAAAAAABmw"); + testRunner.assertValid(); + testRunner.setProperty(ListDropbox.FOLDER, "/"); + testRunner.assertValid(); + testRunner.setProperty(ListDropbox.FOLDER, "/tempFolder"); + testRunner.assertValid(); + testRunner.setProperty(ListDropbox.FOLDER, "/tempFolder/tempSubFolder"); + testRunner.assertValid(); + testRunner.setProperty(ListDropbox.FOLDER, "/tempFolder/tempSubFolder/"); + testRunner.assertValid(); + testRunner.setProperty(ListDropbox.FOLDER, "tempFolder"); + testRunner.assertNotValid(); + testRunner.setProperty(ListDropbox.FOLDER, "odTlUvbpIEAAAAAAAAABmw"); + testRunner.assertNotValid(); + testRunner.setProperty(ListDropbox.FOLDER, ""); + testRunner.assertNotValid(); + } + + @Test + void testRootIsListed() throws Exception { + mockFileListing(); + + String folderName = "/"; + testRunner.setProperty(ListDropbox.FOLDER, folderName); + + //root is listed when "" is used in Dropbox API + when(mockDbxUserFilesRequest.listFolderBuilder("")).thenReturn(mockListFolderBuilder); + when(mockListFolderResult.getEntries()).thenReturn(singletonList( + createFileMetaData(FILENAME_1, folderName, ID_1, CREATED_TIME) + )); + + testRunner.run(); + + testRunner.assertAllFlowFilesTransferred(ListDropbox.REL_SUCCESS, 1); + List flowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS); + MockFlowFile ff0 = flowFiles.get(0); + assertFlowFileAttributes(ff0, folderName); + } + + @Test + void testOnlyFilesAreListedFolderIsFiltered() throws Exception { + mockFileListing(); + + testRunner.setProperty(ListDropbox.FOLDER, TEST_FOLDER); + + when(mockDbxUserFilesRequest.listFolderBuilder(TEST_FOLDER)).thenReturn(mockListFolderBuilder); + when(mockListFolderResult.getEntries()).thenReturn(Arrays.asList( + createFileMetaData(FILENAME_1, TEST_FOLDER, ID_1, CREATED_TIME), + createFolderMetaData("testFolder1", TEST_FOLDER) + )); + + testRunner.run(); + + testRunner.assertAllFlowFilesTransferred(ListDropbox.REL_SUCCESS, 1); + List flowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS); + MockFlowFile ff0 = flowFiles.get(0); + assertFlowFileAttributes(ff0, TEST_FOLDER); + } + + @Test + void testOldItemIsFiltered() throws Exception { + mockFileListing(); + + testRunner.setProperty(ListDropbox.FOLDER, TEST_FOLDER); + + when(mockDbxUserFilesRequest.listFolderBuilder(TEST_FOLDER)).thenReturn(mockListFolderBuilder); + when(mockListFolderResult.getEntries()).thenReturn(Arrays.asList( + createFileMetaData(FILENAME_1, TEST_FOLDER, ID_1, CREATED_TIME), + createFileMetaData(FILENAME_2, TEST_FOLDER, ID_2, OLD_CREATED_TIME) + )); + + testRunner.run(); + + testRunner.assertAllFlowFilesTransferred(ListDropbox.REL_SUCCESS, 1); + List flowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS); + MockFlowFile ff0 = flowFiles.get(0); + assertFlowFileAttributes(ff0, TEST_FOLDER); + } + + @Test + void testRecordWriter() throws Exception { + mockFileListing(); + mockRecordWriter(); + + testRunner.setProperty(ListDropbox.FOLDER, TEST_FOLDER); + + when(mockDbxUserFilesRequest.listFolderBuilder(TEST_FOLDER)).thenReturn(mockListFolderBuilder); + when(mockListFolderResult.getEntries()).thenReturn(Arrays.asList( + createFileMetaData(FILENAME_1, TEST_FOLDER, ID_1, CREATED_TIME), + createFileMetaData(FILENAME_2, TEST_FOLDER, ID_2, CREATED_TIME) + )); + + testRunner.run(); + + testRunner.assertAllFlowFilesTransferred(ListDropbox.REL_SUCCESS, 1); + List flowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS); + MockFlowFile ff0 = flowFiles.get(0); + List expectedFileNames = Arrays.asList(FILENAME_1, FILENAME_2); + List actualFileNames = getFilenames(ff0.getContent()); + + assertEquals(expectedFileNames, actualFileNames); + } + + private void assertFlowFileAttributes(MockFlowFile flowFile, String folderName) { + flowFile.assertAttributeEquals(DropboxFileInfo.ID, ID_1); + flowFile.assertAttributeEquals(DropboxFileInfo.FILENAME, FILENAME_1); + flowFile.assertAttributeEquals(DropboxFileInfo.PATH, folderName); + flowFile.assertAttributeEquals(DropboxFileInfo.TIMESTAMP, Long.toString(CREATED_TIME)); + flowFile.assertAttributeEquals(DropboxFileInfo.SIZE, Long.toString(SIZE)); + flowFile.assertAttributeEquals(DropboxFileInfo.REVISION, REVISION); + } + + private FileMetadata createFileMetaData( + String filename, + String parent, + String id, + long createdTime) { + return FileMetadata.newBuilder(filename, id, + new Date(createdTime), + new Date(createdTime), + REVISION, SIZE) + .withPathDisplay(parent + "/" + filename) + .build(); + } + + private Metadata createFolderMetaData(String folderName, String parent) { + return FolderMetadata.newBuilder(folderName) + .withPathDisplay(parent + "/" + folderName) + .build(); + } + + private void mockStandardDropboxCredentialService() throws Exception { + String credentialServiceId = "dropbox_credentials"; + when(credentialService.getIdentifier()).thenReturn(credentialServiceId); + testRunner.addControllerService(credentialServiceId, credentialService); + testRunner.setProperty(credentialService, APP_KEY, "appKey"); + testRunner.setProperty(credentialService, APP_SECRET, "appSecret"); + testRunner.setProperty(credentialService, ACCESS_TOKEN, "accessToken"); + testRunner.setProperty(credentialService, REFRESH_TOKEN, "refreshToken"); + testRunner.enableControllerService(credentialService); + testRunner.setProperty(ListDropbox.CREDENTIAL_SERVICE, credentialServiceId); + } + + private void mockRecordWriter() throws InitializationException { + RecordSetWriterFactory recordWriter = new JsonRecordSetWriter(); + testRunner.addControllerService("record_writer", recordWriter); + testRunner.enableControllerService(recordWriter); + testRunner.setProperty(ListDropbox.RECORD_WRITER, "record_writer"); + } + + private void mockFileListing() throws DbxException { + when(mockListFolderBuilder.withRecursive(IS_RECURSIVE)).thenReturn(mockListFolderBuilder); + when(mockListFolderBuilder.start()).thenReturn(mockListFolderResult); + when(mockDropboxClient.files()).thenReturn(mockDbxUserFilesRequest); + when(mockListFolderResult.getHasMore()).thenReturn(false); + } + + private List getFilenames(String flowFileContent) { + try { + JsonNode jsonNode = new ObjectMapper().readTree(flowFileContent); + return StreamSupport.stream(spliteratorUnknownSize(jsonNode.iterator(), Spliterator.ORDERED), false) + .map(node -> node.get("filename").asText()) + .collect(toList()); + } catch (JsonProcessingException e) { + return Collections.emptyList(); + } + } +} diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api-nar/pom.xml b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api-nar/pom.xml new file mode 100644 index 0000000000..08c0839c28 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api-nar/pom.xml @@ -0,0 +1,46 @@ + + + + 4.0.0 + + + nifi-dropbox-bundle + org.apache.nifi + 1.18.0-SNAPSHOT + + + nifi-dropbox-services-api-nar + 1.18.0-SNAPSHOT + nar + + true + true + + + + + org.apache.nifi + nifi-dropbox-services-api + 1.18.0-SNAPSHOT + + + org.apache.nifi + nifi-standard-services-api-nar + 1.18.0-SNAPSHOT + nar + + + \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api-nar/src/main/resources/META-INF/LICENSE b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api-nar/src/main/resources/META-INF/LICENSE new file mode 100644 index 0000000000..7a4a3ea242 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api-nar/src/main/resources/META-INF/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api-nar/src/main/resources/META-INF/NOTICE b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api-nar/src/main/resources/META-INF/NOTICE new file mode 100644 index 0000000000..cd10c27e09 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api-nar/src/main/resources/META-INF/NOTICE @@ -0,0 +1,5 @@ +nifi-dropbox-services-api-nar +Copyright 2015-2022 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api/pom.xml b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api/pom.xml new file mode 100644 index 0000000000..4ac6405b45 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api/pom.xml @@ -0,0 +1,34 @@ + + + + 4.0.0 + + + nifi-dropbox-bundle + org.apache.nifi + 1.18.0-SNAPSHOT + + + nifi-dropbox-services-api + jar + + + + org.apache.nifi + nifi-api + + + \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api/src/main/java/org/apache/nifi/dropbox/credentials/service/DropboxCredentialDetails.java b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api/src/main/java/org/apache/nifi/dropbox/credentials/service/DropboxCredentialDetails.java new file mode 100644 index 0000000000..377bd27014 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api/src/main/java/org/apache/nifi/dropbox/credentials/service/DropboxCredentialDetails.java @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.dropbox.credentials.service; + +public class DropboxCredentialDetails { + private final String appKey; + private final String appSecret; + private final String accessToken; + private final String refreshToken; + + public DropboxCredentialDetails(String appKey, String appSecret, String accessToken, String refreshToken) { + this.appKey = appKey; + this.appSecret = appSecret; + this.accessToken = accessToken; + this.refreshToken = refreshToken; + } + + public String getAppKey() { + return appKey; + } + + public String getAppSecret() { + return appSecret; + } + + public String getAccessToken() { + return accessToken; + } + + public String getRefreshToken() { + return refreshToken; + } +} diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api/src/main/java/org/apache/nifi/dropbox/credentials/service/DropboxCredentialService.java b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api/src/main/java/org/apache/nifi/dropbox/credentials/service/DropboxCredentialService.java new file mode 100644 index 0000000000..6d63ba9620 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-api/src/main/java/org/apache/nifi/dropbox/credentials/service/DropboxCredentialService.java @@ -0,0 +1,39 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.dropbox.credentials.service; + +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.controller.ControllerService; + +/** + * DropboxCredentialService interface to support getting Dropbox credentials (app key, app secret, access token, refresh token) + * + * + * @see Dropbox Developers Getting Started + * @see DbxCredential + */ +@Tags({"dropbox", "credentials", "auth", "session"}) +@CapabilityDescription("Provides DropboxCredentialDetails.") +public interface DropboxCredentialService extends ControllerService { + /** + * Get Dropbox credentials + * @return DropboxCredentialDetails object which contains app key, app secret, access and refresh tokens for authorizing + * requests on the Dropbox platform. + */ + DropboxCredentialDetails getDropboxCredential(); +} diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-nar/pom.xml b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-nar/pom.xml new file mode 100644 index 0000000000..1b220119bf --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-nar/pom.xml @@ -0,0 +1,45 @@ + + + + 4.0.0 + + + org.apache.nifi + nifi-dropbox-bundle + 1.18.0-SNAPSHOT + + + nifi-dropbox-services-nar + nar + + true + true + + + + + org.apache.nifi + nifi-dropbox-services-api-nar + 1.18.0-SNAPSHOT + nar + + + org.apache.nifi + nifi-dropbox-services + 1.18.0-SNAPSHOT + + + \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-nar/src/main/resources/META-INF/LICENSE b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-nar/src/main/resources/META-INF/LICENSE new file mode 100644 index 0000000000..7a4a3ea242 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-nar/src/main/resources/META-INF/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-nar/src/main/resources/META-INF/NOTICE b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-nar/src/main/resources/META-INF/NOTICE new file mode 100644 index 0000000000..2075708639 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services-nar/src/main/resources/META-INF/NOTICE @@ -0,0 +1,5 @@ +nifi-dropbox-services-nar +Copyright 2015-2022 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/pom.xml b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/pom.xml new file mode 100644 index 0000000000..3eead7465f --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/pom.xml @@ -0,0 +1,45 @@ + + + + 4.0.0 + + + org.apache.nifi + nifi-dropbox-bundle + 1.18.0-SNAPSHOT + + + nifi-dropbox-services + jar + + + + org.apache.nifi + nifi-api + + + org.apache.nifi + nifi-utils + 1.18.0-SNAPSHOT + + + org.apache.nifi + nifi-dropbox-services-api + 1.18.0-SNAPSHOT + provided + + + \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/src/main/java/org/apache/nifi/services/dropbox/StandardDropboxCredentialService.java b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/src/main/java/org/apache/nifi/services/dropbox/StandardDropboxCredentialService.java new file mode 100644 index 0000000000..33a9715224 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/src/main/java/org/apache/nifi/services/dropbox/StandardDropboxCredentialService.java @@ -0,0 +1,112 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.nifi.services.dropbox; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import org.apache.nifi.annotation.documentation.CapabilityDescription; +import org.apache.nifi.annotation.documentation.Tags; +import org.apache.nifi.annotation.lifecycle.OnEnabled; +import org.apache.nifi.components.PropertyDescriptor; +import org.apache.nifi.controller.AbstractControllerService; +import org.apache.nifi.controller.ConfigurationContext; +import org.apache.nifi.dropbox.credentials.service.DropboxCredentialDetails; +import org.apache.nifi.dropbox.credentials.service.DropboxCredentialService; +import org.apache.nifi.expression.ExpressionLanguageScope; +import org.apache.nifi.processor.util.StandardValidators; + +@CapabilityDescription("Defines credentials for Dropbox processors.") +@Tags({"dropbox", "credentials", "provider"}) +public class StandardDropboxCredentialService extends AbstractControllerService implements DropboxCredentialService { + + public static final PropertyDescriptor APP_KEY = new PropertyDescriptor.Builder() + .name("app-key") + .displayName("App Key") + .description("App Key of the user's Dropbox app." + + " See Additional Details for more information.") + .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .required(true) + .build(); + + public static final PropertyDescriptor APP_SECRET = new PropertyDescriptor.Builder() + .name("app-secret") + .displayName("App Secret") + .description("App Secret of the user's Dropbox app." + + " See Additional Details for more information.") + .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) + .sensitive(true) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .required(true) + .build(); + + public static final PropertyDescriptor ACCESS_TOKEN = new PropertyDescriptor.Builder() + .name("access-token") + .displayName("Access Token") + .description("Access Token of the user's Dropbox app." + + " See Additional Details for more information about Access Token generation.") + .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) + .sensitive(true) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .required(true) + .build(); + + public static final PropertyDescriptor REFRESH_TOKEN = new PropertyDescriptor.Builder() + .name("refresh-token") + .displayName("Refresh Token") + .description("Refresh Token of the user's Dropbox app." + + " See Additional Details for more information about Refresh Token generation.") + .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY) + .sensitive(true) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .required(true) + .build(); + + private static final List PROPERTIES; + + static { + final List props = new ArrayList<>(); + props.add(APP_KEY); + props.add(APP_SECRET); + props.add(ACCESS_TOKEN); + props.add(REFRESH_TOKEN); + PROPERTIES = Collections.unmodifiableList(props); + } + + private DropboxCredentialDetails credential; + + @Override + public final List getSupportedPropertyDescriptors() { + return PROPERTIES; + } + + @OnEnabled + public void onEnabled(final ConfigurationContext context) { + final String appKey = context.getProperty(APP_KEY).evaluateAttributeExpressions().getValue(); + final String appSecret = context.getProperty(APP_SECRET).evaluateAttributeExpressions().getValue(); + final String accessToken = context.getProperty(ACCESS_TOKEN).evaluateAttributeExpressions().getValue(); + final String refreshToken = context.getProperty(REFRESH_TOKEN).evaluateAttributeExpressions().getValue(); + + this.credential = new DropboxCredentialDetails(appKey, appSecret, accessToken, refreshToken); + } + + @Override + public DropboxCredentialDetails getDropboxCredential() { + return credential; + } +} diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/src/main/resources/META-INF/services/org.apache.nifi.controller.ControllerService b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/src/main/resources/META-INF/services/org.apache.nifi.controller.ControllerService new file mode 100644 index 0000000000..f561310fb3 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/src/main/resources/META-INF/services/org.apache.nifi.controller.ControllerService @@ -0,0 +1,15 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +org.apache.nifi.services.dropbox.StandardDropboxCredentialService \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/src/main/resources/docs/org.apache.nifi.services.dropbox.StandardDropboxCredentialService/additionalDetails.html b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/src/main/resources/docs/org.apache.nifi.services.dropbox.StandardDropboxCredentialService/additionalDetails.html new file mode 100644 index 0000000000..e3c330fb5d --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/nifi-dropbox-services/src/main/resources/docs/org.apache.nifi.services.dropbox.StandardDropboxCredentialService/additionalDetails.html @@ -0,0 +1,105 @@ + + + + + + + StandardDropboxCredentialService + + + + +

Generating credentials for Dropbox authentication

+ +

+ StandardDropboxCredentialService requires "App Key", "App Secret", "Access Token" and "Refresh Token".

+ This document describes how to generate these credentials using an existing Dropbox account. +

+ +

Generate App Key and App Secret

+ +

+

    +
  • Login with your Dropbox account.
    • +
  • If you already have an app created, go to Dropbox Developers page, + click on "App Console" button and select your app. On the app's info page you will find the "App key" and "App secret".
    + (See also Dropbox Getting Started, App Console tab, "Navigating the App Console" chapter)
    • + +
  • If you don't have any apps, go to Dropbox Developers page and + click on "Create app" button. (See also Dropbox Getting Started, App Console tab, + "Creating a Dropbox app" chapter.) +
      +
    • On the next page select "Scoped access" and "Full Dropbox" as access type.
      • +
    • Provide a name for your app.
      • +
    • On the app's info page you will find the "App key" and "App secret". + (See also + Dropbox Getting Started, App Console tab, "Navigating the App Console" chapter.) +
      • +
    +
    • +
+

+ +

Set required permissions for your app

+

The "files.content.read" permission has to be enabled for the application to be able to read the files in Dropbox.

+You can set permissions in Dropbox Developers page. +

  • Click on "App Console" button and select your app.
    • +
  • Go to "Permissions" tab and enable the "files.content.read" permission.
    • +
  • Click "Submit" button.
    • +
  • NOTE: In case you already have an Access Token and Refresh Token, those tokens have to be regenerated after the permission change. +See "Generate Access Token and Refresh Token" chapter about token generation.
    • +

    + +

    Generate Access Token and Refresh Token

    +

    +

      +
    • Go to the following web page: + + 
      https://www.dropbox.com/oauth2/authorize?token_access_type=offline&response_type=code&client_id=your_app_key
      • + +
    • Click "Next" and click on "Allow" button on the next page.
      • +
    • An access code will be generated for you, it will be displayed on the next page:
      • + 
      +    "Access Code Generated
+    Enter this code into your_app_name to finish the process
+    your_generated_access_code"
+
      +
    • Execute the following command from terminal to fetch the access and refresh tokens.

      + Make sure you execute the curl command right after the access code generation, since the code expires very quickly.
      + In case the curl command returns "invalid grant" error, please generate a new access code (see previous step)
      + + 
      curl https://api.dropbox.com/oauth2/token -d code=your_generated_access_code -d grant_type=authorization_code -u your_app_key:your_app_secret
      +

      +
      • +
    • The curl command results a json file which contains the "access_token" and "refresh_token":
      • + + 
      +    {
+    "access_token": "sl.xxxxxxxxxxx"
+    "expires_in": 14400,
+    "refresh_token": "xxxxxx",
+    "scope": "files.content.read files.metadata.read",
+    "uid": "xxxxxx",
+    "account_id": "dbid:xxxx"
+    }
+
      +       +
    +

    + + + + \ No newline at end of file diff --git a/nifi-nar-bundles/nifi-dropbox-bundle/pom.xml b/nifi-nar-bundles/nifi-dropbox-bundle/pom.xml new file mode 100644 index 0000000000..c6beccb1d3 --- /dev/null +++ b/nifi-nar-bundles/nifi-dropbox-bundle/pom.xml @@ -0,0 +1,41 @@ + + + + 4.0.0 + + + nifi-nar-bundles + org.apache.nifi + 1.18.0-SNAPSHOT + + + nifi-dropbox-bundle + 1.18.0-SNAPSHOT + pom + + + 4.0.1 + + + + nifi-dropbox-services-api + nifi-dropbox-services-api-nar + nifi-dropbox-processors + nifi-dropbox-processors-nar + nifi-dropbox-services + nifi-dropbox-services-nar + + \ No newline at end of file diff --git a/nifi-nar-bundles/pom.xml b/nifi-nar-bundles/pom.xml index 3c6cba86c0..a0c138216d 100755 --- a/nifi-nar-bundles/pom.xml +++ b/nifi-nar-bundles/pom.xml @@ -111,6 +111,7 @@ nifi-salesforce-bundle nifi-rocksdb-bundle nifi-hubspot-bundle + nifi-dropbox-bundle