Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,578 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24578 Commits

Author SHA1 Message Date
Steve Loughran bb04b60197
HADOOP-17735. Upgrade AWS SDK to 1.11.1026 (#3057) 
Change-Id: Ia9b32ab35454ada5f5e3d47b72cd9f2a2446a309
 2021-05-31 21:49:39 +01:00
zhengchenyu 7feb41b73d
MAPREDUCE-7287. Distcp will delete exists file , If we use "-delete and -update" options and distcp file. (#2852) 
Contributed by zhengchenyu

Change-Id: I61edf9a443c0c6cd5b5dd911901708530cf131ed
 2021-05-28 20:27:00 +01:00
Konstantin V Shvachko c5535caf6e HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f9)
 2021-05-27 18:53:46 -07:00
Takanobu Asanuma d8689f1a08 Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055) 
This reverts commit 0774116756.

Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53ff2dfed3)
 2021-05-27 13:17:24 +09:00
Konstantin V Shvachko 5308d44be4 HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f)
 2021-05-26 15:09:47 -07:00
Akira Ajisaka e14d00469a
MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 8a489ce78e)
 2021-05-26 15:48:51 +09:00
Narges Shadab 41025cd585
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027) 
(cherry picked from commit 3fdeb7435a)
 2021-05-26 14:45:20 +09:00
Jim Brennan 48aa23eb3b YARN-10691. DominantResourceCalculator isInvalidDivisor should consider only countable resource types. Contributed by Bilwa S T. 
(cherry picked from commit c4581827a9)
 2021-05-25 18:10:24 +00:00
Steve Loughran 464bbd5b7c
HADOOP-17511. Add audit/telemetry logging to S3A connector (#2807) 
The S3A connector supports
"an auditor", a plugin which is invoked
at the start of every filesystem API call,
and whose issued "audit span" provides a context
for all REST operations against the S3 object store.

The standard auditor sets the HTTP Referrer header
on the requests with information about the API call,
such as process ID, operation name, path,
and even job ID.

If the S3 bucket is configured to log requests, this
information will be preserved there and so can be used
to analyze and troubleshoot storage IO.

Contributed by Steve Loughran.

Change-Id: Ic0a105c194342ed2d529833ecc42608e8ba2f258
 2021-05-25 12:55:38 +01:00
Akira Ajisaka ad7071c93b
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3)
 2021-05-25 18:29:20 +09:00
Mehakmeet Singh b82a0fa9e6
HADOOP-17705. S3A to add Config to set AWS region (#3020) 
The option `fs.s3a.endpoint.region` can be used
to explicitly set the AWS region of a bucket.

This is needed when using AWS Private Link, as
the region cannot be automatically determined.

Contributed by Mehakmeet Singh

Change-Id: I4b52f85d7af0ddd56b2b0505ac0124d4fcc67ca0
 2021-05-24 13:13:37 +01:00
Mehakmeet Singh a786847b8f
HADOOP-17670. S3AFS and ABFS to log IOStats at DEBUG mode or optionally at INFO level in close() (#2963) 
When the S3A and ABFS filesystems are closed,
their IOStatistics are logged at debug in the log:

org.apache.hadoop.fs.statistics.IOStatisticsLogging

Set `fs.iostatistics.logging.level` to `info` for the statistics
to be logged at info. (also: `warn` or `error` for even higher
log levels).

Contributed by: Mehakmeet Singh

Change-Id: I56d44ad89fc1c0dd4baf701681834e7fd96c544f
 2021-05-24 13:04:20 +01:00
Wei-Chiu Chuang 745a2f5ec3
HADOOP-17723. [build] fix the Dockerfile for ARM (#3037) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1576f81dfe)
 2021-05-24 18:20:49 +08:00
Vinayakumar B dbf1ef4aff
HDFS-15790. Make ProtobufRpcEngineProtos and ProtobufRpcEngineProtos2 Co-Exist (#2767) 
(cherry picked from commit 2bbeae3240)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
 2021-05-24 18:00:38 +08:00
Wei-Chiu Chuang 86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959)" 
This reverts commit 4ffe5eb1dd.
 2021-05-24 17:37:18 +08:00
Yiqun Lin ca52d9946b
HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. 
(cherry picked from commit da1ea2530f)
 2021-05-24 10:48:48 +08:00
Szilard Nemeth 6777acb08c YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja 2021-05-22 14:36:51 +02:00
Viraj Jasani 87e32fbb8d HADOOP-17700. ExitUtil#halt info log should log HaltException (#3015) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c70ee2d548)
 2021-05-22 19:10:28 +09:00
Wei-Chiu Chuang 526dbe4716
HADOOP-17666. Update LICENSE for 3.3.1 (#3011) 
* Inspected the jar files in the produced tarball and updated LICENSE-binary accordingly.

* add LICENSE from hadoop-thirdparty jars.
* remove any dependencies no longer in the tarball.
* Updated the license of thirdparty javascripts and C/C++ files.

Added LICENSE-asio.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/asio-1.10.2/COPYING
Added LICENSE-gmock.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/gmock-1.7.0/LICENSE
Added LICENSE-rapidxml.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/rapidxml-1.13/rapidxml/license.txt
Added LICENSE-uriparser2.txt, copied from hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/uriparser/COPYING
Added LICENSE-tr2.txt, copied from the header of hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/third_party/uriparser2/uriparser2/tr2/optional.hpp
Added LICENSE-cJSON.txt, moved from the bottom of LICENSE.txt

* Generated license report for yarn-managed packages.
* Add LICENSE and NOTICES file of jaxb-api.
* Exclude LICENSE-binary-{yarn-applications-catalog-webapp|yarn-ui} from rat report.
These two files are autogenerated.
 2021-05-21 18:15:48 -07:00
Gautham B A 23ada04dac
HADOOP-17509. Parallelize building of dependencies (#2669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6c891c0dd4)
 2021-05-22 01:32:37 +08:00
Wei-Chiu Chuang 5e09d38b25
HADOOP-17718. Explicitly set locale in the Dockerfile. (#3034) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ad923ad564)

Conflicts:
	dev-support/docker/Dockerfile_centos_7
	dev-support/docker/Dockerfile_centos_8
 2021-05-22 00:31:59 +08:00
Akira Ajisaka 86682a83aa
HADOOP-17663. Remove useless property hadoop.assemblies.version in pom file. (#3025) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3f5a66c158)

 Conflicts:
	hadoop-project/pom.xml
 2021-05-20 10:49:40 +09:00
Jim Brennan 53a1c7653f YARN-10337. Amendment to fix import as in HADOOP-17100 2021-05-19 22:00:55 +00:00
Prabhu Joseph 1b3e4cf9ce YARN-10337. Fix failing testcase TestRMHATimelineCollectors. 
Contributed by Bilwa S T.

(cherry picked from commit 2bbd00dff4)
 2021-05-19 21:19:05 +00:00
Wei-Chiu Chuang fa4915fdbb
Preparing for 3.3.2 development 2021-05-19 21:52:37 +08:00
zhuqi-lucas 7d2eeaecc8 YARN-10701. The yarn.resource-types should support multi types without trimmed. Contributed by Qi Zhu. 2021-05-19 21:24:26 +08:00
Peter Bacsko a45d6437f5 YARN-10120. In Federation Router Nodes/Applications/About pages throws 500 exception when https is enabled. Contributed by Bilwa S T. 2021-05-19 15:07:47 +02:00
Takanobu Asanuma 932a16837a
HADOOP-17224. Install Intel ISA-L library in Dockerfile. (#2537) 
(cherry picked from commit d09e3c929f)
 2021-05-19 20:27:22 +08:00
Peter Bacsko 864b710787 YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B. 2021-05-19 10:31:57 +02:00
Wei-Chiu Chuang 611ed04aff
HADOOP-17426. Upgrade to hadoop-thirdparty-1.1.0. (#3024) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>

Change-Id: Ibf27246b3eb1b074b938f44f4906db9184466303
(cherry picked from commit 2960d83c25)
 2021-05-19 14:57:45 +09:00
Wei-Chiu Chuang 63392376c9
HDFS-16027. Replace abstract methods with default methods in JournalNodeMXBean. (#3018) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-05-18 19:49:00 -07:00
Peter Bacsko 27749ce85d YARN-9615. addendum commit, removing accedentially committed patch file 2021-05-18 22:17:20 +02:00
Wei-Chiu Chuang 0df4743284
HADOOP-17703. checkcompatibility.py errors out when specifying annotations. (#3017) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 110cda3de6)
 2021-05-18 11:36:13 +08:00
Wei-Chiu Chuang 4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 (#2959) 
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.

Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb

* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661). Contributed by Vinayakumar B.

(cherry picked from commit c4c0683dff)

Co-authored-by: Xiaoyu Yao <xyao@apache.org>
Co-authored-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-05-17 18:57:46 -07:00
lujiefsi 137e20cc9b
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790)
 2021-05-17 13:53:27 +09:00
litao 2f56e2b54d
HDFS-15873. Add namenode address in logs for block report (#2743) 
(cherry picked from commit 6a922f9896)
 2021-05-17 12:29:11 +08:00
Xiaoyu Yao 3f9c9ccf46
HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider.… (#3010) 
* HADOOP-17284. Support BCFKS keystores for Hadoop Credential Provider. (#2334)

(cherry picked from commit 4c5ad57818)
 2021-05-13 16:57:58 -07:00
Wei-Chiu Chuang 1b8eb2654a
HADOOP-16822. Provide source artifacts for hadoop-client-api. Contributed by Karel Kolman. 
(cherry picked from commit 2c4ab72a60)
 2021-05-13 20:30:40 +08:00
Aryan Gupta 7142dfec04
HADOOP-17302. Upgrade to jQuery 3.5.1 in hadoop-sls. (#2379) 
Co-authored-by: Aryan Gupta
(cherry picked from commit d60d5fe43d)
 2021-05-13 12:16:24 +08:00
Aryan Gupta 5f147112cf
HADOOP-17286. Updated jQuery in yarn-common. (#2340) 
(cherry picked from commit bbbfa7d415)
 2021-05-13 12:16:20 +08:00
Aryan Gupta 28079e9c30
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 486ddb73f9)
 2021-05-13 12:16:17 +08:00
Ayush Saxena 41b39ff292
HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang. 
(cherry picked from commit 960c9ebaea)

Conflicts:
    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java
 2021-05-13 12:14:09 +08:00
bilaharith 1c6c3920b6
HADOOP-17444. ADLS Gen1: Update adls SDK to 2.3.9 (#2842) 
Contributed by bilaharith
 2021-05-12 18:15:27 +01:00
dependabot[bot] b2897fdd66
HADOOP-17683. Update commons-io to 2.8.0 (#2974) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29105ffb63)
 2021-05-12 10:58:39 +09:00
Peter Bacsko 051a5068dd YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-05-11 19:23:45 +02:00
Viraj Jasani 3176ab028b
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b944084b32)
 2021-05-11 12:39:42 +09:00
Borislav Iordanov f5ef78d46f HADOOP-17665 Ignore missing keystore configuration in reloading mechanism 2021-05-10 15:41:56 -05:00
Ayush Saxena 43db94c7a3 HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w. 2021-05-10 17:11:09 +05:30
sumangala-patki b20bc668d5
HADOOP-17548. ABFS: Toggle Store Mkdirs request overwrite parameter (#2729) (#2781) 
Contributed by Sumangala Patki.

(cherry picked from commit fe633d4739)
 2021-05-10 11:50:01 +05:30
Ayush Saxena 2a95ba29bb HADOOP-17644. Add back the exceptions removed by HADOOP-17432 for compatibility. Contributed by Quan Li. 2021-05-09 04:00:30 +05:30