Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,886 Commits 326 Branches 371 Tags
Commit Graph

17886 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sunil G
78cf8c8099 YARN-7502. Nodemanager restart docs should describe nodemanager supervised property. Contributed by Suma Shivaprasad. 
(cherry picked from commit e50334513c781a10368e7353bec6674f89eb0a7b)
 2018-10-22 14:17:29 +05:30
Wei-Chiu Chuang
8360d15abc HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e2aab8b7c5465719cac53dce407a64c)
(cherry picked from commit a7dd244a4995cbf20feb054530fc5913ccf384ac)

	modified:   hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java
(cherry picked from commit 84726679f0441bfa0788ad5a5ac2bfa4353afedb)
 2018-10-19 13:56:23 -07:00
Wei-Chiu Chuang
8754c0b65e Revert "HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu." 
This reverts commit 437c5187cd82f45ab709f9b87dba75d2911f20e4.
 2018-10-19 13:33:39 -07:00
Ted Yu
437c5187cd HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e2aab8b7c5465719cac53dce407a64c)
(cherry picked from commit a7dd244a4995cbf20feb054530fc5913ccf384ac)
(cherry picked from commit 84726679f0441bfa0788ad5a5ac2bfa4353afedb)
 2018-10-19 13:23:02 -07:00
Takanobu Asanuma
115c039495 HADOOP-15804. upgrade to commons-compress 1.18. Contributed by Akira Ajisaka. 
(cherry picked from commit 9bd18324c7801472409d9ad69ea365aa7a33a9c4)
 2018-10-19 21:41:10 +09:00
Weiwei Yang
37af9580c8 YARN-8907. Fix incorrect logging message in TestCapacityScheduler. Contributed by Zhankun Tang. 
(cherry picked from commit 13cc0f50ea1c5f8978455e34f49716ddb0e3a143)
 2018-10-19 10:03:39 +08:00
Siyao Meng
6a7bf9f850 HDFS-14009. HttpFS: FileStatus#setSnapShotEnabledFlag throws InvocationTargetException when attribute set is emptySet. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-10-18 17:02:38 -07:00
drankye
b34c650a41 HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 
(cherry picked from commit 975d60685eaf9961bdbd3547600b3e38bb088835)
 2018-10-18 08:31:12 -07:00
Inigo Goiri
ea19a36132 HDFS-14000. RBF: Documentation should reflect right scripts for v3.0 and above. Contributed by CR Hota. 
(cherry picked from commit 7347fa2df3cfd48857f4ded3539a60101d026daf)
 2018-10-17 16:43:05 -07:00
Jason Lowe
0aee3a0c3f HADOOP-15859. ZStandardDecompressor.c mistakes a class for an instance. Contributed by Jason Lowe 
(cherry picked from commit 9abda83947a5babfe5a650b3409ad952f6782105)
 2018-10-17 14:44:39 -05:00
Steve Loughran
ae42d59eba
HADOOP-15861. Move DelegationTokenIssuer to the right path. 
Contributed by Wei-Chiu Chuang.

(cherry picked from commit 41b3603b5bcb74a7d78a314a4a5c177d941af27f)
 2018-10-17 11:04:05 +01:00
Xiao Chen
99b447f612 MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko. 
(cherry picked from commit 25f8fcb06476938826cdc92858a61124b18cd98d)
 2018-10-16 10:24:33 -07:00
Xiao Chen
b9f65d717c HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a0250ac3f2341ebe8a14d19b073e6224c8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
 2018-10-15 11:00:39 -07:00
Xiao Chen
ff7ca472d2 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.
 2018-10-12 12:08:38 -07:00
Inigo Goiri
53b522af6d HDFS-13802. RBF: Remove FSCK from Router Web UI. Contributed by Fei Hui. 
(cherry picked from commit f63ee083db7c261207b726f0f44be2e5a92124f3)
 2018-10-12 10:12:07 -07:00
Steve Loughran
2abc4ab60a
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.

(cherry picked from commit f1fbc563719db88c50d57820b5052880cfd6b1aa)
 2018-10-12 09:49:02 +01:00
Wangda Tan
b67dc5e1d1 YARN-7274. Ability to disable elasticity at leaf queue level. (Zian Chen via wangda) 
Change-Id: Ic8d43e297f0f5de788b562f7eff8106c5c35e8d2
(cherry picked from commit 74665e3a7d7f05644d9a5abad5a3f2d47597d6c8)
 2018-10-11 16:00:10 +00:00
Jonathan Eagles
2b6d8c1e29 HADOOP-15835. Reuse Object Mapper in KMSJSONWriter (jeagles) 2018-10-10 10:32:43 -05:00
Hrishikesh Gadre
b170de8be5 HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 323b76bccfa153ef5ba52dc14876283d05618739)
 2018-10-09 19:54:56 -07:00
Weiwei Yang
af85ce6ae4 YARN-8858. CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Contributed by Wangda Tan. 
(cherry picked from commit edce866489d83744f3f47a3b884b0c6136885e4a)
 2018-10-10 09:56:16 +08:00
Inigo Goiri
8a9f61be4e HDFS-13964. RBF: TestRouterWebHDFSContractAppend fails with No Active Namenode under nameservice. Contributed by Ayush Saxena. 
(cherry picked from commit 3bfff0f79e7246cbb2989c720c14cfa2566a9d1a)
 2018-10-06 10:42:51 -07:00
Jason Lowe
bbceae10b9 HADOOP-15820. ZStandardDecompressor native code sets an integer field as a long. Contributed by Jason Lowe 
(cherry picked from commit f13e231025333ebf80b30bbdce1296cef554943b)
 2018-10-05 09:29:41 -05:00
Takanobu Asanuma
4bad89802b HADOOP-15816. Upgrade Apache Zookeeper version due to security concerns. Contributed by Akira Ajisaka. 
(cherry picked from commit 241cbec2dab3c3d49b48f42b86e8bd85cd1f08f3)
 2018-10-05 15:11:31 +09:00
Yiqun Lin
d993a1fc3d HADOOP-15817. Reuse Object Mapper in KMSJSONReader. Contributed by Jonathan Eagles. 
(cherry picked from commit 81f635f47f0737eb551bef1aa55afdf7b268253d)
 2018-10-04 10:36:49 +08:00
Eric E Payne
b412bb2240 YARN-8774. Memory leak when CapacityScheduler allocates from reserved container with non-default label. Contributed by Tao Yang. 
(cherry picked from commit 8598b498bcaf4deffa822f871a26635bdf3d9d5c)
 2018-09-28 15:48:51 +00:00
Jason Lowe
17583e690a YARN-8804. resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues. Contributed by Tao Yang 
(cherry picked from commit 6b988d821e62d29c118e10a7213583b92c302baf)
 2018-09-26 16:54:29 -07:00
Brahma Reddy Battula
b28dacf60c HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula 
(cherry picked from commit e0ff8e2c10f38298b81624df1e62743ac171f18d)
 2018-09-26 23:55:02 +05:30
Sammi Chen
85e00477b8 HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 
(cherry picked from commit 2b635125fb059fc204ed35bc0e264c42dd3a9fe9)
(cherry picked from commit 5da3e8359757c0c1afaccc1d3a0f2bdc453e0311)
 2018-09-25 19:53:03 +08:00
Siyao Meng
2449795b8e HDFS-13830. Backport HDFS-13141 to branch-3.0: WebHDFS: Add support for getting snasphottable directory list. Contributed by Siyao Meng, Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2018-09-21 12:41:03 -07:00
Steve Loughran
fc6d851f70
HADOOP-15748. S3 listing inconsistency can raise NPE in globber. 
Contributed by Steve Loughran.

(cherry picked from commit 646874c326139457b79cf8cfa547b3c91a78c7b4)
 2018-09-20 13:06:20 +01:00
Vinayakumar B
d2655ea60c HADOOP-15736. Trash : Negative Value For Deletion Interval Leads To Abnormal Behaviour. Contributed by Ayush Saxena. 
(cherry picked from commit 7ad27e97f05b13b33fdcef9cb63ace9c1728bfb5)
(cherry picked from commit 2aa385463305beaae8770c8deb22faed6943ef00)
 2018-09-20 09:34:37 +05:30
Chen Liang
78e0c0a1bb HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 14:51:23 -07:00
Arpit Agarwal
ffef819ee8 HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 
(cherry picked from commit 2eb597b1511f8f46866abe4eeec820f4191cc295)
 2018-09-19 13:49:57 -07:00
Inigo Goiri
3fc3a5c01c HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang. 
(cherry picked from commit 7f9a89e1b54a9712af50ffef70bed7cfb91ed34e)
 2018-09-19 12:59:42 -07:00
Inigo Goiri
b93aeb43a4 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 56e0d635e0f49772d001c1fdb385110c705a622a)
 2018-09-19 09:31:58 -07:00
Weiwei Yang
4edcde055b Revert "YARN-8771. CapacityScheduler fails to unreserve when cluster resource contains empty resource type. Contributed by Tao Yang." 
This reverts commit f56e36935da69b5b12086d80bfe1cdb446213882.
 2018-09-19 19:47:06 +08:00
Weiwei Yang
f56e36935d YARN-8771. CapacityScheduler fails to unreserve when cluster resource contains empty resource type. Contributed by Tao Yang. 
(cherry picked from commit 0712537e799bc03855d548d1f4bd690dd478b871)
 2018-09-19 19:41:07 +08:00
Wei-Chiu Chuang
823b1fceee HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 
(cherry picked from commit 28ceb34a725cd06d28fb51361c49bb45464f5368)
(cherry picked from commit 7442b7941dc57ebf4bd0ded4d46cfbf5af6fea81)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 2018-09-19 03:20:26 -07:00
Wei-Chiu Chuang
3a32e84d3e HDFS-13886. HttpFSFileSystem.getFileStatus() doesn't return "snapshot enabled" bit. Contributed by Siyao Meng. 
(cherry picked from commit 44857476fa993fbf9c97f979b91e19d27632c10a)
(cherry picked from commit 595ce945704f37a4df5cb8c9b2c509b8d6a14ec4)
 2018-09-18 15:48:05 -07:00
Wei-Chiu Chuang
2ef3911f7c HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit a1de8cbac5fb9af403db2a02814575f0940d5f39)
(cherry picked from commit 829399a9f38db40b5bb247cb9c0d5f930c094e37)
 2018-09-18 15:48:04 -07:00
Jason Lowe
f5ebb2ccf5 MAPREDUCE-7138. ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler. Contributed by Oleksandr Shevchenko 
(cherry picked from commit 8382b860d4ef4f20d000537ded42a88e98bd2190)
 2018-09-18 17:15:17 -05:00
Jason Lowe
4a2a7d37dd MAPREDUCE-7137. MRAppBenchmark.benchmark1() fails with NullPointerException. Contributed by Oleksandr Shevchenko 
(cherry picked from commit 34b2237e420cfbe3a97ddd44968de8bbe1ed30ab)
 2018-09-18 17:13:00 -05:00
Jason Lowe
f2eeba5c5b HADOOP-15755. StringUtils#createStartupShutdownMessage throws NPE when args is null. Contributed by Lokesh Jain and Dinesh Chitlangia 
(cherry picked from commit e71f61ecb87e04727a5a76e578a75714c9db6706)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestStringUtils.java
 2018-09-18 16:03:42 -05:00
Eric E Payne
373c7a0ceb MAPREDUCE-3801: org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently. Contributed by Jason Lowe 
(cherry picked from commit 2cf8927683d49f4cdc3cba038480128c473c5461)
 2018-09-18 20:42:35 +00:00
Sammi Chen
61847f4ea0 HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu. 
(cherry picked from commit e4fca6aae46a3c04fc56897986a4ab4e5aa98503)
(cherry picked from commit 53528d5ec5ab812484e2d5a6622d58ba8c86a5b0)
 2018-09-18 18:45:49 +08:00
Inigo Goiri
f0537b5161 HDFS-13844. Fix the fmt_bytes function in the dfs-dust.js. Contributed by yanghuafeng. 
(cherry picked from commit a9c1c55cafc0338bb35d2991f217d4157463f1d3)
 2018-09-17 14:39:31 -07:00
Weiwei Yang
14b9e64d80 YARN-8782. Fix exception message in Resource.throwExceptionWhenArrayOutOfBound. Contributed by Gergely Pollak. 
(cherry picked from commit 9a265fa673ef1b8774cfd69c76cdd29bf344e79d)
 2018-09-17 22:27:59 +08:00
Weiwei Yang
ef17728c58 YARN-8720. CapacityScheduler does not enforce max resource allocation check at queue level. Contributed by Tarun Parimi. 
(cherry picked from commit f1a893fdbc2dbe949cae786f08bdb2651b88d673)
 2018-09-14 16:43:28 +08:00
Jason Lowe
f918e4d894 MAPREDUCE-7133. History Server task attempts REST API returns invalid data. Contributed by Oleksandr Shevchenko 
(cherry picked from commit 2886024ac3a8613ecc27f1595b278ce6fc2d03ba)
 2018-09-13 14:47:37 -05:00
Inigo Goiri
1314dc2397 HDFS-13914. Fix DN UI logs link broken when https is enabled after HDFS-13902. Contributed by Jianfei Jiang. 
(cherry picked from commit e1b242a9844d6438c93cdc2ab2443d042b4d5aea)
 2018-09-13 09:18:11 -07:00