Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,246 Commits 326 Branches 371 Tags
Commit Graph

19246 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Akira Ajisaka
3c9d2f5317 YARN-9162. Fix TestRMAdminCLI#testHelp. Contributed by Ayush Saxena. 
(cherry picked from commit 5db7c4906235dd1070b9706ca4971f1eb5d74139)
(cherry picked from commit a453f38015fec454f63e9f5459b0330ac3bc7eb3)
 2019-08-30 17:55:35 -07:00
Istvan Fajth
e3c01e174c HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-30 13:13:48 -07:00
Brian Grunkemeyer
7e9d324bcd HADOOP-15998. Ensure jar validation works on Windows. 
* use a different path separator for artifacts to test
* fail if and commands fail (including the jar listing)
* handle different line endings from jar listing
* make sure we have bash 3.1+

Co-authored-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Abhishek Modi <abmodi@apache.org>
Signed-off-by: Rohith Sharma K S <rohithsharmaks@apache.org>
(cherry picked from commit b86582ce2b8b00c128add054bcb80951c38cb3d3)
 2019-08-29 23:08:39 -05:00
Ayush Saxena
e44693953d HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B. 2019-08-30 08:59:52 +05:30
leosunli
0011136e66 Make DataNodePeerMetrics#minOutlierDetectionSamples configurable (#1314). Contributed by Lisheng Sun. 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
(cherry picked from commit 915271245b9a3f682409db380eb311ffd465b041)
(cherry picked from commit b516d2ea7b2b96a1af350edd7964d3e082ee4fe7)
 2019-08-29 18:46:25 -07:00
Stephen O'Donnell
d42b48d4d0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7bebad61d9c3dbff81fdcf243585fd3e9ae59dde)
(cherry picked from commit 9c0d6e16573a055665318bce8f7d38a92c897393)
 2019-08-29 18:26:14 -07:00
Eric E Payne
51896ff7e6 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 
(cherry picked from commit d562050cec83a2bc2ffb6d109ed3d64b394b870d)
 2019-08-28 21:05:23 +00:00
Jonathan Hung
f73842780e YARN-9438. launchTime not written to state store for running applications 
(cherry picked from commit 9568656cd21d9c02168e18ce35c6726077bbf3a1)
(cherry picked from commit 0c498de6e87c6bdc959afa31deb03d0907e0e1a1)
 2019-08-27 15:45:42 -07:00
Jonathan Hung
79364a9c53 HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns 
(cherry picked from commit 8ab7020e641e65deb002a10732d23bb22802c09d)
(cherry picked from commit d255efa5f8dc57c1411ec2fe040bd22516393edd)
 2019-08-27 10:31:21 -07:00
Jonathan Hung
6baa0d1e4d YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca15098e891c560beb3181c22ef3f80ff)
(cherry picked from commit e4249c320257586384035ea3fc286fe54cc699a1)
 2019-08-26 15:55:11 -07:00
bibinchundatt
eb618e4f22 YARN-9642. Fix Memory Leak in AbstractYarnScheduler caused by timer. Contributed by Bibin A Chundatt. 
(cherry picked from commit d3ce53e5073e35e162f1725836282e4268cd26a5)
 2019-08-26 23:25:16 +05:30
Chen Liang
d2bb764ffa HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:29:23 -07:00
Erik Krogen
9dc921f5e5 HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 
(cherry picked from d699022fce756d25956d33e022100111aa0dd22e)
(cherry picked from 2408c2491f4507f92e4051b72eb0800e5e11f069)
 2019-08-23 11:48:51 -07:00
Chen Liang
0839cf1d4e HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang
ef9b0a1d9a HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang
3135cc35e3 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-23 10:11:22 -07:00
Chen Liang
3835a6c4e8 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang
14265b8f03 HDFS-13547. Add ingress port based sasl resolver. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Akira Ajisaka
83f67cca60
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 
(cherry picked from commit bd7baea5a5d4ff351645e34c0ef09b7ba82f4285)
 2019-08-23 15:29:25 +09:00
Akira Ajisaka
b40aa71cf0
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 
(cherry picked from commit abc8fde4caea0e197568ee28392c46f1ce0d42e1)
 2019-08-23 15:08:34 +09:00
Xiao Chen
a5690b29a7
HDFS-13642. Creating a file with block size smaller than EC policy's cell size should fail. 
(cherry picked from commit cf4108313da83e28d07676078a33016ec8856ff6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
 2019-08-23 15:03:57 +09:00
Akira Ajisaka
42c6119e47
YARN-9774. Fix order of arguments for assertEquals in TestSLSUtils. Contributed by Nikhil Navadiya. 
(cherry picked from commit 84b1982060422760702eca6f1ef515c6ad3e85a5)
 2019-08-23 14:40:34 +09:00
Ayush Saxena
900384cf57 HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 22:54:50 +05:30
Akira Ajisaka
565751e739
HADOOP-16061. Upgrade Yetus to 0.10.0 
This closes #1298

(cherry picked from commit 5e156b9ddec46d6b7d1336bb88136d8826972e7a)

 Conflicts:
	dev-support/bin/yetus-wrapper
 2019-08-22 13:05:19 +09:00
Akira Ajisaka
ec3a5d1db1
HADOOP-16494. Add SHA-512 checksum to release artifact to comply with the release distribution policy. 
This closes #1243

(cherry picked from commit 34dd9ee36674be670013d4fc3d9b8f5b36886812)
 2019-08-22 11:03:40 +09:00
Wei-Chiu Chuang
2ce0914176 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit e978c6c9ed83301fcf8359c35e74ed68045c1a61.
 2019-08-21 09:00:59 -07:00
Surendra Singh Lilhore
c2aeeb01fa HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f95988113da3f06f6d975f99f1ee51d88a793537)
(cherry picked from commit 03c62c7989f818c49d5afca0ac94c7e72a091066)
 2019-08-21 09:00:52 -07:00
bibinchundatt
3547261749 YARN-9765. SLS runner crashes when run with metrics turned off. Contributed by Abhishek Modi. 
(cherry picked from commit 10ec31d20ee1b6a0b1da915acb6b6ec33f2cd415)
 2019-08-21 13:59:57 +05:30
Siyao Meng
e0a5fee7f5 HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1291) Contributed by Siyao Meng. 2019-08-20 10:49:51 -07:00
Wei-Chiu Chuang
7a1f508f58 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 
(cherry picked from commit fbe87eddbc30fe5191c008b496fb83e51ef4ee4a)
(cherry picked from commit f78dcbaae5535e3e05c9f29986b8963ba262baa9)
 2019-08-20 10:36:55 -07:00
Wei-Chiu Chuang
d8c60f829f HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 
(cherry picked from commit aa6995fde289719e0b300e11568c5e68c36b5d05)
(cherry picked from commit b004d305e933bdb588b84062be87c23168640697)
 2019-08-20 10:34:25 -07:00
Wei-Chiu Chuang
fc062b6fef HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 
(cherry picked from commit 360a96f342f3c8cb8246f011abb9bcb0b6ef3eaa)
(cherry picked from commit 6d93886231112c93218066008f957984def3b43b)
 2019-08-19 13:11:21 -07:00
HUAN-PING SU
18f98491e7 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit abae6ff2a2760500b7e7d4414a43069ed4a45930)
(cherry picked from commit abda5030408d12ba44c8b6df4137bdae6ee0690b)
 2019-08-19 11:15:48 -07:00
Wei-Chiu Chuang
e978c6c9ed HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 
(cherry picked from commit 8b802d6b444b95791ba156c7bb307dd584cc9dba)
 2019-08-19 08:39:00 -07:00
KAI XIE
c1a2b29c0f HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919) 
* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode

(cherry picked from commit c765584eb231f8482f5b90b7e8f61f9f7a931d09)
(cherry picked from commit b3c14d4132ed6aa871bb88c4f84f3e3d90da6f93)

Conflicts:
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java
 2019-08-18 18:55:59 -07:00
David Mollitor
0b7f9a09c2 HADOOP-15246. SpanReceiverInfo - Prefer ArrayList over LinkedList. Contributed by David Mollitor. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 901652e26321832ac42ce54daa18ebc1813772b5)
(cherry picked from commit 1b30b3fbe7b5e130778c79465ac7986772a83ab8)
 2019-08-18 18:44:02 -07:00
Surendra Singh Lilhore
92c3a581ec HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a9812023754ed1b3e5b428e161f0add50)
(cherry picked from commit 6b01effd011ab1f88bdbb9f2dbf9bbb0765984e6)
 2019-08-18 18:17:50 -07:00
Shashikant Banerjee
d18b720c70 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0a85af959ce505f0659e5c69d0ca83a5dce0a7c2)
(cherry picked from commit e89413da882e155d8654d5265e180409ccb31e34)
 2019-08-17 03:07:03 -07:00
Szilard Nemeth
fd2e353236 YARN-9100. Add tests for GpuResourceAllocator and do minor code cleanup. Contributed by Peter Bacsko 2019-08-16 15:27:10 +02:00
Szilard Nemeth
0a379e94ba YARN-8586. Extract log aggregation related fields and methods from RMAppImpl. Contributed by Peter Bacsko 2019-08-16 12:15:27 +02:00
Szilard Nemeth
94114378ce YARN-9488. Skip YARNFeatureNotEnabledException from ClientRMService. Contributed by Prabhu Joseph 
(cherry picked from commit 1845a83cec6563482523d8c34b38c4e36c0aa9df)
 2019-08-15 17:16:32 +02:00
Szilard Nemeth
a3fb032010 MAPREDUCE-7230. TestHSWebApp.testLogsViewSingle fails. Contributed by Prabhu Joseph 
(cherry picked from commit 3f4f097f1f29249c8ad5d59ccefcddf0a66b2e6b)
 2019-08-15 17:08:53 +02:00
Christopher Gregorian
96ea7f0562 HDFS-14403. Cost-based extension to the RPC Fair Call Queue. Contributed by Christopher Gregorian. 
(cherry picked from 129576f628d370def74e56112aba3a93e97bbf70)
(cherry picked from e4b650f91e7353f98fee2a725490ca45aa81e1a4)
 2019-08-14 13:37:01 -07:00
Szilard Nemeth
aa0631a042 YARN-9140. Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager. Contributed by Peter Bacsko 2019-08-14 19:04:09 +02:00
Siyao Meng
224643a58c HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b933b7acfe55081f5bad46f9fe05a14b)
(cherry picked from commit 136a97a74dbc12f05b88d0abda101690e7c727d9)
 2019-08-14 07:40:48 -07:00
Wei-Chiu Chuang
6966b76230 HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 
(cherry picked from commit 6a43d0fbd49b3ff1ce75a2334b51a98ae476e473)
(cherry picked from commit 4784165bb24228d13f4e738e0093ab0dade0bff1)
 2019-08-13 17:15:10 -07:00
Masatake Iwasaki
e66ad19395 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe0473e353ee2d489156248a01aa982dfd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63d89e5c8377a035f93485a7c77c3430)
 2019-08-14 09:07:00 +09:00
Eric Badger
a995e6352f YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949f041da2ee04da441c5f9f85e1f2c64)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be76577718b22f936aea9e2b2cd100ea)
 2019-08-13 17:16:57 +00:00
Szilard Nemeth
cb91ab73b0 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko 
(cherry picked from commit 8b3c6791b13fc57891cf81e83d4b626b4f2932e6)
 2019-08-13 15:47:57 +02:00
Szilard Nemeth
a762a6be29 Revert "YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko" 
This reverts commit b20fd9e21295add7e80f07b471bba5c76e433aed.
Commit is reverted since unnecessary files were added, accidentally.
 2019-08-13 15:47:57 +02:00