Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,016 Commits 326 Branches 371 Tags
Commit Graph

15016 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Erik Krogen
a503662d6c HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83c1fb933d409c514d2155e986e4e25b)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
(cherry picked from da548cf263c699932e5a6b60e00893f857713bcf)
(cherry picked from 055f824bcbeb82ee7eb49576ca7b9446dccea401)
 2019-08-02 11:53:34 -07:00
Steve Loughran
e8ccc299f6
HADOOP-15237. In KMS docs there should be one space between KMS_LOG and NOTE. 
Contributed by Snigdhanjali Mishra.

Change-Id: I3abaa658c8786f8afa802ccbb629551313431b0e
 2019-08-01 10:37:28 +01:00
Eric E Payne
77d1aa9376 YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan. 
(cherry picked from commit c04ccf296dab6eaa53f008342e50f8729935a50d)
 2019-07-30 20:31:02 +00:00
Wei-Chiu Chuang
3941b3c708 HDFS-14464. Remove unnecessary log message from DFSInputStream. Contributed by Chao Sun. 2019-07-30 11:36:56 -07:00
Akira Ajisaka
46c00aca21
HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich. 
(cherry picked from commit cbfa3f3e988b8a3f142abadfe4f18201b6438ac9)
 2019-07-29 17:40:14 -07:00
Erik Krogen
61581a86b2 HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 2019-07-29 14:36:57 -07:00
Wei-Chiu Chuang
27ead06a13 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 2019-07-29 14:34:31 -07:00
Erik Krogen
edb46353c6 HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen. 
(cherry-picked from 62efb6300619670d3e0554c3ba14c264fa0c705b)
(cherry-picked from 5ec34bf43adb498797afd24bb02989d4fec25409)
(cherry-picked from e2aab0efef9d504e4695a9c1df2835ed60f6c0c7)
(cherry-picked from f7e517ee2432f3c0a40c9172413753670bf07db4)
 2019-07-26 11:44:12 -07:00
Erik Krogen
653951e1f7 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena." 
This reverts commit 3bfae20e946e40f6936aab54b79fc85df8d3b0b2.
 2019-07-26 08:27:36 -07:00
Steve Loughran
c0fd09c42c
HADOOP-9844. NPE when trying to create an error message response of SASL RPC 
This closes #55

Change-Id: I10a20380565fa89762f4aa564b2f1c83b9aeecdc
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 07530314c2130ecd1525682c59bf51f15b82c024)
 2019-07-26 17:55:45 +09:00
Masatake Iwasaki
3bfae20e94 HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97251267253fa045ba03c4749f95f530)
 2019-07-26 09:54:27 +09:00
Ahmed Hussein
dc4d7f69ce YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 9d63fc7ed61ca6c3d901fe9984a7cca98ce07ea9)
 2019-07-24 11:21:29 -05:00
pingsutw
2cbd7eb5b6
MAPREDUCE-7076. TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build 
This closes #1089

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ee87e9a42e4ff1f27a6f1e5b7c7de97f8989d9b2)
 2019-07-23 10:36:43 +09:00
Haibo Chen
68c24b322d YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonanthan Hung) 2019-07-19 11:00:38 -07:00
Masatake Iwasaki
1f0bf69e16 MAPREDUCE-6521. MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test. 
Contributed by Masatake Iwasaki.

Change-Id: Id74b90eb7cfb9d676188fc5aa47249ee8904a3d5
(cherry picked from commit 79f6118dccf809aed74dd54f46f5d77416e14aa1)
 2019-07-19 09:49:44 +09:00
Masatake Iwasaki
62fdfaf093 HADOOP-16386. FindBugs warning in branch-2: GlobalStorageStatistics defines non-transient non-serializable instance field map. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-07-17 07:50:24 +09:00
Erik Krogen
cf7b9a5bd5 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 2019-07-16 07:22:45 -07:00
Weiwei Yang
3f112e7347 HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 
(cherry picked from commit 6c8cfd135eea15e3dfd76dd33af24a9a8c38fd8d)
 2019-07-11 10:07:47 +08:00
Yiqun Lin
78aebee5c5 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 993dc8726b7d40ac832ae3e23b64e8541b62c4bd)
 2019-07-10 10:58:59 +08:00
stack
a7c6d25dc3 HDFS-14483 Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9"" 
This is a revert of a revert, i.e. a reapplication of the HDFS-14483
patch. Original commit was missing the JIRA number.

Revert "Revert "Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9""

This reverts commit b26ad8a4c38d27894bd42d3fba7abff80bd895f4.
 2019-07-09 08:58:16 -07:00
stack
b26ad8a4c3 Revert "Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9" 
This reverts commit 4f09a6bcfd685bbc9e139497adfeeeff8170e344.
 2019-07-09 08:57:57 -07:00
sunlisheng
4f09a6bcfd Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9 
Signed-off-by: stack <stack@apache.org>
 2019-07-09 08:54:37 -07:00
leosunli
0ff5db746e HDFS-13694. Making md5 computing being in parallel with image loading. 
(cherry picked from commit 9c907294860a48f0d1676a31bda15795b7a6771a)
 2019-07-07 20:04:08 -07:00
Masatake Iwasaki
e61e52805f HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec78c8dab9111f3ba2709cef20794b73a)
 2019-07-04 23:55:13 +09:00
stack
bb7e143d34 HDFS-14585 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch2.9 
Reapply but include JIRA # in the commit message this time.

Revert "Revert "Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9""

This reverts commit 1dd54163fcb40b60efc7ca153d443129e424df7d.
 2019-07-01 11:08:05 -07:00
stack
1dd54163fc Revert "Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9" 
This reverts commit bccb0fd846b7cc69e0d83140193eaa7fc0ac081d.
 2019-07-01 11:07:43 -07:00
sunlisheng
bccb0fd846 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: stack <stack@apache.org>
 2019-07-01 10:59:58 -07:00
Inigo Goiri
0272480e9f HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42a3ef94d6bef160bc58d807f2faa104)
(cherry picked from commit 96371245357bda63b3ede10f37a37f5333a85d69)
(cherry picked from commit 90b88db35d42f2eab4da7f192a5fb99d9c834abb)
(cherry picked from commit 9daa45f646d9eddda21c087c23e6d1498f98c055)
 2019-06-24 21:17:38 -07:00
Inigo Goiri
1373889ac2 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 
(cherry picked from commit 0966407ad6fddb7adf60dba846d1f7c8a13b8ec1)
 2019-06-24 19:04:33 -07:00
Wei-Chiu Chuang
bf63475e7a HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-20 21:06:06 -07:00
Wei-Chiu Chuang
f876e82abc HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 2019-06-20 19:36:44 -07:00
Zsolt Venczel
a68de43957 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff6bf8d5e1d05748ba32b679e397b53f)
(cherry picked from commit 4ea3b04bf36af56612c6b47d9ff8aa49cbbbcc9e)
(cherry picked from commit 76faa41f18f1cb20a6469f53b475f52b662f8107)
 2019-06-18 14:38:42 -07:00
Kitti Nanasi
abd11f9db1 HDFS-13770. dfsadmin -report does not always decrease "missing blocks (with replication factor 1)" metrics when file is deleted. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 00:25:53 -07:00
Wei-Chiu Chuang
ce49f8a5df HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea8c75c134de43a3a925c76b253658e1)
(cherry picked from commit 45543ffd6af723bb83bd44a99f12165363c70abb)
(cherry picked from commit fc8983a5b8c94746d3e58cc203974aa09c1b640b)
(cherry picked from commit 79f8e4d5bc3afaff2f352b918e96a6ef3c2abcce)
 2019-06-17 20:20:34 -07:00
Akira Ajisaka
73a59b2c42 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827381a7f87c01f0befc2475f9478e63b)
(cherry picked from commit f68099aa1316f34591c757bcf4198a3a3c7f691a)
(cherry picked from commit 3d1d8623a11b72c7939d6500537be5fd6565d4d6)
(cherry picked from commit 010a99104b53a23826f8f1f73d0125cb26f2adca)
 2019-06-17 17:04:38 -07:00
huzheng
8b90c69357 HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a184405c3feca024a560807ea215b6858b9)
(cherry picked from commit 9b61ecfcbeeb1d3c320f775b43c68d3e39a271a3)
(cherry picked from commit b082628e5a39fc1bc9d6094beb9c253cc096c701)
(cherry picked from commit 2b9c4fccf4a8930bfff99ab4928b9b3b4980d3b2)
 2019-06-17 07:22:01 -07:00
Chao Sun
9ab3cbf4d3 HADOOP-16359. Bundle ZSTD native in branch-2. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-14 15:21:59 -07:00
Takanobu Asanuma
64102e865f HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 
(cherry picked from commit 54f9f75a443d7d167a7aa7d04a87e3f5af049887)
 2019-06-14 10:21:30 +09:00
Erik Krogen
e9eaf21874 HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 
(cherry-picked from 76b94c274fe9775efcfd51c676d80c88a4f7fdb9)
(cherry-picked from f607efd1950211f85eb571564b774184f8fa1687)
(cherry-picked from cf3c1b9ffc8cfce2e2248a1c1600bd34ff412933)
(cherry-picked from d43219c21d830b198710f70c16ea880302f88d77)
 2019-06-07 14:23:23 -07:00
Eric E Payne
441339a716 YARN-8625. Aggregate Resource Allocation for each job is not present in ATS. Contributed by Prabhu Joseph. 2019-06-04 18:57:44 +00:00
Kihwal Lee
500dc05924 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b6493d315c8910fbc06eaba96273ee335c)
(cherry picked from commit fd8b8c7494d19467485423ce2f3f7459a4fddc29)
 2019-05-30 17:37:38 -05:00
Ayush Saxena
c63049dfd4 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f27e3e4f58ff1b5ef3e93efc56bbecbe)
(cherry picked from commit 2cd7577a5a58db42a14a659b85fbd475091d0cb2)
(cherry picked from commit 90cc5ddd4957fcb27ef5a1e7ee1a67c4eca9a3f5)
(cherry picked from commit 942c5b04f2f7bc33595c4df5c66104ce3c4c2c69)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
 2019-05-29 21:11:15 -07:00
Siyao Meng
cecba551aa HDFS-14514. Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2. Contributed by Siyao Meng, Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Stephen O'Donnell <sodonnell@cloudera.com>
 2019-05-29 17:28:49 -07:00
Takanobu Asanuma
e3d4f6bfed HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:41:21 +09:00
Akira Ajisaka
cea1b0000c
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:54 +09:00
Erik Krogen
f024d27cd9 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f002f74cb4a0360b6682a1b6796d1598)
(cherry-picked from c26cf22d6b3949a4798d8133669b451660a0f050)
(cherry-picked from 9295e95174ad8627874b81a9d43183384a81fb16)
 2019-05-24 13:39:52 -07:00
Konstantin V Shvachko
2afc770443 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:43:12 -07:00
Wei-Chiu Chuang
380289a167 HDFS-13087. Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge, Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Siyao Meng <smeng@cloudera.com>
 2019-05-24 18:36:53 +02:00
Alexis Daboville
b70fdc64d3
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,

(cherry picked from commit 4cb3da6ac703b10de59a8ca371fe957a4a338601)
 2019-05-17 15:27:47 +01:00
Weiwei Yang
11fb89bd68 HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 
(cherry picked from commit 2d8282bb8248e6984878626c4cdc7148aa2e7202)
 2019-05-14 14:09:12 -07:00