Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,964 Commits 326 Branches 371 Tags
Commit Graph

17964 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Konstantin V Shvachko
ea2d862320 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. 2019-07-25 10:07:05 -07:00
Chen Liang
613c9e4f7b HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00
Erik Krogen
34b05a26ac HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-25 10:07:04 -07:00
Konstantin V Shvachko
be8beda8dc HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-07-25 10:07:04 -07:00
Konstantin V Shvachko
ad80383b34 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-07-25 10:07:04 -07:00
Chen Liang
60f0e4df69 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-07-25 10:07:04 -07:00
Chao Sun
b41997ceac HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-07-25 10:07:04 -07:00
Erik Krogen
0f811b0849 HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 2019-07-25 10:07:04 -07:00
Erik Krogen
9976b7bcbe HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-07-25 10:07:04 -07:00
Plamen Jeliazkov
ac0cdb0a8d HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov. 2019-07-25 10:07:04 -07:00
Chao Sun
13e86926a5 HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-25 10:07:04 -07:00
Konstantin V Shvachko
c02f4ccba1 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-07-25 10:07:04 -07:00
Erik Krogen
f847983f2c HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 2019-07-25 10:07:04 -07:00
Erik Krogen
db9cec6ad2 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-07-25 10:07:04 -07:00
Erik Krogen
eacf43252a HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-07-25 10:07:04 -07:00
Plamen Jeliazkov
d502e48a7c HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-07-25 10:07:04 -07:00
Erik Krogen
d7e0011b63 HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. 2019-07-25 10:07:04 -07:00
Erik Krogen
849e1f018e HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-07-25 10:07:04 -07:00
Erik Krogen
c2585f7e28 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-07-25 10:07:04 -07:00
Plamen Jeliazkov
ccad9ce59b HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-07-25 10:07:04 -07:00
Chao Sun
44d8b43697 HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. 2019-07-25 10:07:04 -07:00
Eric E Payne
6aa76eaee3 YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonanthan Hung) 
Addendum to fix 3.0 build
 2019-07-23 21:53:27 +00:00
Jonathan Hung
3b346f50de YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonanthan Hung) 2019-07-22 10:54:38 -07:00
Weiwei Yang
6c8cfd135e HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 2019-07-11 09:53:28 +08:00
Weiwei Yang
ef35954cbe HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 2019-07-09 22:22:50 +08:00
Weiwei Yang
9ab56b2637 Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit 974dd2b4b6103374969fd7cfeb2cee50d4112c6a.
 2019-07-09 12:23:39 +08:00
Erik Krogen
172e0ee0b9 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 
(cherry-picked from 4632708148ed7dec75810feddb22ad98e4670483)
(cherry-picked from 491dc7cc9d6836f34d2647b12dcefa86cc97a927)
(cherry-picked from a59553b5e6bc6fa59ed31306708d899342013191)
 2019-07-08 15:20:41 -07:00
leosunli
14d2c09ea6 HDFS-13694. Making md5 computing being in parallel with image loading. 
(cherry picked from commit 9c907294860a48f0d1676a31bda15795b7a6771a)
 2019-07-07 20:03:29 -07:00
Weiwei Yang
78a6e40994 YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. Contributed by hunshenshi. 2019-07-02 23:05:18 +08:00
Inigo Goiri
9daa45f646 HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42a3ef94d6bef160bc58d807f2faa104)
(cherry picked from commit 96371245357bda63b3ede10f37a37f5333a85d69)
(cherry picked from commit 90b88db35d42f2eab4da7f192a5fb99d9c834abb)
 2019-06-24 21:17:22 -07:00
Inigo Goiri
9aae8b05a7 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 
(cherry picked from commit 0966407ad6fddb7adf60dba846d1f7c8a13b8ec1)
 2019-06-24 19:03:58 -07:00
Santosh Marella
974dd2b4b6 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 026cc2647d4e3c1cffdd154995a5f1f07956c2a0)
 2019-06-23 20:28:14 -07:00
Wei-Chiu Chuang
337a7090b3 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 
(cherry picked from commit a5d76a4965b686ec5532add511c64d5f87c88865)
(cherry picked from commit f33bcbb687e4dd4ab10d5dad89b1b8eae5788efb)
 2019-06-20 20:54:58 -07:00
Zsolt Venczel
76faa41f18 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff6bf8d5e1d05748ba32b679e397b53f)
(cherry picked from commit 4ea3b04bf36af56612c6b47d9ff8aa49cbbbcc9e)
 2019-06-18 14:38:28 -07:00
Zhankun Tang
6da0151081 YARN-9584. Should put initializeProcessTrees method call before get pid. Contributed by Wanqiang Ji. 
(cherry picked from commit 67414a1a80039e70e0afc1de171831a6e981f37a)
 2019-06-18 13:19:28 +08:00
Wei-Chiu Chuang
79f8e4d5bc HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea8c75c134de43a3a925c76b253658e1)
(cherry picked from commit 45543ffd6af723bb83bd44a99f12165363c70abb)
(cherry picked from commit fc8983a5b8c94746d3e58cc203974aa09c1b640b)
 2019-06-17 20:20:21 -07:00
Akira Ajisaka
010a99104b HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827381a7f87c01f0befc2475f9478e63b)
(cherry picked from commit f68099aa1316f34591c757bcf4198a3a3c7f691a)
(cherry picked from commit 3d1d8623a11b72c7939d6500537be5fd6565d4d6)
 2019-06-17 17:04:19 -07:00
Wei-Chiu Chuang
fae98959ad HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
(cherry picked from commit 0eb42e1e05b780dab718bb3adfdd55f527dd5545)
(cherry picked from commit b3e0430c3bee30a52ec08786982522bfcf0bb911)
 2019-06-17 15:18:54 -07:00
huzheng
2b9c4fccf4 HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a184405c3feca024a560807ea215b6858b9)
(cherry picked from commit 9b61ecfcbeeb1d3c320f775b43c68d3e39a271a3)
(cherry picked from commit b082628e5a39fc1bc9d6094beb9c253cc096c701)
 2019-06-17 07:21:46 -07:00
Takanobu Asanuma
1be7c42cb7 HADOOP-16369. Fix zstandard shortname misspelled as zts. Contributed by Jonathan Eagles. 
(cherry picked from commit 54f9f75a443d7d167a7aa7d04a87e3f5af049887)
 2019-06-14 10:20:33 +09:00
Erik Krogen
d43219c21d HADOOP-16345. Fix a potential NPE when instantiating FairCallQueue metrics. Contributed by Erik Krogen. 
(cherry-picked from 76b94c274fe9775efcfd51c676d80c88a4f7fdb9)
(cherry-picked from f607efd1950211f85eb571564b774184f8fa1687)
(cherry-picked from cf3c1b9ffc8cfce2e2248a1c1600bd34ff412933)
 2019-06-07 14:22:48 -07:00
Sean Mackrory
c352b0011e HADOOP-16212. Update guava to 27.0-jre in hadoop-project branch-3.0. Contributed by Gabor Bota. 2019-06-03 07:45:56 -06:00
Kihwal Lee
fd8b8c7494 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b6493d315c8910fbc06eaba96273ee335c)
 2019-05-30 17:36:19 -05:00
Ayush Saxena
942c5b04f2 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f27e3e4f58ff1b5ef3e93efc56bbecbe)
(cherry picked from commit 2cd7577a5a58db42a14a659b85fbd475091d0cb2)
(cherry picked from commit 90cc5ddd4957fcb27ef5a1e7ee1a67c4eca9a3f5)
 2019-05-29 20:58:54 -07:00
Eric E Payne
7da78f06a3 YARN-8625. Aggregate Resource Allocation for each job is not present in ATS. Contributed by Prabhu Joseph. 
(cherry picked from commit 3c63551101dc17038efb9f5345833d2883fce86e)
 2019-05-29 19:23:23 +00:00
Ahmed Hussein
9d63fc7ed6 YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2019-05-29 14:19:20 -05:00
Erik Krogen
2df08761e9 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f002f74cb4a0360b6682a1b6796d1598)
(cherry-picked from c26cf22d6b3949a4798d8133669b451660a0f050)
(cherry-picked from 9295e95174ad8627874b81a9d43183384a81fb16)
 2019-05-24 13:20:01 -07:00
Konstantin V Shvachko
e4ed873594 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:38:07 -07:00
Alexis Daboville
3b09902dd2
HADOOP-16248. MutableQuantiles leak memory under heavy load. 
Contributed by Alexis Daboville,

(cherry picked from commit 4cb3da6ac703b10de59a8ca371fe957a4a338601)
 2019-05-17 15:18:19 +01:00
Weiwei Yang
afe2b98daf HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 
(cherry picked from commit 2d8282bb8248e6984878626c4cdc7148aa2e7202)
 2019-05-14 14:08:36 -07:00