Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,240 Commits 326 Branches 371 Tags
Commit Graph

5167 Commits

Author SHA1 Message Date
Jonathan Hung
d74f64aa99 Setting version to 2.10.1-SNAPSHOT after branch-2 to branch-2.10 rename 2019-12-09 11:49:59 -08:00
Chao Sun
9a385e00a4 HDFS-15017. Remove redundant import of AtomicBoolean in NameNodeConnector. Contributed by Chao Sun. 2019-12-07 13:45:17 -08:00
Chao Sun
c6ce4011d7 HDFS-15005. Backport HDFS-12300 to branch-2. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-12-06 15:25:30 -08:00
Yiqun Lin
fc2cdb6cca HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 
(cherry picked from commit 2b452b4e6063072b2bec491edd3f412eb7ac21f3)
 2019-11-28 10:45:25 +08:00
Erik Krogen
403892264b HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 
(cherry picked from b2cc8b6b4a78f31cdd937dc4d1a2255f80c5881e)
(cherry picked from 60655bfe54e138957ef5bbf480a4541bd83152fd)
(cherry picked from 17779adb32569bd689a93802512003b6a6816bd4)
 2019-11-26 16:46:38 -08:00
He Xiaoqiao
ae7469cb67 HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
(cherry picked from commit 0b50aa29fd5dc114b3e0fc54b5c393bbc9f3102e)
(cherry picked from commit 1d383436f09f8e490171c8f6af4605971ad2f904)
(cherry picked from commit 7909c5df572a5cbc729ebc6c7c6e59f17803d50f)
 2019-11-19 12:23:01 -08:00
Chen Liang
238781ab98 HDFS-14991. Backport better time precision of configuration#getTimeDuration to branch-2 to support SBN read. Contributed by Chen Liang. 2019-11-18 20:26:27 -08:00
Erik Krogen
792e2aecb3 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 
(cherry picked from 586defe7113ed246ed0275bb3833882a3d873d70)
(cherry picked from dec765b329d3947f30273c0e7f0c4eb607ec42c9)
(cherry picked from 5cf36aa2b19aad8677e1a7553ff1f05805f772b7)
 2019-11-15 07:40:20 -08:00
Mukul Kumar Singh
594307fe21 HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Yuval Degani, Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Yuval Degani <ydegani@linkedin.com>
 2019-11-14 20:50:51 -08:00
Ayush Saxena
d1b91ede4c HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:32:13 +05:30
Chen Liang
92c9b1de5b HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 16:23:51 -08:00
Chen Liang
db887f66d8 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 15:47:57 -08:00
Duo Zhang
e649b30b32 HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Reapply but this time with proper patch.

Signed-off-by: stack <stack@apache.org>
 2019-11-01 09:22:23 -07:00
stack
e880347d53 Revert "HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches" 
Wrong patch applied; missed jdk7 handling

This reverts commit a36dbe6f354b0318c944177833f4952a9136ef58.
 2019-11-01 09:21:33 -07:00
Duo Zhang
a36dbe6f35 HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 22:33:51 -07:00
Jonathan Hung
915b795358 Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aac608746946dc2ed5711549a01a2055)
(cherry picked from commit 4d274f60bc4bf2800477800ec58b2b2c6f4f656e)
 2019-10-30 21:01:40 -07:00
Wei-Chiu Chuang
a805d80ebe HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 2019-10-22 16:43:25 -07:00
Kihwal Lee
b634bd225b HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef7735acd6f91b73d2ecb0227f8dd75a2e4)
 2019-10-21 16:36:42 -05:00
Jonathan Hung
108127238b Preparing for 2.11.0 development 2019-10-15 10:01:45 -07:00
Chen Liang
6566402a1b HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 2019-10-10 13:29:30 -07:00
Erik Krogen
200c52f78b HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e0014343f60f853cb796781ca1fa03a72efd)
(cherry picked from 6630c9b75d65deefb5550e355eef7783909a57bc)
(cherry picked from 9fdb849e034573bb44abd593eefa1e13a3261376)
 2019-10-10 09:09:50 -07:00
Erik Krogen
8337d1a489 HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 2019-10-10 09:07:18 -07:00
Ayush Saxena
ca777db3ca HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-10-08 13:30:30 +05:30
Shweta Yakkali
56e2ad2dce HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-10-03 16:43:27 -07:00
Ayush Saxena
a3b9b955be HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-10-03 16:36:50 -07:00
Shweta Yakkali
c87b1f6f5b HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 16:25:31 -07:00
Chen Liang
72b37e8833 HDFS-12979. StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang. 2019-10-03 12:23:25 -07:00
Chen Liang
ed8f1ae602 HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:51:59 -07:00
Konstantin V Shvachko
dc2b838a8e HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 18:04:16 -07:00
Masatake Iwasaki
9482da7053 HADOOP-16544. update io.netty in branch-2. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-10-01 05:24:32 +09:00
Ayush Saxena
66e60942e9 HDFS-14853. Addendum: NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-28 09:14:50 +05:30
Ayush Saxena
40becd37d0 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-28 08:30:55 +05:30
Chen Liang
071d199774 HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 16:06:35 -07:00
Chen Liang
dc72b2e65a HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang. 2019-09-18 17:22:49 -07:00
Kihwal Lee
d4038e7a39 HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 1851d06eb3b70f39f3054a7c06f0ad2bc664aaec)
 2019-09-18 16:51:21 -05:00
He Xiaoqiao
6fe8261fa0 HDFS-14771. Backport HDFS-14617 to branch-2 (Improve fsimage load time by writing sub-sections to the fsimage index). Contributed by He Xiaoqiao and Stephen O'Donnel. 
Reviewed-by: Stephen O'Donnell <sodonnell@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-17 09:34:54 -07:00
Ayush Saxena
b9e832bf73 HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-09-14 10:30:46 +05:30
Xiaoyu Yao
c44e53e7a3
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-05 16:24:17 -07:00
Xiaoyu Yao
1c9255aebf
Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun." 
This reverts commit a524608d1e0a276fd822e9574f0a4a6d51298b13.
 2019-09-05 16:22:15 -07:00
Ayush Saxena
ad250a8cbb HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang 2019-09-05 22:05:47 +05:30
Masatake Iwasaki
3ded8ece6b HADOOP-16439. Upgrade bundled Tomcat in branch-2. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-09-04 17:26:33 +09:00
Chen Liang
1ff68d0e47 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang 2019-09-03 17:18:41 -07:00
Chen Liang
163cb4d5ae HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-09-03 17:11:29 -07:00
Chen Liang
e63619a6dd HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-09-03 17:11:29 -07:00
Chen Liang
a06e35a197 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-09-03 17:11:28 -07:00
Xiaoyu Yao
a524608d1e
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-03 08:27:23 -07:00
Chen Liang
f05d87b32e HDFS-14726. Fix JN incompatibility issue in branch-2 due to backport of HDFS-10519. Contributed by Chen Liang. 2019-08-30 15:55:35 -07:00
Chen Liang
e786147096 HDFS-13977. NameNode can kill itself if it tries to send too many txns to a QJM simultaneously. Contributed by Erik Krogen. 2019-08-26 14:32:27 -07:00
Wei-Chiu Chuang
69b80f1ee8 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit 8755de108d0e711cbfa784a20bd89e28fe79266b.
 2019-08-21 09:04:07 -07:00
Wei-Chiu Chuang
bdab03cf71 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 2019-08-20 10:38:52 -07:00