Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,308 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

15308 Commits

Author SHA1 Message Date
Yiqun Lin fc2cdb6cca HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 
(cherry picked from commit 2b452b4e60)
 2019-11-28 10:45:25 +08:00
Erik Krogen 403892264b HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 
(cherry picked from b2cc8b6b4a)
(cherry picked from 60655bfe54)
(cherry picked from 17779adb32)
 2019-11-26 16:46:38 -08:00
Eric E Payne 3a7121b0b6 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f)
 2019-11-22 19:03:42 +00:00
Eric E Payne bfb76e60cf YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988)
 2019-11-22 18:57:05 +00:00
Tao Yang 5b2a1b641a YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 
(cherry picked from commit 8afabe41e4)
 2019-11-22 16:18:59 +08:00
Jason Lowe dbbe23f269 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 
(cherry picked from commit 25df505421)
 2019-11-21 11:55:20 -08:00
Eric E Payne a9f51e0372 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) (refactored patch for branch-2) 2019-11-21 19:23:48 +00:00
Wangda Tan 835bc686e0 YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f)

Conflicts:
  hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
  I did not backport the changes to ResourcePBImpl.java because the code was removed by mistake and soon added back as part of YARN-7483 (Eric Payne).
 2019-11-19 22:39:14 +00:00
He Xiaoqiao ae7469cb67 HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
(cherry picked from commit 0b50aa29fd)
(cherry picked from commit 1d383436f09f8e490171c8f6af4605971ad2f904)
(cherry picked from commit 7909c5df572a5cbc729ebc6c7c6e59f17803d50f)
 2019-11-19 12:23:01 -08:00
Chen Liang 238781ab98 HDFS-14991. Backport better time precision of configuration#getTimeDuration to branch-2 to support SBN read. Contributed by Chen Liang. 2019-11-18 20:26:27 -08:00
Eric E Payne a74411ebaa Revert "YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda)" 
Pulling this back into branch-2 prevented nodemanagers from coming up if extended resources are present.

This reverts commit ab83765a6c.
 2019-11-15 22:01:28 +00:00
Eric E Payne 6973f78a73 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 
(cherry picked from commit 0b4c44bdee)
 2019-11-15 19:00:06 +00:00
Wangda Tan ab83765a6c YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f)
 2019-11-15 17:21:30 +00:00
Erik Krogen 792e2aecb3 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 
(cherry picked from 586defe711)
(cherry picked from dec765b329)
(cherry picked from 5cf36aa2b1)
 2019-11-15 07:40:20 -08:00
Akira Ajisaka 81060b3413
HADOOP-15097. AbstractContractDeleteTest::testDeleteNonEmptyDirRecursive with misleading path. Contributed by Xieming Li. 
(cherry picked from commit 92c28c100e)
 2019-11-15 13:53:39 +09:00
Mukul Kumar Singh 594307fe21 HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Yuval Degani, Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Yuval Degani <ydegani@linkedin.com>
 2019-11-14 20:50:51 -08:00
Robert Kanter 235d6c73e5 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter) 
(cherry picked from commit c8b53c4364)
(cherry picked from commit 0506c762b2)
 2019-11-14 15:47:24 -08:00
Daniel Templeton 7f1d2e196c YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 
(cherry picked from commit 8498d287cd)
 2019-11-14 14:56:23 -08:00
Wangda Tan b5128b2ae2 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
(cherry picked from commit d02e42cee4)
 2019-11-14 14:44:30 -08:00
Ayush Saxena d1b91ede4c HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:32:13 +05:30
Akira Ajisaka 5e100ada27 HDFS-14590. [SBN Read] Add the document link to the top page. Contributed by Takanobu Asanuma. 
(cherry picked from commit 48d7f00da8)
 2019-11-07 22:32:03 +09:00
Chen Liang 92c9b1de5b HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 16:23:51 -08:00
Chen Liang db887f66d8 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 15:47:57 -08:00
Ahmed Hussein c59b1b66a5 MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2019-11-05 15:48:27 -06:00
Sunil G ec5c102ba7 YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen. 
(cherry picked from commit 71220d218d)
 2019-11-05 16:03:52 +00:00
Duo Zhang e649b30b32 HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Reapply but this time with proper patch.

Signed-off-by: stack <stack@apache.org>
 2019-11-01 09:22:23 -07:00
stack e880347d53 Revert "HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches" 
Wrong patch applied; missed jdk7 handling

This reverts commit a36dbe6f35.
 2019-11-01 09:21:33 -07:00
Jonathan Hung fab62a145d Add 2.10.0 release notes for HDFS-12943 
(cherry picked from commit ef9d12df24)
(cherry picked from commit 9ccfc2d6d3)
(cherry picked from commit f29481f046)
 2019-10-31 20:35:49 -07:00
Jonathan Hung ae3f5d66e4 YARN-9945. Fix javadoc in FederationProxyProviderUtil in branch-2 2019-10-31 15:30:23 -07:00
Duo Zhang a36dbe6f35 HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 22:33:51 -07:00
Jonathan Hung 915b795358 Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aa)
(cherry picked from commit 4d274f60bc)
 2019-10-30 21:01:40 -07:00
Eric Badger 128631c5ac YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87)
 2019-10-25 17:45:40 +00:00
Wei-Chiu Chuang a805d80ebe HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 2019-10-22 16:43:25 -07:00
Jonathan Hung 339ab2347d HDFS-14667. Backport [HDFS-14403] "Cost-based FairCallQueue" to branch-2. Contributed by Erik Krogen 2019-10-22 11:46:25 -07:00
Kihwal Lee b634bd225b HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef773)
 2019-10-21 16:36:42 -05:00
Ayush Saxena c78eb006f9 HADOOP-16662. Remove unnecessary InnerNode check in NetworkTopology#add(). Contributed by Lisheng Sun. 2019-10-20 11:58:30 +05:30
Da Zhou 7f71473888
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI 
(cherry picked from commit 9a8edb0aed)
 2019-10-16 22:04:36 -07:00
Bilahari T H 6cb9ee32e1
HADOOP-16652. Backport of HADOOP-16587: Make ABFS AAD endpoints configurable 2019-10-16 15:39:39 -07:00
Jonathan Hung f555aa740b HADOOP-16655. Change cipher suite when fetching tomcat tarball for branch-2 2019-10-15 17:40:29 -07:00
Jonathan Hung 108127238b Preparing for 2.11.0 development 2019-10-15 10:01:45 -07:00
Haibo Chen 600d54fcd6 YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88)
 2019-10-15 15:52:25 +00:00
Chen Liang 6566402a1b HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 2019-10-10 13:29:30 -07:00
Erik Krogen 200c52f78b HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e00143)
(cherry picked from 6630c9b75d)
(cherry picked from 9fdb849e03)
 2019-10-10 09:09:50 -07:00
Erik Krogen 8337d1a489 HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 2019-10-10 09:07:18 -07:00
Sneha Vijayarajan 27d7d09fa4
HADOOP-16630 : Backport of Hadoop-16548 : Disable Flush() over config 2019-10-09 16:50:06 -07:00
Sneha Vijayarajan 2ccb1683ba
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists 2019-10-09 16:23:25 -07:00
Jonathan Hung ab7181e9af YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:19:39 -07:00
Ayush Saxena ca777db3ca HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-10-08 13:30:30 +05:30
Shweta Yakkali 56e2ad2dce HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-10-03 16:43:27 -07:00
Ayush Saxena a3b9b955be HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-10-03 16:36:50 -07:00