Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,968 Commits 326 Branches 371 Tags
Commit Graph

14968 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sunil G
c5e77e8b84 YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 
(cherry picked from commit 7805deed4896e470ebd2f6bbd1ba9962947c63cd)
 2019-08-29 17:02:46 -07:00
Arun Suresh
6db8c36ba9 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh) 
(cherry picked from commit 6708ac330147b2d3816a31f2ee83e09c41fe0dd9)
 2019-08-29 17:02:46 -07:00
Rohith Sharma K S
f99d603aaf YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev. 
(cherry picked from commit 239c1824a07fede71dd751ed2c1f40036b22170e)
 2019-08-29 17:02:46 -07:00
Rohith Sharma K S
015a55a5c6 YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev. 
(cherry picked from commit 9e4ba6aff595b7a5b53d016c97c8334e10f1f2cb)
 2019-08-29 17:02:46 -07:00
Varun Vasudev
7d5050a768 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 
(cherry picked from commit 759114b0063907d4c07ea6ee261e861bf5cc3a9a)
 2019-08-29 17:02:46 -07:00
Arun Suresh
817837e65c YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh) 
(cherry picked from commit 7ba698997b6845b664b2f8c82e384b457c30d176)
 2019-08-29 17:02:46 -07:00
Varun Vasudev
133eb31d54 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev. 
(cherry picked from commit ceb12c59a34677518a198e19acb4e29cdfc8a0e1)
 2019-08-29 17:02:46 -07:00
Wangda Tan
be2afb4a5d YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda) 
(cherry picked from commit 32c91223f1bd06561ea4ce2d1944e8d9a847f18c)
 2019-08-29 17:02:46 -07:00
Wangda Tan
afcf9c55b5 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda) 
(cherry picked from commit 1bbab7c1570a2438b2fa6da70397dd1d5211a137)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
 2019-08-29 17:02:46 -07:00
Sneha Vijayarajan
f56972bfe1 HADOOP-16460: ABFS: fix for Sever Name Indication (SNI) 
Contributed by Sneha Vijayarajan <snvijaya@microsoft.com>
 2019-08-28 22:42:28 +00:00
Da Zhou
6cb74629ec HADOOP-16315. ABFS: transform full UPN for named user in AclStatus 
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
 2019-08-28 22:42:28 +00:00
bilaharith
1376f2ed3d HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC. 
Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
 2019-08-28 22:42:28 +00:00
Arun Singh
8044415ea1 HADOOP-16404. ABFS default blocksize change(256MB from 512MB) 
Contributed by: Arun Singh
 2019-08-28 22:42:28 +00:00
Da Zhou
42816bf698 HADOOP-16376. ABFS: Override access() to no-op. 
Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
 2019-08-28 22:42:28 +00:00
Aaron Fabbri
67fe5398fd Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest" 
This reverts commit 7c2d7c05a9a9cc981674f97cc3825e917a17b1f7.

Git Commit name and email were incorrect. Will re-commit.
 2019-08-28 22:42:28 +00:00
DadanielZ
605c812749 HADOOP-16251. ABFS: add FSMainOperationsBaseTest 
Author: Da Zhou
 2019-08-28 22:42:28 +00:00
Da Zhou
50953c586b HADOOP-16269. ABFS: add listFileStatus with StartFrom. 
Author:    Da Zhou
 2019-08-28 22:42:28 +00:00
Da Zhou
e5bb8498df HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream. 
Contributed by Da Zhou.
 2019-08-28 22:42:28 +00:00
Eric E Payne
8e8c16dbc6 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 
(cherry picked from commit d562050cec83a2bc2ffb6d109ed3d64b394b870d)
 2019-08-28 21:26:03 +00:00
Jonathan Hung
b6be5f869a YARN-9438. launchTime not written to state store for running applications 
(cherry picked from commit 9568656cd21d9c02168e18ce35c6726077bbf3a1)
(cherry picked from commit 0c498de6e87c6bdc959afa31deb03d0907e0e1a1)
(cherry picked from commit f73842780e3e5daff2db4d30771a194f3bc47d6d)
 2019-08-27 15:49:58 -07:00
Jonathan Hung
f083f7fef1 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca15098e891c560beb3181c22ef3f80ff)
(cherry picked from commit e4249c320257586384035ea3fc286fe54cc699a1)
(cherry picked from commit 6baa0d1e4d78c15833257d98bdd2b38806faed9f)
 2019-08-26 16:05:25 -07:00
Chen Liang
e786147096 HDFS-13977. NameNode can kill itself if it tries to send too many txns to a QJM simultaneously. Contributed by Erik Krogen. 2019-08-26 14:32:27 -07:00
Weiwei Yang
97f207d1a9 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang. 
(Cherry picked from commit 67f239c42f676237290d18ddbbc9aec369267692)

(cherry picked from commit a0b7abf278c147d3ae7821c6783a85cb972c6ec2)
 2019-08-23 09:59:47 -07:00
Akira Ajisaka
786df37429
HADOOP-16494. Add SHA-512 checksum to release artifact to comply with the release distribution policy. 
This closes #1243

(cherry picked from commit 34dd9ee36674be670013d4fc3d9b8f5b36886812)
 2019-08-22 11:04:05 +09:00
Wei-Chiu Chuang
69b80f1ee8 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit 8755de108d0e711cbfa784a20bd89e28fe79266b.
 2019-08-21 09:04:07 -07:00
Wei-Chiu Chuang
bdab03cf71 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 2019-08-20 10:38:52 -07:00
Shashikant Banerjee
ab7509efe7 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-18 19:01:12 -07:00
Wei-Chiu Chuang
8755de108d HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-18 17:51:44 -07:00
He Xiaoqiao
b020c5ccfd HDFS-14725. Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report). Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-16 17:53:45 -07:00
Masatake Iwasaki
61bd868e80 HDFS-14707. Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-08-15 11:21:17 +09:00
Masatake Iwasaki
670de354b3 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe0473e353ee2d489156248a01aa982dfd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63d89e5c8377a035f93485a7c77c3430)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
 2019-08-14 12:35:14 +09:00
Christopher Gregorian
99cd181a78 HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a4e3b8f165806dfd2966f0951a5e575.

(cherry-picked from f96a2df38d889f29314c57f4d94227b2e419a11f)
(cherry-picked from d4492bdd9edec60c236aff85de50b963097e5a7f)
(cherry-picked from 7b8f08f59e5c8906930ccc67b967b7cfcbd41768)
(cherry picked from ec00431eaaa25eff5bb5e0cafb52de685187a159)
 2019-08-13 14:28:18 -07:00
Eric Badger
afa9a4084d YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949f041da2ee04da441c5f9f85e1f2c64)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be76577718b22f936aea9e2b2cd100ea)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c

(cherry picked from commit db88224e8f9d164ac811fcca9efe4a350cebecd1)
 2019-08-13 17:27:43 +00:00
Erik Krogen
719214ff30 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e24be294b40088a8aa46086ba8ca4ba8)
(cherry picked from 016aa139406d1a151fbec3fb8912eb08e0f638d7)
(cherry picked from f6ce2f4a50898b16556089fe18a6717a15570d85)
(cherry picked from 5657e45fb2b904bb31a4e3b70a360477ba288c15)
 2019-08-13 08:30:33 -07:00
Chen Liang
885b7f4c98 Fix JDK7 compatibility in branch-2. 2019-08-12 17:42:01 -07:00
Chen Liang
0d153dcb8c HDFS-13898 Addendum: add a missing test helper method 2019-08-12 10:31:37 -07:00
Wei-Chiu Chuang
585b6de637 Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit 567e1178d88ccfc258ce2ade4f8af66cc5a4daa7.
 2019-08-09 18:29:00 -07:00
Chen Liang
097710131a [SBN read] Addendum: Misc fix of issues, javadoc, whitespace unused backported fields etc 2019-08-09 15:49:58 -07:00
Wei-Chiu Chuang
12dc7ff566 HDFS-12836. startTxId could be greater than endTxId when tailing in-progress edit log. Contributed by Chao Sun. 2019-08-09 15:49:58 -07:00
Wei-Chiu Chuang
9599c91518 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-08-09 15:49:58 -07:00
Erik Krogen
2fe15053ab HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen
c377540704 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen
5190fd64be HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen
c055e7812c HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen
3b2b97a6e6 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3f74968c92751f0972746b012e72810)
(cherry-picked from commit 6d076dd5e8be8fee5062a04d4cd2aa0c5da0ea56)
 2019-08-09 15:49:58 -07:00
Erik Krogen
a0327f58cb HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-08-09 15:49:58 -07:00
Chen Liang
b247a8b0de [SBN read] Addendum: Misc fix to be Java 7 compatible 2019-08-09 15:49:58 -07:00
Konstantin V Shvachko
3804b8e7be HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2019-08-09 15:49:57 -07:00
Chao Sun
8cf1c0b91b HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko
3ec2ff756d HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-08-09 15:49:57 -07:00