Commit Graph

14042 Commits

Author SHA1 Message Date
John Zhuge 2442a8d716 HADOOP-14824. Update ADLS SDK to 2.2.2 for MSI fix. Contributed by Atul Sikaria.
(cherry picked from commit 7ecc6dbed6)
2017-08-31 21:13:50 -07:00
Yiqun Lin 41d8e4e9b3 HDFS-12317. HDFS metrics render error in the page of Github. Contributed by Yiqun Lin. 2017-09-01 10:13:01 +08:00
Daniel Templeton 004231dc08 YARN-6780. ResourceWeights.toString() cleanup (Contributed by weiyuan via Daniel Templeton)
(cherry picked from commit d4417dae4f)
2017-08-31 09:38:31 -07:00
Haibo Chen 59f1ca7258 MAPREDUCE-6937. Backport MAPREDUCE-6870 to branch-2 while preserving compatibility. (Peter Bacsko via Haibo Chen) 2017-08-31 09:28:19 -07:00
Sunil G e8b4427259 YARN-7116. CapacityScheduler Web UI: Queue's AM usage is always show on per-user's AM usage. Contributed by Wangda Tan.
(cherry picked from commit f9e0cc8cdc)
2017-08-31 18:43:17 +05:30
Yufei Gu 626fa7c8e1 YARN-7077. TestAMSimulator and TestNMSimulator fail. (Contributed by Akira Ajisaka via Yufei Gu) 2017-08-30 23:08:02 -07:00
Robert Kanter 8523319e15 YARN-7094. Document the current known issue with server-side NM graceful decom (rkanter)
(cherry picked from commit ce79f7be29)
2017-08-30 18:06:03 -07:00
Konstantin V Shvachko 9a0aac4170 MAPREDUCE-6931. Remove TestDFSIO "Total Throughput" calculation. Contributed by Dennis Huo.
(cherry picked from commit 3e0e2033cc)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
2017-08-30 14:42:36 -07:00
Steve Loughran 25e9c29608
HADOOP-14802. Add support for using container saskeys for all accesses.
Contributed by Sivaguru Sankaridurg

(cherry picked from commit 021974f4cb)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
2017-08-30 22:40:50 +01:00
Junping Du 7dd5bb9492 HADOOP-14814. Fix incompatible API change on FsServerDefaults to HADOOP-14104. Contributed by Junping Du.
(cherry picked from commit 41480233a9)
2017-08-30 13:30:48 -07:00
Xiao Chen 858715d054 HDFS-12336. Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory. Contributed by Wellington Chevreuil. 2017-08-30 09:43:12 -07:00
Rohith Sharma K S 877d96b83d YARN-6982. Potential issue on setting AMContainerSpec#tokenConf to null before app is completed. Contributed by Manikandan R.
(cherry picked from commit 4cae120c61)
2017-08-30 10:50:57 +05:30
Yongjun Zhang 870ef0b2fd HDFS-12191. Provide option to not capture the accessTime change of a file to snapshot if no other modification has been done to this file. Contributed by Yongjun Zhang.
(cherry picked from commit cf93d60d3f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirAttrOp.java
2017-08-29 21:57:25 -07:00
Jian He 036cbd7453 YARN-6756. ContainerRequest#executionTypeRequest causes NPE. Contributed by Jian He 2017-08-29 20:36:23 -07:00
Wangda Tan 9bcd64d233 YARN-5219. When an export var command fails in launch_container.sh, the full container launch should fail. (Sunil G via wangda)
Change-Id: Iaa6b978bb89482e9d1d77ba57f4adfdc48e39a3c
(cherry picked from commit f59332b97b)
2017-08-29 15:29:53 -07:00
Junping Du b906bddd95 YARN-7037. Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices. Contributed by Tao Yang.
(cherry picked from commit ad45d19998)
2017-08-29 15:17:21 -07:00
Jason Lowe cd08aa72a6 MAPREDUCE-6641. TestTaskAttempt fails in trunk. Contributed by Haibo Chen. 2017-08-29 12:24:42 -05:00
Jason Lowe c930aa3ff9 YARN-5816. TestDelegationTokenRenewer#testCancelWithMultipleAppSubmissions is still flakey. Contributed by Robert Kanter 2017-08-29 10:02:44 -05:00
Junping Du f252e1fe28 YARN-7076. yarn application -list -appTypes is not working. Contributed by Jian He.
(cherry picked from commit 312b1fd9da)
2017-08-28 18:27:38 -07:00
Arpit Agarwal e19f004f65 HDFS-11998. Enable DFSNetworkTopology as default. Contributed by Chen Liang. 2017-08-28 16:22:42 -07:00
Arpit Agarwal 98ba68b05d HDFS-12293. DataNode should log file name on disk error. Contributed by Ajay Kumar. 2017-08-28 15:50:06 -07:00
Wangda Tan 82e1cd08eb YARN-7112. TestAMRMProxy is failing with invalid request. (Jason Lowe via wangda)
Change-Id: I432f4523921666ac218fcf9936f826a0d8d635e0
(cherry picked from commit 51881a863c)
2017-08-28 11:38:05 -07:00
Yufei Gu 221c1bdf7e YARN-7099. ResourceHandlerModule.parseConfiguredCGroupPath only works for privileged yarn users. (Contributed by Miklos Szegedi via Yufei Gu)
(cherry picked from commit d5bc958279)
2017-08-28 10:11:01 -07:00
Yiqun Lin d63e98ad5f HDFS-11530. Use HDFS specific network topology to choose datanode in BlockPlacementPolicyDefault. Contributed by Chen Liang. 2017-08-28 20:47:38 +08:00
Sunil G 80570737a8 YARN-7051. Avoid concurrent modification exception in FifoIntraQueuePreemptionPlugin. Contributed by Eric Payne.
(cherry picked from commit 02599bda04)
2017-08-28 12:27:29 +05:30
Li Lu b7f319c53c YARN-6999. Add log about how to solve Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster. (Linlin Zhou via gtcarrera9)
(cherry picked from commit 3bee7046c5)
2017-08-25 22:45:20 -07:00
Lei Xu c22b5108df HDFS-12215. DataNode.transferBlock does not create its daemon in the xceiver thread group. (Lei Xu)
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
2017-08-25 16:06:34 -07:00
Eric Payne 1617744801 YARN-7087. NM failed to perform log aggregation due to absent container. Contributed by Jason Lowe.
(cherry picked from commit e864f81471)
2017-08-25 15:52:29 -05:00
Jason Lowe ea5bb5898f YARN-7052. RM SchedulingMonitor gives no indication why the spawned thread crashed. Contributed by Eric Payne
(cherry picked from commit 39a9dc8e4a)
2017-08-25 15:32:29 -05:00
Kihwal Lee 5a83ffa396 HDFS-12299. Race Between update pipeline and DN Re-Registration. Contributed by Brahma Reddy Battula. 2017-08-25 14:53:53 -05:00
Arpit Agarwal 2cae387402 HDFS-12319. DirectoryScanner will throw IllegalStateException when Multiple BP's are present. Contributed by Brahma Reddy Battula. 2017-08-25 10:41:59 -07:00
Jason Lowe 82957ce0d0 YARN-6640. AM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang
(cherry picked from commit 3a4e861169)
2017-08-25 09:20:50 -05:00
Yufei Gu 389b659c94 YARN-7049. FSAppAttempt preemption related fields have confusing names. (Contributed by Karthik Kambatla via Yufei Gu)
(cherry picked from commit 9e2699ac2c)
2017-08-24 17:54:00 -07:00
Junping Du e12358c5ce YARN-6876. Create an abstract log writer for extendability. Contributed by Xuan Gong.
(cherry picked from commit c2cb7ea1ef)
2017-08-24 15:27:40 -07:00
Akira Ajisaka bf903396a0
YARN-7047. Moving logging APIs over to slf4j in hadoop-yarn-server-nodemanager. Contributed by Yeliang Cang. 2017-08-25 03:28:02 +09:00
Arpit Agarwal ae69c99068 HDFS-11577. Combine the old and the new chooseRandom for better performance. Contributed by Chen Liang. 2017-08-24 11:17:36 -07:00
bibinchundatt cc50ca0722 YARN-7074. Fix NM state store update comment. Contributed by Botong Huang
(cherry picked from commit de0cba700b)
2017-08-24 16:40:18 +05:30
Junping Du c98917f798 YARN-7090. testRMRestartAfterNodeLabelDisabled get failed when CapacityScheduler is configured. Contributed by Wangda Tan.
(cherry picked from commit 652dd434d9)
2017-08-23 18:07:07 -07:00
Arpit Agarwal 8b2d5a4faa HDFS-11514. DFSTopologyNodeImpl#chooseRandom optimizations. Contributed by Chen Liang. 2017-08-23 12:49:13 -07:00
John Zhuge 6e8c696651 HADOOP-14251. Credential provider should handle property key deprecation. Contributed by John Zhuge.
(cherry picked from commit 7e6463d2fb)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java
2017-08-23 11:59:47 -07:00
Wangda Tan 245b49332d YARN-6251. Do async container release to prevent deadlock during container updates. (Arun Suresh via wangda)
Change-Id: I6c67d20c5dd4d22752830ebf0ed2340824976ecb
(cherry picked from commit f49843a988)
2017-08-23 10:35:37 -07:00
John Zhuge c54c3500ed HDFS-11738. Hedged pread takes more time when block moved from initial locations. Contributed by Vinayakumar B.
(cherry picked from commit b6bfb2fcb2)
2017-08-23 19:44:52 +05:30
Subru Krishnan a45ffdcddc YARN-7053. Move curator transaction support to ZKCuratorManager. (Jonathan Hung via Subru).
(cherry picked from commit 4249172e14)
2017-08-22 19:21:35 -07:00
Ray Chiang 2299c8dffa YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang)
(cherry picked from commit f5f14a2ad6)
2017-08-22 15:24:02 -07:00
Jason Lowe 45c1f8ac8e HADOOP-14687. AuthenticatedURL will reuse bad/expired session cookies. Contributed by Daryn Sharp
(cherry picked from commit c379310212)
2017-08-22 16:53:51 -05:00
Jason Lowe b15a393440 YARN-6988. container-executor fails for docker when command length > 4096 B. Contributed by Eric Badger 2017-08-22 16:28:08 -05:00
Jason Lowe a7893033fe YARN-7048. Fix tests faking kerberos to explicitly set ugi auth type. Contributed by Daryn Sharp
(cherry picked from commit 657dd59cc8)
2017-08-22 13:17:31 -05:00
Jason Lowe fc7df25a11 YARN-2416. InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time. Contributed by Jonathan Eagles
(cherry picked from commit 3efcd51c3b)
2017-08-22 12:59:47 -05:00
Arpit Agarwal b3ea11dfdb HDFS-11482. Add storage type demand to into DFSNetworkTopology#chooseRandom. Contributed by Chen Liang. 2017-08-21 14:07:59 -07:00
Manoj Govindassamy 89fc7fe670 HDFS-11988. Verify HDFS Snapshots with open files captured are consistent across truncates and appends to current version file. 2017-08-21 11:14:44 -07:00