Commit Graph

2711 Commits

Author SHA1 Message Date
Junping Du 2703ec6871 YARN-5076. YARN web interfaces lack XFS protection. Contributed by Jonathan Maron. 2016-05-19 14:10:15 -07:00
Jian He 2d846fdcbd YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. Contributed by Rohith Sharma K S & Zhiguo Hong
(cherry picked from commit feb90ffcca)
2016-05-19 13:02:05 -07:00
Junping Du 416274b53e YARN-5100. The YarnApplicationState is always running in ATS even application is finished. Contributed by Xuan Gong.
(cherry picked from commit 141873ca7d)
2016-05-19 09:14:42 -07:00
Steve Loughran 151ecdf26d HADOOP-12767. Update apache httpclient version to 4.5.2; httpcore to 4.4.4. Artem Aliev via stevel. 2016-05-19 14:44:44 +01:00
Arun Suresh 15808717f3 YARN-5110. Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 1597630681)
2016-05-18 23:20:54 -07:00
Arun Suresh a555a320e8 YARN-5090. Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster. (asuresh)
(cherry picked from commit 8a9ecb7584)
2016-05-18 23:20:36 -07:00
Arun Suresh f31602893e YARN-5075. Fix findbugs warnings in hadoop-yarn-common module. (asuresh)
(cherry picked from commit ccc93e7812)
2016-05-18 23:20:08 -07:00
Arun Suresh bbcea3abf3 YARN-4738. Notify the RM about the status of OPPORTUNISTIC containers (Konstantinos Karanasos via asuresh)
(cherry picked from commit f45bc5a83e)
2016-05-18 23:12:19 -07:00
Arun Suresh 69a9ce3494 YARN-2888. Corrective mechanisms for rebalancing NM container queues. (asuresh)
(cherry picked from commit f0ac18d001)
2016-05-18 23:09:11 -07:00
Arun Suresh 005dae521c YARN-5073. Refactor startContainerInternal() in ContainerManager to remove unused parameter. (Konstantinos Karanasos via asuresh)
(cherry picked from commit d971bf252f)
2016-05-18 22:36:28 -07:00
Arun Suresh 1b4600abcf YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
(cherry picked from commit 341888a0aa)
2016-05-18 22:35:55 -07:00
Arun Suresh 307cda70db YARN-5049. Extend NMStateStore to save queued container information. (Konstantinos Karanasos via asuresh)
(cherry picked from commit d464f4d1c4)
2016-05-18 22:09:30 -07:00
Arun Suresh d80d24aabf YARN-2885. Create AMRMProxy request interceptor and ContainerAllocator to distribute OPPORTUNISTIC containers to appropriate Nodes (asuresh)
(cherry picked from commit 2bf025278a318b0452fdc9ece4427b4c42124e39)
(cherry picked from commit c282a08f38)
2016-05-18 22:08:55 -07:00
Karthik Kambatla b56fc51b70 YARN-2883. Queuing of container requests in the NM. (Konstantinos Karanasos and Arun Suresh via kasha)
(cherry picked from commit c8172f5f14)
2016-05-18 22:08:20 -07:00
Arun Suresh 68964bbaa2 YARN-4335. Allow ResourceRequests to specify ExecutionType of a request ask (kkaranasos via asuresh)
(cherry picked from commit 8ffabfdf4f)
(cherry picked from commit b2a654c5ee)
2016-05-18 22:02:43 -07:00
Karthik Kambatla 2c6c701aff YARN-2882. Add an OPPORTUNISTIC ExecutionType. (Konstantinos Karanasos and Inigo Goiri via kasha)
(cherry picked from commit fb00794368)
2016-05-18 22:02:28 -07:00
Naganarasimha a166286b9e YARN-4925. ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression. Contributed by Bibin A Chundatt
(cherry picked from commit f04c81c9ce)
2016-05-19 01:35:47 +08:00
Varun Vasudev 3c9392a31d YARN-4913. Yarn logs should take a -out option to write to a directory. Contributed by Xuan Gong.
(cherry picked from commit ef1757790d)
2016-05-18 22:45:06 +05:30
Jian He 264c06a438 YARN-4832. NM side resource value should get updated if change applied in RM side. Contributed by Junping Du 2016-05-17 15:02:14 -07:00
Eric Payne 42b2a34ed8 YARN-5069. TestFifoScheduler.testResourceOverCommit race condition. Contributed by Eric Badger.
(cherry picked from commit 1217c8f6b4)
2016-05-16 20:38:10 +00:00
Jason Lowe 575c056357 YARN-4325. Nodemanager log handlers fail to send finished/failed events in some cases. Contributed by Junping Du
(cherry picked from commit 81effb7dcd)
2016-05-16 15:43:20 +00:00
Rohith Sharma K S a37b3694ea YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks) 2016-05-16 15:25:15 +05:30
Xuan 3fe4107909 YARN-5080. Cannot obtain logs using YARN CLI -am for either KILLED or
RUNNING AM. Contributed by Xuan Gong

(cherry picked from commit 7be53b65f9)
2016-05-14 12:22:06 -07:00
Jason Lowe bb95a5e18f YARN-5053. More informative diagnostics when applications killed by a user. Contributed by Eric Badger
(cherry picked from commit 013000fbc2)
2016-05-12 20:30:10 +00:00
Sangjin Lee d6d13ec670 YARN-4577. Enable aux services to have their own custom classpath/jar file (Xuan Gong via sale)
(cherry picked from commit 0bbe01f8d5)
2016-05-12 10:14:24 -07:00
Karthik Kambatla f96b2b6fac YARN-4995. FairScheduler: Display per-queue demand on the scheduler page. (xupeng via kasha)
(cherry picked from commit 4b4e4c6ba8)
2016-05-11 17:36:42 -07:00
Junping Du 9dc3683d87 YARN-5029. RM needs to send update event with YarnApplicationState as Running to ATS/AHS. Contributed by Xuan Gong.
(cherry picked from commit 39f2bac38b)
2016-05-11 09:33:16 -07:00
Vinod Kumar Vavilapalli e0d6a9632d YARN-4842. Fixed "yarn logs" command to guess (and thus not require) the appOwner argument when viewing another user's logs. Contributed by Ram Venkatesh and Xuan Gong.
(cherry picked from commit 87f5e35133)
2016-05-09 22:47:58 -07:00
Naganarasimha 3732a1e985 YARN-4926. Change nodelabel rest API invalid reponse status to 400. Contributed by Bibin A Chundatt
(cherry picked from commit 2750fb900f)
2016-05-08 23:02:07 +05:30
Yongjun Zhang 5172d0e7b1 YARN-5048. DelegationTokenRenewer#skipTokenRenewal may throw NPE (Jian He via Yongjun Zhang)
(cherry picked from commit 47c41e7ac7)
2016-05-06 22:38:56 -07:00
Jason Lowe 3895058a67 YARN-4747. AHS error 500 due to NPE when container start event is missing. Contributed by Varun Saxena
(cherry picked from commit b2ed6ae731)
2016-05-06 23:00:25 +00:00
Wangda Tan b68e6b1d6d getApplicationReport call may raise NPE for removed queues. (Jian He via wangda)
(cherry picked from commit 23248f63aa)
2016-05-06 15:32:15 -07:00
Jian He a6b24c62ab YARN-4390. Do surgical preemption based on reserved container in CapacityScheduler. Contributed by Wangda Tan
(cherry picked from commit bb62e05925)
2016-05-05 12:56:55 -07:00
Jason Lowe ee86cef2fe YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
(cherry picked from commit d0da13229c)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
2016-05-05 14:33:01 +00:00
Varun Vasudev 38a3b86141 YARN-4595. Add support for configurable read-only mounts when launching Docker containers. Contributed by Billie Rinaldi.
(cherry picked from commit 72b047715c)
2016-05-05 13:02:38 +05:30
Vinod Kumar Vavilapalli 8262ef8318 YARN-4905. Improved "yarn logs" command-line to optionally show log metadata also. Contributed by Xuan Gong.
(cherry picked from commit 9e37fe3b7a)
2016-05-04 14:17:09 -07:00
Wangda Tan 585299146a YARN-4984. LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak. (Junping Du via wangda)
(cherry picked from commit 7bd418e48c)
2016-05-04 11:39:25 -07:00
Junping Du 1ffb0c43d6 YARN-4920. ATS/NM should support a link to dowload/get the logs in text format. Contributed by Xuan Gong.
(cherry picked from commit 3cf223166d452a0f58f92676837a9edb8ddc1139)
2016-05-04 10:36:31 -07:00
Rohith Sharma K S 5aad4070b2 YARN-4947. Test timeout is happening for TestRMWebServicesNodes. Contributed by Bibin A Chundatt
(cherry picked from commit 75e0450593)
2016-05-04 10:26:25 +05:30
Jason Lowe baac4e7db1 YARN-5003. Add container resource to RM audit log. Contributed by Nathan Roberts
(cherry picked from commit ed54f5f1ff)
2016-05-03 22:16:17 +00:00
Jason Lowe d8fcf02069 YARN-4834. ProcfsBasedProcessTree doesn't track daemonized processes. Contributed by Nathan Roberts
(cherry picked from commit c6b4839168)
2016-05-03 22:14:50 +00:00
Junping Du 47f67ae447 YARN-4851. Metric improvements for ATS v1.5 storage components. Li Lu via junping_du.
(cherry picked from commit 06413da72e)
2016-05-03 04:18:01 -07:00
Robert Kanter ac8fb579c6 Remove parent's env vars from child processes 2016-04-29 09:26:09 -07:00
Varun Vasudev 6561e3b500 YARN-3998. Add support in the NodeManager to re-launch containers. Contributed by Jun Gong.
(cherry picked from commit 0f25a1bb52)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
2016-04-29 16:22:06 +05:30
Jian He 6ba39a1597 YARN-5009. NMLeveldbStateStoreService database can grow substantially leading to longer recovery times. Contributed by Jason Lowe
(cherry picked from commit 4a8508501b)
2016-04-28 21:54:30 -07:00
Jian He a9707dceaf YARN-5008. LeveldbRMStateStore database can grow substantially leading to long recovery times. Contributed by Jason Lowe 2016-04-28 21:28:03 -07:00
Li Lu 50b7a35d56 YARN-4956. findbug issue on LevelDBCacheTimelineStore. (Zhiyuan Yang via gtcarrera9)
(cherry picked from commit f16722d2ef)
2016-04-27 10:58:12 -07:00
Karthik Kambatla 864ecb4434 YARN-4807. MockAM#waitForState sleep duration is too long. (Yufei Gu via kasha)
(cherry picked from commit 185c3d4de1)
2016-04-27 09:43:42 -07:00
Varun Vasudev cf3e93ee73 YARN-4966. Improve yarn logs to fetch container logs without specifying nodeId. Contributed by Xuan Gong.
(cherry picked from commit 66b07d8374)
2016-04-27 15:57:17 +05:30
Jian He 9d3ddb0b4d YARN-4983. JVM and UGI metrics disappear after RM transitioned to standby mode
(cherry picked from commit 4beff01354)
2016-04-26 21:02:04 -07:00
Karthik Kambatla 52bfa90fed YARN-4795. ContainerMetrics drops records. (Daniel Templeton via kasha)
(cherry picked from commit 1a3f1482e2)
2016-04-26 06:18:27 -07:00
Karthik Kambatla a5edb45b18 YARN-1297. FairScheduler: Move some logs to debug and check if debug logging is enabled
(cherry picked from commit 4b1dcbbe0c)
2016-04-26 05:10:29 -07:00
Wangda Tan 45ff579bfa YARN-4846. Fix random failures for TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers. (Bibin A Chundatt via wangda)
(cherry picked from commit 7cb3a3da96)
2016-04-22 11:41:08 -07:00
Eric Payne a7f903b2ba YARN-4556. TestFifoScheduler.testResourceOverCommit fails. Contributed by Akihiro Suda
(cherry picked from commit 3dce486d88)
2016-04-21 21:27:10 +00:00
Li Lu 054fa104c5 YARN-4968. A couple of AM retry unit tests need to wait SchedulerApplicationAttempt stopped. (Wangda Tan via gtcarrera9)
(cherry picked from commit 7c6339f66a)
2016-04-21 13:27:47 -07:00
Chris Douglas 878924189d YARN-4976. Missing NullPointer check in ContainerLaunchContextPBImpl. Contributed by Giovanni Matteo Fumarola
(cherry picked from commit 95a5046607)
2016-04-21 10:46:35 -07:00
Karthik Kambatla 75cf238354 YARN-4784. Fairscheduler: defaultQueueSchedulingPolicy should not accept FIFO. (Yufei Gu via kasha)
(cherry picked from commit 170c4fd4cd)
2016-04-20 23:58:30 -07:00
Karthik Kambatla cd74cf9525 YARN-4935. TestYarnClient#testSubmitIncorrectQueue fails when FairScheduler is the default. (Yufei Gu via kasha)
(cherry picked from commit 6e297836d6)
2016-04-20 23:41:03 -07:00
Wangda Tan 83a5cdc400 YARN-4890. Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate. (Sunil G via wangda)
(cherry picked from commit 33fd95a99c)
2016-04-20 17:38:22 -07:00
Vinod Kumar Vavilapalli 342bb438f2 Fixed TimelineClient to retry SocketTimeoutException too. Contributed by Xuan Gong.
(cherry picked from commit 477003730e)
2016-04-18 11:49:00 -07:00
Wangda Tan 41cafeb5a1 YARN-4934. Reserved Resource for QueueMetrics needs to be handled correctly in few cases. (Sunil G via wangda)
(cherry picked from commit fdc46bfb37)
2016-04-16 22:50:00 -07:00
Xuan 94e4d349b0 YARN-4965. Distributed shell AM failed due to ClientHandlerException thrown by jersey. Contributed by Junping Du
(cherry picked from commit e6c0742012)
2016-04-16 19:40:11 -07:00
Arun Suresh 8460a323ce YARN-4468. Document the general ReservationSystem functionality, and the REST API. (subru and carlo via asuresh) 2016-04-15 17:20:05 -07:00
Jason Lowe cd148cb347 YARN-4940. yarn node -list -all failed if RM start with decommissioned node. Contributed by sandflee
(cherry picked from commit 69f3d428d5)
2016-04-15 20:38:04 +00:00
Naganarasimha 2213a40f30 YARN-4909. Fix intermittent failures of TestRMWebServices And TestRMWithCSRFFilter. Contributed by Bibin A Chundatt
(cherry picked from commit fdbafbc9e5)
2016-04-15 23:40:07 +05:30
Jason Lowe ece01478c5 YARN-4924. NM recovery race can lead to container not cleaned up. Contributed by sandflee
(cherry picked from commit 3150ae8108)
2016-04-14 19:19:46 +00:00
Robert Kanter e79a47670b YARN-4541. Change log message in LocalizedResource#handle() to DEBUG (rchiang via rkanter)
(cherry picked from commit 0d9194df00)
2016-04-13 17:45:36 -07:00
Xuan 5bc64dafc3 YARN-4886. Add HDFS caller context for EntityGroupFSTimelineStore. Contributed by Li Lu
(cherry picked from commit e0cb426758)
2016-04-13 10:39:31 -07:00
Naganarasimha 53c24e00e8 YARN-4810. NM applicationpage cause internal error 500. Contributed by Bibin A Chundatt.
(cherry picked from commit 437e9d6475)
2016-04-12 18:25:11 +05:30
Rohith Sharma K S d93cfefea5 YARN-4897. dataTables_wrapper change min height. Contributed by Bibin A Chundatt.
(cherry picked from commit 4ee7f0a3e7)
2016-04-12 10:28:18 +05:30
Rohith Sharma K S e9d1d8afd7 YARN-4794. Deadlock in NMClientImpl. Contributed by Jian He.
(cherry picked from commit ff722bbbdf)
2016-04-12 10:03:29 +05:30
Vinod Kumar Vavilapalli f1dcd40294 YARN-4168. Fixed a failing test TestLogAggregationService.testLocalFileDeletionOnDiskFull. Contributed by Takashi Ohnishi.
(cherry picked from commit 44bbc50d91)
2016-04-11 12:12:30 -07:00
Jason Lowe f1a370ce8b Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla"
This reverts commit 814ceeb489.
2016-04-11 15:56:29 +00:00
Junping Du a3f8491410 YARN-4928. Some yarn.server.timeline.* tests fail on Windows attempting to use a test root path containing a colon. Contributed by Gergely Novák.
(cherry picked from commit 08ddb3ac6d)
2016-04-11 08:51:22 -07:00
Akira Ajisaka 8cf6630fc6 YARN-4630. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
(cherry picked from commit 1ff27f9d12)
2016-04-11 14:55:37 +09:00
Varun Vasudev 4baee3eb67 YARN-4562. YARN WebApp ignores the configuration passed to it for keystore settings. Contributed by Sergey Shelukhin.
(cherry picked from commit 07920acc16)
2016-04-11 09:40:28 +05:30
Akira Ajisaka 88556294e2 YARN-4938. MiniYarnCluster should not request transitionToActive to RM on non-HA environment. Contributed by Eric Badger.
(cherry picked from commit 1b78b2ba17)
2016-04-11 01:32:49 +09:00
Karthik Kambatla 94a88ae87b YARN-4927. TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler. (Bibin A Chundatt via kasha)
(cherry picked from commit ff95fd547b)
2016-04-09 10:31:29 -07:00
Wangda Tan dac2021f87 YARN-4826. Document configuration of ReservationSystem for CapacityScheduler. (Subru Krishnan via wangda)
(cherry picked from commit fcb3fcd4c6)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
2016-04-08 16:28:14 -07:00
Wangda Tan 12ccdd6540 YARN-3215. Respect labels in CapacityScheduler when computing headroom. (Naganarasimha G R via wangda)
(cherry picked from commit ec06957941)
2016-04-08 15:34:24 -07:00
Jian He 77a75de319 YARN-4740. AM may not receive the container complete msg when it restarts. Contributed by Jun Gong 2016-04-08 11:21:07 -07:00
Karthik Kambatla 2b97a50eec YARN-4756. Unnecessary wait in Node Status Updater during reboot. (Eric Badger via kasha)
(cherry picked from commit e82f961a39)
2016-04-07 17:30:54 -07:00
Jian He 42bc565630 YARN-4769. Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI. Contributed by Varun Vasudev 2016-04-06 16:14:13 -07:00
Varun Vasudev 8f9b97ccce YARN-4906. Capture container start/finish time in container metrics. Contributed by Jian He.
(cherry picked from commit b41e65e5bc)
2016-04-06 13:42:06 +05:30
Wangda Tan 11e796b5cd YARN-4699. Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node. (Sunil G via wangda)
(cherry picked from commit 21eb428448)
2016-04-05 16:25:55 -07:00
Masatake Iwasaki e98bb0279b YARN-4917. Fix typos in documentation of Capacity Scheduler. (Takashi Ohnishi via iwasakims)
(cherry picked from commit 500e5a5952)
2016-04-06 04:01:34 +09:00
Masatake Iwasaki 36a3fe0334 YARN-4915. Fix typo in YARN Secure Containers documentation (Takashi Ohnishi via iwasakims)
(cherry picked from commit 30206346cf)
2016-04-06 03:47:42 +09:00
Junping Du 0907ce8c93 YARN-4916. TestNMProxy.tesNMProxyRPCRetry fails. Contributed by Tibor Kiss.
(cherry picked from commit 0005816743)
2016-04-05 09:02:50 -07:00
Junping Du eeff2e35f8 YARN-4893. Fix some intermittent test failures in TestRMAdminService. Contributed by Brahma Reddy Battula.
(cherry picked from commit 6be28bcc46)
2016-04-05 07:05:06 -07:00
Jason Lowe 814ceeb489 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
(cherry picked from commit 1cbcd4a491)
2016-04-05 13:41:18 +00:00
Rohith Sharma K S 13a4e25f26 YARN-4609. RM Nodes list page takes too much time to load. Contributed by Bibin A Chundatt
(cherry picked from commit 776b549e2a)
2016-04-05 14:53:24 +05:30
Rohith Sharma K S eec23580b4 YARN-4880. Running TestZKRMStateStorePerf with real zookeeper cluster throws NPE. Contributed by Sunil G
(cherry picked from commit 552237d4a3)
2016-04-05 14:37:31 +05:30
Li Lu 236d1a5093 YARN-4706. UI Hosting Configuration in TimelineServer doc is broken. (Akira AJISAKA via gtcarrera9)
(cherry picked from commit f61de41736)
2016-04-04 14:41:40 -07:00
naganarasimha 3772602848 YARN-4746. yarn web services should convert parse failures of appId, appAttemptId and containerId to 400. Contributed by Bibin A Chundatt
(cherry picked from commit 5092c94195)
2016-04-04 18:08:18 +05:30
Rohith Sharma K S c8271cd117 YARN-4607. Pagination support for AppAttempt page TotalOutstandingResource Requests table. Contributed by Bibin A Chundatt
(cherry picked from commit 1e6f92977d)
2016-04-04 08:13:03 +05:30
Allen Wittenauer 92a3dbe44f YARN-4850. test-fair-scheduler.xml isn't valid xml (Yufei Gu via aw)
(cherry picked from commit b1394d6307)
2016-04-01 16:57:31 -07:00
Karthik Kambatla ecfd25dc65 YARN-4657. Javadoc comment is broken for Resources.multiplyByAndAddTo(). (Daniel Templeton via kasha)
(cherry picked from commit 81d04cae41)
2016-04-01 16:20:18 -07:00
Arun Suresh f56266fccc Missing file for YARN-4895.
(cherry picked from commit 5686caa9fc)
2016-04-01 15:58:38 -07:00
Arun Suresh 24ce9ea456 YARN-4895. Add subtractFrom method to ResourceUtilization class. Contributed by Konstantinos Karanasos.
(cherry picked from commit 82621e38a0)
2016-04-01 15:00:02 -07:00
Varun Vasudev 10d8f8a39c Revert "YARN-4857. Add missing default configuration regarding preemption of CapacityScheduler. Contributed by Kai Sasaki."
This reverts commit 0064cba169.

(cherry picked from commit 3488c4f2c9)
2016-04-01 12:21:49 +05:30