16810 Commits

Author SHA1 Message Date
Akira Ajisaka
2fa1d39952
YARN-6916. Moving logging APIs over to slf4j in hadoop-yarn-server-common. Contributed by Bibin A Chundatt and Akira Ajisaka.
(cherry picked from commit 4a877737182808fe3392a116762ba59973a477a6)
2017-10-04 06:08:00 +09:00
Jian He
d2c3b93ee5 YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. Contributed by Wangda Tan
(cherry picked from commit 4111e6c7819d66e17e0932302c589777a355dbae)
2017-10-03 12:23:03 -07:00
Akira Ajisaka
47aa044fcc
MAPREDUCE-6971. Moving logging APIs over to slf4j in hadoop-mapreduce-client-app. Contributed by Jinjiang Ling.
(cherry picked from commit 453d48bdfbb67ed3e66c33c4aef239c3d7bdd3bc)
2017-10-03 12:15:43 +09:00
Sidharta S
dcc454b4a0 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe
(cherry picked from commit 7eb8499996869cdb63743f1c9eca2ba91d57ad08)
2017-10-02 19:32:19 -07:00
Wangda Tan
c2f751cb0f YARN-7259. Add size-based rolling policy to LogAggregationIndexedFileController. (xgong via wangda)
Change-Id: Ifaf82c0aee6b73b9b6ebf103aa72e131e3942f31
(cherry picked from commit 280080fad01304c85a9ede4d4f7b707eb36c0155)
2017-10-02 16:37:27 -07:00
Junping Du
aef3e4b6af YARN-7194. Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat. Contributed by Xuan Gong.
(cherry picked from commit c92c1d521eadfd8a4cd8205cc6aee74816f353f4)
2017-10-02 16:35:27 -07:00
Lei Xu
eb76d3dbec HDFS-12569. Unset EC policy logs empty payload in edit log. (Lei (Eddy) Xu)
(cherry picked from commit 0eab2fcac615228447e7f8bdf4b3ee742ea796ae)
2017-10-02 15:33:35 -07:00
Xiao Chen
3dd3f924fc HADOOP-14095. Document caveats about the default JavaKeyStoreProvider in KMS.
(cherry picked from commit d7ecac379a02876919d3e6081d42f0937f54e664)
2017-09-29 19:34:42 -07:00
Wangda Tan
d58d75ca50 YARN-6509. Add a size threshold beyond which yarn logs will require a force option. (Xuan Gong via wangda)
Change-Id: I755fe903337d4ff9ec35dae5b9cce638794e1d0f
(cherry picked from commit ec2ae3060a807c8754826af2135a68c08b2e4f13)
2017-09-29 16:49:59 -07:00
Junping Du
c3eade44d0 YARN-7072. Add a new log aggregation file format controller. Contributed by Xuan Gong.
(cherry picked from commit 3fddabc2fe4fbdb8ef3f9ce7558955c4f0794dcc)
2017-09-29 16:49:44 -07:00
Wangda Tan
cdaab892d8 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda)
Change-Id: I0ee0b1bb459437432a22cf68861a6354f0decabb
(cherry picked from commit febeead5f95c6fc245ea3735f5b538d4bb4dc8a4)
2017-09-29 16:46:34 -07:00
Andrew Wang
575763fb7f HADOOP-14909. Fix the word of erasure encoding in the top page. Contributed by Takanobu Asanuma.
(cherry picked from commit 373d0a51955cabff77e934a28ba2de308207374a)
2017-09-29 13:34:39 -07:00
Lei Xu
337506190b HDFS-12453. TestDataNodeHotSwapVolumes fails in trunk Jenkins runs. (Lei (Eddy) Xu) 2017-09-29 10:47:37 -07:00
Eric Payne
7a9479597f YARN-7084. TestSchedulingMonitor#testRMStarts fails sporadically. Contributed by Jason Lowe.
(cherry picked from commit 7f6118f918e0fbe81cb64e8b9f6959b8d66679f6)
2017-09-29 10:39:01 -05:00
Wangda Tan
091fc32ce8 YARN-6623. Add support to turn off launching privileged containers in the container-executor. (Varun Vasudev via wangda)
Change-Id: I76aec68cbfb42d239279d7245a03290d691e87a4
(cherry picked from commit d3b1c6319546706c41a2011ead6c3fe208883200)
2017-09-28 16:46:23 -07:00
Chris Trezzo
3d23522110 YARN-7250. Update Shared cache client api to use URLs.
(cherry picked from commit c114da5e64d14b1d9e614081c4171ea0391cb1aa)
2017-09-28 15:31:23 -07:00
Jason Lowe
a6630f703c HADOOP-14902. LoadGenerator#genFile write close timing is incorrectly calculated. Contributed by Hanisha Koneru
(cherry picked from commit 6f789fe05766a61b12ca10df3f26ee354eac84aa)
2017-09-28 16:39:47 -05:00
Jason Lowe
a1e9739730 YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh
(cherry picked from commit 85d81ae58ec4361a944c84753a900460a0888b9b)
2017-09-28 14:14:14 -05:00
Andrew Wang
5a4f37019a Preparing for 3.0.0 GA development 2017-09-28 11:32:01 -07:00
Arun Suresh
59453dad8c YARN-7240. Add more states and transitions to stabilize the NM Container state machine. (Kartheek Muthyala via asuresh)
(cherry picked from commit df800f6cf3ea663daf4081ebe784808b08d9366d)
2017-09-28 11:29:26 -07:00
Arun Suresh
ed855e2eef YARN-6059. Update paused container state in the NM state store. (Hitesh Sharma via asuresh)
(cherry picked from commit 66ca0a65408521d5f9b080dd16b353b49fb8eaea)
2017-09-28 11:27:43 -07:00
Arun Suresh
8eebee5094 YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh)
(cherry picked from commit 4f8194430fc6a69d9cc99b78828fd7045d5683e8)
2017-09-28 11:24:20 -07:00
Arun Suresh
dd51a74601 YARN-5292. NM Container lifecycle and state transitions to support for PAUSED container state. (Hitesh Sharma via asuresh)
(cherry picked from commit 864fbacd4548004b1de8b0812627976acd22aff5)
2017-09-28 11:22:58 -07:00
Aaron Fabbri
2223393ad1 HADOOP-14851 LambdaTestUtils.eventually() doesn't spin on Assertion failures. Contributed by Steve Loughran
(cherry picked from commit 180e814b081d3707c95641171d649b547db41a04)
2017-09-28 10:25:17 -07:00
Manoj Govindassamy
46031d84fb HDFS-12404. Rename hdfs config authorization.provider.bypass.users to attributes.provider.bypass.users.
(cherry picked from commit 3b3be355b35d08a78d9dcd647650812a2d28207b)
2017-09-28 10:22:25 -07:00
Lei Xu
d9dd856739 HDFS-12409. Add metrics of execution time of different stages in EC recovery task. (Lei (Eddy) Xu)
(cherry picked from commit 73aed34dffa5e79f6f819137b69054c1dee2d4dd)
2017-09-28 10:21:54 -07:00
Lei Xu
58eacdbb42 HDFS-12412. Change ErasureCodingWorker.stripedReadPool to cached thread pool. (Lei (Eddy) Xu)
(cherry picked from commit 123342cd0759ff88801d4f5ab10987f6e3f344b0)
2017-09-28 10:21:24 -07:00
Arpit Agarwal
7f371da52a HDFS-12470. DiskBalancer: Some tests create plan files under system directory. Contributed by Hanisha Koneru.
(cherry picked from commit a2dcba18531c6fa4b76325f5132773f12ddfc6d5)
2017-09-28 10:19:49 -07:00
Arpit Agarwal
14a05ee4c1 HDFS-12496. Make QuorumJournalManager timeout properties configurable. Contributed by Ajay Kumar.
(cherry picked from commit b9e423fa8d30ea89244f6ec018a8064cc87d94a9)
2017-09-28 10:19:11 -07:00
Arpit Agarwal
f8df655f31 HDFS-12530. Processor argument in Offline Image Viewer should be case insensitive. Contributed by Hanisha Koneru.
(cherry picked from commit 08fca508e66e8eddc5d8fd1608ec0c9cd54fc990)
2017-09-28 10:18:22 -07:00
Daniel Templeton
7b49de5e30 YARN-6622. Document Docker work as experimental (Contributed by Varun Vasudev)
(cherry picked from commit 6651cbcc72d92caf86b744fa76fba513b36b12c7)
2017-09-28 10:16:45 -07:00
Wei-Chiu Chuang
2ab2a9438f HDFS-12458. TestReencryptionWithKMS fails regularly. Contributed by Xiao Chen.
(cherry picked from commit 7c34ceaf4fa28e2ecabd6626860bb1c7418e4b8d)
2017-09-28 05:11:49 -07:00
Xuan
81e7872793 YARN-7257. AggregatedLogsBlock reports a bad 'end' value as a bad
'start' value. Contributed by Jason Lowe

(cherry picked from commit 28c4957fccebe2d7e63ec9fe9af58313b4f21d4f)
2017-09-27 15:47:45 -07:00
Chris Trezzo
ad7ed9d044 YARN-7253. Shared Cache Manager daemon command listed as admin subcmd in yarn script.
(cherry picked from commit c87db8d154ab2501e786b4f1669b205759ece5c3)
2017-09-27 11:36:42 -07:00
Akira Ajisaka
e5629ee713
HADOOP-14822. hadoop-project/pom.xml is executable. Contributed by Ajay Kumar.
(cherry picked from commit dd378775df5bd79e23c12120535404c4ca0fb386)
2017-09-27 14:26:46 +09:00
Yiqun Lin
91e866ee6e HDFS-12495. TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks fails intermittently. Contributed by Eric Badger.
(cherry picked from commit e9b790db9de7c3fbe451bcdc3fd928a27a0d8b4e)
2017-09-27 10:53:33 +08:00
Daryn Sharp
61dc4bcd61 [HDFS-12386] Add fsserver defaults call to WebhdfsFileSystem. (Rushabh Shah via daryn) 2017-09-26 15:27:32 -05:00
Jason Lowe
bf03363d83 MAPREDUCE-6968. Staging directory erasure coding config property has a typo. Contributed by Jason Lowe
(cherry picked from commit 9df05005ac258441eeceec25ecef7b28f9f50a02)
2017-09-26 14:53:47 -05:00
Jason Lowe
8526bfa19d Revert "YARN-6570. No logs were found for running application, running"
This reverts commit e91447ccc9ff438dd300e2564c251b2c4b0f369a.
2017-09-26 10:41:44 -05:00
Rohith Sharma K S
52571d9fa3 YARN-65. Reduce RM app memory footprint once app has completed. Contributed by Manikandan R.
(cherry picked from commit 06e5a7b5cf141420d3a411088b87acba72e68cad)
2017-09-26 14:46:47 +05:30
Akira Ajisaka
739d3ef809
HADOOP-14893. WritableRpcEngine should use Time.monotonicNow. Contributed by Chetna Chaudhari.
(cherry picked from commit d08b8c801a908b4242e7b21a54f3b1e4072f1eae)
2017-09-26 16:46:46 +09:00
Aaron Fabbri
f3dbcd6a89
HADOOP-14890 Move up to AWS SDK 1.11.199. Contributed by Steve Loughran. 2017-09-25 17:19:46 -07:00
Aaron Fabbri
60b966cb45
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran 2017-09-25 16:01:15 -07:00
Ray Chiang
da6ed36edb HADOOP-14903. Add json-smart explicitly to pom.xml. (rchiang)
(cherry picked from commit cde804b91f2fed514935d186d89ea7cbbf2701f3)
2017-09-25 15:32:15 -07:00
Jason Lowe
bfad0ca3b0 HADOOP-14881. LoadGenerator should use Time.monotonicNow() to measure durations. Contributed by Bharat Viswanadham
(cherry picked from commit ac05a51bbb2a3fad4e85f9334a3408571967900a)
2017-09-25 15:38:24 -05:00
Akira Ajisaka
149d3ad6ba
YARN-7153. Remove duplicated code in AMRMClientAsyncImpl.java. Contributed by Sen Zhao.
(cherry picked from commit e928ee583c5a1367e24eab34057f8d8496891452)
2017-09-26 02:04:23 +09:00
Akira Ajisaka
366ce11ac0
HADOOP-14892. MetricsSystemImpl should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari.
(cherry picked from commit 3a10367a17b99e2c3035ec54048f94f334d93227)
2017-09-26 01:47:59 +09:00
Akira Ajisaka
51b3a38064
HDFS-12304. Remove unused parameter from FsDatasetImpl#addVolume. Contributed by Chen Liang.
(cherry picked from commit 02e2a9b1152b0e144fcf43bec2fce26d8a6c6dbc)
2017-09-26 01:26:13 +09:00
Daniel Templeton
dca2fb5ac5 HADOOP-14897. Loosen compatibility guidelines for native dependencies
(cherry picked from commit 0807470808156af13e0992285cac070052d86c26)
2017-09-25 16:15:56 +02:00
Daniel Templeton
972250923f HADOOP-13714. Tighten up our compatibility guidelines for Hadoop 3
(cherry picked from commit 7618fa9194b40454405f11a25bec4e2d79506912)
2017-09-23 00:05:25 +02:00