Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,062 Commits 326 Branches 371 Tags
Commit Graph

19062 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Siyao Meng
86fb5e1c8e HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7cb46f035a92056783bad23a9abc6a264d71285d)
(cherry picked from commit 0550aa32f1a4b1809c234f8ed785db08b2e33d13)
 2019-05-01 16:49:36 -07:00
Arpit Agarwal
7b51edbab9
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 
(cherry picked from commit 4b4200f1f87ad40d9c19ba160f706ffd0470a8d4)
 2019-04-29 14:54:11 -07:00
Weiwei Yang
94a895b94f YARN-9307. node_partitions constraint does not work. Contributed by kyungwan nam. 2019-04-26 13:16:43 +08:00
Weiwei Yang
d242b166ed YARN-9325. TestQueueManagementDynamicEditPolicy fails intermittent. Contributed by Prabhu Joseph. 
(cherry picked from commit 1c8046d67ec10710e7749ed1929b09fac4b1ba94)
 2019-04-23 14:25:33 +08:00
Erik Krogen
9da70a1b4b HADOOP-16265. Fix bug causing Configuration#getTimeDuration to use incorrect units when the default value is used. Contributed by starphin. 
(cherry-picked from 1ddb48872f6a4985f4d0baadbb183899226cff68)
(cherry-picked from 28ff96f367ab2260666c2177980bce7c66e6dc57)
 2019-04-22 08:19:25 -07:00
Eric Yang
8b228a42e9 YARN-8587. Added retries for fetching docker exit code. 
Contributed by Charo Zhang

(cherry picked from commit c16c49b8c3b8e2e42c00e79a50e7ae029ebe98e2)
 2019-04-19 15:40:56 -04:00
Eric Yang
68a98be8a2 YARN-6695. Fixed NPE in publishing appFinished events to ATSv2. 
Contributed by Prabhu Joseph

(cherry picked from commit df76cdc8959c51b71704ab5c38335f745a6f35d8)
 2019-04-18 12:31:34 -04:00
Puleya7
546ed4937c HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d847986aa14a51cc2a3cc83151f22c43781)
 2019-04-17 21:57:06 +09:00
Akira Ajisaka
b45813b259
HADOOP-16227. Upgrade checkstyle to 8.19 
(cherry picked from commit 4b4fef2f0e0ed1e185ea1058db7a65d68d4970b9)
 2019-04-15 10:58:41 +09:00
Masatake Iwasaki
fd90af892c HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 
(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)
 2019-04-12 12:03:11 +09:00
Weiwei Yang
c37065eae9 YARN-9463. Add queueName info when failing with queue capacity sanity check. Contributed by Aihua Xu. 
(cherry picked from commit 8c1bba375b144fd515b389174ddb349f2d9246fa)
 2019-04-10 23:04:27 +08:00
Weiwei Yang
bd0c9bc160 YARN-9413. Queue resource leak after app fail for CapacityScheduler. Contributed by Tao Yang. 
(cherry picked from commit ec143cbf678bd65f87fdd464c23022a2d2c54c07)
 2019-04-06 20:38:06 +08:00
Akira Ajisaka
a70076b3a5
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f09502a4f07342dee457e47bb52b03441ae)
 2019-04-05 12:31:14 +09:00
David Mollitor
41f1bc2056
HADOOP-16208. Do Not Log InterruptedException in Client. 
Contributed by David Mollitor.

(cherry picked from commit c90736350ba158c7872a39426e7a29c5e5e0bb48)
 2019-04-04 21:13:09 +01:00
Stephen O'Donnell
d9899015eb HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
(cherry picked from commit 388f445dde577999b2d81f809adcfca8f0958499)
 2019-04-04 04:06:33 -07:00
Steve Loughran
3210ae372d
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
 2019-04-03 21:42:08 +01:00
Wei-Chiu Chuang
084fb9de22 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070a124234c77f16193ee925d32ca9a20)
(cherry picked from commit c8703dda0727e17d759d7ad27f0caee88103a530)
 2019-04-03 12:41:07 -07:00
Akira Ajisaka
875435dc7d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)
 2019-04-03 19:40:23 +09:00
Akira Ajisaka
10d6e68324
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d2153884bf78a159534ea711c0b70acc1c05)
 2019-04-02 10:55:07 +09:00
Adam Antal
52536ff887 MAPREDUCE-7190. Add SleepJob additional parameter to make parallel runs distinguishable. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 856cbf62d3fd1c885ae8374f08d7c18c83941791)
(cherry picked from commit ddf88596550be13889ff7002aadd0ebf797996a1)
 2019-04-01 10:39:18 -07:00
Gabor Bota
f01d592f3b
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8ecb83b666d4ed223fc270e1a46642c1)
 2019-04-01 13:52:44 +09:00
Takanobu Asanuma
1fc74178eb HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb48ca26a629d4d5f3f07e2858d09594)
 2019-03-27 03:29:42 +09:00
Eric Yang
dbc02bcda7 YARN-9391. Fixed node manager environment leaks into Docker containers. 
Contributed by Jim Brennan

(cherry picked from commit 3c45762a0bfb403e069a03e30d35dd11432ee8b0)
 2019-03-25 15:55:46 -04:00
Weiwei Yang
f34d6b937f HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 
(cherry picked from commit 568d3ab8b65d1348dec9c971feffe200e6cba2ef)
 2019-03-19 10:11:17 +08:00
Erik Krogen
d4fbbc83ad HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6bef369fef6a8364f0c0764eba90e44a)
(cherry-picked from 0de8b55a095ada2b98c0a41899651bd8e524f42e)
 2019-03-18 08:53:13 -07:00
Inigo Goiri
80732146c6 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631cb8a0f67bbc39b5b5a1b0a81ca3b3c)
 2019-03-15 13:58:39 -07:00
Erik Krogen
cb76fc5618 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae1da09ced735da36bf7d80a40c3fa1b)
(cherry picked from commit fec7c5f3ebbaea7b290e0904570c3a485b541a22)
 2019-03-13 13:20:51 -07:00
Shweta Yakkali
a324db2cce HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020a538115b3e08f41ff88167033d2775)
(cherry picked from commit 1ceefa726e1b531ec92c2ee2212b25c327644ef6)
 2019-03-12 16:06:16 -07:00
Stephen O'Donnell
327e57ae6e HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b38dccab25058dff1b8743d8a3f82734)
(cherry picked from commit a21e2e4dbc1681175130242138ced0d36b6fafae)
 2019-03-12 10:19:37 -07:00
Steve Loughran
1bace86501
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 16:53:06 +00:00
Weiwei Yang
fdd56b5b05 MAPREDUCE-7191. JobHistoryServer should log exception when loading/parsing history file failed. Contributed by Jiandan Yang. 
(cherry picked from commit f0605146b34ddbd19fc9e52d4761f9a347d5173e)
 2019-03-11 16:40:01 +08:00
Erik Krogen
b0b8000240 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3f74968c92751f0972746b012e72810)
(cherry-picked from commit 6d076dd5e8be8fee5062a04d4cd2aa0c5da0ea56)
 2019-03-07 08:43:39 -08:00
Sunil G
6941033396 YARN-8803. [UI2] Show flow runs in the order of recently created time in graph widgets. Contributed by Akhil PB. 
(cherry picked from commit c79f139519e9b2486de31b307d7811b4c2a6b5b0)
 2019-03-06 16:50:19 +05:30
Sunil G
379a9bfd9a YARN-9138. Improve test coverage for nvidia-smi binary execution of GpuDiscoverer. Contributed by Szilard Nemeth. 
(cherry picked from commit 46045c5cb3ab06a35df27879afbd1bc3c2a384dd)
 2019-03-06 16:02:39 +05:30
Wei-Chiu Chuang
d951497f57 HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587aa66ac99ec5f5b50827ec3e0a327613)
(cherry picked from commit e58ccca3ce131c955ceb115cd0b75e452eea828b)
 2019-03-04 10:46:00 -08:00
bibinchundatt
e663a6af89 Revert "YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph" 
This reverts commit 7db50ffcebf5413bbd1a80ee2cd8e2ffc11befe1.
 2019-03-04 17:03:45 +05:30
Erik Krogen
b7eb9116eb HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-03-01 08:12:54 -08:00
Sunil G
80d507d1a4 YARN-9139. Simplify initializer code of GpuDiscoverer. Contributed by Szilard Nemeth. 2019-03-01 19:28:33 +05:30
Eric Yang
72eacb3e10 YARN-9334. Allow YARN Service client to send SPNEGO challenge header when authentication type is not simple. 
Contributed by Billie Rinaldi

(cherry picked from commit 04b228e43b728d574d7ad97330aa4218cb7f8bf8)
 2019-02-28 10:12:09 -08:00
Sunil G
817028364a YARN-9121. Replace GpuDiscoverer.getInstance() to a readable object for easy access control. Contributed by Szilard Nemeth. 2019-02-27 17:46:43 +05:30
Weiwei Yang
10d4a9a7fb YARN-9248. RMContainerImpl:Invalid event: ACQUIRED at KILLED. Contributed by lujie. 
(cherry picked from commit 8c30114b0055bdb44608d4a6d1fa838a04821ff6)
 2019-02-27 17:39:37 +08:00
Sunil G
51b010b19f YARN-9087. Improve logging for initialization of Resource plugins. Contributed by Szilard Nemeth. 2019-02-27 11:57:32 +05:30
Sunil G
84928ba3d1 YARN-9168. DistributedShell client timeout should be -1 by default. Contributed by Zhankun Tang. 
(cherry picked from commit 6cec90653dc2ae60a10262745997ea91dd872a8b)
 2019-02-25 15:29:53 +05:30
Sunil G
cb0f45b75b YARN-9213. RM Web UI v1 does not show custom resource allocations for containers page. Contributed by Szilard Nemeth. 
(cherry picked from commit f282f9c362dc5a2b7a86b7e2f78d53835c2a6188)
 2019-02-25 11:39:05 +05:30
Weiwei Yang
dab22e74a4 YARN-9316. TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently. Contributed by Prabhu Joseph. 
(cherry picked from commit 9cd5c5447f008b38be43b21ca596d8a36ad0cd55)
 2019-02-24 22:57:40 +08:00
bibinchundatt
6d1cf7b395 YARN-9317. Avoid repeated YarnConfiguration#timelineServiceV2Enabled check. Contributed by Prabhu Joseph 2019-02-23 08:03:28 +05:30
bibinchundatt
7db50ffceb YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph 2019-02-22 20:52:40 +05:30
Sunil G
d6377c8b68 YARN-9118. Handle exceptions with parsing user defined GPU devices in GpuDiscoverer. Contributed by Szilard Nemeth. 
(cherry picked from commit 95fbbfed75dd309b5d56032ece64996165572287)
 2019-02-22 20:23:51 +05:30
Weiwei Yang
040d475030 YARN-9238. Avoid allocating opportunistic containers to previous/removed/non-exist application attempt. Contributed by lujie. 
(cherry picked from commit 9c88695bcda0ffe4c7f49d643c649dfa1dce9bde)
 2019-02-22 21:43:53 +08:00
Steve Loughran
a1d383fc14
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS. 
Contributed by Steve Loughran.

(cherry picked from commit 9cb2f470b759bbe7609a00e8f8f72779e2daae80)
 2019-02-21 13:58:34 +00:00