Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,589 Commits 326 Branches 371 Tags
Commit Graph

20589 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric Yang
4a64dab0dd YARN-8622. Fixed container-executor compilation on MacOSX. 
Contributed by Siyao Meng

(cherry picked from commit ef97a20831677c055aa6bff6ad0649cbb3a56a86)
 2019-04-18 19:01:11 -04:00
Eric Yang
2503409977 YARN-6695. Fixed NPE in publishing appFinished events to ATSv2. 
Contributed by Prabhu Joseph

(cherry picked from commit df76cdc8959c51b71704ab5c38335f745a6f35d8)
 2019-04-18 12:30:55 -04:00
Siyao Meng
742a3ad24b YARN-9487. NodeManager native build shouldn't link against librt on macOS. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6e4399ea61eb4301f4b6c59633d77c0a04efb12b)
 2019-04-17 22:57:33 -07:00
Puleya7
d724f74f6b HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d847986aa14a51cc2a3cc83151f22c43781)
 2019-04-17 21:55:45 +09:00
Akira Ajisaka
df5d8f05d9
HADOOP-16227. Upgrade checkstyle to 8.19 
(cherry picked from commit 4b4fef2f0e0ed1e185ea1058db7a65d68d4970b9)
 2019-04-15 10:47:02 +09:00
Masatake Iwasaki
03079be707 HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 
(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)
 2019-04-12 11:59:14 +09:00
Weiwei Yang
db185de31c YARN-9463. Add queueName info when failing with queue capacity sanity check. Contributed by Aihua Xu. 
(cherry picked from commit 8c1bba375b144fd515b389174ddb349f2d9246fa)
 2019-04-10 23:02:24 +08:00
Akira Ajisaka
b28fdb98ee
HADOOP-14635. Javadoc correction for AccessControlList#buildACL. Contributed by Yeliang Cang. 
(cherry picked from commit ab645b3caa0313f08a26bdd70fa14434d09e6cb5)
 2019-04-08 15:19:17 +09:00
Weiwei Yang
7a80b1b481 YARN-9413. Queue resource leak after app fail for CapacityScheduler. Contributed by Tao Yang. 
(cherry picked from commit ec143cbf678bd65f87fdd464c23022a2d2c54c07)
 2019-04-06 20:19:03 +08:00
Akira Ajisaka
f020ac2c06
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f09502a4f07342dee457e47bb52b03441ae)
 2019-04-05 12:30:46 +09:00
David Mollitor
50fc9577c0
HADOOP-16208. Do Not Log InterruptedException in Client. 
Contributed by David Mollitor.

(cherry picked from commit c90736350ba158c7872a39426e7a29c5e5e0bb48)
 2019-04-04 16:16:57 +01:00
Stephen O'Donnell
388f445dde HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
 2019-04-04 04:06:15 -07:00
Weiwei Yang
06cf8ce30f YARN-9394. Use new API of RackResolver to get better performance. Contributed by Lantao Jin. 
(cherry picked from commit 945e8c60640ceb938ad8d27767d44eec53a15038)
 2019-04-04 18:41:04 +08:00
Steve Loughran
b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d9990ef9059b6ac9a19ad24310d6f36d04)
 2019-04-03 21:35:19 +01:00
Wei-Chiu Chuang
c8703dda07 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070a124234c77f16193ee925d32ca9a20)
 2019-04-03 11:01:44 -07:00
Akira Ajisaka
80a8d3310e
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)
 2019-04-03 19:36:17 +09:00
Akira Ajisaka
b6039e241d
HADOOP-16226. new Path(String str) does not remove all the trailing slashes of str 
(cherry picked from commit aaaf856f4b7b53d424eb1eab010311de0d5fbe1e)
 2019-04-03 13:18:36 +09:00
Akira Ajisaka
8037052c2b
HADOOP-16225. Fix links to the developer mailing lists in DownstreamDev.md. Contributed by Wanqiang Ji. 
(cherry picked from commit ebd0d2153884bf78a159534ea711c0b70acc1c05)
 2019-04-02 10:54:16 +09:00
Adam Antal
ddf8859655 MAPREDUCE-7190. Add SleepJob additional parameter to make parallel runs distinguishable. Contributed by Adam Antal. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 856cbf62d3fd1c885ae8374f08d7c18c83941791)
 2019-04-01 10:38:58 -07:00
Gabor Bota
67cdf807a2
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8ecb83b666d4ed223fc270e1a46642c1)
 2019-04-01 13:52:19 +09:00
Steve Loughran
60c9042286
HADOOP-16058. S3A tests to include Terasort. 
Contributed by Steve Loughran.

This includes
 - HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
 - MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
 - MAPREDUCE-7091. Terasort on S3A to switch to new committers
 - MAPREDUCE-7092. MR examples to work better against cloud stores
 2019-03-29 15:25:45 +00:00
Siyao Meng
52cfbc39cc
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail. 
Contributed by Siyao Meng

(cherry picked from commit ce4bafdf442c004b6deb25eaa2fa7e947b8ad269)
 2019-03-26 18:43:43 +00:00
Takanobu Asanuma
162e9999c7 HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb48ca26a629d4d5f3f07e2858d09594)
 2019-03-27 03:28:54 +09:00
Eric Yang
10642a6205 YARN-9391. Fixed node manager environment leaks into Docker containers. 
Contributed by Jim Brennan

(cherry picked from commit 3c45762a0bfb403e069a03e30d35dd11432ee8b0)
 2019-03-25 15:54:52 -04:00
Andrew Olson
ade3af6ef2
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized. 
Author:    Andrew Olson
(cherry picked from commit faba3591d32f2e4808c2faeb9472348d52619c8a)
 2019-03-22 10:36:34 +00:00
David Mollitor
397b63ad0b
HADOOP-16181. HadoopExecutors shutdown Cleanup. 
Author:    David Mollitor <david.mollitor@cloudera.com>
(cherry picked from commit d18d0859ebfc46a18fd9140b42fb95f1da96380e)
 2019-03-22 10:30:21 +00:00
David Mollitor
9a449ac075
HADOOP-16196. Path Parameterize Comparable. 
Author:    David Mollitor <david.mollitor@cloudera.com>

(cherry picked from commit 246ab77f281614b2206bfb1dae70a8dec62147a7)
 2019-03-22 10:27:17 +00:00
Weiwei Yang
39f60faa60 HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 
(cherry picked from commit 568d3ab8b65d1348dec9c971feffe200e6cba2ef)
 2019-03-19 10:08:11 +08:00
Adam Antal
81a6ba1825
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants. 
Contributed by Adam Antal.

(cherry picked from commit c0427c84dddf942529dfdfc5cc7a3e25e3f12c5e)
 2019-03-18 19:14:43 +00:00
Erik Krogen
0de8b55a09 HADOOP-16192. Fix CallQueue backoff bugs: perform backoff when add() is used and update backoff when refreshed. 
(cherry-picked from 8c95cb9d6bef369fef6a8364f0c0764eba90e44a)
 2019-03-18 08:46:53 -07:00
Inigo Goiri
4eb0497091 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631cb8a0f67bbc39b5b5a1b0a81ca3b3c)
 2019-03-15 13:58:03 -07:00
Ben Roling
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files. 
Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.
 2019-03-14 19:46:34 +00:00
Erik Krogen
fec7c5f3eb HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae1da09ced735da36bf7d80a40c3fa1b)
 2019-03-13 13:19:18 -07:00
Weiwei Yang
3d86dd8931 MAPREDUCE-7192. JobHistoryServer attempts page support jump to containers log page in NM when logAggregation is disable. Contributed by Jiandan Yang. 
(cherry picked from commit 159a715eef23046ee9909289defe62e68a5df533)
 2019-03-13 17:41:31 +08:00
Shweta Yakkali
1ceefa726e HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020a538115b3e08f41ff88167033d2775)
 2019-03-12 16:05:55 -07:00
Stephen O'Donnell
a21e2e4dbc HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b38dccab25058dff1b8743d8a3f82734)
 2019-03-12 10:18:56 -07:00
Steve Loughran
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 11:15:25 +00:00
Weiwei Yang
9a2b24642d MAPREDUCE-7191. JobHistoryServer should log exception when loading/parsing history file failed. Contributed by Jiandan Yang. 
(cherry picked from commit f0605146b34ddbd19fc9e52d4761f9a347d5173e)
 2019-03-11 16:03:38 +08:00
Da Zhou
cfaf21a4ba
HADOOP-16169. ABFS: Bug fix for getPathProperties. 
Author:    Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit e0260417ade5568ae37dcc3753aea0d1e0fd701b)
 2019-03-08 13:53:44 +00:00
Erik Krogen
6d076dd5e8 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3f74968c92751f0972746b012e72810)
 2019-03-07 08:42:41 -08:00
Praveen Krishna
451844fee5
HADOOP-16114. NetUtils#canonicalizeHost gives different value for same host. 
Author:    Praveen Krishna <praveenkrishna@tutanota.com>
(cherry picked from commit 2b94e51a8f17548e5ef6af8e1e934a94031bb1dc)
 2019-03-07 11:08:48 +00:00
Sunil G
aff5973401 YARN-8803. [UI2] Show flow runs in the order of recently created time in graph widgets. Contributed by Akhil PB. 
(cherry picked from commit c79f139519e9b2486de31b307d7811b4c2a6b5b0)
 2019-03-06 16:49:49 +05:30
Sunil G
d721634fea YARN-9138. Improve test coverage for nvidia-smi binary execution of GpuDiscoverer. Contributed by Szilard Nemeth. 
(cherry picked from commit 46045c5cb3ab06a35df27879afbd1bc3c2a384dd)
 2019-03-06 16:01:56 +05:30
Stephen O'Donnell
3fe31b36fa
HADOOP-16140. hadoop fs expunge to add -immediate option to purge trash immediately. 
Contributed by Stephen O'Donnell.

(cherry picked from commit 686c0141eff0886c285b8e52fddade43c1ce4570)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 14:11:49 +00:00
Da Zhou
dc38fc598d
HADOOP-16136. ABFS: Should only transform username to short name 
Contributed by Da Zhou.

(cherry picked from commit 3988e75ca385aec31ca1fc49d6cffce1ea935825)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 10:47:58 +00:00
Da Zhou
075f6b061c
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.

(cherry picked from commit 1f1655028eede24197705a594b6ef19e6737db35)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 10:44:46 +00:00
Da Zhou
ae832ccffe
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. 
Contributed by Shweta Yakkali.

Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
(cherry picked from commit 02eb91856e7e8477c62e0f8bf1bac6de3e00a8a4)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 10:39:37 +00:00
Wei-Chiu Chuang
e58ccca3ce HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587aa66ac99ec5f5b50827ec3e0a327613)
 2019-03-04 10:45:31 -08:00
bibinchundatt
63ed16e076 Revert "YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph" 
This reverts commit cf1944eb6e18e00ac18346b92d5f6373de1c1765.
 2019-03-04 17:01:40 +05:30
Weiwei Yang
4ceb4e4f05 YARN-9332. RackResolver tool should accept multiple hosts. Contributed by Lantao Jin. 
(cherry picked from commit fe6b2b2f23e69f0643e870d9c500117088983209)
 2019-03-02 16:04:24 +00:00