Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,031 Commits 326 Branches 371 Tags
Commit Graph

1117 Commits

Author SHA1 Message Date
Szilard Nemeth
62622ab9c1 YARN-9836. General usability improvements in showSimulationTrace.html. Contributed by Adam Antal 2019-11-19 21:21:17 +01:00
Andras Bokor
89e95370a4 HADOOP-16710. Testing_azure.md documentation is misleading. 
Contributed by Andras Bokor.

Change-Id: Icf07a53145936953629c7dace2e9648b7b21588d
 2019-11-17 17:06:10 +00:00
Siyao Meng
e0cf1735e1 HADOOP-16676. Backport HADOOP-16152 to branch-3.2. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-11-12 11:38:42 -08:00
Da Zhou
fe96407451
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI 
(cherry picked from commit 9a8edb0aeddd7787b2654f6e2a8465c325e048a2)
 2019-10-16 14:27:11 -07:00
Rohith Sharma K S
7d5bb2ebb7 Preparing for 3.2.2-SNAPSHOT development. 2019-09-07 08:52:08 +05:30
bilaharith
3b3c0c4b87 HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC. 
Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
 2019-08-27 19:08:38 +00:00
Robert Levas
ce23e971b4 HADOOP-16340. ABFS driver continues to retry on IOException responses from REST operations. 
Contributed by Robert Levas.

This makes the HttpException constructor protected rather than public, so it is possible
to implement custom subclasses of this exception -exceptions which will not be retried.

Change-Id: Ie8aaa23a707233c2db35948784908b6778ff3a8f
 2019-08-27 19:08:29 +00:00
Da Zhou
a6d50a9054 HADOOP-16376. ABFS: Override access() to no-op. 
Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
 2019-08-27 19:04:16 +00:00
Da Zhou
dd636127e9 HADOOP-16269. ABFS: add listFileStatus with StartFrom. 
Author:    Da Zhou
 2019-08-27 19:01:21 +00:00
Da Zhou
006ae258b3 HADOOP-16163. NPE in setup/teardown of ITestAbfsDelegationTokens. 
Contributed by Da Zhou.

Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-08-27 19:01:21 +00:00
Akira Ajisaka
afb3f329fd
YARN-9774. Fix order of arguments for assertEquals in TestSLSUtils. Contributed by Nikhil Navadiya. 
(cherry picked from commit 84b1982060422760702eca6f1ef515c6ad3e85a5)
 2019-08-23 14:40:15 +09:00
bibinchundatt
69255fa1b9 YARN-9765. SLS runner crashes when run with metrics turned off. Contributed by Abhishek Modi. 
(cherry picked from commit 10ec31d20ee1b6a0b1da915acb6b6ec33f2cd415)
 2019-08-21 13:57:53 +05:30
KAI XIE
b3c14d4132 HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel (#919) 
* DistCp to support checksum validation when copy blocks in parallel

* address review comments

* add checksums comparison test for combine mode

(cherry picked from commit c765584eb231f8482f5b90b7e8f61f9f7a931d09)
 2019-08-18 18:48:21 -07:00
Da Zhou
330e450397
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus 
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
 2019-08-12 09:41:52 +08:00
Ayush Saxena
35ff1ce42c HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun. 2019-07-20 13:29:45 +05:30
Arun Singh
5f2d07af1b
HADOOP-16404. ABFS default blocksize change(256MB from 512MB) 
Contributed by: Arun Singh
 2019-07-19 20:34:28 -07:00
Masatake Iwasaki
b6718c754a HADOOP-16401. ABFS: port Azure doc to 3.2 branch. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-07-10 17:16:43 +09:00
Takanobu Asanuma
6dffad028e HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d20656433cdec76c2108d24ff3b935657c1e80)
 2019-06-20 20:17:45 -07:00
DadanielZ
9c8e40fbdb
HADOOP-16251. ABFS: add FSMainOperationsBaseTest. Re-commit to fix git metadata. 
Author: Da Zhou
(cherry picked from commit ff27e8eabded6b1de9860da95155b304c07b4c6e)
 2019-06-07 18:09:38 +01:00
Da Zhou
bf0bb2470f
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream. 
Contributed by Da Zhou.

(cherry picked from commit 1cef194a28086991cd39fb62092d2b2105ece57b)
 2019-06-07 18:09:38 +01:00
Vishwajeet Dusane
907a016142
HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit 1edf1914acb74e45f6717c703f519cb382aae173)
 2019-06-07 18:09:37 +01:00
Shweta Yakkali
6b115966bc
HADOOP-16157. [Clean-up] Remove NULL check before instanceof in AzureNativeFileSystemStore 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I6269ae66378e46eed440a76f847ae1af1fa95450
(cherry picked from commit bb8ad096e785f7127a5c0de15167255d9b119578)
 2019-06-07 18:09:37 +01:00
Shweta Yakkali
57c6060c3a
HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). 
Contributed by Shweta Yakkali.

(cherry picked from commit 13f0ee21f2c17ebacaa35e14ee01f39624f38a8d)

Change-Id: Ibd010d2e6adc15f53a9c5357482e57313bf84d2e
 2019-06-07 18:09:37 +01:00
Da Zhou
3593b66693
HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI 
(Contributed by Da Zhou via Daniel Templeton)

Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
(cherry picked from commit e374584479b687e41d5379bb6d827dcae620e123)
 2019-06-07 18:09:37 +01:00
Denes Gerencser
ede5cbd707
HADOOP-16174. Disable wildfly logs to the console. 
Follow-on to HADOOP-15851.

Author:    Denes Gerencser <dgerencser@cloudera.com>
(cherry picked from commit ddede7ae6fbbadbe08861bc85a664b73d66f77c7)
 2019-06-07 18:09:37 +01:00
Steve Loughran
96489069b0
HADOOP-15851. Disable wildfly logs to the console. 
Contributed by Vishwajeet Dusane.

(cherry picked from commit ef9dc6c44c686e836bb25e31ff355cff80572d23)
 2019-06-07 18:09:37 +01:00
Steve Loughran
baa8670256
HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth. 
Contributed by Da Zhou.

(cherry picked from commit bd50fa956b1ca25bb2136977b98a6aa6895eff8b)
 2019-06-07 18:09:37 +01:00
Takanobu Asanuma
a9a3450560 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:34:16 +09:00
Akira Ajisaka
855dc997d6
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:27:33 +09:00
Andrew Olson
55603529d0
HADOOP-16294: Enable access to input options by DistCp subclasses. 
Adding a protected-scope getter for the DistCpOptions, so that a subclass does
not need to save its own copy of the inputOptions supplied to its constructor,
if it wishes to override the createInputFileListing method with logic similar
to the original implementation, i.e. calling CopyListing#buildListing with a path and input options.

Author:    Andrew Olson
(cherry picked from commit c15b3bca86a0f973ccdddd020f3ff2d5767ff1bd)
 2019-05-16 16:13:12 +02:00
Weiwei Yang
26eb9f52fb HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 
(cherry picked from commit 2d8282bb8248e6984878626c4cdc7148aa2e7202)
 2019-05-14 14:06:42 -07:00
Rajat Khandelwal
12e0053932
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die. 
Contributed by Rajat Khandelwal

(cherry picked from commit 591ca698230f25217c10c7549aff8097baa11f1e)
 2019-05-09 21:14:37 +01:00
Akira Ajisaka
df5d8f05d9
HADOOP-16227. Upgrade checkstyle to 8.19 
(cherry picked from commit 4b4fef2f0e0ed1e185ea1058db7a65d68d4970b9)
 2019-04-15 10:47:02 +09:00
Masatake Iwasaki
03079be707 HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 
(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)
 2019-04-12 11:59:14 +09:00
Steve Loughran
b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685) 
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d9990ef9059b6ac9a19ad24310d6f36d04)
 2019-04-03 21:35:19 +01:00
Akira Ajisaka
80a8d3310e
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)
 2019-04-03 19:36:17 +09:00
Steve Loughran
60c9042286
HADOOP-16058. S3A tests to include Terasort. 
Contributed by Steve Loughran.

This includes
 - HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
 - MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
 - MAPREDUCE-7091. Terasort on S3A to switch to new committers
 - MAPREDUCE-7092. MR examples to work better against cloud stores
 2019-03-29 15:25:45 +00:00
Siyao Meng
52cfbc39cc
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail. 
Contributed by Siyao Meng

(cherry picked from commit ce4bafdf442c004b6deb25eaa2fa7e947b8ad269)
 2019-03-26 18:43:43 +00:00
Andrew Olson
ade3af6ef2
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized. 
Author:    Andrew Olson
(cherry picked from commit faba3591d32f2e4808c2faeb9472348d52619c8a)
 2019-03-22 10:36:34 +00:00
Weiwei Yang
39f60faa60 HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 
(cherry picked from commit 568d3ab8b65d1348dec9c971feffe200e6cba2ef)
 2019-03-19 10:08:11 +08:00
Adam Antal
81a6ba1825
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants. 
Contributed by Adam Antal.

(cherry picked from commit c0427c84dddf942529dfdfc5cc7a3e25e3f12c5e)
 2019-03-18 19:14:43 +00:00
Ben Roling
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files. 
Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.
 2019-03-14 19:46:34 +00:00
Steve Loughran
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 11:15:25 +00:00
Da Zhou
cfaf21a4ba
HADOOP-16169. ABFS: Bug fix for getPathProperties. 
Author:    Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit e0260417ade5568ae37dcc3753aea0d1e0fd701b)
 2019-03-08 13:53:44 +00:00
Da Zhou
dc38fc598d
HADOOP-16136. ABFS: Should only transform username to short name 
Contributed by Da Zhou.

(cherry picked from commit 3988e75ca385aec31ca1fc49d6cffce1ea935825)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 10:47:58 +00:00
Da Zhou
075f6b061c
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.

(cherry picked from commit 1f1655028eede24197705a594b6ef19e6737db35)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 10:44:46 +00:00
Da Zhou
ae832ccffe
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. 
Contributed by Shweta Yakkali.

Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
(cherry picked from commit 02eb91856e7e8477c62e0f8bf1bac6de3e00a8a4)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 10:39:37 +00:00
Steve Loughran
685a41f449
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS. 
Contributed by Steve Loughran.

(cherry picked from commit 9cb2f470b759bbe7609a00e8f8f72779e2daae80)
 2019-02-21 13:39:37 +00:00
Masatake Iwasaki
dc9c3ce30b HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki. 
(cherry picked from commit aa3ad3660506382884324c4b8997973f5a68e29a)
 2019-02-20 22:17:18 +09:00
bibinchundatt
bdfdf12178 YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T. 
(cherry picked from commit 779dae4de7e518938d58badcef065ea457be911c)
 2019-02-20 00:37:47 +05:30