Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,690 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24690 Commits

Author SHA1 Message Date
Fei Hui 984875c624
HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc (#2345) 2020-09-28 09:54:01 -07:00
Inigo Goiri eae0035a2d HDFS-15530. RBF: Fix typo in DFS_ROUTER_QUOTA_CACHE_UPDATE_INTERVAL var definition. Contributed by Sha Fanghao. 2020-09-28 09:40:39 -07:00
Takanobu Asanuma d89af79507
HDFS-15577. Refactor TestTracing. (#2302) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2020-09-28 18:27:41 +09:00
Vinayakumar B 82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng 2020-09-27 19:27:13 +05:30
Aryan Gupta bbbfa7d415
HADOOP-17286. Updated jQuery in yarn-common. (#2340) 2020-09-27 03:18:01 +09:00
Uma Maheswara Rao G 899dea2a21
HDFS-15598: ViewHDFS#canonicalizeUri should not be restricted to DFS only API. (#2339). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-25 21:21:01 -07:00
Ye Ni 00c4de63cf
HDFS-15594. Lazy calculate live datanodes in safe mode tip (#2332) 2020-09-25 09:47:54 -07:00
Shubham Gupta e3cd627069
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
 2020-09-25 12:20:39 +05:30
tkuramoto33 3fc007a37b
HADOOP-17282. libzstd-dev should be used instead of libzstd1-dev on Ubuntu 18.04 or higher. (#2336) 2020-09-25 14:17:50 +09:00
Uma Maheswara Rao G 3ccc962b99
HDFS-15596: ViewHDFS#create(f, permission, cflags, bufferSize, replication, blockSize, progress, checksumOpt) should not be restricted to DFS only. (#2333). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-24 07:07:48 -07:00
Aryan Gupta 486ddb73f9
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2020-09-24 21:15:55 +09:00
huangtianhua ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS (#2189) 
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
 2020-09-24 14:27:04 +05:30
bshashikant 368f2f637e
HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled (#2326) 2020-09-24 14:00:41 +05:30
Sneha Vijayarajan c3a90dd918
HADOOP-17279: ABFS: testNegativeScenariosForCreateOverwriteDisabled fails for non-HNS account. 
Contributed by Sneha Vijayarajan

Testing:

namespace.enabled=false
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 246
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=SharedKey
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 33
Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

namespace.enabled=true
auth.type=OAuth
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify

Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-09-23 15:59:00 +00:00
Adam Antal 7ec16ac7f6 YARN-10443. Document options of logs CLI. Contributed by Ankit Kumar. 
Fixes #2325
 2020-09-23 17:42:56 +02:00
tkuramoto33 18f9d77c46
YARN-6754. Fair scheduler docs should explain meaning of weight=0 for a queue. (#2300) 2020-09-23 14:50:28 +09:00
Ye Ni ae089f2db7
HDFS-15557. Log the reason why a storage log file can't be deleted (#2274) 2020-09-22 13:23:20 -07:00
Steve Loughran 7fae4133e0
HADOOP-17261. s3a rename() needs s3:deleteObjectVersion permission (#2303) 
Contributed by Steve Loughran.
 2020-09-22 17:22:04 +01:00
Hui Fei 474fa80bfb
HADOOP-17277. Correct spelling errors for separator (#2322) 
Contributed by Hui Fei.
 2020-09-22 17:10:33 +01:00
Kihwal Lee dfc2682213 HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross. 2020-09-22 10:53:04 -05:00
Xiaoyu Yao 6b5d9e2334
HADOOP-17259. Allow SSLFactory fallback to input config if ssl-*.xml … (#2301) 2020-09-21 12:41:06 -07:00
crossfire c8c1cc43d3
HADOOP-17088.Failed to load XInclude files with relative path. (#2097) 
Contributed by Yushi Hayasaka.
 2020-09-21 19:10:51 +01:00
lfengnan 3e8b1e7426
HDFS-15554. RBF: force router check file existence in destinations before adding/updating mount points (#2266). Contributed by Fengnan Li. 2020-09-21 23:24:13 +05:30
Mukund Thakur 83c7c2b4c4
HADOOP-17023. Tune S3AFileSystem.listStatus() (#2257) 
S3AFileSystem.listStatus() is optimized for invocations
where the path supplied is a non-empty directory.
The number of S3 requests is significantly reduced, saving
time, money, and reducing the risk of S3 throttling.

Contributed by Mukund Thakur.
 2020-09-21 17:20:16 +01:00
Ayush Saxena 7a6265ac42 HDFS-14811. RBF: TestRouterRpc#testErasureCoding is flaky. Contributed by Chen Zhang. 2020-09-21 08:50:08 +05:30
zz 95dfc875d3
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223) 
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-09-19 23:10:05 -07:00
Ayush Saxena 8eba922bb3 HDFS-15579. RBF: The constructor of PathLocation may got some misunderstanding. Contributed by Janus Chow. 2020-09-20 08:54:25 +05:30
Masatake Iwasaki 364b958085
HADOOP-17270. Fix testCompressorDecompressorWithExeedBufferLimit to c… (#2311) 2020-09-20 00:47:02 +09:00
Sneha Vijayarajan e31a636e92
HADOOP-17215: Support for conditional overwrite. 
Contributed by Sneha Vijayarajan

DETAILS:

    This change adds config key "fs.azure.enable.conditional.create.overwrite" with
    a default of true.  When enabled, if create(path, overwrite: true) is invoked
    and the file exists, the ABFS driver will first obtain its etag and then attempt
    to overwrite the file on the condition that the etag matches. The purpose of this
    is to mitigate the non-idempotency of this method.  Specifically, in the event of
    a network error or similar, the client will retry and this can result in the file
    being created more than once which may result in data loss.  In essense this is
    like a poor man's file handle, and will be addressed more thoroughly in the future
    when support for lease is added to ABFS.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 42
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 87, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 457, Failures: 0, Errors: 0, Skipped: 74
    Tests run: 207, Failures: 0, Errors: 0, Skipped: 140
 2020-09-19 01:28:44 +00:00
ThomasMarquardt 0dc54d0247
HADOOP-17203: Revert HADOOP-17183. ABFS: Enabling checkaccess on ABFS 
This reverts commit a2610e21ed.
 2020-09-18 17:52:11 -07:00
David Tucker fc2435cb5c
HADOOP-15136. Correct typos in filesystem.md (#2314) 
Contributed by David Tucker
 2020-09-18 18:28:14 +01:00
Ayush Saxena 2377df38ad HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 2020-09-18 17:28:11 +05:30
Uma Maheswara Rao G 7bba4c609c
HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. (#2312). Contributed by Uma Maheswara Rao G. 2020-09-18 15:18:10 +05:30
Szilard Nemeth eacbe07b56 YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 2020-09-17 19:57:19 +02:00
Xiaoyu Yao 6adf8462ba
HADOOP-17208. LoadBalanceKMSClientProvider#deleteKey should invalidateCache via all KMSClientProvider instances. (#2259) 2020-09-17 10:39:19 -07:00
S O'Donnell 20a0e6278d HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell 2020-09-17 14:11:42 +01:00
bshashikant 425f48799c
HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. (#2296) 2020-09-17 14:50:08 +05:30
Uma Maheswara Rao G e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled (#2305). Contributed by Uma Maheswara Rao G. 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2020-09-16 22:43:00 -07:00
Wanqiang Ji ce86183691
HADOOP-17246. Fix build the hadoop-build Docker image failed (#2277) 2020-09-16 16:23:57 +09:00
Takanobu Asanuma 5c5b2ed7c7 HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui. 2020-09-16 01:36:46 +09:00
hemanthboyina f4ed9f3f91 HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 2020-09-15 10:10:21 +05:30
He Xiaoqiao 924959088e HDFS-15559. Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables. Contributed by Lisheng Sun. 2020-09-15 12:05:04 +08:00
Akira Ajisaka 65df30bad6
HADOOP-17262. Switch to Yetus main branch. (#2304) 2020-09-15 11:12:00 +09:00
Jim Brennan 90894ea641 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 2020-09-14 21:19:14 +00:00
Siyao Meng 6d3312de47
HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty (#2258) 2020-09-14 13:31:34 -07:00
Akira Ajisaka c78d18023d
HDFS-15555. RBF: Refresh cacheNS when SocketException occurs. (#2267) 2020-09-14 11:34:52 +09:00
Uma Maheswara Rao G d2779de3f5
HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. (#2298). Contributed by Uma Maheswara Rao G. 2020-09-12 17:06:39 -07:00
Mingliang Liu f59f7f2175
HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell 2020-09-12 01:41:38 -07:00
Steve Loughran 958cab804e
Revert "HADOOP-17244. S3A directory delete tombstones dir markers prematurely. (#2280)" 
This reverts commit 9960c01a25.

Change-Id: I820534c3292f2a343693d835f625488c325fb5d6
 2020-09-11 18:07:49 +01:00
Eric E Payne 9afec2ed17 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 2020-09-11 13:29:26 +00:00