Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,384 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24384 Commits

Author SHA1 Message Date
Szilard Nemeth f6b9f82b3f YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:41:17 +01:00
Steve Loughran a2ae0d7079
Revert "HADOOP-17430. Restore ability to set Text to empty byte array (#2545)" 
This reverts commit 9e85eb9a2e.

Change-Id: Id1ac803b29931b0f643cb37bbe58534726c36f1e
 2021-01-08 10:50:28 +00:00
srinivasst 98565b6c60 YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a)
 2021-01-08 10:57:37 +05:30
Akira Ajisaka 3748c258f9
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. (#2592) 
(cherry picked from commit ae4945fb2c)
 2021-01-06 12:14:09 +09:00
dgzdot 9e85eb9a2e HADOOP-17430. Restore ability to set Text to empty byte array (#2545) 
Contributed by gaozhan.ding

Change-Id: Ib2ad9120c15c46a3fa2de9e3206875cbbc2363c2
 2021-01-05 21:15:14 +00:00
Wei-Chiu Chuang 94c126cc9e HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf57)
 2021-01-04 20:56:18 -08:00
Wei-Chiu Chuang 6340ac857b HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 
(cherry picked from commit 66ee0a6df0)
 2021-01-04 11:28:26 -08:00
He Xiaoqiao cfcd17ffe7
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:32:24 +08:00
Liang-Chi Hsieh 060594259c HDFS-15690. Add lz4-java as test dependency (#2481) 2020-12-29 13:36:04 -08:00
Gautham B A 98fe00e208 HDFS-15699 Remove lz4 references in vcxproj (#2498) 2020-12-29 13:34:54 -08:00
dengzh bc0c3ece0f HADOOP-17390. Skip license check on lz4 code files (#2478) 
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license

Contributed by Zhihua Deng.
 2020-12-29 13:32:26 -08:00
Liang-Chi Hsieh 87064df1f2 HADOOP-17292. Using lz4-java in Lz4Codec (#2350) 
Contributed by Liang-Chi Hsieh.
 2020-12-29 13:17:26 -08:00
Masatake Iwasaki b8a4361d7b HADOOP-17270. Fix testCompressorDecompressorWithExeedBufferLimit to c… (#2311) 2020-12-29 13:11:51 -08:00
He Xiaoqiao 3736f6eb57
HADOOP-17445. Update the year to 2021. Contributed by He Xiaoqiao. 2020-12-24 20:47:16 +08:00
Eric Badger 1b5b2bea83 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230)
 2020-12-21 23:11:47 +00:00
yzhangal adf6ca18b4
HADOOP-17338. Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc (#2497) 
Yongjun Zhang <yongjunzhang@pinterest.com>

Change-Id: Ibbc6a39afb82de1208e6ed6a63ede224cc425466
 2020-12-19 12:24:16 +00:00
Ahmed Hussein be508718d8 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234)
 2020-12-17 17:50:47 +00:00
He Xiaoqiao 3a860e876e HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 
(cherry picked from commit fa14e4bc00)
 2020-12-15 14:01:48 -08:00
Ayush Saxena faf26f70aa HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
 2020-12-14 20:41:06 -08:00
Chao Sun 81e533de8f
HADOOP-16080. hadoop-aws does not work with hadoop-client-api. Contributed by Chao Sun (#2522) 2020-12-12 09:37:13 -08:00
S O'Donnell 1a63df86e2 HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 9ed737001c)
 2020-12-11 21:05:00 +00:00
Jim Brennan e5f11ea5b2 HADOOP-13571. ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0. Contributed by Eric Badger 
(cherry picked from commit 6de1a8eb67)
 2020-12-11 20:19:08 +00:00
Ahmed Hussein 16ca8b7d20 HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186c)
 2020-12-11 17:04:40 +00:00
Akira Ajisaka 71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2538) 
(cherry picked from commit 1b29c9bfee)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-12-11 13:58:02 +09:00
Neil bc5458bbd4 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50)
 2020-12-10 13:50:32 -08:00
Jim Brennan 27e455a1a0 HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 20:57:43 +00:00
Liang-Chi Hsieh 485e4e3348 HADOOP-17425. Bump up snappy-java to 1.1.8.2. (#2536) 
(cherry picked from commit c2cecfc9b9)

 Conflicts:
	hadoop-project/pom.xml
 2020-12-10 13:58:49 +00:00
Masatake Iwasaki 3a50148809 HADOOP-17369. Bump up snappy-java to 1.1.8.1. (#2451) 
(cherry picked from commit 0946f58cae)
 2020-12-10 13:57:35 +00:00
Mukund Thakur e4cab4b7a3
HADOOP-17186. Fixing javadoc in ListingOperationCallbacks (#2196) 
(cherry picked from commit ac697571a1)
 2020-12-10 18:32:22 +09:00
Akira Ajisaka b84bc63aa2 HADOOP-17344. Harmonize guava version and shade guava in yarn-csi. (#2434) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-12-10 05:51:45 +05:30
Ayush Saxena 8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Ankit Kumar f04a9dfda1
YARN-10491. Fix deprecation warnings in SLSWebApp.java (#2519) 
Signed-off-by: Akira Ajisaka <ajisaka@apache.org>
(cherry picked from commit aaf9e3d320)
 2020-12-09 10:53:42 +09:00
Ahmed Hussein e02b179c4c HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 
(cherry picked from commit 01383a2172)
 2020-12-08 19:28:44 +00:00
Hui Fei cb2dce30d4 HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 10:40:14 +08:00
crossfire edd9b659ca HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d)
 2020-12-07 15:50:38 -08:00
Jim Brennan 6a5864ee4a HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 2020-12-07 21:19:36 +00:00
Ahmed Hussein 30c9f3920f HADOOP-17389. KMS should log full UGI principal. (#2476) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Jim Brennan <jbrennan@apache.org>
(cherry picked from commit 8c234fc0d4)
 2020-12-07 17:12:52 +00:00
Chao Sun 4628647067 HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2 (#2517) 2020-12-04 10:43:25 -08:00
Akira Ajisaka efd74cc8ae
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136) 
(cherry picked from commit 3e7b65c3e5)
 2020-12-04 10:07:58 +09:00
Ahmed Hussein a7dbd3b6f9 HDFS-15706. HttpFS: Log more information on request failures. (#2515) 
(cherry picked from commit 07655a7040)
 2020-12-03 15:15:18 -08:00
Jim Brennan 5bfb97bc7d HADOOP-17392. Remote exception messages should not include the exception class (#2486). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-03 17:59:01 +00:00
Andrea Scarpino c5b9c5dfe5
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b)
 2020-12-04 00:16:45 +09:00
Thomas Marquardt a5695057b1
HADOOP-17397: ABFS: SAS Test updates for version and permission update 
DETAILS:

    The previous commit for HADOOP-17397 was not the correct fix.  DelegationSASGenerator.getDelegationSAS
    should return sp=p for the set-permission and set-acl operations.  The tests have also been updated as
    follows:

    1. When saoid and suoid are not specified, skoid must have an RBAC role assignment which grants
       Microsoft.Storage/storageAccounts/blobServices/containers/blobs/modifyPermissions/action and sp=p
       to set permissions or set ACL.

    2. When saoid or suiod is specified, same as 1) but furthermore the saoid or suoid must be an owner of
       the file or directory in order for the operation to succeed.

    3. When saoid or suiod is specified, the ownership check is bypassed by also including 'o' (ownership)
       in the SAS permission (for example, sp=op).  Note that 'o' grants the saoid or suoid the ability to
       change the file or directory owner to themself, and they can also change the owning group. Generally
       speaking, if a trusted authorizer would like to give a user the ability to change the permissions or
       ACL, then that user should be the file or directory owner.

TEST RESULTS:

    namespace.enabled=true
    auth.type=SharedKey
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 89, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 461, Failures: 0, Errors: 0, Skipped: 24
    Tests run: 208, Failures: 0, Errors: 0, Skipped: 24

    namespace.enabled=true
    auth.type=OAuth
    -------------------
    $mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
    Tests run: 89, Failures: 0, Errors: 0, Skipped: 0
    Tests run: 461, Failures: 0, Errors: 0, Skipped: 70
    Tests run: 208, Failures: 0, Errors: 0, Skipped: 141
 2020-12-03 14:31:06 +00:00
Ahmed Hussein 2249edb001 HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 60201cbf69)
 2020-12-02 21:40:17 +00:00
Ahmed Hussein 0f18a19d97 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7)
 2020-12-02 20:12:31 +00:00
Eric Payne 1fd6d81617 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 
(cherry picked from commit fa773a8326)
 2020-12-01 22:51:20 +00:00
Ahmed Hussein a660714387 HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82f)
 2020-12-01 21:16:49 +00:00
kevinzhao1661 7d9c6edadf
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398)
 2020-11-30 11:17:59 +09:00
Mukund Thakur 3ef0e3d615 HADOOP-17398. Skipping network I/O in S3A getFileStatus(/) breaks some tests (#2493) 
Follow-on to HADOOP-17323.

Contributed by Mukund Thakur.
 2020-11-26 20:26:44 +00:00
Steve Loughran 1e59bf7394
HADOOP-17385. ITestS3ADeleteCost.testDirMarkersFileCreation failure (#2473). 
Contributed by Steve Loughran

The addition of deprecated S3A configuration options in HADOOP-17318
triggered a reload of default (xml resource) configurations, which breaks
tests which fail if there's a per-bucket setting inconsistent with test
setup.

Creating an S3AFS instance before creating the Configuration() instance
for test runs gets that reload out the way before test setup takes
place.

Along with the fix, extra changes in the failing test suite to fail
fast when marker policy isn't as expected, and to log FS state better.

Rather than create and discard an instance, add a new static method
to S3AFS and invoke it in test setup. This forces the load

Change-Id: Id52b1c46912c6fedd2ae270e2b1eb2222a360329
 2020-11-26 17:28:01 +00:00