Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,772 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24772 Commits

Author SHA1 Message Date
Steve Loughran a1c0673526
HADOOP-18198. Preparing for 3.3.3 release 
Change-Id: Idebf79191dc91dad52073f2c63ee9ab3a99464d9
 2022-04-14 17:19:16 +01:00
Chao Sun f91452b289 Make upstream aware of 3.3.2 release 2022-03-02 19:09:35 -08:00
Steve Loughran 7f33a4e992
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: I3856afa821dbc8c2e3cb1cbe33793ec1734e2e24
 2022-02-21 18:01:53 +00:00
Akira Ajisaka 0bcb014209 YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp (#2591) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9cb535caf2)
 2022-02-10 15:14:26 -08:00
Ayush Saxena f9cccfa7ea
HADOOP-18096. Distcp: Sync moves filtered file to home directory rather than deleting. (#3940). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: stack <stack@apache.org>
 2022-02-11 02:10:02 +05:30
Petre Bogdan Stolojan 8cd8e435fb
HADOOP-17198. Support S3 Access Points (#3260) (branch-3.3.2) (#3955) 
Add support for S3 Access Points. This provides extra security as it
ensures applications are not working with buckets belong to third parties.

To bind a bucket to an access point, set the access point (ap) ARN,
which must be done for each specific bucket, using the pattern

fs.s3a.bucket.$BUCKET.accesspoint.arn = ARN

* The global/bucket option `fs.s3a.accesspoint.required` to
mandate that buckets must declare their access point.
* This is not compatible with S3Guard.

Consult the documentation for further details.
 2022-02-04 10:09:00 -08:00
Steve Loughran 35e352a88b
HADOOP-18094. Disable S3A auditing by default. 
See HADOOP-18091. S3A auditing leaks memory through ThreadLocal references

* Adds a new option fs.s3a.audit.enabled to controls whether or not auditing
is enabled. This is false by default.

* When false, the S3A auditing manager is NoopAuditManagerS3A,
which was formerly only used for unit tests and
during filsystem initialization.

* When true, ActiveAuditManagerS3A is used for managing auditing,
allowing auditing events to be reported.

* updates documentation and tests.

This patch does not fix the underlying leak. When auditing is enabled,
long-lived threads will retain references to the audit managers
of S3A filesystem instances which have already been closed.

Contributed by Steve Loughran.

Change-Id: I671e594cd59e8ca77a1f65be791ad0ae9530b8d9
 2022-01-24 15:06:01 +00:00
Xiangyi Zhu 6da346a358 HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3882). Contributed by Xiangyi Zhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-15 09:26:05 -08:00
ahmarsuhail 78d57c8e58 HADOOP-16223. Remove misleading fs.s3a.delegation.tokens.enabled prompt (#3879) 
Contributed by Ahmar Suhail

Change-Id: I6a33043831a059325c58b0f76c925e52c6ae14f7
 2022-01-15 09:25:45 -08:00
Mukund Thakur 472a28f91d HADOOP-18065. ExecutorHelper.logThrowableFromAfterExecute() is too noisy. (#3860) 
Downgrading warn logs to debug in case of InterruptedException

Contributed By: Mukund Thakur
 2022-01-15 09:25:38 -08:00
monthonk d101e4d7fa HADOOP-14334. S3 SSEC tests to downgrade when running against a mandatory encryption object store (#3870) 
Contributed by Monthon Klongklaew

Change-Id: Ib275c9690bbc90170c6a442ded198fe006c20bc1
 2022-01-15 09:25:32 -08:00
Ayush Saxena 46b1411189 HADOOP-18056. DistCp: Filter duplicates in the source paths. (#3825). Contributed by Ayush Saxena. 
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Steve Loughran <stevel@apache.org>
 2022-01-15 09:25:25 -08:00
Jackson Wang d7cacea07b HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 09:07:32 -08:00
Brahma Reddy Battula 8af28b7cca Updated the index as per 3.3.2 release 2022-01-05 10:44:28 -08:00
Ashutosh Gupta 6ea9906a8b HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 10:01:38 -08:00
liever18 d167ef4850 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5)
 2022-01-05 10:01:20 -08:00
Wei-Chiu Chuang e42e483d00 Make upstream aware of 3.3.1 release 2022-01-04 14:51:49 -08:00
Ashutosh Gupta b9ed31f9ec HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 14:38:52 -08:00
jianghuazhu b8fff25645 HADOOP-18063. Remove unused import AbstractJavaKeyStoreProvider in Shell class. (#3846) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7398a0f1b2)
 2022-01-04 14:38:41 -08:00
Igor Dvorzhak 9a24995d73 HADOOP-13464. Upgrade Gson dependency to version 2.8.9 (#2524) 
Change-Id: Ifd3fb9ec6ebfc8874bb799bc198219511fe55a2f

Update pom.xml

Update pom.xml

(cherry picked from commit 795054882a)
 2022-01-04 14:38:38 -08:00
Ashutosh Gupta 80a19c8a48 HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29ec88)
 2022-01-04 14:38:35 -08:00
Akira Ajisaka 6db1165380 Revert "HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836)" 
This reverts commit 05b43f2057.
 2022-01-04 14:38:31 -08:00
Ashutosh Gupta 8aac1c7dc9 HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29ec88)
 2022-01-04 14:38:28 -08:00
Akira Ajisaka 84a767ea24 HADOOP-18045. Disable TestDynamometerInfra (#3829) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
(cherry picked from commit dba139cd0f)
 2022-01-04 14:38:24 -08:00
jianghuazhu 2ddc66fc0c HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2022-01-04 14:38:19 -08:00
Akira Ajisaka 5a7efbfbb5 HADOOP-18052. Support Apple Silicon in start-build-env.sh (#3817) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c8725de547)
 2022-01-04 14:38:07 -08:00
Ashutosh Gupta c63509127f YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405fbd)
 2022-01-04 14:38:02 -08:00
Eric Payne 1be9d60f6c YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 
(cherry picked from commit e2d6fd075d)
 2022-01-04 14:37:59 -08:00
wangzhaohui bceab2e439 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629)
 2022-01-04 14:37:55 -08:00
jianghuazhu 82cf0aa296 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2022-01-04 14:37:52 -08:00
Szilard Nemeth d3fded12dc YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke 2022-01-04 14:37:47 -08:00
Dhananjay Badaya 4201cdb44e HADOOP-13500. Synchronizing iteration of Configuration properties object (#3775) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4483607a4e)
 2022-01-04 14:37:43 -08:00
litao c0ae38c6b4 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d)
 2022-01-04 14:37:40 -08:00
litao 24917bae9f HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 
(cherry picked from commit 10cdde59b4)
 2022-01-04 14:37:37 -08:00
Ayush Saxena 57c9a8509f HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-04 14:37:26 -08:00
litao f16d9dfc31 HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 
(cherry picked from commit c56a07f36b)
 2022-01-04 14:37:22 -08:00
Viraj Jasani a45df81822 HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 (#3764) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 14:37:19 -08:00
GuoPhilipse 45e6d22462 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7)
 2022-01-04 14:37:15 -08:00
Anoop Sam John de27fa0097 HADOOP-17643 WASB : Make metadata checks case insensitive (#3103) 2022-01-04 14:37:12 -08:00
Akira Ajisaka c257abca21 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
 2022-01-04 14:37:08 -08:00
qinyuren 04f0025ad2 HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2022-01-04 14:37:04 -08:00
Szilard Nemeth 52c5bbc993 YARN-11020. [UI2] No container is found for an application attempt with a single AM container. Contributed by Andras Gyori 2022-01-04 14:37:01 -08:00
Takanobu Asanuma 1e3f94fa3c Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 55c0e676fd.
 2022-01-04 14:36:57 -08:00
qinyuren 5214140123 HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043)
 2022-01-04 14:36:54 -08:00
lzx404243 47436346b4 HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7ba91c1235)
 2022-01-04 14:36:50 -08:00
Ashutosh Gupta b410289745 YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 5a950b8900)
 2022-01-04 14:36:46 -08:00
LeonGao d02bf2dcce HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661)
 2022-01-04 14:36:42 -08:00
Prabhu Joseph 77c1c482ef YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 
(cherry picked from commit 6b4987f09c)
 2022-01-04 14:36:39 -08:00
Takanobu Asanuma 906eb831c6 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106)
 2022-01-04 14:36:36 -08:00
huhaiyang e50c9224fe HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b)
 2022-01-04 14:36:32 -08:00