Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,346 Commits 326 Branches 371 Tags
Commit Graph

24346 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wei-Chiu Chuang
cc0cc7ab5f HADOOP-17614. Bump netty to the latest 4.1.61. (#2850) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 29a6769b68f32fc3eb0686a955040b9412a24a66)
 2021-04-05 08:37:10 +08:00
Surendra Singh Lilhore
e079aaa820 MAPREDUCE-7199. HsJobsBlock reuse JobACLsManager for checkAccess. Contributed by Bilwa S T 
(cherry picked from commit a1b0697d379d33223ec1a46dfef31d6d226169bb)
 2021-04-02 21:31:45 +05:30
Inigo Goiri
d973f37054 HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina. 
(cherry picked from commit 2148a8fe645333444c4e8110bb56acf0fb8e41b4)
 2021-04-02 17:14:34 +08:00
Inigo Goiri
887948d127 HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina. 
(cherry picked from commit 719b53a79dc169a8c52229831dcb011935a8a151)
 2021-04-02 17:14:24 +08:00
Surendra Singh Lilhore
02d37c7d22 HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina 
(cherry picked from commit 743c2e9071f4a73e0196ad4ca005b767758642b9)
 2021-04-02 17:14:14 +08:00
Takanobu Asanuma
14fddba715 HDFS-15252. HttpFS: setWorkingDirectory should not accept invalid paths. Contributed by hemanthboyina. 
(cherry picked from commit 736659e0e1ab2882313e3a41d9a20d4b0f5b0816)
 2021-04-02 17:13:59 +08:00
Ayush Saxena
2da3356428 HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui. 
(cherry picked from commit fc8a6dd8a92f8e18436d1a73d6e8ef808c583aee)
 2021-04-02 17:13:00 +08:00
Takanobu Asanuma
f5cc1540b4 HDFS-15510. RBF: Quota and Content Summary was not correct in Multiple Destinations. Contributed by Hemanth Boyina. 
(cherry picked from commit ca8e7a77256003e11ab7e3d079ee4cf9f50080dd)
 2021-04-02 17:11:33 +08:00
Ayush Saxena
a549b4a82e HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq. 
(cherry picked from commit 936bf09c3745cfec26fa9cfa0562f88b1f8be133)
 2021-04-02 17:11:16 +08:00
Ayush Saxena
5bfc3a4c3a HDFS-15198. RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by zhengchenyu. 
(cherry picked from commit 8a9a674ef10a951c073ef17ba6db1ff07cff52cd)
 2021-04-02 17:10:56 +08:00
Inigo Goiri
327488d688 HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li. 
(cherry picked from commit 80b877a72f52ef0f4acafe15db55b8ed61fbe6d2)
 2021-04-02 14:12:17 +08:00
Inigo Goiri
57d2fff84b HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina. 
(cherry picked from commit 8b7695bb2628574b4450bac19c12b29db9ee0628)
 2021-04-02 14:06:52 +08:00
Inigo Goiri
1148d3895f HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina. 
(cherry picked from commit 130f89e068ca346a44fa6619ae0726c1e5cc5d06)
 2021-04-02 14:06:52 +08:00
Inigo Goiri
41fad5056c HDFS-15265. HttpFS: validate content-type in HttpFSUtils. Contributed by hemanthboyina. 
(cherry picked from commit 31b2f687eff5a77ea83c2279cc94c4c7158160dc)
 2021-04-02 14:06:52 +08:00
Ayush Saxena
5187bd37ae HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena. 
(cherry picked from commit 37d65822235fe8285d10232589aba39ededd3be1)
 2021-04-02 14:06:52 +08:00
maobaolong
2b207ea402 HDFS-15667. Audit log record the unexpected allowed result when delete (#2437) 
(cherry picked from commit 95c96605b30cc31839a04bd5d4061a2c89e4c09c)
 2021-04-02 14:06:52 +08:00
Brahma Reddy Battula
90bbaca88b HADOOP-17587. Kinit with keytab should not display the keytab file's full path in any logs. Contributed by Ravuri Sushma sree. 
(cherry picked from commit bc7689abf5723fb6ec763266227801636105f5a1)
 2021-04-02 10:04:19 +05:30
Brahma Reddy Battula
c60e81b5a8 HADOOP-17610. DelegationTokenAuthenticator prints token information. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 478402cc740fa21123b2a332d3ac7e66170a5535)
 2021-04-02 09:56:47 +05:30
litao
62937d15bb HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d64595ab0831626c80d8d1b788a35cd4)
 2021-04-02 10:57:05 +09:00
Viraj Jasani
f707d8407a
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:35:44 -07:00
He Xiaoqiao
e5d7b22b8b
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:59:53 +08:00
Brahma Reddy Battula
13878fc06b HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 0665ce99308aba1277d8f36bad9308062ad4b6ea)
 2021-04-01 09:22:18 +05:30
Borislav Iordanov
c365149e16 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures.

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:50:28 -07:00
Surendra Singh Lilhore
c70f5eb8fa MAPREDUCE-6826. Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING. Contributed by Bilwa S T. 
(cherry picked from commit d4e36409d40d9f0783234a3b98394962ae0da87e)
 2021-03-31 21:35:06 +05:30
He Xiaoqiao
ae2d856515
HDFS-15932. Improve the balancer error message when process exits abnormally. Contributed by Renukaprasad C. 2021-03-31 12:24:01 +08:00
He Xiaoqiao
ab451b8e42 HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBPOfferService.java
 2021-03-30 22:29:49 +08:00
Wanqiang Ji
872839416b HDFS-15331. Remove invalid exclusions that minicluster dependency on HDFS (#1996). Contributed by Wanqiang Ji 
(cherry picked from commit e877118454b97ae72d3b52dd256999ca073df91d)
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
53b024a2b7 HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module 
(cherry picked from commit 9062814bab7d6087c584e03b0b16eaa60b31f60a)
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
a407f7de5a HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li. 
(cherry picked from commit 32895f4f7ea51522e779e68a030bfc9515f40798)
 2021-03-30 22:29:49 +08:00
Akira Ajisaka
c4f7b8578b HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li. 
(cherry picked from commit 7938ebfb9dd12b874ced1798f0c3ad0c160cb465)
 2021-03-30 22:29:49 +08:00
Karthik Palanisamy
769df1e964 HDFS-15253 Default checkpoint transfer speed, 50mb per second (#2366) 
(cherry picked from commit df4006eb8139a08b92b5a94adfde544de978137c)
 2021-03-30 22:29:49 +08:00
Ayush Saxena
7f20fad419 HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena. 
(cherry picked from commit 2e8cafac3b071fe5b943542827fd8a496b137fa9)
 2021-03-30 22:29:48 +08:00
Stephen O'Donnell
56ef16468a
HADOOP-17222. Create socket address leveraging URI cache (#2817) 
Contributed by fanrui.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-30 11:59:44 +01:00
Akira Ajisaka
c96fbb9199
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9f674e734e7f4c917ee2d51c77e527d)
 2021-03-30 17:47:54 +09:00
Brahma Reddy Battula
5358313f97 YARN-10466.Fix NullPointerException in yarn-services Component.java. Contributed by D M Murali Krishna Reddy 
(cherry picked from commit 413a4c3c05d317090c706385c51e4cabcfd92b0e)
 2021-03-30 13:52:35 +05:30
Brahma Reddy Battula
bac1326e4e YARN-10439. addendum fix for shaded guva. 
(cherry picked from commit 6577bf1891b11c9271d73491b311059677dfb376)
 2021-03-30 13:49:23 +05:30
Brahma Reddy Battula
616a41ee32 YARN-10441. Add support for hadoop.http.rmwebapp.scheduler.page.class. Contributed by D M Murali Krishna Reddy 
(cherry picked from commit b61f52ec565b84306ec8d9e0b53f5d0390e1b597)
 2021-03-30 09:52:28 +05:30
Brahma Reddy Battula
5181b2004b YARN-10439. Yarn Service AM listens on all IP's on the machine. Contributed by D M Murali Krishna Reddy 
(cherry picked from commit d0dcfc405c624f73ed1af9527bbf456a10337a6d)
 2021-03-30 09:46:52 +05:30
Brahma Reddy Battula
e9d8f16a70 YARN-10437. Destroy yarn service if any YarnException occurs during submitApp.Contributed by D M Murali Krishna Reddy 
(cherry picked from commit 2d62dced4b60938cab630321830a0510d5391338)
 2021-03-30 09:40:06 +05:30
Viraj Jasani
d6aaf516d9 HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:25:14 +05:30
Harunobu Daikoku
db8af483d1 HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b68ac462aec0d493718d7a10fbda0b6d)
 2021-03-29 11:47:16 +09:00
Ayush Saxena
9c9b16c957
HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2808). Contributed by Ayush Saxena. 
* HADOOP-17531. DistCp: Reduce memory usage on copying huge directories. (#2732).

* HADOOP-17531.Addendum: DistCp: Reduce memory usage on copying huge directories. (#2820)

Signed-off-by: Steve Loughran <stevel@apache.org>
 2021-03-27 09:25:25 +05:30
Eric Badger
65bba8c3ed YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu. 
(cherry picked from commit 19e418c10d8fbd994e5ca326bf894cab706b8a4f)
 2021-03-25 22:35:19 +00:00
Xiaoyu Yao
67d52af225 HADOOP-16828. Zookeeper Delegation Token Manager fetch sequence number by batch. Contributed by Fengnan Li. 
(cherry picked from commit 6288e15118fab65a9a1452898e639313c6996769)
 2021-03-25 14:44:02 +00:00
Steve Loughran
a07e3c41ca
HADOOP-13551. AWS metrics wire-up (#2778) 
Moves to the builder API for AWS S3 client creation, and
offers a similar style of API to the S3A FileSystem and tests, hiding
the details of which options are client, which are in AWS Conf,
and doing the wiring up of S3A statistics interfaces to the AWS
SDK internals. S3A Statistics, including IOStatistics, should now
count throttling events handled in the AWS SDK itself.

This patch restores endpoint determination by probes to US-East-1
if the client isn't configured with fs.s3a.endpoint.

Explicitly setting the endpoint will save the cost of these probe
HTTP requests.

Contributed by Steve Loughran.

Change-Id: Ifa6caa8ff56369ad30e4fd01a42bc74f7b8b3d6b
 2021-03-25 13:59:33 +00:00
Mukund Thakur
4c3324ca1a
HADOOP-17305. Fix ITestCustomSigner to work with s3 compatible endpoints (#2395) 
Contributed by Mukund Thakur

Change-Id: Ia5def405056691c349cf05530fd3172047d2058b
 2021-03-25 13:59:33 +00:00
leosunli
b3e968d21e HDFS-13639. SlotReleaser is not fast enough (#1885) 
(cherry picked from commit be374faf429d28561dd9c582f5c55451213d89a4)
 2021-03-25 13:25:17 +00:00
Akira Ajisaka
80d4636067
HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein. 
(cherry picked from commit c5929c00a35a8e58bcf1c80d9d414c3cbdce4483)

 Conflicts:
	hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml
 2021-03-25 19:54:51 +09:00
He Xiaoqiao
bf018e7237
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:25:22 +08:00
LeonGao91
4db5643ff9 HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently (#2299) 
(cherry picked from commit de055ec8aad1b5a131d366aa5395f2904f0f6ecd)
 2021-03-25 05:09:11 +08:00