Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,380 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

9402 Commits

Author SHA1 Message Date
huhaiyang b38b00e528
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
 2021-06-01 00:26:47 -07:00
Ayush Saxena b8ab19373d Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)" 
This reverts commit 041488e8f5.
 2021-06-01 08:38:13 +05:30
Ayush Saxena bce14e746b HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng. 2021-06-01 08:34:56 +05:30
Jinglun e90c41af34 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 19:28:40 +08:00
Jinglun 2b11e160e2 Revert "HDFS-15973." because the user.name and user.email message is not correct. 
This reverts commit b7f69467c1.
 2021-05-30 19:27:10 +08:00
李经纶 b7f69467c1 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 18:50:03 +08:00
Konstantin V Shvachko 8ce30f51f9 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 2021-05-27 18:47:20 -07:00
Konstantin V Shvachko 1abd03d68f HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 2021-05-26 14:59:50 -07:00
Gautham B A bcaeb1ac8c
Revert "Revert "HDFS-15971. Make mkstemp cross platform (#2898)"" (#3044) 2021-05-26 14:12:53 -07:00
Hui Fei 98a74e2351
HDFS-16041. TestErasureCodingCLI fails (#3052) 2021-05-26 20:12:17 +09:00
Viraj Jasani 986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Narges Shadab 3fdeb7435a
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027) 2021-05-26 14:43:41 +09:00
zhuxiangyi e18d806212
HDFS-16024: RBF: Rename data to the Trash should be based on src loca… (#3009) 2021-05-26 13:41:49 +08:00
Viraj Jasani 028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:55:58 +09:00
lfengnan 43bf009112
HDFS-15757 RBF: Improving Router Connection Management (#2651) 2021-05-19 10:53:42 -07:00
huhaiyang acd712ca92
HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747) 2021-05-19 09:04:41 +08:00
Ayush Saxena fcd4140e5f
HDFS-15988. Stabilise HDFS Pre-Commit. (#2860). Contributed by Ayush Saxena. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-05-16 00:45:11 +05:30
Viraj Jasani b944084b32
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-11 12:38:15 +09:00
Siyao Meng 8d5cc98b42
HDFS-15997. Implement dfsadmin -provisionSnapshotTrash -all (#2958) 2021-05-10 12:41:07 -07:00
Ayush Saxena 359c0c1d1e HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w. 2021-05-10 17:09:51 +05:30
Jinglun 1b6994287a HDFS-15923. RBF: Authentication failed when rename accross sub clusters (#2819). Contributed by zhuobin zheng. 
Reviewed-by: Jinglun <jinglun@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-08 19:50:17 +08:00
Akira Ajisaka 9aa651f0b4
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980) 
Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>
 2021-05-08 00:24:06 +09:00
zhuqi 76c69c39d9
HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. (#2833) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-05-05 21:02:13 +05:30
Viraj Jasani 041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-05-04 01:31:23 -07:00
Akira Ajisaka 7c561e2a02
HDFS-15952. TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky. (#2970) 2021-05-04 00:25:29 +09:00
lfengnan 6e525ab81c
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 19:18:47 +09:00
Wei-Chiu Chuang f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
Karthik Palanisamy bd3da73a0f
HDFS-15865. Interrupt DataStreamer thread if no ack (#2728) 2021-05-01 23:35:31 +05:30
lfengnan 552e9dcc6c
HDFS-15561. RBF: Remove NPE when local namenode is not configured (#2954). Contributed by Fengnan Li. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-29 21:14:18 +08:00
Wei-Chiu Chuang b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
 2021-04-28 23:54:39 -07:00
Wei-Chiu Chuang f41a368c14
HADOOP-11245. Update NFS gateway to use Netty4 (#2832) 
Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>
 2021-04-28 23:43:11 -07:00
bshashikant ef13f8ad6b
HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881) 2021-04-27 09:15:06 +05:30
Stephen O'Donnell 605ed85c29
HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 2021-04-26 11:00:23 +01:00
litao b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-26 18:38:43 +09:00
Takanobu Asanuma 01bad0e92a HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel. 2021-04-26 13:29:28 +09:00
Brahma Reddy Battula 2621d3f15b HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang 90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
jianghuazhu c486dc396c
HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913) Contributed by JiangHua Zhu. 2021-04-25 14:31:51 +08:00
zhuxiangyi dff95c5eca
HDFS-15974. RBF: Unable to display the datanode UI of the router (#2915) 
Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 10:19:11 +09:00
Viraj Jasani acb49f2fc3
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-21 19:45:49 +09:00
litao 85a3532849
HDFS-15970. Print network topology on the web (#2896) 2021-04-19 20:42:30 +09:00
litao af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong (#2907) 2021-04-17 16:49:09 +09:00
Eric Badger aed13f0f42 Revert "HDFS-15971. Make mkstemp cross platform (#2898)" 
This reverts commit b088f46b68.
 2021-04-16 17:15:33 +00:00
zhangshuyan0 14816be0b1
HDFS-15963. Unreleased volume references cause an infinite loop. (#2889) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-17 00:08:31 +08:00
Akira Ajisaka f0241ec216
HDFS-15977. Call explicit_bzero only if it is available. (#2914) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-04-16 13:26:45 +09:00
lfengnan 17be99f9f9
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2903) 2021-04-15 16:11:47 -07:00
Vivek Ratnavel Subramanian c821008836
HDFS-15850. Superuser actions should be reported to external enforcers (#2784) 2021-04-15 14:45:51 -07:00
Masatake Iwasaki 2717203f85
HADOOP-17569. Building native code fails on Fedora 33. (#2886) 2021-04-15 21:33:18 +09:00
Ayush Saxena 7d6f0ca0bd Revert "HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754). Contributed by tomscut." 
This reverts commit e565b05c80.

The removed method needs to be used in HDFS-15423.
 2021-04-13 23:28:42 +05:30
Gautham B A b088f46b68
HDFS-15971. Make mkstemp cross platform (#2898) 2021-04-13 08:54:45 -07:00
Inigo Goiri 82462739f8
Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605)" (#2900) 
This reverts commit cb3ed32fe0.
 2021-04-12 20:44:31 -07:00
lfengnan cb3ed32fe0
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605) 2021-04-12 12:42:33 -07:00
Gautham B A 3148791da4
HDFS-15962. Make strcasecmp cross platform (#2883) 2021-04-09 09:01:33 -07:00
Viraj Jasani 56bd968fb4
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 2021-04-09 11:10:18 +09:00
Gautham B A bf66116407
HDFS-15955. Make explicit_bzero cross platform (#2875) 2021-04-08 09:44:47 -07:00
Ayush Saxena c6539e3289
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-08 20:49:08 +05:30
Stephen O'Donnell 4c567fcff7
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 2021-04-08 11:59:02 +01:00
Gautham B A 42ddb5c6fe
HDFS-15909. Make fnmatch cross platform (#2792) 2021-04-06 09:23:21 -07:00
He Xiaoqiao 459df41997
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 2021-04-06 21:22:48 +08:00
Gautham B A 36014b8282
HDFS-15949. Fix integer overflow (#2857) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-06 18:22:54 +09:00
Viraj Jasani 3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 13:39:10 +09:00
Viraj Jasani de05cefbae
HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 00:19:21 +05:30
Gautham B A 6e1df1c048
HDFS-15947. Replace deprecated protobuf APIs (#2856) 2021-04-05 09:52:48 -07:00
litao 9e2d5d6d05
HDFS-15951. Remove unused parameters in NameNodeProxiesClient (#2859) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-05 17:47:10 +09:00
Gautham B A ca5e13fd74
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-05 11:42:05 +09:00
zhuqi 96e410a127
HDFS-15930: Fix some @param errors in DirectoryScanner. (#2829). Contributed by Qi Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-04 03:11:02 +05:30
litao 3cb76447f5
HDFS-15946. Fix java doc in FSPermissionChecker (#2855). Contributed by tomscut. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-02 10:37:04 -07:00
Gautham B A 70536ba1f9
HDFS-15944. Prevent truncation by snprintf (#2853) 2021-04-02 09:37:33 -07:00
Brahma Reddy Battula ed74479ea5 HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree. 2021-04-02 09:47:20 +05:30
litao 7c83f140dc
HDFS-15938. Fix java doc in FSEditLog (#2837). Contributed by tomscut. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 19:28:17 -07:00
litao 4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-02 10:33:40 +09:00
Viraj Jasani 4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:34:31 -07:00
Brahma Reddy Battula 0665ce9930 HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 2021-04-01 09:19:39 +05:30
He Xiaoqiao 5eca748970
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:32:03 +08:00
Borislav Iordanov 2c482fbacf HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted
(and yet again because of commit message format).

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:46:35 -07:00
stack 22961a615d Revert "HADOOP-16524. Automatic keystore reloading for HttpServer2" 
This reverts commit a2975d2153.
 2021-03-31 10:43:09 -07:00
stack a2975d2153 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted.
 2021-03-31 10:40:20 -07:00
stack 5183aaeda2 Revert "Hadoop 16524 - resubmission following some unit test fixes (#2693)" 
Revert to fix the summary message.

This reverts commit 9509bebf7f.
 2021-03-31 10:39:55 -07:00
Borislav Iordanov 9509bebf7f
Hadoop 16524 - resubmission following some unit test fixes (#2693) 
Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:07:42 -07:00
He Xiaoqiao da4ceba4aa
HDFS-15863. RBF: Validation message to be corrected in FairnessPolicyController. Contributed by Renukaprasad C. 2021-03-31 13:00:13 +08:00
Gautham B A 7dfff496fa
HDFS-15929. Replace RAND_pseudo_bytes in util.cc (#2826) 2021-03-30 13:36:02 -07:00
Gautham B A 8668abf87e
HDFS-15927. Catch polymorphic type by reference (#2824) 2021-03-30 13:35:45 -07:00
Gautham B A 50b1e1c0c4
HDFS-15928. Replace RAND_pseudo_bytes in rpc_engine.cc (#2825) 2021-03-30 10:37:10 -07:00
Viraj Jasani 90fdd04f54
HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:22:54 +05:30
Gautham B A 6a3e135316
HDFS-15922. Use memcpy for copying non-null terminated string. (#2827) 2021-03-29 10:38:59 -07:00
Harunobu Daikoku ea6595d3b6
HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
 2021-03-29 11:43:29 +09:00
Ayush Saxena cd2501e54b HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun. 2021-03-28 16:13:59 +05:30
litao 72037a63b1
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2748) 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-03-27 17:57:09 +09:00
Gautham B A 1fed18bb2d
HDFS-15922. Use memcpy for copying non-null terminated string in jni_helper.c (#2818) 2021-03-26 09:26:35 -07:00
Akira Ajisaka af1f9f43ea
HADOOP-17133. Implement HttpServer2 metrics (#2145) 2021-03-25 12:09:43 -07:00
He Xiaoqiao ed5814deff
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:24:14 +08:00
Gautham B A 654555783d
HDFS-15918. Replace deprecated RAND_pseudo_bytes (#2811) 2021-03-24 10:52:33 -07:00
touchida 95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 2021-03-24 16:56:09 +08:00
Takanobu Asanuma b4d97a8dc7 HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 2021-03-24 17:51:35 +09:00
Viraj Jasani 4b4ccce02f
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2794) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 22:47:45 -07:00
Ayush Saxena 569e407f64 HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell. 2021-03-24 02:10:02 +05:30
Gautham B A 714427c414
HDFS-15910. Improve security with explicit_bzero (#2793) 2021-03-23 09:58:19 -07:00
Jack Jiang 9c2a712597
HDFS-15913. Remove useless NULL checks before instanceof (#2805) 2021-03-23 08:51:35 -07:00
Narges Shadab d77c7ab4e0
HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790). Contributed by Narges Shadab. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 17:06:58 +05:30
Gautham B A 61bfa42389
HDFS-15903. Refactor X-Platform lib (#2783) 2021-03-22 08:40:54 -07:00
litao 343ce8a8c9
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788) 2021-03-22 20:29:17 +09:00
He Xiaoqiao 03a3a04c06
HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-22 18:37:00 +08:00
Viraj Jasani 261191cbc0
HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() (#2785) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-19 20:37:44 -07:00
Narges Shadab f5f3fc6fe9
HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736). Contributed by Narges Shadab. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-19 16:36:48 +08:00
Viraj Jasani 9c43b60348
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 2021-03-17 15:12:36 +09:00
Takanobu Asanuma 9ba60c33d8 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 2021-03-17 14:14:26 +09:00
touchida 25390babc7
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-17 12:00:05 +09:00
Hui Fei 0d42104119
HDFS-15898. Test case TestOfflineImageViewer fails (#2779) 2021-03-17 09:11:30 +08:00
Gautham B A 47620f8821
HDFS-15843. Make write cross-platform (#2710) 2021-03-16 09:06:51 -07:00
sunlisheng 7025f39944 HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun. 2021-03-15 11:34:13 +08:00
He Xiaoqiao 970455c917
HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun. 2021-03-14 17:48:18 +08:00
litao e565b05c80
HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754). Contributed by tomscut. 2021-03-14 04:07:31 +05:30
lamberken 6921ec8b40
HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options (#2751). Contributed by Xie Lei. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-13 14:22:11 +08:00
Xiaoyu Yao 7cf5969f0b
HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. (#2764) 2021-03-12 11:27:53 -08:00
Takanobu Asanuma ecd3335187 HDFS-15848. Snapshot Operations: Add debug logs at the entry point. Contributed by Bhavik Patel. 2021-03-12 15:42:15 +09:00
Akira Ajisaka 23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-11 10:56:07 +09:00
Hui Fei 6a55baeee4
HDFS-15875. Check whether file is being truncated before truncate (#2746) 2021-03-10 14:11:29 +08:00
litao 6a922f9896
HDFS-15873. Add namenode address in logs for block report (#2743) 2021-03-08 11:50:51 -05:00
He Xiaoqiao 4a0b7f7ebe
HDFS-15384. Document getLocatedBlocks(String src, long start) of DFSClient only return partial blocks. Contributed by Yang Yun. 2021-03-08 12:18:46 +08:00
tomscut 9cb51bf106 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 2021-03-06 13:40:44 -08:00
litao cdba06e380
HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
 2021-03-03 22:34:02 +09:00
JJiaguodong 8af56de1fa
HADOOP-17560. Fix some spelling errors (#2730) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-03-03 11:41:05 +09:00
zhuqi 1f1a1ef52d
HDFS-15856: Make write pipeline retry times configurable. (#2721). Contributed by Qi Zhu 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-02 13:16:11 +08:00
Konstantin V Shvachko 9501c698f4 HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 2021-03-01 17:17:23 -08:00
litao 32353eb38a
HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker (#2718) 
Authored-by: tomscut <litao@bigo.sg>
 2021-03-01 07:52:59 -08:00
S O'Donnell c3b3b36dee HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 2021-03-01 11:36:41 +00:00
S O'Donnell 9c583f36d5 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 2021-02-24 10:29:36 +00:00
LeonGao 940c780feb
HDFS-15781. Add metrics for how blocks are moved in replaceBlock. (#2704) 2021-02-23 22:29:48 -08:00
Ayush Saxena 647d23b611 HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel. 2021-02-23 19:08:38 +05:30
Takanobu Asanuma a57282540f
HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. (#2714) 2021-02-23 15:12:06 +09:00
Wang Yuxuan 1e32ab7d2f HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
 2021-02-22 18:13:36 +08:00
Wei-Chiu Chuang f8059d0bdf Revert "first commit (#2559)" 
This reverts commit 1b94569d89.
 2021-02-22 18:13:06 +08:00
Wang Yuxuan 1b94569d89
first commit (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
 2021-02-22 02:10:39 -08:00
jianghuazhu 98ffd65d8b
HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
 2021-02-21 23:22:32 -08:00
Ayush Saxena ff84a57483 HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun. 2021-02-20 19:48:00 +05:30
Ayush Saxena 2ce5752fa8 HDFS-15839. RBF: Cannot get method setBalancerBandwidth on Router Client. Contributed by Yang Yun. 2021-02-20 19:34:29 +05:30
Takanobu Asanuma 25af8901a9 HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel. 2021-02-19 14:44:58 +09:00
Akira Ajisaka 9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)" 
This reverts commit 4cf3531583.
 2021-02-19 11:09:10 +09:00
Hui Fei 2970bd93f3
HDFS-15830. Support to make dfs.image.parallel.load reconfigurable (#2694) 2021-02-19 09:07:22 +08:00
Ayush Saxena e391844e8e HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. 2021-02-18 01:39:04 +05:30
Akira Ajisaka 0ed9e36282
HDFS-15834. Remove the usage of org.apache.log4j.Level (#2696) 2021-02-17 01:59:24 -05:00
Akira Ajisaka 4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454) 
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
 2021-02-17 10:38:20 +09:00
Akira Ajisaka 2b3c5b1733
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 (#2702) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-02-16 14:50:50 +09:00
Zehao Chen 07a4220cd2
HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 2021-02-15 11:14:32 -06:00
litao c3134ab3a9
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics (#2676) 2021-02-14 00:12:24 -05:00
Steve Loughran 798df6d699
HADOOP-13327 Output Stream Specification. (#2587) 
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.

The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.

Contributed by Steve Loughran.
 2021-02-10 10:28:59 +00:00
LeonGao 9434c1eccc
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 2021-02-09 10:49:28 +00:00
bshashikant 2df2dfb9ed
HDFS-15817. Rename snapshots while marking them deleted. (#2677) 2021-02-09 10:27:51 +05:30
LeonGao 0e2b3086e3
HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. (#2625) 2021-02-08 11:01:21 -08:00
bshashikant 19ae0faacc
HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit (#2682) 2021-02-06 08:26:12 -08:00
Jim Brennan c4918fb298 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 21:20:30 +00:00
fermi 7ef2875c89 HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 2021-02-05 11:01:49 +08:00
maobaolong b99dba38b9
HDFS-15819. Fix import style #2681 2021-02-05 10:11:27 +08:00
Hui Fei 15a1f7adfc HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 2021-02-04 09:19:51 +08:00
Kihwal Lee 182623d2bc HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross. 2021-02-03 08:55:50 -06:00
Hui Fei 66ecee333e HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang 2021-02-03 20:00:45 +08:00
YaYun-Wang 394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type (#2377) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-02-02 22:44:34 -08:00
crossfire 18978f2e20
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 2021-02-02 09:02:09 +00:00
He Xiaoqiao 1b893e10ae
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-02 13:50:54 +08:00
belugabehr 21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
He Xiaoqiao b54134661b
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C. 2021-02-01 23:28:29 +08:00
Narges Shadab 115623a6ee
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. (#2652) 2021-02-01 01:56:59 -08:00
Siyao Meng 1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer (#2645) 2021-02-01 13:42:44 +09:00
Gautham B A ad483fd66e
HDFS-15740. Add x-platform utilities (#2567) 2021-01-29 10:11:38 -08:00
Jim Brennan f565476939 [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 2021-01-28 18:17:15 +00:00
Jim Brennan 6184ab07f6 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 2021-01-28 16:47:52 +00:00
sunlisheng f8769e0f4b HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun. 2021-01-28 10:10:39 +08:00
Ayush Saxena e40f99f6d5 HDFS-15767. RBF: Router federation rename of directory. Contributed by Jinglun. 2021-01-26 14:25:27 +05:30
Akira Ajisaka 45b3a84709
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 2021-01-21 10:07:46 +09:00
He Xiaoqiao 97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-16 14:06:56 +08:00
LeonGao 7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount (#2583). Contributed by Leon Gao. 2021-01-15 16:28:11 -08:00
Masatake Iwasaki 630f8ddd2c
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-14 16:19:35 +09:00
touchida 8ec824f2ba
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-01-13 11:23:07 +09:00
Yiqun Lin 0d7ac54510 HDFS-14558. RBF: Isolation/Fairness documentation. Contributed by Fengnan Li. 2021-01-12 23:38:55 +08:00
Masatake Iwasaki 32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. (#2507) 2021-01-11 11:59:32 -08:00
He Xiaoqiao ec22850dbe
Make upstream aware of 3.2.2 release. 2021-01-09 18:06:08 +08:00
Ahmed Hussein 77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. (#2601). Contributed by Ahmed Hussein and Daryn Sharp. 2021-01-08 13:10:09 -06:00
Ayush Saxena 2ba7ec2b48 HDFS-15766. RBF: MockResolver.getMountPoints() breaks the semantic of FileSubclusterResolver. Contributed by Jinglun. 2021-01-08 16:27:54 +05:30
lfengnan 87bd4d2aca
HDFS-15754. Add DataNode packet metrics (#2578) 
Contributed by Fengnan Li.
 2021-01-07 23:46:23 -08:00
Wei-Chiu Chuang 2b4febcf57
HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 2021-01-04 20:54:44 -08:00
Ayush Saxena 77299ae992 HDFS-15748. RBF: Move the router related part from hadoop-federation-balance module to hadoop-hdfs-rbf. Contributed by Jinglun. 2021-01-05 00:05:03 +05:30
Wei-Chiu Chuang 66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453) 2021-01-04 09:43:58 -08:00
Masatake Iwasaki 2825d060cf
HDFS-15648. TestFileChecksum should be parameterized. (#2501) 2021-01-04 14:00:57 +09:00
Ayush Saxena 1cd96e8dd8 HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. 2021-01-01 11:37:52 +05:30
maobaolong 3b77cf4446
HDFS-15749. Make size of editPendingQ can be configurable (#2572) 2020-12-28 14:18:18 +08:00
hemanthboyina 16a20503ca HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina. 2020-12-22 17:27:31 +05:30
langlaile1221 4ae561bcb4
HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType (#2566) 2020-12-22 09:25:11 +08:00
Ayush Saxena ba496f61f8 HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina. 2020-12-22 02:16:00 +05:30
Ayush Saxena 5ab1db9e21 HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut. 2020-12-21 09:58:30 +05:30
Ayush Saxena 2aea43bf4f HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun. 2020-12-20 23:06:19 +05:30
Ayush Saxena 3ecd3628d4 HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang. 2020-12-20 15:18:25 +05:30
Ayush Saxena 70789fc6b9 HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao. 2020-12-20 15:12:32 +05:30
Ahmed Hussein c2672bb234
HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-17 11:33:29 -06:00
Ayush Saxena 0da00fc743 HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-12-14 20:42:23 -08:00
liuyanpunk 3363cc380b HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.(#2534) 2020-12-15 10:06:11 +05:30
S O'Donnell 9ed737001c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 2020-12-11 20:53:10 +00:00
Ahmed Hussein be35fa186c
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 2020-12-11 11:02:42 -06:00
Neil 9bd3c9bc50
HDFS-15720 namenode audit async logger should add some log4j config (#2532) 2020-12-10 13:47:00 -08:00
Ahmed Hussein 3ec01b1bb3
HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 13:09:53 -06:00
Ahmed Hussein 01383a2172
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 2020-12-08 13:25:24 -06:00