Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,861 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

19861 Commits

Author SHA1 Message Date
Akira Ajisaka 3e2534c032
HADOOP-16748. Migrate to Python 3 and upgrade Yetus to 0.13.0 (branch-3.2) (#2717) 
- Upgrade Yetus to 0.13.0 to support Python 3 for the release scripts.
- Removed determine-flaky-tests-hadoop.py.
- Temporarily disabled shelldocs check due to YETUS-1099.

Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
(cherry picked from commit b9b49ed956)

 Conflicts:
	dev-support/Jenkinsfile
	dev-support/docker/Dockerfile

(cherry picked from commit ff8b79f890)
(cherry picked from commit 08652eeee7)
 2021-03-27 00:33:16 +09:00
Akira Ajisaka ce08b9d2eb
HADOOP-16061. Upgrade Yetus to 0.10.0 
This closes #1298

(cherry picked from commit 5e156b9dde)
(cherry picked from commit 43b8396b77)
 2021-03-27 00:33:00 +09:00
Akira Ajisaka db94908212
HADOOP-16054. Update Dockerfile to use Bionic (#2713) 
(cherry picked from commit 81d8b71534)

 Conflicts:
	dev-support/docker/Dockerfile

(cherry picked from commit fa5c9a1886)
 2021-03-27 00:19:55 +09:00
Akira Ajisaka a8ddc179df
HADOOP-17602. Upgrade JUnit to 4.13.1. Contributed by Ahmed Hussein. 
(cherry picked from commit c5929c00a3)

 Conflicts:
	hadoop-cloud-storage-project/hadoop-huaweicloud/pom.xml

(cherry picked from commit 80d4636067)

 Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit 819239f971)
 2021-03-25 20:29:18 +09:00
He Xiaoqiao b33e3083ec
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:32:34 +08:00
Viraj Jasani a0dd4e0722
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2799) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-24 10:19:56 -07:00
Akira Ajisaka 332c2a637d
YARN-10036. Install yarnpkg and upgrade nodejs in Dockerfile (#1772) 
(cherry picked from commit bc825b9628)
(cherry picked from commit 710a24b6cd)

 Conflicts:
	dev-support/docker/Dockerfile
 2021-03-24 16:29:33 +09:00
Kevin Su 65b1945f22
HADOOP-16754. Fix docker failed to build yetus/hadoop. 
Contributed by Kevin Su.

Change-Id: Ie9cf27b7a7ba5923405f24d39b95b0248f3718b7
(cherry picked from commit 0f25cbbb25)
 2021-03-24 16:25:07 +09:00
Allen Wittenauer 1bcde89983
HADOOP-14671. Upgrade Apache Yetus to 0.8.0 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4b8c2b1c1c)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-common-project/hadoop-common/src/site/markdown/release/2.10.0/RELEASENOTES.2.10.0.md
	hadoop-common-project/hadoop-common/src/site/markdown/release/3.0.2/CHANGELOG.3.0.2.md
	hadoop-common-project/hadoop-common/src/site/markdown/release/3.1.2/RELEASENOTES.3.1.2.md
 2021-03-24 16:04:11 +09:00
Jim Brennan 34e507cb8c YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 2021-03-23 18:57:23 +00:00
litao 75f2d47ff5 HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2800) 
(cherry picked from commit d05d15620e)
 2021-03-23 18:46:19 +09:00
Jim Brennan c1b25422a5 MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein 
(cherry picked from commit ede490d131)
 2021-03-22 19:28:23 +00:00
Ayush Saxena c9c471e04e HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58)
 2021-03-22 13:10:02 +00:00
Mingliang Liu f2de8cc5cf
YARN-10706. Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2 (#2791) 
Contributed by Mingliang Liu.

Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-21 23:03:33 -07:00
kwangsun 7af7eb3e9f
HADOOP-17952. Fix the wrong CIDR range example in Proxy User documentation. (#2780) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c8d327a4f1)
 2021-03-22 11:46:44 +09:00
Ayush Saxena 30a1b17b22 HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad)
 2021-03-19 19:50:19 +00:00
Ayush Saxena 8354ce680b HDFS-15633. Avoid redundant RPC calls for getDiskStatus. (#2386). Contributed by Ayush Saxena. 2021-03-19 23:14:37 +05:30
Ayush Saxena 162923e799 HADOOP-17594. DistCp: Expose the JobId for applications executing through run method (#2786). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2021-03-19 14:21:02 +05:30
Ayush Saxena 8ec256db04 HADOOP-17310. Touch command with -c option is broken. (#2393). Contributed by Ayush Saxena. 2021-03-19 00:14:11 +05:30
Viraj Jasani 503101c1f1 HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348)
 2021-03-17 15:14:29 +09:00
Takanobu Asanuma b898058e45 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8)
 2021-03-17 14:18:54 +09:00
touchida b4c07209fc
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7)
 2021-03-17 13:12:40 +09:00
Jim Brennan a92300e0a9 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein. 
(cherry picked from commit 38f86cc8c7)
 2021-03-16 19:19:01 +00:00
Eric Payne 2439e07708 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7)
 2021-03-15 19:55:13 +00:00
He Xiaoqiao 0224dd87af HADOOP-17585. Correct timestamp format in the docs for the touch command. Contributed by Stephen O'Donnell. 
(cherry picked from commit b1dc6c40a0)
 2021-03-14 15:20:16 +00:00
Ayush Saxena 77aedb7d4c HADOOP-17586. Upgrade org.codehaus.woodstox:stax2-api to 4.2.1. (#2769). Contributed by Ayush Saxena. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 Conflicts:
	LICENSE-binary
 2021-03-13 15:20:50 +05:30
Jim Brennan a6c2fced53 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f3)
 2021-03-12 17:47:43 +00:00
Viraj Jasani 6ecf829e79
HADOOP-17571 : Bump up woodstox-core to 5.3.0 due to security concerns (#2757) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-03-11 12:28:02 -08:00
Hui Fei 7250f0e327 HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee4)
 2021-03-11 10:45:57 +08:00
Akira Ajisaka be253c2fd0
HADOOP-17557. skip-dir option is not processed by Yetus. Contributed by Ahmed Hussein. 
(cherry picked from commit 63aa142b77)
 2021-03-09 11:11:32 +09:00
tomscut 7b925bc486 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106)
 2021-03-06 14:18:20 -08:00
Akira Ajisaka 5dda43c061
HADOOP-17546. Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md. Contributed by Ravuri Sushma sree. 
(cherry picked from commit 9fd2198daa)
 2021-03-04 14:57:41 +09:00
litao c503130d8b HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380)
 2021-03-03 22:38:04 +09:00
Eric Badger 14dea40154 [MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file. 
Contributed by Eric Badger.

(cherry picked from commit a17dc4e821)
 2021-03-02 21:50:12 +00:00
Konstantin V Shvachko 78bd68a0b8 HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4)
 2021-03-01 17:27:38 -08:00
S O'Donnell 53f46214e4 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java

(cherry picked from commit 707806092f)
 2021-03-01 12:26:41 +00:00
Jim Brennan 4d323cd47c MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 21:39:04 +00:00
Eric Badger 5862543c89 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec)
 2021-02-26 19:36:55 +00:00
Surendra Singh Lilhore 379a7a5726 HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit db6252b6c3)
(cherry picked from commit 7856af2cc65377352db5ebc93a778f373f71d215)
(cherry picked from commit 5315de3d437f78d81dc5fa9f43849de6198220e6)
 2021-02-26 10:49:59 +08:00
Jonathan Hung eb6c08e423 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
(cherry picked from commit be6e99963d)
(cherry picked from commit 6863a5bb8a)
 2021-02-25 15:35:01 -08:00
Jim Brennan 0a45b5460d [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit 3795f66364)
 2021-02-25 20:09:27 +00:00
S O'Donnell bf229135ca HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5)
 2021-02-24 11:33:43 +00:00
Renukaprasad C 2cf469eb74 HADOOP-17510. Hadoop prints sensitive Cookie information. (#2673) 
(cherry picked from commit 924fa87010)
(cherry picked from commit 9951ea9fa8)
 2021-02-24 17:34:26 +08:00
Eric Badger ea73b6ad5c YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3ed)
 2021-02-22 19:15:51 +00:00
Xieming Li d6797520b9
HADOOP-16052. Remove Subversion and Forrest from Dockerfile 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit bdbca0e53b)
 2021-02-22 23:34:17 +09:00
Akira Ajisaka 4cbab66547
MAPREDUCE-7323. Remove job_history_summary.py. (#2712) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655)
 2021-02-22 23:16:53 +09:00
Wang Yuxuan 09bbbc8b86 HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f)
(cherry picked from commit 52a46f064a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
(cherry picked from commit fe822b525e)
 2021-02-22 18:16:12 +08:00
jianghuazhu cab9172d2e HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b)
(cherry picked from commit ee0e32abdc)
(cherry picked from commit efb796f45f)
 2021-02-22 15:25:08 +08:00
Kengo Seki 7bcb1a83a3
HADOOP-16764. Rewrite Python example codes using Python3 (#1762) 
(cherry picked from commit fd7de2b82a)
 2021-02-20 19:16:07 +09:00
Eric Badger 9327039fb8 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b)
 2021-02-19 23:44:28 +00:00