Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,986 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

25986 Commits

Author SHA1 Message Date
Ashutosh Gupta 9c3330c22f
MAPREDUCE-7377. Remove unused imports in MapReduce project (#4299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:34:19 +09:00
Jian Chen d2c9eb653a
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases (#4110) 
Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-14 01:11:42 +09:00
Szilard Nemeth eced5bec68 YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth 2022-05-13 13:16:12 +02:00
Hui Fei dad2900b87
Revert "HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199)" (#4306) 2022-05-13 14:44:41 +08:00
Felix Nguyen 680af27aa6
HDFS-14750. RBF: Support dynamic handler allocation in routers (#4199) 2022-05-13 14:28:53 +08:00
Szilard Nemeth 5e2f4339fa YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-12 13:42:06 +02:00
singer-bin d74f328b00
HDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmin class (#4122) 2022-05-12 16:53:09 +08:00
Lei Yang 6a95c3a039
HADOOP-18193:Support nested mount points in INodeTree 
Fixes #4181

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-05-11 17:01:21 -07:00
Gautham B A 1350539f2d
HDFS-16465. Remove redundant strings.h inclusions (#4279) 
* Remove redundant strings.h inclusions

* strings.h was included in a bunch of
  C/C++ files and were redundant.
* Also, strings.h is not available on
  Windows and thus isn't cross-platform
  compatible.

* Build for all platforms in CI

* Revert "Build for all platforms in CI"

This reverts commit 2650f047bd6791a5908cfbe50cc8e70d42c512cb.

* Debug failure on Centos 8

* Skipping pipeline run on
  Centos 7 to debug the
  failure on Centos 8.

* Revert "Debug failure on Centos 8"

This reverts commit e365e34d6f.
 2022-05-11 23:04:22 +05:30
Szilard Nemeth 9af3eabdca YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth 2022-05-11 18:01:31 +02:00
Szilard Nemeth f143e99428 MAPREDUCE-7379. RMContainerRequestor#makeRemoteRequest has confusing log message. Contributed by Ashutosh Gupta 2022-05-11 16:55:19 +02:00
Szilard Nemeth eeedc0c7e4 YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke 2022-05-11 14:39:42 +02:00
Szilard Nemeth b62d6ce6fd YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori 2022-05-11 14:29:05 +02:00
slfan1989 09a69f996b
YARN-11130. removed unused import (#4276) 2022-05-11 12:14:52 +05:30
Viraj Jasani 6983aef945
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-05-11 08:10:03 +08:00
hchaverr 99a83fd4bd
HADOOP-18222. Prevent DelegationTokenSecretManagerMetrics from registering multiple times 
Fixes #4266

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-05-10 13:58:39 -07:00
daimin d486ae8c0f
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4291) 2022-05-10 08:01:16 +08:00
Ayush Saxena 665ada6d21
MAPREDUCE-7376. AggregateWordCount fetches wrong results. (#4257). Contributed by Ayush Saxena. 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2022-05-09 22:56:14 +05:30
Gautham B A a394c2b031
HDFS-16572. Fix typo in readme of hadoop-project-dist 2022-05-08 23:47:13 +05:30
Ashutosh Gupta 40a8b9a6a5
HADOOP-17479. Fix the examples of hadoop config prefix (#4197) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 08:09:24 +09:00
Ashutosh Gupta 01027e52a9
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 08:05:34 +09:00
Ashutosh Gupta ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-08 07:38:32 +09:00
daimin 29401c8203
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-07 01:30:32 +08:00
Gautham B A d346be9bd2
HDFS-16564. Use uint32_t for hdfs_find (#4245) 
* hdfs_find uses u_int32_t type for
  storing the value for the max-depth
  command line argument.
* The type u_int32_t isn't standard,
  isn't available on Windows and thus
  breaks cross-platform compatibility.
  We need to replace this with uint32_t
  which is available on all platforms since
  it's part of the C++ standard.
 2022-05-04 20:19:08 +05:30
Daniel Carl Jones 4230162a76
HADOOP-18168. Fix S3A ITestMarkerTool use of purged public bucket. (#4140) 
This moves off use of the purged s3a://landsat-pds bucket, so fixing tests
which had started failing.
* Adds a new class, PublicDatasetTestUtils to manage the use of public datasets.
* The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon
  HADOOP-14661.

Consult the updated test documentation when running against other S3 stores.

Contributed by Daniel Carl Jones

Change-Id: Ie8585e4d9b67667f8cb80b2970225d79a4f8d257
 2022-05-03 14:28:08 +01:00
Steve Loughran 6ec39d45c9 Revert "HADOOP-18168. . (#4140)" 
This reverts commit 6ab7b72cd6.
 2022-05-03 14:27:52 +01:00
Daniel Carl Jones 6ab7b72cd6
HADOOP-18168. . (#4140) 
This moves off use of the purged s3a://landsat-pds bucket, so fixing tests
which had started failing.
* Adds a new class, PublicDatasetTestUtils to manage the use of public datasets.
* The new test bucket s3a://usgs-landsat/ is requester pays, so depends upon
  HADOOP-14661.

Consult the updated  test documentation when running against other S3 stores.

Contributed by Daniel Carl Jones
 2022-05-03 14:26:52 +01:00
Steve Loughran aa917b1cd4
HADOOP-18198. add -mvnargs option to create-release command line (#4239) 
This allows for builds to be run with options like
--mvnargs="-Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false"

Contributed by Ayush Saxena.

Change-Id: I396e82d0915d679657d063a948f865041bcdde29
 2022-05-03 14:08:39 +01:00
Murali Krishna 9807dfd25c
HDFS-16562. Upgrade moment.min.js to 2.29.2 (#4240) 2022-05-03 17:42:45 +05:30
Viraj Jasani 2dfa928a20
HDFS-16521. DFS API to retrieve slow datanodes (#4107) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-05-03 05:05:40 +08:00
jteagles d4a91bd0c0
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class (#4242) 
Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-03 03:14:26 +09:00
Ashutosh Gupta a74acc755e
YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. (#4222) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-05-03 03:08:30 +09:00
Akira Ajisaka 4653fcd704
HADOOP-18219. Fix shaded client test failure (#4254) 
This patch MUST be cherrypicked into all branches with
HADOOP-15983 Use jersey-json that is built to use jackson2

Contributed by Akira Ajisaka
 2022-05-02 12:25:09 +01:00
Ashutosh Gupta 23e1511cd0
HDFS-16256. Minor fix in HDFS Fedbalance document (#4192) 2022-05-02 08:08:12 +08:00
Gautham B A 88155cebe9
HDFS-16468. Define ssize_t for Windows (#4228) 
* Some C/C++ files use ssize_t data type.
  This isn't available for Windows and we
  need to define an alias for this and set it
  to an appropriate type to make it cross
  platform compatible.
 2022-04-29 22:25:09 +05:30
smarthan 7bd7725532
HDFS-16553. Fix checkstyle for the length of BlockManager construction method over limit. (#4211). Contributed by Chengwei Wang. 2022-04-29 22:59:23 +08:00
Viraj Jasani b9ade7a4cd
HDFS-16528. Reconfigure slow peer enable for Namenode (#4186) 2022-04-29 17:00:17 +08:00
stack 9ed8d60511 HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
This reverts the previous commit 4e47eb66d1
undone so I could reapply with the '.' after the HDFS-16540 as is done
in all other commits.
 2022-04-28 10:28:01 -07:00
stack 4e47eb66d1 Revert "HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)" 
Revert to add the '.' after HDFS-16540 so commit message format matches
precedent

This reverts commit bda0881bf9.
 2022-04-28 10:26:46 -07:00
huaxiangsun bda0881bf9
HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170) 
When DN with the same UUID is registered with a different IP, host2DatanodeMap needs to be updated accordingly.
 2022-04-28 10:24:37 -07:00
PJ Fanning 63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-28 14:18:19 +09:00
Felix Nguyen b4ff49a394
HDFS-16539. RBF: Support refreshing/changing router fairness policy controller without rebooting router (#4168) 2022-04-27 14:42:30 +08:00
smarthan f187e9bcd5
HDFS-16554. Remove unused configuration dfs.namenode.block.deletion.increment. (#4213). Contributed by Chengwei Wang. 2022-04-27 11:44:38 +08:00
liever18 2632d49282
HADOOP-18216. io.file.buffer.size must be positive (#4220) 
Documents that the buffer size must be > 0

Contributed by Jingxuan Fu.
 2022-04-26 19:53:11 +01:00
hchaverri d60262fe00
HADOOP-18167. Add metrics to track delegation token secret manager op… (#4092) 
* HADOOP-18167. Add metrics to track delegation token secret manager operations
 2022-04-26 16:20:11 +00:00
litao f1e5f8e764
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4234) 2022-04-26 11:48:06 +08:00
Ashutosh Gupta 006767eb94
YARN-10303. Fixing rest api example, status code error and other typos in ResourceManagerRest.md (#4221) 2022-04-26 09:51:11 +08:00
Ashutosh Gupta fb13c1e4a8
MAPREDUCE-7246. In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, updating the datatype of appId to "string". (#4223) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-25 14:29:35 +09:00
Ashutosh Gupta 89279f0b90
HDFS-16255. RBF: Fix dead link to fedbalance document (#4193) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-25 14:11:41 +09:00
Steve Loughran 44ae2fa8e5
HADOOP-16202. Enhanced openFile(): hadoop-azure changes. (#2584/4) 
Stops the abfs connector warning if openFile().withFileStatus()
is invoked with a FileStatus is not an abfs VersionedFileStatus.

Contributed by Steve Loughran.

Change-Id: I85076b365eb30aaef2ed35139fa8714efd4d048e
 2022-04-24 17:33:05 +01:00