Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,004 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

26004 Commits

Author SHA1 Message Date
Dongjoon Hyun f5148ca542
MAPREDUCE-7363. Rename JobClientUnitTest to TestJobClients (#3487) 2021-09-28 09:50:01 -07:00
huhaiyang 1f8f53f7d7
HDFS-16231. Fix TestDataNodeMetrics#testReceivePacketSlowMetrics (#3471) 2021-09-28 13:09:18 +08:00
jianghuazhu 02993bc11e
HDFS-16237.Record the BPServiceActor information that communicates with Standby. (#3479) 2021-09-28 13:07:36 +08:00
Masatake Iwasaki 86e80752d1
HDFS-16240. Replace unshaded guava in HttpFSServerWebServer. (#3492) 2021-09-28 13:35:35 +09:00
Takanobu Asanuma 2068b0041c
HADOOP-17940. Upgrade Kafka to 2.8.1 (#3488) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-09-28 13:21:55 +09:00
Chao Sun 7ed949600d
HADOOP-17936. Fix test failure after reverting HADOOP-16878 (#3482) 2021-09-27 13:56:22 -07:00
Gautham B A ae6cd4cea5
HADOOP-17931. Fix typos in usage message in winutils.exe (#3490) 2021-09-27 13:41:55 -07:00
Inigo Goiri 1e8c25d528
Revert "Fix winutils typos (#3484)" (#3489) 
Reverting to include JIRA number HADOOP-17931.
 2021-09-27 09:40:47 -07:00
Gautham B A 0f3734a99e
Fix winutils typos (#3484) 2021-09-27 09:37:45 -07:00
Dongjoon Hyun b79cacf97b
HADOOP-17939. Support building on Apple Silicon (#3486) 
Support building on Apple Silicon with ARM CPUs by using the x86_64 version of protoc.

Contributed by  Dongjoon Hyun
 2021-09-27 13:27:10 +01:00
Viraj Jasani 9d44f503db
HADOOP-17910. [JDK 17] TestNetUtils fails (#3481) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-27 10:12:47 +09:00
Chao Sun 1397cf2496 Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same" 
This reverts commit 59f01a548e.
 2021-09-25 21:32:43 -07:00
Chao Sun 2ee294b1b1 Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko." 
This reverts commit 7bc305db5d.
 2021-09-25 09:29:33 -07:00
Renukaprasad C 4c516536be
HDFS-16236. Example command for daemonlog is not correct (#3476) 2021-09-25 18:32:52 +08:00
AngersZhuuuu 5f9321a5d4
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 2021-09-25 18:17:44 +08:00
Wei-Chiu Chuang 87632bbacf
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 2021-09-24 08:35:23 -07:00
litao 3113a119af
HDFS-16232. Fix java doc for BlockReaderRemote#newBlockReader (#3456) 2021-09-24 13:14:25 +08:00
Ayush Saxena 98e2781dde
HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov. 2021-09-23 10:45:02 +05:30
Viraj Jasani 329b95b620
HADOOP-17892. Add Hadoop code formatter in dev-support (#3387) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Gautham B A <gautham.bangalore@gmail.com>
 2021-09-23 12:34:19 +08:00
Gautham B A 35a8d48872
HDFS-16205. Make hdfs_allowSnapshot tool cross platform (#3388) 2021-09-22 10:29:38 -07:00
Liang-Chi Hsieh 138add2cb2
HADOOP-17868. Add more tests for BuiltInGzipCompressor (#3336) 2021-09-22 07:59:28 -07:00
Viraj Jasani c7e7b2f907
HDFS-16219. RBF: Set default map tasks and bandwidth in RouterFederationRename (#3452) 2021-09-22 15:34:51 +09:00
Rintaro Ikeda 962068d2d8
HADOOP-17926. Maven-eclipse-plugin is no longer needed since Eclipse can import Maven projects by itself. (#3465) 2021-09-22 15:26:38 +09:00
LeonGao b1431813d0
HDFS-10648. Expose Balancer metrics through Metrics2 (#3427) 2021-09-21 12:26:17 -07:00
Tamas Domok 8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-21 16:13:04 +02:00
9uapaw 9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori 2021-09-21 16:08:24 +02:00
Mehakmeet Singh acffe203b8
HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446) 
Addresses the problem of processes running out of memory when
there are many ABFS output streams queuing data to upload,
especially when the network upload bandwidth is less than the rate
data is generated.

ABFS Output streams now buffer their blocks of data to
"disk", "bytebuffer" or "array", as set in
"fs.azure.data.blocks.buffer"

When buffering via disk, the location for temporary storage
is set in "fs.azure.buffer.dir"

For safe scaling: use "disk" (default); for performance, when
confident that upload bandwidth will never be a bottleneck,
experiment with the memory options.

The number of blocks a single stream can have queued for uploading
is set in "fs.azure.block.upload.active.blocks".
The default value is 20.

Contributed by Mehakmeet Singh.
 2021-09-21 12:48:06 +01:00
Neil ae2c5ccfcf
HADOOP-17893. Improve PrometheusSink for Namenode TopMetrics (#3426) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-21 10:43:50 +09:00
Viraj Jasani feee41aa00
HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink (#3386) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-21 10:11:20 +09:00
Szilard Nemeth 4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth 2021-09-20 16:47:46 +02:00
Tamas Domok f93e8fbf2d
HDFS-16129. Fixing the signature secret file misusage in HttpFS. Contributed by Tamas Domok 
* HDFS-16129. Fixing the signature secret file misusage in HttpFS.

The signature secret file was not used in HttpFs.
 - if the configuration did not contain the deprecated
httpfs.authentication.signature.secret.file option then it
used the random secret provider
 - if both option (httpfs. and hadoop.http.) was set then
the HttpFSAuthenticationFilter could not read the file
because the file path was not substituted properly

!NOTE! behavioral change: the deprecated httpfs. configuration
values are overwritten with the hadoop.http. values.

The commit also contains a follow up change to the YARN-10814,
empty secret files will result in a random secret provider.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-20 14:29:50 +02:00
Thomas Leplus f92c6750ec
HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. (#3455). Contributed by Thomas Leplus. 2021-09-20 00:09:05 +05:30
Adam Antal a9b2469a53
YARN-10950. Code cleanup in QueueCapacities (#3454) 2021-09-19 14:42:02 +02:00
TiborKovacsCloudera 5dcb23c1f0
YARN-10937. Fix log message arguments in LogAggregationFileController (#3450) 2021-09-19 14:35:34 +02:00
Szilard Nemeth aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class (#3420) 2021-09-19 13:11:56 +02:00
Gautham B A d6eb077247
HADOOP-17902. Fix Hadoop build on Debian 10 (#3408) 2021-09-18 11:51:33 -07:00
Viraj Jasani c9763a99c1
HDFS-16227. De-flake TestMover#testMoverWithStripedFile (#3429) 2021-09-18 19:02:45 +08:00
jianghuazhu cd5c6395e8
HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. (#3451). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-18 11:06:51 +05:30
Rintaro Ikeda 607c20c612
HADOOP-17919. Fix command line example in Hadoop Cluster Setup documentation. (#3453) 2021-09-17 22:24:44 +09:00
Steve Loughran 5ebcd4bb92
HADOOP-17126. implement non-guava Precondition checkNotNull 
This adds a new class org.apache.hadoop.util.Preconditions which is

* @Private/@Unstable
* Intended to allow us to move off Google Guava
* Is designed to be trivially backportable
  (i.e contains no references to guava classes internally)

Please use this instead of the guava equivalents, where possible.

Contributed by: Ahmed Hussein

Change-Id: Ic392451bcfe7d446184b7c995734bcca8c07286e
 2021-09-17 11:06:13 +01:00
Viraj Jasani 7c25a77911
HDFS-16217. RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources (#3434) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-17 16:39:29 +09:00
LeonGao deb52cdcbb
HDFS-16224. testBalancerWithObserverWithFailedNode times out (#3425) 2021-09-16 23:59:36 -07:00
litao 71a601241c
HADOOP-17914. Print RPC response length in the exception message (#3436) 2021-09-17 14:45:14 +08:00
adamantal f5c76c8e31 YARN-10936. Log typo corrected. (#3449) 
Contributed by: Tibor Kovacs
 2021-09-17 08:27:51 +02:00
Liang-Chi Hsieh a424878e0e
HADOOP-17891. Fix compilation error under skipShade (ADDENDUM) (#3441) 2021-09-16 09:58:28 -07:00
Gautham B A 16ca362564
HADOOP-17913. Filter deps with release labels (#3437) 2021-09-16 09:18:58 -07:00
lfengnan 4d21655d04
HDFS-16218. Use HdfsConfiguration for passing in Router principal (#3444) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-16 14:58:02 +09:00
litao 2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation (#3366) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-09-16 10:00:02 +09:00
Mehakmeet Singh c54bf19978
HADOOP-17871. S3A CSE: minor tuning (#3412) 
This migrates the fs.s3a-server-side encryption configuration options
to a name which covers client-side encryption too.

fs.s3a.server-side-encryption-algorithm becomes fs.s3a.encryption.algorithm
fs.s3a.server-side-encryption.key becomes fs.s3a.encryption.key

The existing keys remain valid, simply deprecated and remapped
to the new values. If you want server-side encryption options
to be picked up regardless of hadoop versions, use
the old keys.

(the old key also works for CSE, though as no version of Hadoop
with CSE support has shipped without this remapping, it's less
relevant)


Contributed by: Mehakmeet Singh
 2021-09-15 22:29:22 +01:00
Steve Loughran 10f3abeae7
Revert "HADOOP-17195. OutOfMemory error while performing hdfs CopyFromLocal to ABFS (#3406)" (#3443) 
This reverts commit 52c024cc3a.
 2021-09-15 22:27:49 +01:00