hadoop/hadoop-tools
Mehakmeet Singh acffe203b8
HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446)
Addresses the problem of processes running out of memory when
there are many ABFS output streams queuing data to upload,
especially when the network upload bandwidth is less than the rate
data is generated.

ABFS Output streams now buffer their blocks of data to
"disk", "bytebuffer" or "array", as set in
"fs.azure.data.blocks.buffer"

When buffering via disk, the location for temporary storage
is set in "fs.azure.buffer.dir"

For safe scaling: use "disk" (default); for performance, when
confident that upload bandwidth will never be a bottleneck,
experiment with the memory options.

The number of blocks a single stream can have queued for uploading
is set in "fs.azure.block.upload.active.blocks".
The default value is 20.

Contributed by Mehakmeet Singh.
2021-09-21 12:48:06 +01:00
..
hadoop-aliyun HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-archive-logs HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-archives HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
hadoop-aws HADOOP-17871. S3A CSE: minor tuning (#3412) 2021-09-15 22:29:22 +01:00
hadoop-azure HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446) 2021-09-21 12:48:06 +01:00
hadoop-azure-datalake HADOOP-13327 Output Stream Specification. (#2587) 2021-02-10 10:28:59 +00:00
hadoop-datajoin HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-distcp HADOOP-17628. Distcp contract test is really slow with ABFS and S3A; timing out. (#3240) 2021-08-02 11:36:43 +01:00
hadoop-dynamometer HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
hadoop-extras Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-federation-balance HDFS-16087. Fix stuck issue in rbfbalance tool (#3141). Contributed by Eric Yin. 2021-07-21 00:01:55 +08:00
hadoop-fs2img HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-gridmix HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 2021-04-06 13:39:10 +09:00
hadoop-kafka HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
hadoop-openstack HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-pipes Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-resourceestimator Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-rumen HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 2021-04-06 13:39:10 +09:00
hadoop-sls YARN-10878. move TestNMSimulator off com.google (#3268) 2021-08-05 11:34:10 +01:00
hadoop-streaming HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 2021-03-11 10:56:07 +09:00
hadoop-tools-dist HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
pom.xml HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00