Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,278 Commits 326 Branches 371 Tags 1.5 GiB
Java 92.7%
C++ 2.9%
C 1.9%
JavaScript 1.2%
Shell 0.5%
Other 0.5%
Go to file
Mehakmeet Singh acffe203b8
HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446) 
Addresses the problem of processes running out of memory when
there are many ABFS output streams queuing data to upload,
especially when the network upload bandwidth is less than the rate
data is generated.

ABFS Output streams now buffer their blocks of data to
"disk", "bytebuffer" or "array", as set in
"fs.azure.data.blocks.buffer"

When buffering via disk, the location for temporary storage
is set in "fs.azure.buffer.dir"

For safe scaling: use "disk" (default); for performance, when
confident that upload bandwidth will never be a bottleneck,
experiment with the memory options.

The number of blocks a single stream can have queued for uploading
is set in "fs.azure.block.upload.active.blocks".
The default value is 20.

Contributed by Mehakmeet Singh.
 		2021-09-21 12:48:06 +01:00
.github HADOOP-17799. Improve the GitHub pull request template (#3277) 2021-08-14 21:16:15 +09:00
dev-support HADOOP-17902. Fix Hadoop build on Debian 10 (#3408) 2021-09-18 11:51:33 -07:00
hadoop-assemblies HDFS-15346. FedBalance tool implementation. Contributed by Jinglun. 2020-06-18 13:33:25 +08:00
hadoop-build-tools HADOOP-17897. Allow nested blocks in switch case in checkstyle settings. (#3394) 2021-09-08 13:55:48 +09:00
hadoop-client-modules HADOOP-17891. Fix compilation error under skipShade (ADDENDUM) (#3441) 2021-09-16 09:58:28 -07:00
hadoop-cloud-storage-project HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
hadoop-common-project HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446) 2021-09-21 12:48:06 +01:00
hadoop-dist Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
hadoop-hdfs-project HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink (#3386) 2021-09-21 10:11:20 +09:00
hadoop-mapreduce-project MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter (#2500) 2021-09-01 13:47:09 +09:00
hadoop-maven-plugins HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1 (#3152) 2021-06-29 18:32:16 -07:00
hadoop-minicluster HDFS-15331. Remove invalid exclusions that minicluster dependency on HDFS (#1996). Contributed by Wanqiang Ji 2020-05-06 02:06:38 +05:30
hadoop-project HADOOP-17886. Upgrade ant to 1.10.11 (#3371) 2021-09-02 16:11:07 -05:00
hadoop-project-dist Make upstream aware of 3.3.1 release 2021-06-15 15:43:36 +00:00
hadoop-tools HADOOP-17195. ABFS: OutOfMemory error while uploading huge files (#3446) 2021-09-21 12:48:06 +01:00
hadoop-yarn-project YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth 2021-09-20 16:47:46 +02:00
licenses HADOOP-17144. Update Hadoop's lz4 to v1.9.2. Contributed by Hemanth Boyina. 2020-10-18 18:37:46 +05:30
licenses-binary HADOOP-15993. Upgrade Kafka to 2.4.0 in hadoop-kafka module. (#1796) 2020-01-09 16:24:58 +09:00
.asf.yaml HADOOP-17234. Add .asf.yaml to allow Github to Jira integration. (#2253). Contributed by Ayush Saxena. 2020-08-28 17:22:46 +05:30
.gitattributes HADOOP-13598. Add eol=lf for unix format files in .gitattributes. Contributed by Yiqun Lin. 2016-09-14 11:14:31 +09:00
.gitignore YARN-10407. Add phantomjsdriver.log to gitignore. (#2244) 2020-09-01 10:44:55 +09:00
BUILDING.txt HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1 (#3152) 2021-06-29 18:32:16 -07:00
LICENSE-binary HADOOP-17834. Bump aliyun-sdk-oss to 3.13.0 (#3261) 2021-08-14 21:19:10 +09:00
LICENSE.txt HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 (#2330) 2020-09-24 21:15:55 +09:00
NOTICE-binary HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
NOTICE.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
README.txt HADOOP-15958. Revisiting LICENSE and NOTICE files. 2019-08-27 13:47:12 +09:00
pom.xml HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1 (#3152) 2021-06-29 18:32:16 -07:00
start-build-env.sh HADOOP-17298. Backslash in username causes build failure in the environment started by start-build-env.sh. (#2367) 2020-10-20 10:09:03 +09:00

README.txt 

For the latest information about Hadoop, please visit our website at:

   http://hadoop.apache.org/

and our wiki, at:

   https://cwiki.apache.org/confluence/display/HADOOP/