hadoop/hadoop-tools/hadoop-aws
Steve Loughran 1ef34d0819
HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396)
This adds a semaphore to throttle the number of FileSystem instances which
can be created simultaneously, set in "fs.creation.parallel.count".

This is designed to reduce the impact of many threads in an application calling
FileSystem.get() on a filesystem which takes time to instantiate -for example
to an object where HTTPS connections are set up during initialization.
Many threads trying to do this may create spurious delays by conflicting
for access to synchronized blocks, when simply limiting the parallelism
diminishes the conflict, so speeds up all threads trying to access
the store.

The default value, 64, is larger than is likely to deliver any speedup -but
it does mean that there should be no adverse effects from the change.

If a service appears to be blocking on all threads initializing connections to
abfs, s3a or store, try a smaller (possibly significantly smaller) value.

Contributed by Steve Loughran.

Change-Id: I57161b026f28349e339dc8b9d74f6567a62ce196
2020-11-25 14:55:29 +00:00
..
dev-support HADOOP-16697. Tune/audit S3A authoritative mode. 2020-01-10 11:11:56 +00:00
src HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients. (#2396) 2020-11-25 14:55:29 +00:00
pom.xml HADOOP-13230. S3A to optionally retain directory markers. 2020-08-15 20:19:49 +01:00