hadoop/hadoop-tools/hadoop-resourceestimator/README.md

20 lines
1.1 KiB
Markdown

Resource Estimator Service
==========================
Resource Estimator Service can parse the history logs of production jobs, extract their resource consumption skylines in the past runs and predict their resource requirements for the new run.
## Current Status
* Support [Hadoop YARN ResourceManager](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) logs.
* In-memory store for parsed history resource skyline and estimation.
* A [Linear Programming](https://github.com/optimatika/ojAlgo) based estimator.
* Provides REST interface to parse logs, query history store and estimations.
## Upcoming features
* UI to query history and edit and save estimations.
* Persisent store implementation for store (either DB based or distributed key-value like HBase).
* Integrate directly with the [Hadoop YARN Reservation System](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ReservationSystem.html) to make a recurring reservation based on the estimated resources.
Refer to the [design document](https://issues.apache.org/jira/secure/attachment/12886714/ResourceEstimator-design-v1.pdf) for more details.