mirror of https://github.com/apache/druid.git
a0234c4e13
There are two type of DeterminePartitionsJob: - When the input data is not assume grouped, there may be duplicate rows. In this case, two MR jobs are launched. The first one do group job to remove duplicate rows. And a second one to perform global sorting to find lower and upper bound for target segments. - When the input data is assume grouped, we only need to launch the global sorting MR job to find lower and upper bound for segments. Sampling strategy: - If the input data is assume grouped, sample by random at the mapper side of the global sort mr job. - If the input data is not assume grouped, sample at the mapper of the group job. Use hash on time and all dimensions and mod by sampling factor to sample, don't use random method because there may be duplicate rows. |
||
---|---|---|
.. | ||
src | ||
pom.xml |