Commit Graph

3161 Commits

Author SHA1 Message Date
Jason Tedor c147fe5691 Do not lose CacheTest failure stack traces 2016-01-06 20:00:11 -05:00
Nik Everett 91464c7687 Remove some raw from ActionRequest subclasses
Renames "T" used in many subclasses of ActionRequest to "Self" and tightens
the type bounds.
2016-01-06 19:01:58 -05:00
Nik Everett d54f1a8f20 Merge pull request #15796 from nik9000/boundary_chars
Add test for boundary chars
2016-01-06 18:26:38 -05:00
Nik Everett 9935ae921e Version.LATEST instead of Lucene.VERSION
There was a TODO for it.
2016-01-06 17:36:10 -05:00
Jason Tedor 4c0f5bda47 Use CyclicBarriers for sychronizing driver and test threads
This commit modifies some tests to use CyclicBarriers to correctly and
simply sychronize driver and test threads.
2016-01-06 15:07:05 -05:00
Jason Tedor 22abf14812 Visible failures in cluster state update task execution ordering test 2016-01-06 14:43:24 -05:00
Jason Tedor 557b11cc2b Sychronize threads in cluster state update task execution ordering test
This commit uses a CyclicBarrier to correctly and simply sychronize the
driver and test threads in
ClusterServiceIT#testClusterStateUpdateTasksAreExecutedInOrder.
2016-01-06 14:41:43 -05:00
Jason Tedor d1b4cf6778 Further simplify cluster state update task execution ordering test 2016-01-06 14:41:42 -05:00
Jason Tedor 18b42ce798 Simplify cluster state task execution ordering test 2016-01-06 14:41:42 -05:00
Jason Tedor 270b08b302 Add test that cluster state update tasks are executed in order
This commit adds a test that ensures that cluster state update tasks
are executed in order from the perspective of a single thread.
2016-01-06 14:41:38 -05:00
Jason Tedor ef16113697 Merge pull request #15735 from jasontedor/master-node-change-predicate
Refactor master node change predicate for reuse
2016-01-06 13:58:13 -05:00
Nik Everett add60a7560 [highlighting] Another test for boundary chars 2016-01-06 13:42:15 -05:00
Nicholas Knize 7df9ba6053 [TEST] Speed up GeoShapeQueryTests
This commit speeds up GeoShapeQueryTests by reducing the size of the random generated shapes and defaulting geo_shape indexes to use quadtree (more efficient for shapes) over geohash.
2016-01-06 12:41:04 -06:00
Martijn van Groningen 04b79c112f test: unmuted test
test failed, because now the percolator returns upto 10 matches whereas before this was unbounded. The test has been updated to take this in account by checking the total count instead of the number of matches
2016-01-06 19:10:55 +01:00
Jason Tedor 3b192cfc74 Merge pull request #15791 from jasontedor/relocating-shard-failure
Only fail the relocation target when a replication request on it fails

Closes #15790
2016-01-06 12:56:49 -05:00
Jason Tedor bb4d857e44 Redundant assertion in TransportReplicationActionTests#runReplicateTest 2016-01-06 12:53:45 -05:00
Jason Tedor c291c17142 Cleanup TransportReplicationActionTests#runReplicateTest
This commit cleans up some of the assertions in
TransportReplicationActionTests#runReplicateTest:
 - use a Map to track actual vs. expected requests
 - assert that no request was sent to the local node
 - use RoutingTable#shardRoutingTable convenience method
 - explicitly use false in boolean conditions
 - clarify requests are expected on replica shards when assigned and
   execution on replicas is true
 - test ShardRouting equality when checking the failed shard request
2016-01-06 12:53:45 -05:00
Jason Tedor 6413adb5bc Assert that replication requests are sent to the correct shard copies
This commit adds tighter assertions in
TransportReplicationActionTests#runReplicateTest that replication
requests are sent to the correct shard copies.
2016-01-06 12:53:45 -05:00
Jason Tedor 75106daf9c Only fail the relocation target when a replication request on it fails
This commit addresses an issue when handling a failed replication
request against a relocating target shard. Namely, if a replication
request fails against the target of a relocation we currently fail both
the source and the target. This leads to an unnecessary
recovery. Instead, only the target of the relocation should be failed.
2016-01-06 12:53:41 -05:00
Nik Everett f5898fb07f [highlighting] Test for boundary chars 2016-01-06 12:32:09 -05:00
Martijn van Groningen 81cffd1be3 test: mute test 2016-01-06 18:30:04 +01:00
Martijn van Groningen 247ce06fc3 percolator: if size is 0 then use TotalHitCountCollector
Fixes PercolateIT#testPercolateSizingWithQueryAndFilter test
2016-01-06 18:00:00 +01:00
Nik Everett 664d67d41d Add test for alias filter leaking into highlighter 2016-01-06 11:32:03 -05:00
Jason Tedor cd56366378 Assert that we fail the correct shard when a replication request fails
This commit adds an assertion to
TransportReplicationActionTests#runReplicateTest that when a replication
request fails, we fail the correct shard.
2016-01-06 11:01:02 -05:00
Martijn van Groningen 2d6adf6428 Percolator refactoring:
* Added percolator field mapper that extracts the query terms and indexes these terms with the percolator query.
* At percolate time these extracted terms are used to query percolator queries that are like to be evaluated. This can significantly cut down the time it takes to percolate. Whereas before all percolator queries were evaluated if they matches with the document being percolated.
* Changes made to percolator queries are no longer immediately visible, a refresh needs to happen before the changes are visible.
* By default the percolate api only returns upto 10 matches instead of returning all matching percolator queries.
* Made percolate more modular, so that it is easier to add unit tests.
* Added unit tests for the percolator.

Closes #12664
Closes #13646
2016-01-06 16:08:10 +01:00
Nik Everett 646d24ebdc Add a test that the typename isn't highlighted 2016-01-06 09:20:15 -05:00
Yannick Welsch de6dfe15a7 Add PathHierarchy type back to path_hierarchy tokenizer for backward compatibility with 1.x
Closes #15785
2016-01-06 14:37:33 +01:00
Yannick Welsch a6ec1434d6 [TEST] Reduce log level in NodeVersionAllocationDeciderTests 2016-01-06 14:35:47 +01:00
Simon Willnauer 8a90c8085d Merge pull request #15788 from s1monw/dont_delete_tlog_file
Never delete translog-N.tlog file when creation fails
2016-01-06 14:31:22 +01:00
Simon Willnauer 5c833750d7 apply feedback from @bleskes 2016-01-06 14:19:58 +01:00
Simon Willnauer 12b93e72f0 Never delete translog-N.tlog file when creation fails
We today delete the translog-N.tlog file if any subsequent operation fails
but we might actually be in a good state if for instance the creation of the writer
failes after we sucessfully baked the new translog generation into the checkpoint. In this situation
we used to delete the translog-N.tlog file and failed on the next recovery of the translog with a
NoSuchFileException | FileNotFoundException just like in https://discuss.elastic.co/t/cannot-recover-index-because-of-missing-tanslog-files/38336

This commit changes the behavior and cleans up that limbo state on recovery if we already have a generation+1 file written but not baked into
the checkpoint we remove that file but only if the previous ckp file has already been renamed otherwise we know we can't be in this state.
2016-01-06 13:10:21 +01:00
Simon Willnauer 56329d0f53 Never call a listerner under lock in InternalEngine
We has a postIndex|DeleteUnderLock listener callback to load percolator
queries which is entirely private to the index shard in the meanwhile. Yet,
it still calls an external callback while holding an indexing lock which is scary
since we have no control over how long the operation could possibly take.

This commit decouples the percolator registry entirely from the ShardIndexingService
by pessimistically fetching percolator documents from the the engine using realtime get.
Even in situations where the same document is changed concurrently we will eventually end up
in the correct state without loosing an update. This also moves the index throtteling stats directly into
the engine to entirely remove the need for the dependency between InternalEngine and ShardIndexingService.
2016-01-06 11:38:34 +01:00
Yannick Welsch 55cc88e1ae Fix version-based allocation decider to prevent peer recovery from node with older version
Relocating a non-primary shard from one node to another is actually done by recovering from the active
primary shard in the cluster, and not the node that we are logically relocating from.

Closes #15775
2016-01-06 10:07:39 +01:00
Michael McCandless 3744fb9dc0 merge master 2016-01-06 04:03:42 -05:00
Michael McCandless 5f4afe842c remove nocommits; fix test case 2016-01-06 03:41:23 -05:00
Adrien Grand 7e3ccf2ee3 Merge pull request #15746 from jpountz/fix/missing_terms_agg
Make `missing` on terms aggs work with all execution modes.
2016-01-06 09:32:39 +01:00
Jason Tedor d032dabed5 Merge pull request #15777 from jasontedor/safer-cluster-state-task-notifications
Safe cluster state task notifications
2016-01-05 16:56:24 -05:00
Jason Tedor 05c46c9d35 Safe cluster state task notifications
This commit addresses an issue where a cluster state task listener
throwing an exception could prevent other listeners from being notified,
and could prevent the executor from receiving notifications that a new
cluster state was published. Additionally, this commit also addresses a
similar issue for executors handling cluster state publication
notifications.
2016-01-05 16:44:59 -05:00
Igor Motov 9ca4386fba Lower logging level for registering/unregistering tasks with the task manager
At the debug level the task manager is too talkative.
2016-01-05 15:15:02 -05:00
Igor Motov a89dba27c2 Task Management: Add framework for registering and communicating with tasks
Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to #15117
2016-01-05 12:24:43 -05:00
Simon Willnauer dff30ece05 Double check if stream must be flush to allow tests to make better assumptions of what is visible and what isn't after tragic events 2016-01-05 17:58:20 +01:00
Michael McCandless 1d46a00d43 move async-ness upwards 2016-01-05 11:27:05 -05:00
Michael McCandless 8e7719d136 add logger.warn if thread pool size is clipped; fix test failure 2016-01-05 10:59:40 -05:00
Simon Willnauer ea6718d878 Add pending review from @bleskes on #15771 2016-01-05 16:34:26 +01:00
Michael McCandless 99d6ec53fa fold in feedback 2016-01-05 09:53:13 -05:00
Simon Willnauer 5272c98136 format finally block 2016-01-05 15:50:10 +01:00
Simon Willnauer 469a179ab1 remove stale comments 2016-01-05 15:48:35 +01:00
Simon Willnauer 2370ace922 assert that tragic even exception is never null 2016-01-05 15:45:41 +01:00
Simon Willnauer f69502dd04 Simplify TranslogWriter to always write to a stream
We used to write into an in-memory buffer and if necessary also allow reading
from the memory buffer if the some translog locations that are not flushed to
the channel need to be read. This commit hides all writing behind a buffered output
stream and if ncecessary flushes all buffered data to the channel for reading. This allows
for several simplifcations like reusing javas build in BufferedOutputStream and removes the
need for read write locks on the translog writer. All thread safety is now achived using
the synchronized primitive.
2016-01-05 15:22:56 +01:00
Simon Willnauer 107859f347 Merge pull request #15766 from s1monw/add_more_io_exceptions
Add serialization support for more important IOExceptions
2016-01-05 12:00:35 +01:00
Simon Willnauer 3a41dfe1dd Add serialization support for more important IOExceptions
Several IOExceptions are always wrapped in an NotSerializableWrapper which is
annoying to read. These exceptions are important to get right across the network
and we should support the important ones that indicate problems on the Filesystem.

This commit also adds general support for IOException to preserve the parent type
across the network if no specific type is serializable.
2016-01-05 10:30:26 +01:00
Adrien Grand 1fcf9958b2 Fix GeoPointFieldMapperTests expectations. 2016-01-05 10:26:39 +01:00
Simon Willnauer bf7e2c333a Merge pull request #14632 from s1monw/indexing_stats_javadocs_
Add javadocs to IndexingStats.Stats
2016-01-05 09:37:42 +01:00
Simon Willnauer d4de8dbcfe Merge pull request #15762 from s1monw/issues/15754
Close recovered translog readers if createWriter fails
2016-01-05 08:23:11 +01:00
Simon Willnauer f6020a6ff2 Close recovered translog readers if createWriter fails
If we fail to create a writer all recovered translog readers are not
closed today which causes all open files to leak.

Closes #15754
2016-01-04 23:40:20 +01:00
Jason Tedor 06851b7224 Merge pull request #15736 from jasontedor/shard-state-action-cluster-state-refactoring
Make cluster state external to o.e.c.a.s.ShardStateAction
2016-01-04 16:22:12 -05:00
Jason Tedor 8a793b6564 Rename Shard(.*)ClusterStateHandler to Shard$1ClusterStateTaskExecutor 2016-01-04 16:13:39 -05:00
Nicholas Knize ff1aed80cc Revert GeoPointField.stored default back to false
As a default in V2, the GeoPointField.stored option was set to true. Since this consumes disk space with no positive benefit the default stored option is being reverted back to false.
2016-01-04 11:11:02 -06:00
Jason Tedor af523c4236 Fix order of logging parameters in ShardStateAction 2016-01-04 11:29:00 -05:00
Jason Tedor 7591f2047a Modify nested classes in ShardStateAction to be static 2016-01-04 11:17:18 -05:00
Jason Tedor fdb0c909ec Inline Shard(.*)TransportHandler#handleShard\1OnMaster methods 2016-01-04 11:10:52 -05:00
Jason Tedor 754bd66b63 Always log shard ID in ShardStateAction
This commit restores logging the ShardRouting#shardId at the front of
the log messages in ShardStateAction. The reason for this is so that
shard-level log messages have the format "[component][node][shard]
message".
2016-01-04 11:10:41 -05:00
Jason Tedor 1a131cf917 Inline ShardStateAction#innerShardFailed method 2016-01-04 11:10:23 -05:00
Adrien Grand c934f859c7 Make `missing` on terms aggs work with all execution modes.
There are two bugs:
 - the 'global_ordinals_low_cardinality' mode requires a fielddata-based impl so
   that it can extract the segment to global ordinal mapping
 - the 'global_ordinals_hash' mode abusively casts to the values source to a
   fielddata-based impl while it is not needed

Closes #14882
2016-01-04 11:13:17 +01:00
Adrien Grand 1a47226d9a Merge pull request #15663 from jpountz/remove/mapping_backcompat
Remove mapping backward compatibilit with pre-2.0.
2016-01-04 10:05:39 +01:00
Jason Tedor a70f76f763 Make cluster state external to o.e.c.a.s.ShardStateAction
This commit modifies the handling of cluster states in
o.e.c.a.s.ShardStateAction so that all necessary state is obtained
externally to the ShardStateAction#shardFailed and
ShardStateAction#shardStarted methods. This refactoring permits the
removal of the ClusterService field from ShardStateAction.
2016-01-03 19:15:49 -05:00
Jason Tedor c47340f2f1 Refactor master node change predicate for reuse
This commit migrates a ClusterStateObserver.ChangePredicate for
detecting a master node change into a separate class for reuse
elsewhere.
2016-01-03 12:48:21 -05:00
Jason Tedor 6a12b5e59a Cleanup o/e/c/ClusterStateObserver.java
This commit applies a minor code cleanup to
o/e/c/ClusterStateObserver.java. In particular
 - employ the diamond operator instead of explicitly specifying a
   generic type parameter
 - use 'L' instead of 'l' for specifying a long literal
 - remove redundant static modifier on a nested interface
 - remove redundant public access modifiers on interface methods
 - reformat the declaration of the four-argument ChangePredicate#apply
 - simplify the bodies of ValidationPredicate#apply
2016-01-03 12:35:25 -05:00
Jason Tedor 110fe6d36a Remove unused constant from o.e.c.n.DiscoveryNode 2016-01-03 11:56:32 -05:00
Jason Tedor 265f8f8512 Address unchecked warnings in o/e/c/c/Cache.java 2016-01-02 18:38:40 -05:00
Jason Tedor 61b7014023 Null parameter checks in BulkProcessor.Builder#builder 2016-01-01 19:52:58 -05:00
Nicholas Knize dc07affff1 Reconcile GeoPoint toString and fromString methods
GeoPoint.toString prints as a json array of values, but resetFromString expects comma delimited. This commit reconciles the methods.
2015-12-31 10:56:04 -06:00
Nicholas Knize ef33a74286 Fix multi-field support for GeoPoint types
This commit fixes multiField support for GeoPointFieldMapper by passing an externalValueContext to the multiField parser. Unit testing is added for multi field coverage.
2015-12-31 09:34:56 -06:00
Adrien Grand cb08c52a2a Merge pull request #15715 from jpountz/fix/dyn_mapping_and_template_multi_field
Fix dynamic mapping corner case.
2015-12-31 09:44:39 +01:00
Dave 3f9c0fbb58 MapperService: check index.mapper.dynamic during index creation
The MapperService doesn't currently check the
index.mapper.dynamic setting during index creation,
so indices can be created with dynamic mappings even
if this setting is false. Add a check that throws an
exception in this case. Fixes #15381
2015-12-30 21:34:02 +01:00
Jim Ferenczi 4986817c6d Fix NPE 2015-12-30 19:55:47 +01:00
Jim Ferenczi cce600ae57 Update test which assumes that an unknown similarity type is accepted. 2015-12-30 19:12:03 +01:00
Jim Ferenczi a4df067524 Fixes test with wrong similarity type (bm25 => BM25) 2015-12-30 18:27:06 +01:00
Adrien Grand 1576965831 Fix dynamic mapping corner case.
Today we fail if the field exists in another type and multi fields are defined
in a template.
2015-12-30 18:12:17 +01:00
Jim Ferenczi 992ffac509 Merge pull request #15446 from jimferenczi/classic_similarity
Renames `default` similarity into `classic`
2015-12-30 08:42:20 -08:00
Adrien Grand 6d3c9b074c Remove support for the `multi_field` type.
It is officially unsupported since version 1.0.
2015-12-30 12:03:15 +01:00
Adrien Grand a7e1eeb13d Merge pull request #15684 from jpountz/deprecate/field_boost
Remove object notation for core types.
2015-12-30 09:58:01 +01:00
Jim Ferenczi d898c0a6b0 Removes not relevant part of the bw compat test. 2015-12-30 03:54:26 +01:00
Nicholas Knize d027ceb76d [TEST] Update GeoDistanceQuery to use Lucene's maxRadialDistance
Removing maxRadialDistance method from ES GeoUtils in favor of Lucene 5.4 GeoDistanceUtils.maxRadialDistanceMeters.
2015-12-29 09:47:59 -06:00
Robert Muir 25914ae879 Merge pull request #15688 from rmuir/thirdPartyAudit3
Improve thirdPartyAudit check, round 3
2015-12-29 09:24:51 -05:00
Simon Willnauer 39cec9f2ff [TEST] Improve test speed
DedicatedClusterSnapshotRestoreIT#testRestoreIndexWithMissingShards took ~1.5 min to finish
due to timeouts that are applied if not all shards are allocated. Now that the index that has
unallocated shareds is not refreshed the test is more reasonable and runs in 15 sec
2015-12-29 14:53:15 +01:00
Daniel Mitterdorfer c813d21ffb Replace * import with explicit imports 2015-12-29 12:08:30 +01:00
Daniel Mitterdorfer 46a4aa9704 Tighten assertions in BulkProcessorRetryIT
With this commit we check more precisely on the result of a bulk
request. It could either be ok, fail or be rejected due to resource
constraints. Previously, we have relied that by default we never
get rejected.

However, this is a valid condition even when retrying. With this
commit we check that we either retried often enough that we don't
get rejected *and* if we got rejected that we maxed out the number
of specified retries.
2015-12-29 11:43:41 +01:00
Simon Willnauer 60cbb2d7bc [TEST] Protect UpgradeIT from using too many replicas 2015-12-29 10:41:04 +01:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Yannick Welsch c6182cbd37 Fail replica shards before primary shards
As failing primaries also fail associated replicas, we must fail replicas first so that their nodes are properly added to ignore list

Closes #15686
2015-12-28 22:37:18 +01:00
Simon Willnauer 387bdbd322 Don't limit recoveries in RoutingTableTests 2015-12-28 21:36:34 +01:00
Adrien Grand bffbad3c19 Remove object notation for core types.
When specifying a string field, you can either do:

```
{
  "foo": "bar"
}
```

or

```
{
  "foo": {
    "value": "bar",
    "boost": 42
  }
}
```

The latter option is now removed.

Closes #15388
2015-12-28 19:49:18 +01:00
Yannick Welsch 7e134da7d3 Cache result of RoutingNodes.node(...) in ModelNode 2015-12-28 19:14:48 +01:00
Yannick Welsch eea791de15 Remove superfluous method numPrimaries 2015-12-28 19:14:48 +01:00
Yannick Welsch e855282258 Simplify numShards calculation 2015-12-28 19:14:48 +01:00
Yannick Welsch 1536d7fe37 For rebalancing an index, only consider nodes that currently have a shard of that index or where the index can be allocated
This allows to prune a large number of nodes in case of hot/warm setup
2015-12-28 19:14:48 +01:00
Yannick Welsch 207dfc457d Remove superfluous method 2015-12-28 19:14:48 +01:00
Yannick Welsch fc0a33be05 Eliminate adding/removing shard to simulate weight of added shard / removed shard
Removal of the pattern node.addShard() -> calculate weight -> node.removeShard() which is expensive as, beside map lookups, it invalidates caching of precomputed values in ModelNode and ModelIndex. Replaced by adding an additional parameter to the weight function which accounts for the added / removed shard.
2015-12-28 19:14:48 +01:00
Yannick Welsch 5bd31a6cca Fix typo in method name 2015-12-28 19:14:48 +01:00
Yannick Welsch 67905b384f Remove unused class 2015-12-28 19:14:48 +01:00
Yannick Welsch 1bb7ca8a85 Removed unused methods 2015-12-28 19:14:48 +01:00
Yannick Welsch 0fd19008b1 Precalculate avgShardsPerNode 2015-12-28 19:14:48 +01:00
Yannick Welsch 507bb11345 Split two-element array into proper variables 2015-12-28 19:14:48 +01:00
Adrien Grand c6cf84336f Merge pull request #15679 from jpountz/fix/text_parsing
Make text parsing less lenient.
2015-12-28 16:47:05 +01:00
Adrien Grand 5eb7555ffb Make text parsing less lenient.
It now requires that the parser is on a value.
2015-12-28 16:06:42 +01:00
Simon Willnauer ba755a554f Merge pull request #15372 from s1monw/trash_recovery_threads
Remove recovery threadpools and throttle outgoing recoveries on the master
2015-12-28 15:43:14 +01:00
Simon Willnauer dcb164db3b Merge pull request #15598 from lks21c/master
Put space into last comment line
2015-12-28 15:41:10 +01:00
Jason Tedor 84c4ab6c18 Correctly release threads from starting gate in o.e.c.ClusterServiceIT 2015-12-28 08:13:55 -05:00
Jason Tedor 35cc749c9a Correctly release threads from starting gate in o.e.c.c.CacheTests 2015-12-28 08:07:37 -05:00
Simon Willnauer e8daad66d1 apply more feedback from @dakrone 2015-12-28 12:52:38 +01:00
Simon Willnauer a20b07d581 Merge branch 'master' into trash_recovery_threads 2015-12-28 12:45:14 +01:00
Simon Willnauer 0a816cd343 Add tests for equals and hashCode and fix FiltersFunctionScoreQuery equals and hashCode impls
Relates to #15676
2015-12-28 12:34:34 +01:00
Adrien Grand 23f736a0c1 Remove ParseContext.ignoredValue.
This API is unused.
2015-12-24 15:28:09 +01:00
Adrien Grand af122f4151 Remove mapping backward compatibilit with pre-2.0.
This removes the backward compatibility layer with pre-2.0 indices, notably
the extraction of _id, _routing or _timestamp from the source document when a
path is defined.
2015-12-24 13:47:37 +01:00
Adrien Grand e2fbdcfb4f Merge pull request #15633 from jpountz/fix/dynamic_mappings
Improve cross-type dynamic mapping updates.
2015-12-24 10:24:08 +01:00
Adrien Grand 3015eb3088 Improve cross-type dynamic mapping updates.
Today when dynamically mapping a field that is already defined in another type,
we use the regular dynamic mapping logic and try to copy some settings to avoid
introducing conflicts. However this is quite fragile as we don't deal with every
existing setting. This proposes a different approach that will just reuse the
shared field type.

Close #15568
2015-12-24 10:22:35 +01:00
Adrien Grand 79b14215aa Merge pull request #15636 from jpountz/feature/bool_fields
Add sub-fields support to `bool` fields.
2015-12-24 10:21:49 +01:00
Robert Muir d144ba24a5 Merge pull request #15588 from rmuir/hdfs2-only
merge current hdfs improvements to master
2015-12-23 18:17:22 -05:00
Jason Tedor 6ab922c5a0 Reduce runtime of CidrsTests#testValidCombinations
This commit reduces the running time of CidrsTests#testValidCombinations
by hoisting some costly repeated operations outside of the inner test
loop.
2015-12-23 18:02:41 -05:00
Adrien Grand f40b72633b Add sub-fields support to `bool` fields.
`bool` is our only core mapper that does not support sub fields.

Close #6587
2015-12-23 17:45:46 +01:00
Adrien Grand d8d8666877 Remove `index_name` back compat.
Since 2.0 we enforce that fields have the same full and index names. So in 3.x
we can remove the ability to have different names on the same field.
2015-12-23 14:55:26 +01:00
Adrien Grand 07658f58a8 FunctionScoreQuery should implement two-phase iteration.
FunctionScoreQuery should do two things that it doesn't do today:
 - propagate the two-phase iterator from the wrapped scorer so that things are
   still executed efficiently eg. if a phrase or geo-distance query is wrapped
 - filter out docs that don't have a high enough score using two-phase
   iteration: this way the score is only checked when everything else matches

While doing these changes, I noticed that minScore was ignored when scores were
not needed and that explain did not take it into account, so I fixed these
issues as well.
2015-12-23 12:02:21 +01:00
Adrien Grand a2072fe927 Merge pull request #15539 from jpountz/fix/immutable_document_mapper
Make mapping updates more robust.
2015-12-23 09:55:42 +01:00
Adrien Grand f535c27024 Make mapping updates more robust.
This changes a couple of things:

Mappings are truly immutable. Before, each field mapper stored a
MappedFieldTypeReference that was shared across fields that have the same name
across types. This means that a mapping update could have the side-effect of
changing the field type in other types when updateAllTypes is true. This works
differently now: after a mapping update, a new copy of the mappings is created
in such a way that fields across different types have the same MappedFieldType.
See the new Mapper.updateFieldType API which replaces MappedFieldTypeReference.

DocumentMapper is now immutable and MapperService.merge has been refactored in
such a way that if an exception is thrown while eg. lookup structures are being
updated, then the whole mapping update will be aborted. As a consequence,
FieldTypeLookup's checkCompatibility has been folded into copyAndAddAll.

Synchronization was simplified: given that mappings are truly immutable, we
don't need the read/write lock so that no documents can be parsed while a
mapping update is being processed. Document parsing is not performed under a
lock anymore, and mapping merging uses a simple synchronized block.
2015-12-23 09:55:07 +01:00
Lee Hinman 862cdad8dc Merge remote-tracking branch 'dakrone/freebsd-test-fix' 2015-12-22 15:41:35 -07:00
Jason Tedor 4c9a9b4b2d Reduce number of threads in o.e.c.c.CacheTests
This commit reduces the randomized number of threads in the cache tests
to reduce the amount of time that the tests take to execute.
2015-12-22 16:55:55 -05:00
Lee Hinman 482843e27b Fix build to run correctly on FreeBSD
This adds the required changes/checks so that the build can run on
FreeBSD.

There are a few things that differ between FreeBSD and Linux:

- CPU probes return -1 for CPU usage
- `hot_threads` cannot be supported on FreeBSD

From OpenJDK's `os_bsd.cpp`:

```c++
bool os::is_thread_cpu_time_supported() {
  #ifdef __APPLE__
  return true;
  #else
  return false;
  #endif
}
```

So this API now returns (for each FreeBSD node):

```
curl -s localhost:9200/_nodes/hot_threads
::: {Devil Hunter Gabriel}{q8OJnKCcQS6EB9fygU4R4g}{127.0.0.1}{127.0.0.1:9300}
   hot_threads is not supported on FreeBSD
```

- multicast fails in native `join` method - known bug:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

Which causes:

```
1> Caused by: java.net.SocketException: Invalid argument
1>    at java.net.PlainDatagramSocketImpl.join(Native Method)
1>    at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:179)
1>    at java.net.MulticastSocket.joinGroup(MulticastSocket.java:323)
1>    at org.elasticsearch.plugin.discovery.multicast.MulticastChannel$Plain.buildMulticastSocket(MulticastChannel.java:309)
```

So these tests are skipped on FreeBSD.

Resolves #15562
2015-12-22 12:36:04 -07:00
Daniel Mitterdorfer f9a601c7da Make BulkProcessorRetryIT less sensitive
In this commit we increase the queue size of the bulk pool in
BulkProcessorRetryIT to make it less sensitive.

As this test case should stress the pool so bulk processor needs to
back off but not so much that the backoff policy will give up at
some point (which is a valid condition), we still keep it below the
default queue size of 50.
2015-12-22 17:37:30 +01:00
Adrien Grand 6f3fc5d75d Speed up CodecTests.
Some tests, but in particular CodecTests, are slow because they test all
versions that ever existed even though they should only test supported
versions.
2015-12-22 15:16:08 +01:00
Simon Willnauer f5e4cd4616 Remove recovery threadpools and throttle outgoing recoveries on the master
Today we throttle recoveries only for incoming recoveries. Nodes that have a lot
of primaries can get overloaded due to too many recoveries. To still keep that at bay
we limit the number of threads that are sending files to the target to overcome this problem.

The right solution here is to also throttle the outgoing recoveries that are today unbounded on
the master and don't start the recovery until we have enough resources on both source and target nodes.

The concurrency aspects of the recovery source also added a lot of complexity and additional threadpools
that are hard to configure. This commit removes the concurrent streamns notion completely and sends files
in the thread that drives the recovery simplifying the recovery code considerably.
Outgoing recoveries are not throttled on the master via a allocation decider.
2015-12-22 14:59:43 +01:00
Simon Willnauer d353dcb138 Merge pull request #15592 from s1monw/remove_deprecated_query_cache_settings
Remove deprecated query cache settings
2015-12-22 10:34:00 +01:00
kwangsik 70d38760e6 Put space into last comment line 2015-12-22 18:27:50 +09:00
Simon Willnauer 8135a4ac9f Remove `index.compound_on_flush` setting and default to `true`
We added this undocumented realtime setting as backup plan long ago
but to date we haven't had a situation where it was a problem. It's reducing
the number of filehandles in the NRT case dramatically and should always be enabled.
2015-12-22 09:56:26 +01:00
Simon Willnauer 3a5d3a3bb0 Remove deprecated query cache settings 2015-12-22 09:39:12 +01:00
Simon Willnauer 8c898048bc Merge pull request #15584 from s1monw/move_translog_syn_to_index_service
Move async translog sync logic into IndexService
2015-12-22 09:28:11 +01:00
Robert Muir 010d1a89c5 Merge branch 'master' into hdfs2-only 2015-12-22 00:40:54 -05:00
Jun Ohtani 51f6519e71 Analysis : Fix no response from Analyze API without specified index
fix test
2015-12-22 11:53:29 +09:00
Jun Ohtani 267cd65506 Analysis : Fix no response from Analyze API without specified index
Fix error handling in TransportSingleShardAction without shardIt

Closes #15148
2015-12-22 10:13:52 +09:00
Michael McCandless 7ef43a22ff limit the max size of bulk and index thread pools to bounded number of processors 2015-12-21 19:18:19 -05:00
Simon Willnauer 98f9749ee1 fix visibility 2015-12-21 22:12:32 +01:00
Simon Willnauer 87494dd7db fix spelling and remove wildcard imports 2015-12-21 22:06:51 +01:00
Simon Willnauer c3f901400c Move async translog sync logic into IndexService
Today the logic to async - commit the translog is in every translog instance
itself. While the setting is a per index setting we manageing it per shard. This
polluts the translog code and can more easily be managed in IndexService.
2015-12-21 21:36:02 +01:00
Robert Muir 0ed45c5bfb remove filesystem leniency 2015-12-21 14:16:53 -05:00
Simon Willnauer 445be98e4c Merge pull request #15574 from s1monw/drop_simple_translog
Drop support for simple translog and hard-wire buffer to 8kb
2015-12-21 17:28:55 +01:00
Adrien Grand a6bf860cb5 Merge pull request #15575 from jpountz/upgrade/5.5.0-snapshot-1721183
Upgrade to lucene-5.5.0-snapshot-1721183.
2015-12-21 17:17:42 +01:00
Adrien Grand cf52e96c42 Upgrade to lucene-5.5.0-snapshot-1721183.
Some files that implement or use the Scorer API had to be changed because of
https://issues.apache.org/jira/browse/LUCENE-6919.
2015-12-21 17:02:08 +01:00
Simon Willnauer fcfd98e9e8 Drop support for simple translog and hard-wire buffer to 8kb
Today we have two variants of translogs for indexing. We only recommend the buffered
one which also has a 20% advantage in indexing speed. This commit removes the option and defaults
to the buffered case. It also hard-wires the translog buffer to 8kb instead of 64kb. We used to
adjust that buffer based on if the shard is active or not, this code has also been removed and
instead we just keep an 8kb buffer arround.
2015-12-21 16:44:35 +01:00
Jim Ferenczi 81fd2169cf Renames "default" similarity into "classic".
Replaces deprecated DefaultSimilarity by ClassicSimilarity.
Fixes #15102
2015-12-21 16:22:53 +01:00
Jim Ferenczi 96e29be7c6 Merge pull request #15571 from jimferenczi/min_should_match
Min should match greater than the number of optional clauses should return no result
2015-12-21 16:06:51 +01:00
Jim Ferenczi 4d32cc0b9f Queries.calculateMinShouldMatch returns the number of "min should match" clauses that the user wanted
even if the number of optional clauses is smaller than the provided number.
In such case the query now returns no result.
Closes #15521
2015-12-21 16:06:06 +01:00
Simon Willnauer 0cdbcdab64 added 2.3.0-SNAPSHOT as a Version constant 2015-12-21 15:48:16 +01:00
Simon Willnauer 7fe2eddfec fix test 2015-12-21 15:26:56 +01:00
Simon Willnauer afc1cc19af Simplify translog-based flush settings
This commit removes `index.translog.flush_threshold_ops` and `index.translog.disable_flush`
in favor of `index.translog.flush_threshold_size`. The number of operations is meaningless by itself and
can easily be turned into a size value with knowledge of the data. Disabling the flush is only useful in
tests and we can set the size value to a really high value. If users really need to do this they can
also apply a very high value like `1PB`.
2015-12-21 15:15:00 +01:00
Simon Willnauer eb64a81d05 Remove `index.merge.scheduler.notify_on_failure` and default to `true`
This setting was undocumented and should not be set by any user. We should
fail the shard instead.

Closes  #15570
2015-12-21 14:39:58 +01:00
Adrien Grand ac393b7a31 Make mappings tests more realistic.
DocumentMapperParser has both parse and parseCompressed methods. Except that the
parse methods are ONLY used from the unit tests. This commit removes the parse
method and moves all tests to parseCompressed so that they test more
realistically how mappings are managed.

Then I renamed parseCompressed to parse given that this is the only alternative
anyway.
2015-12-21 10:44:00 +01:00
Jim Ferenczi 1ec44dcdda Merge pull request #15371 from jimferenczi/alias_routing
Resolves the conflict between alias routing and parent routing by applying the alias routing and ignoring the parent routing.
2015-12-21 09:58:45 +01:00
Jim Ferenczi 776e5d8096 Separates routing and parent in all documentrequest in order to be able to distinguish an explicit routing value from a parent routing.
Resolves conflicts between parent routing and alias routing with the following rule:
    * The parent routing is ignored if there is an alias routing that matches the request.
Closes #3068
2015-12-21 09:44:57 +01:00
Robert Muir 5d976f1431 add sniper for broken hadoop 2015-12-18 20:16:15 -05:00
Zachary Tong 5cfefe08b0 Fix collector's class name on response output 2015-12-18 16:47:06 -05:00
Ryan Ernst 690fb2cd3f Rename InternalFilters.Bucket to InternalFilters.InternalBucket to avoid name collision 2015-12-18 13:22:20 -08:00
Ryan Ernst beec7ca9db Merge branch 'master' into wildcard_imports 2015-12-18 13:18:08 -08:00
Ryan Ernst 4ea19995cf Remove wildcard imports 2015-12-18 12:43:47 -08:00
Zachary Tong 4700cb9409 [TEST] Check length of profiled results in tests 2015-12-18 15:19:04 -05:00
Zachary Tong 41139c6b15 Ensure profile results are immutable after serialization 2015-12-18 15:19:00 -05:00
Daniel Mitterdorfer 56e4752d28 Align handling of interrupts in BulkProcessor
With this commit we implement a cancellation policy in
BulkProcessor which is aligned for the sync and the async case
and also document it.

Closes #14833.
2015-12-18 19:34:23 +01:00
Robert Muir 7e53076112 Merge pull request #15545 from rmuir/rat
add gradle licenseHeaders to precommit
2015-12-18 13:27:37 -05:00
Robert Muir 266bece2fe add missing license headers 2015-12-18 13:19:09 -05:00
Robert Muir 6ba374d632 add missing license headers 2015-12-18 13:15:34 -05:00
Areek Zillur 9d9b557cea Don't allow nodes with missing custom data types to join cluster 2015-12-18 11:33:29 -05:00
Colin Goodheart-Smithe 15588a4991 Aggregations: Run pipeline aggregations for empty buckets added in the Range Aggregation
Closes #15471
2015-12-18 14:42:05 +00:00
Christoph Büscher 2f97ff0925 Merge pull request #15010 from cbuescher/refactor-shapebuilders
Make remaining ShapeBuilders implement Writeable
2015-12-18 11:11:34 +01:00
Simon Willnauer 55f77db45b Merge pull request #15278 from s1monw/settings_prototype
Add infrastructure to transactionally apply and reset dynamic settings
2015-12-18 10:57:29 +01:00
Simon Willnauer 186242145f [TEST] use new settings infra 2015-12-18 10:26:57 +01:00
Simon Willnauer 5b991b9d5e Check for tragic event on all kinds of exceptions not only ACE and IOException
It's important to close not matter what exception caused a tragic event. Today
we only check on IOException and AlreadyClosedExceptions. The test had a bug and
threw an IAE instead causing the translog not to be closed.
2015-12-18 10:16:11 +01:00
Simon Willnauer eca2435838 Merge branch 'master' into settings_prototype 2015-12-18 09:15:58 +01:00
Simon Willnauer 711f949ca2 Convert `transport.tcp.compress` setting 2015-12-18 09:15:10 +01:00
Daniel Mitterdorfer 801de8f2ba Fix usage of latch in RetryTests 2015-12-18 06:49:17 +01:00
Ryan Ernst 6a99796b02 Build: Move test framework under a "test" top level dir
This allows adding more test projects, eg integ test fixtures that will
be coming soon.
2015-12-17 21:25:06 -08:00
Jason Tedor b62f8f11d4 Address possible race condition in test
This commit addresses a potential race condition in
ClusterServiceIT#testClusterStateBatchedUpdates. The potential race
condition is that the main test thread could be released to execute the
final test assertions before the cluster state publication callbacks had
completed thereby causing a situtation where the test assertions could
be executed before the final test state had been realized.
2015-12-17 16:57:41 -05:00
Zachary Tong 9a70dbb51a Add ability to profile query and collectors
Provides a new flag which can be enabled on a per-request basis.
When `"profile": true` is set, the search request will execute in a
mode that collects detailed timing information for query components.

```
GET /test/test/_search
{
   "profile": true,
   "query": {
      "match": {
         "foo": "bar"
      }
   }
}
```

Closes #14889

Squashed commit of the following:

commit a92db5723d2c61b8449bd163d2f006d12f9889ad
Merge: 117dd99 3f87b08
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Dec 17 09:44:10 2015 -0500

    Merge remote-tracking branch 'upstream/master' into query_profiler

commit 117dd9992e8014b70203c6110925643769d80e62
Merge: 9b29d68 82a64fd
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Dec 15 13:27:18 2015 -0500

    Merge remote-tracking branch 'upstream/master' into query_profiler

    Conflicts:
    	core/src/main/java/org/elasticsearch/search/SearchService.java

commit 9b29d6823a71140ecd872df25ff9f7478e7fe766
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Dec 14 16:13:23 2015 -0500

    [TEST] Profile flag needs to be set, ensure searches go against primary only for consistency

commit 4d602d8ad1f8cbc7b475450921fa3bc7d395b34f
Merge: 8b48e87 7742c1e
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Dec 14 10:56:25 2015 -0500

    Merge remote-tracking branch 'upstream/master' into query_profiler

commit 8b48e876348b163ab730eeca7fa35142165b05f9
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Dec 14 10:56:01 2015 -0500

    Delegate straight to in.matchCost, no need for profiling

commit fde3b0587911f0b5f15e779c671d0510cbd568a9
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Dec 14 10:28:23 2015 -0500

    Documentation tweaks, renaming build_weight -> create_weight

commit 46f5e011ee23fe9bb8a1f11ceb4fa9d19fe48e2e
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Dec 14 10:27:52 2015 -0500

    Profile TwoPhaseIterator should override matchCost()

commit b59f894ddb11b2a7beebba06c4ec5583ff91a7b2
Merge: 9aa1a3a b4e0c87
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Dec 9 14:23:26 2015 -0500

    Merge remote-tracking branch 'upstream/master' into query_profiler

commit 9aa1a3a25c34c9cd9fffaa6114c25a0ec791307d
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Dec 9 13:41:48 2015 -0500

    Revert "Move some of the collector wrapping logic into ProfileCollectorBuilder"

    This reverts commit 02cc31767fb76a7ecd44a302435e93a05fb4220e.

commit 57f7c04cea66b3f98ba2bec4879b98e4fba0b3c0
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Dec 9 13:41:31 2015 -0500

    Revert "Rearrange if/else to make intent clearer"

    This reverts commit 59b63c533fcaddcdfe4656e86a6f6c4cb1bc4a00.

commit 2874791b9c9cd807113e75e38be465f3785c154e
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Dec 9 13:38:13 2015 -0500

    Revert "Move state into ProfileCollectorBuilder"

    This reverts commit 0bb3ee0dd96170b06f07ec9e2435423d686a5ae6.

commit 0bb3ee0dd96170b06f07ec9e2435423d686a5ae6
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Dec 3 11:21:55 2015 -0500

    Move state into ProfileCollectorBuilder

commit 59b63c533fcaddcdfe4656e86a6f6c4cb1bc4a00
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Dec 2 17:21:12 2015 -0500

    Rearrange if/else to make intent clearer

commit 511db0af2f3a86328028b88a6b25fa3dfbab963b
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Dec 2 17:12:06 2015 -0500

    Rename WEIGHT -> BUILD_WEIGHT

commit 02cc31767fb76a7ecd44a302435e93a05fb4220e
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Dec 2 17:11:22 2015 -0500

    Move some of the collector wrapping logic into ProfileCollectorBuilder

commit e69356d3cb4c60fa281dad36d84faa64f5c32bc4
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 30 15:12:35 2015 -0500

    Cleanup imports

commit c1b4f284f16712be60cd881f7e4a3e8175667d62
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 30 15:11:25 2015 -0500

    Review cleanup: Make InternalProfileShardResults writeable

commit 9e61c72f7e1787540f511777050a572b7d297636
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 30 15:01:22 2015 -0500

    Review cleanup: Merge ProfileShardResult, InternalProfileShardResult.  Convert to writeable

commit 709184e1554f567c645690250131afe8568a5799
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 30 14:38:08 2015 -0500

    Review cleanup: Merge ProfileResult, InternalProfileResult.  Convert to writeable

commit 7d72690c44f626c34e9c608754bc7843dd7fd8fe
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 30 14:01:34 2015 -0500

    Review cleanup: use primitive (and default) for profile flag

commit 97d557388541bbd3388cdcce7d9718914d88de6d
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 30 13:09:12 2015 -0500

    Review cleanup: Use Collections.emptyMap() instead of building an empty one explicitly

commit 219585b8729a8b0982e653d99eb959efd0bef84e
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 30 13:08:12 2015 -0500

    Add todo to revisit profiler architecture in the future

commit b712edb2160e032ee4b2f2630fadf131a0936886
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 30 13:05:32 2015 -0500

    Split collector serialization from timing, use writeable instead of streamable

    Previously, the collector timing was done in the same class that was serialized, which required
    leaving the collector null when serializing.  Besides being a bit gross, this made it difficult to
    change the class to Writeable.

    This splits up the timing (InternalProfileCollector + ProfileCollector) and the serialization of
    the times (CollectorResult).  CollectorResult is writeable, and also acts as the public interface.

commit 6ddd77d066262d4400e3d338b11cebe7dd27ca78
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Nov 25 13:15:12 2015 -0500

    Remove dead code

commit 06033f8a056e2121d157654a65895c82bbe93a51
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Nov 25 12:49:51 2015 -0500

    Review cleanup:  Delegate to in.getProfilers()

    Note:  Need to investigate how this is used exactly so we can add a test, it isn't touched by a
    normal inner_hits query...

commit a77e13da21b4bad1176ca2b5d5b76034fb12802f
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Nov 25 11:59:58 2015 -0500

    Review cleanup:  collapse to single `if` statement

commit e97bb6215a5ebb508b0293ac3acd60d5ae479be1
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Nov 25 11:39:43 2015 -0500

    Review cleanup: Return empty map instead of null for profile results

    Note: we still need to check for nullness in SearchPhaseController, since an empty/no-hits result
    won't have profiling instantiated (or any other component like aggs or suggest).  Therefore
    QuerySearchResult.profileResults() is still @Nullable

commit db8e691de2a727389378b459fa76c942572e6015
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Nov 25 10:14:47 2015 -0500

    Review cleanup: renaming, formatting fixes, assertions

commit 9011775fe80ba22c2fd948ca64df634b4e32772d
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Nov 19 20:09:52 2015 -0500

    [DOCS] Add missing annotation

commit 4b58560b06f08d4b99b149af20916ee839baabd7
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Nov 19 20:07:17 2015 -0500

    [DOCS] Update documentation for new format

commit f0458c58e5538ed8ec94849d4baf3250aa9ec841
Author: Adrien Grand <jpountz@gmail.com>
Date:   Tue Nov 17 10:14:09 2015 +0100

    Reduce visibility of internal classes.

commit d0a7d319098e60b028fa772bf8a99b2df9cf6146
Merge: e158070 1bdf29e
Author: Adrien Grand <jpountz@gmail.com>
Date:   Tue Nov 17 10:09:18 2015 +0100

    Merge branch 'master' into query_profiler

commit e158070a48cb096551f3bb3ecdcf2b53bbc5e3c5
Author: Adrien Grand <jpountz@gmail.com>
Date:   Tue Nov 17 10:08:48 2015 +0100

    Fix compile error due to bad html in javadocs.

commit a566b5d08d659daccb087a9afbe908ec3d96cd6e
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 16 17:48:37 2015 -0500

    Remove unused collector

commit 4060cd72d150cc68573dbde62ca7321c47f75703
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 16 17:48:10 2015 -0500

    Comment cleanup

commit 43137952bf74728f5f5d5a8d1bfc073e0f9fe4f9
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Nov 16 17:32:06 2015 -0500

    Fix negative formatted time

commit 5ef3a980266326aff12d4fe380f73455ff28209f
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Nov 13 17:10:17 2015 +0100

    Fix javadocs.

commit 276114d29e4b17a0cc0982cfff51434f712dc59e
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Nov 13 16:25:23 2015 +0100

    Fix: include rewrite time as well...

commit 21d9e17d05487bf4903ae3d2ab6f429bece2ffef
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Nov 13 15:10:15 2015 +0100

    Remove TODO about profiling explain.

commit 105a31e8e570efb879447159c3852871f5cf7db4
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Nov 13 14:59:30 2015 +0100

    Fix nocommit now that the global collector is a root collector.

commit 2e8fc5cf84adb1bfaba296808c329e5f982c9635
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Nov 13 14:53:38 2015 +0100

    Make collector wrapping more explicit/robust (and a bit less magical).

commit 5e30b570b0835e1ce79a57933a31b6a2d0d58e2d
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Nov 13 12:44:03 2015 +0100

    Simplify recording API a bit.

commit 9b453afced6adc0a59ca1d67d90c28796b105185
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Nov 13 10:54:25 2015 +0100

    Fix serialization-related nocommits.

commit ad97b200bb123d4e9255e7c8e02f7e43804057a5
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Nov 13 10:46:30 2015 +0100

    Fix DFS.

commit a6de06986cd348a831bd45e4f524d2e14d9e03c3
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Nov 12 19:29:16 2015 +0100

    Remove forbidden @Test annotation.

commit 4991a28e19501109af98026e14756cb25a56f4f4
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Nov 12 19:25:59 2015 +0100

    Limit the impact of query profiling on the SearchContext API.

    Rule is: we can put as much as we want in the search.profile package but should
    aim at touching as little as possible other areas of the code base.

commit 353d8d75a5ce04d9c3908a0a63d4ca6e884c519a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Nov 12 18:05:09 2015 +0100

    Remove dead code.

commit a3ffafb5ddbb5a2acf43403c946e5ed128f47528
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Nov 12 15:30:35 2015 +0100

    Remove call to forbidden String.toLowerCase() API.

commit 1fa8c7a00324fa4e32bd24135ebba5ecf07606f1
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Nov 12 15:30:27 2015 +0100

    Fix compilation.

commit 2067f1797e53bef0e1a8c9268956bc5fb8f8ad97
Merge: 22e631f fac472f
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Nov 12 15:21:12 2015 +0100

    Merge branch 'master' into query_profiler

commit 22e631fe6471fed19236578e97c628d5cda401a9
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Nov 3 18:52:05 2015 -0500

    Fix and simplify serialization of shard profile results

commit 461da250809451cd2b47daf647343afbb4b327f2
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Nov 3 18:32:22 2015 -0500

    Remove helper methods, simpler without them

commit 5687aa1c93d45416d895c2eecc0e6a6b302139f2
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Nov 3 18:29:32 2015 -0500

    [TESTS] Fix tests for new rewrite format

commit ba9e82857fc6d4c7b72ef4d962d2102459365299
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Oct 30 15:28:14 2015 -0400

    Rewrites begone! Record all rewrites as a single time metric

commit 5f28d7cdff9ee736651d564f71f713bf45fb1d91
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Oct 29 15:36:06 2015 -0400

    Merge duplicate rewrites into one entry

    By using the Query as the key in a map, we can easily merge rewrites together.  This means
    the preProcess(), assertion and main query rewrites all get merged together.  Downside is that
    rewrites of the same Query (hashcode) but in different places get lumped together.  I think the
    simplicity of the solution is better than the slight loss in output fidelity.

commit 9a601ea46bb21052746157a45dcc6de6bc350e9c
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Oct 29 15:28:27 2015 -0400

    Allow multiple "searches" per profile (e.g. query + global agg)

commit ee30217328381cd83f9e653d3a4d870c1d2bdfce
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Oct 29 11:29:18 2015 -0400

    Update comment, add nullable annotation

commit 405c6463a64e118f170959827931e8c6a1661f13
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Oct 29 11:04:30 2015 -0400

    remove out-dated comment

commit 2819ae8f4cf1bfd5670dbd1c0e06195ae457b58f
Author: Adrien Grand <jpountz@gmail.com>
Date:   Tue Oct 27 19:50:47 2015 +0100

    Don't render children in the profiles when there are no children.

commit 7677c2ddefef321bbe74660471603d202a4ab66f
Author: Adrien Grand <jpountz@gmail.com>
Date:   Tue Oct 27 19:50:35 2015 +0100

    Set the profiler on the ContextIndexSearcher.

commit 74a4338c35dfed779adc025ec17cfd4d1c9f66f5
Author: Adrien Grand <jpountz@gmail.com>
Date:   Tue Oct 27 19:50:01 2015 +0100

    Fix json rendering.

commit 6674d5bebe187b0b0d8b424797606fdf2617dd27
Author: Adrien Grand <jpountz@gmail.com>
Date:   Tue Oct 27 19:20:19 2015 +0100

    Revert "nocommit - profiling never enabled because setProfile() on ContextIndexSearcher never called"

    This reverts commit d3dc10949024342055f0d4fb7e16c7a43423bfab.

commit d3dc10949024342055f0d4fb7e16c7a43423bfab
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Oct 23 17:20:57 2015 -0400

    nocommit - profiling never enabled because setProfile() on ContextIndexSearcher never called

    Previously, it was enabled by using DefaultSearchContext as a third-party "proxy", but since
    the refactor to make it unit testable, setProfile() needs to be called explicitly.  Unfortunately,
    this is not possible because SearchService only has access to an IndexSearcher.  And it is not
    cast'able to a DefaultIndexSearcher.

commit b9ba9c5d1f93b9c45e97b0a4e35da6f472c9ea53
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Oct 23 16:27:00 2015 -0400

    [TESTS] Fix unit tests

commit cf5d1e016b2b4a583175e07c16c7152f167695ce
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Oct 23 16:22:34 2015 -0400

    Increment token after recording a rewrite

commit b7d08f64034e498533c4a81bff8727dd8ac2843e
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Oct 23 16:14:09 2015 -0400

    Fix NPE if a top-level root doesn't have children

commit e4d3b514bafe2a3a9db08438c89f0ed68628f2d6
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Oct 23 16:05:47 2015 -0400

    Fix NPE when profiling is disabled

commit 445384fe48ed62fdd01f7fc9bf3e8361796d9593
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Oct 23 16:05:37 2015 -0400

    [TESTS] Fix integration tests

commit b478296bb04fece827a169e7522df0a5ea7840a3
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Oct 23 15:43:24 2015 -0400

    Move rewrites to their own section, remove reconciliation

    Big commit because the structural change affected a lot of the wrapping code.  Major changes:

    - Rewrites are now in their own section in the response
    - Reconciliation is gone...we don't attempt to roll the rewrites into the query tree structure
    - InternalProfileShardResults (plural) simply holds a Map<String, InternalProfileShardResult> and
    helps to serialize / ToXContent
    - InternalProfileShardResult (singular) is now the holder for all shard-level profiling details. Currently
    this includes query, collectors and rewrite.  In the future it would hold suggest, aggs, etc
    - When the user requests the profiled results, they get back a Map<String, ProfileShardResult>
    instead of doing silly helper methods to convert to maps, etc
    - Shard details are baked into a string instead of serializing the object

commit 24819ad094b208d0e94f17ce9c3f7c92f7414124
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Oct 23 10:25:38 2015 -0400

    Make Profile results immutable by removing relative_time

commit bfaf095f45fed74194ef78160a8e5dcae1850f9e
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Oct 23 10:54:59 2015 +0200

    Add nocommits.

commit e9a128d0d26d5b383b52135ca886f2c987850179
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Oct 23 10:39:37 2015 +0200

    Move all profile-related classes to the same package.

commit f20b7c7fdf85384ecc37701bb65310fb8c20844f
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Oct 23 10:33:14 2015 +0200

    Reorganize code a bit to ease unit testing of ProfileCollector.

commit 3261306edad6a0c70f59eaee8fe58560f61a75fd
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 18:07:28 2015 +0200

    Remove irrelevant nocommit.

commit a6ac868dad12a2e17929878681f66dbd0948d322
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 18:06:45 2015 +0200

    Make CollectorResult's reason a free-text field to ease bw compat.

commit 5d0bf170781a950d08b81871cd1e403e49f3cc12
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 16:50:52 2015 +0200

    Add unit tests for ProfileWeight/ProfileScorer.

commit 2cd88c412c6e62252504ef69a59216adbb574ce4
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 15:55:17 2015 +0200

    Rename InternalQueryProfiler to Profiler.

commit 84f5718fa6779f710da129d9e0e6ff914fd85e36
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 15:53:58 2015 +0200

    Merge InternalProfileBreakdown into ProfileBreakdown.

commit 135168eaeb8999c8117ea25288104b0961ce9b35
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 13:56:57 2015 +0200

    Make it possible to instantiate a ContextIndexSearcher without SearchContext.

commit 5493fb52376b48460c4ce2dedbe00cc5f6620499
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 11:53:29 2015 +0200

    Move ProfileWeight/Scorer to their own files.

commit bf2d917b9dae3b32dfc29c35a7cac4ccb7556cce
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 11:38:24 2015 +0200

    Fix bug that caused phrase queries to fail.

commit b2bb0c92c343334ec1703a221af24a1b55e36d53
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 11:36:17 2015 +0200

    Parsing happens on the coordinating node now.

commit 416cabb8621acb5cd8dfa77374fd23e428f52fe9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 11:22:17 2015 +0200

    Fix compilation (in particular remove guava deps).

commit f996508645f842629d403fc2e71c1890c0e2cac9
Merge: 4616a25 bc3b91e
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Oct 22 10:44:38 2015 +0200

    Merge branch 'master' into query_profiler

commit 4616a25afffe9c24c6531028f7fccca4303d2893
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Oct 20 12:11:32 2015 -0400

    Make Java Count API compatible with profiling

commit cbfba74e16083d719722500ac226efdb5cb2ff55
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Oct 20 12:11:19 2015 -0400

    Fix serialization of profile query param, NPE

commit e33ffac383b03247046913da78c8a27e457fae78
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Oct 20 11:17:48 2015 -0400

    TestSearchContext should return null Profiler instead of exception

commit 73a02d69b466dc1a5b8a5f022464d6c99e6c2ac3
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Oct 19 12:07:29 2015 -0400

    [DOCS] Update docs to reflect new ID format

commit 36248e388c354f954349ecd498db7b66f84ce813
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Oct 19 12:03:03 2015 -0400

    Use the full [node][index][shard] string as profile result ID

commit 5cfcc4a6a6b0bcd6ebaa7c8a2d0acc32529a80e1
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Oct 15 17:51:40 2015 -0400

    Add failing test for phrase matching

    Stack trace generated:

    [2015-10-15 17:50:54,438][ERROR][org.elasticsearch.search.profile] shard [[JNj7RX_oSJikcnX72aGBoA][test][2]], reason [RemoteTransportException[[node_s0][local[1]][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: AssertionError[nextPosition() called more than freq() times!]; ], cause [java.lang.AssertionError: nextPosition() called more than freq() times!
    	at org.apache.lucene.index.AssertingLeafReader$AssertingPostingsEnum.nextPosition(AssertingLeafReader.java:353)
    	at org.apache.lucene.search.ExactPhraseScorer.phraseFreq(ExactPhraseScorer.java:132)
    	at org.apache.lucene.search.ExactPhraseScorer.access$000(ExactPhraseScorer.java:27)
    	at org.apache.lucene.search.ExactPhraseScorer$1.matches(ExactPhraseScorer.java:69)
    	at org.elasticsearch.common.lucene.search.ProfileQuery$ProfileScorer$2.matches(ProfileQuery.java:226)
    	at org.apache.lucene.search.ConjunctionDISI$TwoPhaseConjunctionDISI.matches(ConjunctionDISI.java:175)
    	at org.apache.lucene.search.ConjunctionDISI$TwoPhase.matches(ConjunctionDISI.java:213)
    	at org.apache.lucene.search.ConjunctionDISI.doNext(ConjunctionDISI.java:128)
    	at org.apache.lucene.search.ConjunctionDISI.nextDoc(ConjunctionDISI.java:151)
    	at org.apache.lucene.search.ConjunctionScorer.nextDoc(ConjunctionScorer.java:62)
    	at org.elasticsearch.common.lucene.search.ProfileQuery$ProfileScorer$1.nextDoc(ProfileQuery.java:205)
    	at org.apache.lucene.search.Weight$DefaultBulkScorer.scoreAll(Weight.java:224)
    	at org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:169)
    	at org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
    	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:795)
    	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:509)
    	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:347)
    	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:111)
    	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:366)
    	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:378)
    	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:368)
    	at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:365)
    	at org.elasticsearch.transport.local.LocalTransport$2.doRun(LocalTransport.java:280)
    	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    	at java.lang.Thread.run(Thread.java:745)

commit 889fe6383370fe919aaa9f0af398e3040209e40b
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Oct 15 17:30:38 2015 -0400

    [DOCS] More docs

commit 89177965d031d84937753538b88ea5ebae2956b0
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Oct 15 09:59:09 2015 -0400

    Fix multi-stage rewrites to recursively find most appropriate descendant rewrite

    Previously, we chose the first rewrite that matched.  But in situations where a query may
    rewrite several times, this won't build the tree correctly.  Instead we need to recurse
    down all the rewrites until we find the most appropriate "leaf" rewrite

    The implementation of this is kinda gross: we recursively call getRewrittenParentToken(),
    which does a linear scan over the rewriteMap and tries to find rewrites with a larger token
    value (since we know child tokens are always larger).  Can almost certainly find a better
    way to do this...

commit 0b4d782b5348e5d03fd26f7d91bc4a3fbcb7f6a5
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Oct 14 19:30:06 2015 -0400

    [Docs] Documentation checkpoint

commit 383636453f6610fcfef9070c21ae7ca11346793e
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Sep 16 16:02:22 2015 -0400

    Comments

commit a81e8f31e681be16e89ceab9ba3c3e0a018f18ef
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Sep 16 15:48:49 2015 -0400

    [TESTS] Ensure all tests use QUERY_THEN_FETCH, DFS does not profile

commit 1255c2d790d85fcb9cbb78bf2a53195138c6bc24
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Sep 15 16:43:46 2015 -0400

    Refactor rewrite handling to handle identical rewrites

commit 85b7ec82eb0b26a6fe87266b38f5f86f9ac0c44f
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Sep 15 08:51:14 2015 -0400

    Don't update parent when a token is added as root -- Fixes NPE

commit 109d02bdbc49741a3b61e8624521669b0968b839
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Sep 15 08:50:40 2015 -0400

    Don't set the rewritten query if not profiling -- Fixes NPE

commit 233cf5e85f6f2c39ed0a2a33d7edd3bbd40856e8
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Sep 14 18:04:51 2015 -0400

    Update tests to new response format

commit a930b1fc19de3a329abc8ffddc6711c1246a4b15
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Sep 14 18:03:58 2015 -0400

    Fix serialization

commit 69afdd303660510c597df9bada5531b19d134f3d
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Sep 14 15:11:31 2015 -0400

    Comments and cleanup

commit 64e7ca7f78187875378382ec5d5aa2462ff71df5
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Sep 14 14:40:21 2015 -0400

    Move timing into dedicated class, add proper rewrite integration

commit b44ff85ddbba0a080e65f2e7cc8c50d30e95df8e
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Sep 14 12:00:38 2015 -0400

    Checkpoint - Refactoring to use a token-based dependency tree

commit 52cedd5266d6a87445c6a4cff3be8ff2087cd1b7
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Fri Sep 4 19:18:19 2015 -0400

    Need to set context profiling flag before calling queryPhase.preProcess

commit c524670cb1ce29b4b3a531fa2bff0c403b756f46
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Sep 4 18:00:37 2015 +0200

    Reduce profiling overhead a bit.

    This removes hash-table lookups everytime we start/stop a profiling clock.

commit 111444ff8418737082236492b37321fc96041e09
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Sep 4 16:18:59 2015 +0200

    Add profiling of two-phase iterators.

    This is useful for eg. phrase queries or script filters, since they are
    typically consumed through their two-phase iterator instead of the scorer.

commit f275e690459e73211bc8494c6de595c0320f4c0b
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Sep 4 16:03:21 2015 +0200

    Some more improvements.

    I changed profiling to disable bulk scoring, since it makes it impossible to
    know where time is spent. Also I removed profiling of operations that are
    always fast (eg. normalization) and added nextDoc/advance.

commit 3c8dcd872744de8fd76ce13b6f18f36f8de44068
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Sep 4 14:39:50 2015 +0200

    Remove println.

commit d68304862fb38a3823aebed35a263bd9e2176c2f
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Sep 4 14:36:03 2015 +0200

    Fix some test failures introduced by the rebase...

commit 04d53ca89fb34b7a21515d770c32aaffcc513b90
Author: Adrien Grand <jpountz@gmail.com>
Date:   Fri Sep 4 13:57:35 2015 +0200

    Reconcile conflicting changes after rebase

commit fed03ec8e2989a0678685cd6c50a566cec42ea4f
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Thu Aug 20 22:40:39 2015 -0400

    Add Collectors to profile results

    Profile response element has now been re-arranged so that everything is listed per-shard to
    facilitate grouping elements together.  The new `collector` element looks like this:

    ```
    "profile": {
      "shards": [
         {
            "shard_id": "keP4YFywSXWALCl4m4k24Q",
            "query": [...],
            "collector": [
               {
                  "name": "MultiCollector",
                  "purpose": "search_multi",
                  "time": "16.44504400ms",
                  "relative_time": "100.0000000%",
                  "children": [
                     {
                        "name": "FilteredCollector",
                        "purpose": "search_post_filter",
                        "time": "4.556013000ms",
                        "relative_time": "27.70447437%",
                        "children": [
                           {
                              "name": "SimpleTopScoreDocCollector",
                              "purpose": "search_sorted",
                              "time": "1.352166000ms",
                              "relative_time": "8.222331299%",
                              "children": []
                           }
                        ]
                     },
                     {
                        "name": "BucketCollector: [[non_global_term, another_agg]]",
                        "purpose": "aggregation",
                        "time": "10.40379400ms",
                        "relative_time": "63.26400829%",
                        "children": []
                     },
           ...
    ```

commit 1368b495c934be642c00f6cbf9fc875d7e6c07ff
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Aug 19 12:43:03 2015 -0400

    Move InternalProfiler to profile package

commit 53584de910db6d4a6bb374c9ebb954f204882996
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 18 18:34:58 2015 -0400

    Only reconcile rewrite timing when rewritten query is different from original

commit 9804c3b29d2107cd97f1c7e34d77171b62cb33d0
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 18 16:40:15 2015 -0400

    Comments and cleanup

commit 8e898cc7c59c0c1cc5ed576dfed8e3034ca0967f
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 18 14:19:07 2015 -0400

    [TESTS] Fix comparison test to ensure results sort identically

commit f402a29001933eef29d5a62e81c8563f1c8d0969
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 18 14:17:59 2015 -0400

    Add note about DFS being unable to profile

commit d446e08d3bc91cd85b24fc908e2d82fc5739d598
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 18 14:17:23 2015 -0400

    Implement some missing methods

commit 13ca94fb86fb037a30d181b73d9296153a63d6e4
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 18 13:10:54 2015 -0400

    [TESTS] Comments & cleanup

commit c76c8c771fdeee807761c25938a642612a6ed8e7
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 18 13:06:08 2015 -0400

    [TESTS] Fix profileMatchesRegular to handle NaN scores and nearlyEqual floats

commit 7e7a10ecd26677b2239149468e24938ce5cc18e1
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 18 12:22:16 2015 -0400

    Move nearlyEquals() utility function to shared location

commit 842222900095df4b27ff3593dbb55a42549f2697
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 18 12:04:35 2015 -0400

    Fixup rebase conflicts

commit 674f162d7704dd2034b8361358decdefce1f76ce
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Aug 17 15:29:35 2015 -0400

    [TESTS] Update match and bool tests

commit 520380a85456d7137734aed0b06a740e18c9cdec
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Aug 17 15:28:09 2015 -0400

    Make naming consistent re: plural

commit b9221501d839bb24d6db575d08e9bee34043fc65
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Aug 17 15:27:39 2015 -0400

    Children need to be added to list after serialization

commit 05fa51df940c332fbc140517ee56e849f2d40a72
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Aug 17 15:22:41 2015 -0400

    Re-enable bypass for non-profiled queries

commit f132204d264af77a75bd26a02d4e251a19eb411d
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Aug 17 15:21:14 2015 -0400

    Fix serialization of QuerySearchResult, InternalProfileResult

commit 27b98fd475fc2e9508c91436ef30624bdbee54ba
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Aug 10 17:39:17 2015 -0400

    Start to add back tests, refactor Java api

commit bcfc9fefd49307045108408dc160774666510e85
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Aug 4 17:08:10 2015 -0400

    Checkpoint

commit 26a530e0101ce252450eb23e746e48c2fd1bfcae
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Tue Jul 14 13:30:32 2015 -0400

    Add createWeight() checkpoint

commit f0dd61de809c5c13682aa213c0be65972537a0df
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Mon Jul 13 12:36:27 2015 -0400

    checkpoint

commit 377ee8ce5729b8d388c4719913b48fae77a16686
Author: Zachary Tong <zacharyjtong@gmail.com>
Date:   Wed Mar 18 10:45:01 2015 -0400

    checkpoint
2015-12-17 15:29:00 -05:00
Michael McCandless 485f4171bb a few cleanups 2015-12-17 13:28:59 -05:00
Clinton Gormley c31117f1e8 Added BWC indices for 2.0.2 and 2.1.1 and versions 2.0.3-SNAPSHOT and 2.1.2-SNAPSHOT 2015-12-17 18:39:08 +01:00
Christoph Büscher 8a02295b9a Make sure both JTS and Spatial4J are present before registering shape builders, query or mapper 2015-12-17 18:22:05 +01:00
Jason Tedor c867c7c699 Merge pull request #15509 from jasontedor/simpler-compressed-oops-representation
Simpler using compressed oops flag representation
2015-12-17 12:21:24 -05:00
Christoph Büscher 3aabf335ec Adding integration test checking that empty geometry collection returns zero docs 2015-12-17 18:21:17 +01:00
Christoph Büscher 7d2509cd63 Only Polygon and MultiPolygon need `orientation` property
Removing the `orientation` field from ShapeBuilder, only leaving it in
PolygonBuilder and MultiPolygonBuilder which are the only places where
it is actually used and parsed at the moment.
2015-12-17 18:21:17 +01:00
Christoph Büscher 081e0e9d61 Changes after rebase on master 2015-12-17 18:21:16 +01:00
Christoph Büscher f07e61c05b Make GeoShapeQueryBuilder serialize shape 2015-12-17 18:21:16 +01:00
Christoph Büscher 2a810d680d Adding serialization, equals and hashCode to GeometryCollectionBuilder 2015-12-17 18:21:16 +01:00
Christoph Büscher 1f5ee642fd Making MultiPolygonBuilder writable and adding equals/hashCode 2015-12-17 18:20:32 +01:00
Christoph Büscher ae7e8bbaf0 Making PolygonBuilder writable and add equals/hashCode 2015-12-17 18:20:32 +01:00
Christoph Büscher 31f90c91af Making LineStringBuilder and MultiLineStringBuilder writable and add equals/hashCode 2015-12-17 18:20:32 +01:00
Christoph Büscher 315b0c263d Geo: Make rest of ShapeBuilders implement Writable
This is the second part of making ShapeBuilders implement Writable.

This PR add serialization, equality and hashCode to all remaining
ShapeBuilders that don't implement it yet.
2015-12-17 18:20:32 +01:00
Jason Tedor bf7a89e304 Remove optionality from streaming useCompressedOops field 2015-12-17 12:18:39 -05:00
Adrien Grand 4afc4f3dc0 Merge pull request #15511 from jpountz/fix/simplify_Text
Simplify the Text API.
2015-12-17 18:07:39 +01:00
Adrien Grand 76abf43f7f Merge pull request #15508 from jpountz/fix/MetaDataMappingService
MetaDataMappingService should call MapperService.merge with the original mapping update.
2015-12-17 18:04:46 +01:00
Britta Weber 51fc09b264 Temporarily deactivate spuriously failing test 2015-12-17 17:57:55 +01:00
Simon Willnauer 87f800cdca make it updateable 2015-12-17 17:26:11 +01:00
Adrien Grand 6ea16671f4 Simplify the Text API.
We have the Text API, which is essentially a wrapper around a String and a
BytesReference and then we have 3 implementations depending on whether the
String view should be cached, the BytesReference view should be cached, or both
should be cached.

This commit merges everything into a single Text that is essentially the old
StringAndBytesText impl.

Long term we should look into whether this API has any performance benefit or
if we could just use plain strings. This would greatly simplify all our other
APIs that currently use Text.
2015-12-17 17:22:38 +01:00