In order to be more consistent with what they do, the query cache has been
renamed to request cache and the filter cache has been renamed to query
cache.
A known issue is that package/logger names do no longer match settings names,
please speak up if you think this is an issue.
Here are the settings for which I kept backward compatibility. Note that they
are a bit different from what was discussed on #11569 but putting `cache` before
the name of what is cached has the benefit of making these settings consistent
with the fielddata cache whose size is configured by
`indices.fielddata.cache.size`:
* index.cache.query.enable -> index.requests.cache.enable
* indices.cache.query.size -> indices.requests.cache.size
* indices.cache.filter.size -> indices.queries.cache.size
Close#11569
This commit consolidates several abstractions on the shard level in
ordinary classes not managed by the shard level guice injector.
Several classes have been collapsed into IndexShard and IndexShardGatewayService
was cleaned up to be more lightweight and self-contained. It has also been moved into
the index.shard package and it's operation is renamed from recovery from "gateway" to recovery
from "store" or "shard_store".
Closes#11847
To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget) to other active shards. The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as a trigger to attempt adding a sync id marker (which will speed up future recoveries).
We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.
Closes#11479
In #11072 we are adding a check that will prevent opening of old indices. However, this check doesn't take into consideration the fact that indices can be made compatible with the current version through upgrade API. In order to make compatibility check aware of the upgrade, the upgrade API should write a new setting `index.version.minimum_compatible` that will indicate the minimum compatible version of lucene this index is compatible with and `index.version.upgraded` that will indicate the version of elasticsearch that performed the upgrade.
Closes#11095
#10032 introduced the notion of sealing an index by marking it with a special read only marker, allowing for a couple of optimization to happen. The most important one was to speed up recoveries of shards where we know nothing has changed since they were online by skipping the file based sync phase. During the implementation we came up with a light notion which achieves the same recovery benefits but without the read only aspects which we dubbed synced flush. The fact that it was light weight and didn't put the index in read only mode, allowed us to do it automatically in the background which has great advantage. However we also felt the need to allow users to manually trigger this operation.
The implementation at #11179 added the sync flush internal logic and the manual (rest) rest API. The name of the API was modeled after the sealing terminology which may end up being confusing. This commit changes the API name to match the internal synced flush naming, namely `{index}/_flush/synced'.
On top of that it contains a couple other changes:
- Remove all java client API. This feature is not supposed to be called programtically by applications but rather by admins.
- Improve rest responses making structure similar to other (flush) API
- Change IndexShard#getOperationsCount to exclude the internal +1 on open shard . it's confusing to get 1 while there are actually no ongoing operations
- Some minor other clean ups
Mappings conflicts should not be ignored. If I read the history correctly, this
option was added when a mapping update to an existing field was considered a
conflict, even if the new mapping was exactly the same. Now that mapping updates
are smart enough to detect conflicting options, we don't need an option to
ignore conflicts.
As a follow up to #10870, this removes support for
index templates on disk. It also removes a missed
place still allowing disk based mappings.
closes#11052
* Removed the docs for `index.compound_format` and `index.compound_on_flush` - these are expert settings which should probably be removed (see https://github.com/elastic/elasticsearch/issues/10778)
* Removed the docs for `index.index_concurrency` - another expert setting
* Labelled the segments verbose output as experimental
* Marked the `compression`, `precision_threshold` and `rehash` options as experimental in the cardinality and percentile aggs
* Improved the experimental text on `significant_terms`, `execution_hint` in the terms agg, and `terminate_after` param on count and search
* Removed the experimental flag on the `geobounds` agg
* Marked the settings in the `merge` and `store` modules as experimental, rather than the modules themselves
Closes#10782
This option defaults to false, because it is also important to upgrade
the "merely old" segments since many Lucene improvements happen within
minor releases.
But you can pass true to do the minimal work necessary to upgrade to
the next major Elasticsearch release.
The HTTP GET upgrade request now also breaks out how many bytes of
ancient segments need upgrading.
Closes#10213Closes#10540
Conflicts:
dev-tools/create_bwc_index.py
rest-api-spec/api/indices.upgrade.json
src/main/java/org/elasticsearch/action/admin/indices/optimize/OptimizeRequest.java
src/main/java/org/elasticsearch/action/admin/indices/optimize/ShardOptimizeRequest.java
src/main/java/org/elasticsearch/action/admin/indices/optimize/TransportOptimizeAction.java
src/main/java/org/elasticsearch/index/engine/InternalEngine.java
src/test/java/org/elasticsearch/bwcompat/StaticIndexBackwardCompatibilityTest.java
src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
src/test/java/org/elasticsearch/rest/action/admin/indices/upgrade/UpgradeReallyOldIndexTest.java
Deleting a type from an index is inherently dangerous because
the type can be recreated with new mappings which may conflict
with existing segments still using the old mappings. This
removes the ability to delete a type (similar to how deleting
fields within a type is not allowed, for the same reason).
closes#8877closes#10231
I've been attempting to programatically verify that adding index templates via the `{path.conf}/templates/` directory works fine although I was never able to validate this via an API call to the `/_template/`. It seems that these templates do not appear in that API call, which I discovered in the following mail thread:
http://elasticsearch-users.115913.n3.nabble.com/Loading-of-index-settings-template-from-file-in-config-templates-td4024923.html#d1366317284000-912
My question is why wouldn't the `/_template/*` method return these templates? This tends to complicate things for those that want to perform automated tests to verify that they are in fact being recognized and used by Elasticsearch.
This commit adds the current total number of translog operations to the recovery reporting API. We also expose the recovered / total percentage:
```
"translog": {
"recovered": 536,
"total": 986,
"percent": "54.3%",
"total_time": "2ms",
"total_time_in_millis": 2
},
```
Closes#9368Closes#10042
To support the `_recovery` API, the recovery process keeps track of current progress in a class called RecoveryState. This class currently have some issues, mostly around concurrency (see #6644 ). This PR cleans it up as well as other issues around it:
- Make the Index subsection API cleaner:
- remove redundant information - all calculation is done based on the underlying file map
- clearer definition of what is what: total files, vs reused files (local files that match the source) vs recovered files (copied over). % based progress is reported based on recovered files only.
- cleaned up json response to match other API (sadly this breaks the structure). We now properly report human values for dates and other units.
- Add more robust unit testing
- Detail flag was passed along as state (it's now a ToXContent param)
- State lookup during reporting is now always done via the IndexShard , no more fall backs to many other classes.
- Cleanup APIs around time and move the little computations to the state class as opposed to doing them out of the API
I also improved error messages out of the REST testing infra for things I run into.
Closes#6644Closes#9811
Squashed commit of the following:
commit 20835037c98e7d2fac4206c372717a05a27c4790
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 15:27:17 2015 -0700
Use Enum for "_primary" preference
commit 325acbe4585179190a959ba3101ee63b99f1931a
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 14:32:41 2015 -0700
Use ?preference=_primary automatically for realtime GET operations
commit edd49434af5de7e55928f27a1c9ed0fddb1fb133
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 14:32:06 2015 -0700
Move engine creation into protected createNewEngine method
commit 67a797a9235d4aa376ff4af16f3944d907df4577
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 13:14:01 2015 -0700
Factor out AssertingSearcher so it can be used by mock Engines
commit 62b0c28df8c23cc0b8205b33f7595c68ff940e2b
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 11:43:17 2015 -0700
Use IndexMetaData.isIndexUsingShadowReplicas helper
commit 1a0d45629457578a60ae5bccbeba05acf5d79ddd
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 09:59:31 2015 -0700
Rename usesSharedFilesystem -> isOnSharedFilesystem
commit 73c62df4fc7da8a5ed557620a83910d89b313aa1
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 09:58:02 2015 -0700
Add MockShadowEngine and hook it up to be used
commit c8e8db473830fce1bdca3c4df80a685e782383bc
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 09:45:50 2015 -0700
Clarify comment about pre-defined mappings
commit 60a4d5374af5262bd415f4ef40f635278ed12a03
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 09:18:22 2015 -0700
Add a test for shadow replicas that uses field data
commit 7346f9f382f83a21cd2445b3386fe67472bc3184
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 08:37:14 2015 -0700
Revert changes to RecoveryTarget.java
commit d90d6980c9b737bd8c0f4339613a5373b1645e95
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 08:35:44 2015 -0700
Rename `ownsShard` to `canDeleteShardContent`
commit 23001af834d66278ac84d9a72c37b5d1f3a10a7b
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 08:35:25 2015 -0700
Remove ShadowEngineFactory, add .newReadOnlyEngine method in EngineFactory
commit b64fef1d2c5e167713e869b22d388ff479252173
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 18 08:25:19 2015 -0700
Add warning that predefined mappings should be used
commit a1b8b8cf0db49d1bd1aeb84e51491f7f0de43b59
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 17 14:31:50 2015 -0700
Remove unused import and fix index creation example in docs
commit 0b1b852365ceafc0df86866ac3a4ffb6988b08e4
Merge: b9d1fed a22bd49
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 17 10:56:02 2015 -0700
Merge remote-tracking branch 'refs/remotes/origin/master' into shadow-replicas
commit b9d1fed25ae472a9dce1904eb806702fba4d9786
Merge: 4473e63 41fd4d8
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 17 09:02:27 2015 -0700
Merge remote-tracking branch 'refs/remotes/origin/master' into shadow-replicas
commit 4473e630460e2f0ca2a2e2478f3712f39a64c919
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 17 09:00:39 2015 -0700
Add asciidoc documentation for shadow replicas
commit eb699c19f04965952ae45e2caf107124837c4654
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 16:15:39 2015 +0100
remove last nocommit
commit c5ece6d16d423fbdd36f5d789bd8daa5724d77b0
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 16:13:12 2015 +0100
simplify shadow engine
commit 45cd34a12a442080477da3ef14ab2fe7947ea97e
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 11:32:57 2015 +0100
fix tests
commit 744f228c192602a6737051571e040731d413ba8b
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 11:28:12 2015 +0100
revert changes to IndexShardGateway - these are leftovers from previous iterations
commit 11886b7653dabc23655ec76d112f291301f98f4a
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 11:26:48 2015 +0100
Back out non-shared FS code. this will go in in a second iteration
commit 77fba571f150a0ca7fb340603669522c3ed65363
Merge: e8ad614 2e3c6a9
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 11:16:46 2015 +0100
Merge branch 'master' into shadow-replicas
Conflicts:
src/main/java/org/elasticsearch/index/engine/Engine.java
commit e8ad61467304e6d175257e389b8406d2a6cf8dba
Merge: 48a700d 1b8d8da
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 10:54:20 2015 +0100
Merge branch 'master' into shadow-replicas
commit 48a700d23cff117b8e4851d4008364f92b8272a0
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 10:50:59 2015 +0100
add test for failing shadow engine / remove nocommit
commit d77414c5e7b2cde830a8e3f70fe463ccc904d4d0
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 17 10:27:56 2015 +0100
remove nocommits in IndexMetaData
commit abb696563a9e418d3f842a790fcb832f91150be2
Author: Simon Willnauer <simonw@apache.org>
Date: Mon Feb 16 17:05:02 2015 +0100
remove nocommit and simplify delete logic
commit 82b9f0449108cd4741568d9b4495bf6c10a5b019
Author: Simon Willnauer <simonw@apache.org>
Date: Mon Feb 16 16:45:27 2015 +0100
reduce the changes compared to master
commit 28f069b6d99a65e285ac8c821e6a332a1d8eb315
Author: Simon Willnauer <simonw@apache.org>
Date: Mon Feb 16 16:43:46 2015 +0100
fix primary relocation
commit c4c999dd61a44a7a0db9798275a622f2b85b1039
Merge: 2ae80f9 455a85d
Author: Simon Willnauer <simonw@apache.org>
Date: Mon Feb 16 15:04:26 2015 +0100
Merge branch 'master' into shadow-replicas
commit 2ae80f9689346f8fd346a0d3775a6341874d8bef
Author: Lee Hinman <lee@writequit.org>
Date: Fri Feb 13 16:25:34 2015 -0700
throw UnsupportedOperationException on write operations in ShadowEngine
commit 740c28dd9ef987bf56b670fa1a8bcc6de2845819
Merge: e5bc047 305ba33
Author: Lee Hinman <lee@writequit.org>
Date: Fri Feb 13 15:38:39 2015 -0700
Merge branch 'master' into shadow-replicas
commit e5bc047d7c872ae960d397b1ae7b4b78d6a1ea10
Author: Lee Hinman <lee@writequit.org>
Date: Fri Feb 13 11:38:09 2015 -0700
Don't replicate document request when using shadow replicas
commit 213292e0679d8ae1492ea11861178236f4abd8ea
Author: Simon Willnauer <simonw@apache.org>
Date: Fri Feb 13 13:58:05 2015 +0100
add one more nocommit
commit 83d171cf632f9b77cca9de58505f7db8fcda5599
Merge: aea9692 09eb8d1
Author: Simon Willnauer <simonw@apache.org>
Date: Fri Feb 13 13:52:29 2015 +0100
Merge branch 'master' into shadow-replicas
commit aea96920d995dacef294e48e719ba18f1ecf5860
Author: Simon Willnauer <simonw@apache.org>
Date: Fri Feb 13 09:56:41 2015 +0100
revert unneeded changes on Store
commit ea4e3e58dc6959a92c06d5990276268d586735f3
Author: Lee Hinman <lee@writequit.org>
Date: Thu Feb 12 14:26:30 2015 -0700
Add documentation to ShadowIndexShard, remove nocommit
commit 4f71c8d9f706a0c1c39aa3a370efb1604559d928
Author: Lee Hinman <lee@writequit.org>
Date: Thu Feb 12 14:17:22 2015 -0700
Add documentation to ShadowEngine
commit 28a9d1842722acba7ea69e0fa65200444532a30c
Author: Lee Hinman <lee@writequit.org>
Date: Thu Feb 12 14:08:25 2015 -0700
Remove nocommit, document canDeleteIndexContents
commit d8d59dbf6d0525cd823d97268d035820e5727ac9
Author: Lee Hinman <lee@writequit.org>
Date: Thu Feb 12 10:34:32 2015 -0700
Refactor more shared methods into the abstract Engine
commit a7eb53c1e8b8fbfd9281b43ae39eacbe3cd1a0a6
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 17:38:59 2015 +0100
Simplify shared filesystem recovery by using a dedicated recovery handler that skip
most phases and enforces shard closing on the soruce before the target opens it's engine
commit a62b9a70adad87d7492c526f4daf868cb05018d9
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 15:59:54 2015 +0100
fix compile error after upstream changes
commit abda7807bc3328a89fd783ca7ad8c6deac35f16f
Merge: f229719 35f6496
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 15:57:28 2015 +0100
Merge branch 'master' into shadow-replicas
Conflicts:
src/main/java/org/elasticsearch/index/engine/Engine.java
commit f2297199b7dd5d3f9f1f109d0ddf3dd83390b0d1
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 12:41:32 2015 +0100
first cut at catchup from primary
make flush to a refresh
factor our ShadowIndexShard to have IndexShard be idential to the master and least intrusive
cleanup abstractions
commit 4a367c07505b84b452807a58890f1cbe21711f27
Author: Simon Willnauer <simonw@apache.org>
Date: Thu Feb 12 09:50:36 2015 +0100
fix primary promotion
commit cf2fb807e7e243f1ad603a79bc9d5f31a499b769
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 16:45:41 2015 -0700
Make assertPathHasBeenCleared recursive
commit 5689b7d2f84ca1c41e4459030af56cb9c0151eff
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 15:58:19 2015 -0700
Add testShadowReplicaNaturalRelocation
commit fdbe4133537eaeb768747c2200cfc91878afeb97
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 15:28:57 2015 -0700
Use check for shared filesystem in primary -> primary relocation
Also adds a nocommit
commit 06e2eb4496762130af87ce68a47d360962091697
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 15:21:32 2015 -0700
Add a test checking that indices with shadow replicas clean up after themselves
commit e4dbfb09a689b449f0edf6ee24222d7eaba2a215
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 15:08:18 2015 -0700
Fix segment info for ShadowEngine, remove test nocommit
commit 80cf0e884c66eda7d59ac5d59235e1ce215af8f5
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 14:30:13 2015 -0700
Remove nocommit in ShadowEngineTests#testFailStart()
commit 5e33eeaca971807b342f9be51a6a566eee005251
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 14:22:59 2015 -0700
Remove overly-complex test
commit 2378fbb917b467e79c0262d7a41c23321bbeb147
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 13:45:44 2015 -0700
Fix missing import
commit 52e9cd1b8334a5dd228d5d68bd03fd0040e9c8e9
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 13:45:05 2015 -0700
Add a test for replica -> primary promotion
commit a95adbeded426d7f69f6ddc4cbd6712b6f6380b4
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 12:54:14 2015 -0700
Remove tests that don't apply to ShadowEngine
commit 1896feda9de69e4f9cf774ef6748a5c50e953946
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 10:29:12 2015 -0700
Add testShadowEngineIgnoresWriteOperations and testSearchResultRelease
commit 67d7df41eac5e10a1dd63ddb31de74e326e9d38b
Author: Lee Hinman <lee@writequit.org>
Date: Wed Feb 11 10:06:05 2015 -0700
Add start of ShadowEngine unit tests
commit ca9beb2d93d9b5af9aa6c75dbc0ead4ef57e220d
Merge: 2d42736 57a4646
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 18:03:53 2015 +0100
Merge branch 'master' into shadow-replicas
commit 2d42736fed3ed8afda7e4aff10b65d292e1c6f92
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 17:51:22 2015 +0100
shortcut recovery if we are on a shared FS - no need to compare files etc.
commit 24d36c92dd82adce650e7ac8e9f0b43c83b2dc53
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 17:08:08 2015 +0100
utilize the new delete code
commit 2a2eed10f58825aae29ffe4cf01aefa5743a97c7
Merge: 343dc0b 173cfc1
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 16:07:41 2015 +0100
Merge branch 'master' into shadow-replicas
Conflicts:
src/main/java/org/elasticsearch/gateway/GatewayMetaState.java
commit 343dc0b527a7052acdc783ac5abcaad1ef78dbda
Author: Simon Willnauer <simonw@apache.org>
Date: Wed Feb 11 16:05:28 2015 +0100
long adder is not available in java7
commit be02cabfeebaea74b51b212957a2a466cfbfb716
Author: Lee Hinman <lee@writequit.org>
Date: Tue Feb 10 22:04:24 2015 -0700
Add test that restarts nodes to ensure shadow replicas recover
commit 7fcb373f0617050ca1a5a577b8cf32e32dc612b0
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 10 23:19:21 2015 +0100
make test more evil
commit 38135af0c1991b88f168ece0efb72ffe9498ff59
Author: Simon Willnauer <simonw@apache.org>
Date: Tue Feb 10 22:25:11 2015 +0100
make tests pass
commit 05975af69e6db63cb95f3e40d25bfa7174e006ea
Author: Lee Hinman <lee@writequit.org>
Date: Mon Jan 12 18:44:29 2015 +0100
Add ShadowEngine
This has been very trappy. Rather than continue to allow buggy behavior
of having upgrade/optimize requests sidestep the single shard per node
limits optimize is supposed to be subject to, this removes
the ability to run the upgrade/optimize async.
closes#9638
Issue #9566 raises the point that setting the number of shards on a closed index can lead to this index not beeing able to open again. This change in documentation is ment to warn the user about this issue.
We now have a very useful annotation to mark features or parameters as
experimental. Let's use it! This commit replaces some custom text warnings with
this annotation and adds this annotation to some existing features/parameters:
- inner_hits (unreleased yet)
- terminate_after (released in 1.4)
- per-bucket doc count errors in the terms agg (released in 1.4)
I also tagged with this annotation settings which should either be not needed
(like the ability to evict entries from the filter cache based on time) or that
are too deep into the way that Elasticsearch works like the Directory
implementation or merge settings.
Close#9563
The `full` option and `FlushType.NEW_WRITER` only exists to allow
realtime changes to two settings (`index.codec` and `index.concurrency`).
Those settings are very expert and don't really need to be updateable
in realtime.
This adds a new boolean (index.merge.scheduler.auto_throttle) dynamic
setting, default true (matching Lucene), to adaptively set the IO rate
limit for merges over time.
This is more flexible than the previous fixed rate throttling because
it responds depending on the incoming merge rate, so search-heavy
applications that are not doing much indexing will see merges heavily
throttled while indexing-heavy cases will lighten the throttle so
merges can keep up within incoming indexing.
The fixed rate throttling is still available as a fallback if things
go horribly wrong.
Closes#9243Closes#9133
additional element per segment.
This commit adds a verbose flag to the _segments api. Currently the
only additional information returned when set to true is the full
ram tree from lucene for each segment.
make the "es090" postings format read-only, just to support old segments. There is a test version that subclasses it with write-capability for testing.
Closes#8571
Adrien Grand
Simon Willnauer
Clinton Gormley
Boaz Leskes
Lee Hinman
Britta Weber
Igor Motov
Nik Everett
Martijn van Groningen
Jun Ohtani
Ryan Ernst
Michael McCandless
javanna
Corey Daley
Florian Hopf
Al Lefebvre
Christoph Büscher
Sebastian Ziebell
mdzor
Robert Muir