Start line:  
End line:  

Snippet Preview

Snippet HTML Code

Stack Overflow Questions
   * Copyright 2013 Google Inc. All Rights Reserved.
   * Licensed under the Apache License, Version 2.0 (the "License");
   * you may not use this file except in compliance with the License.
   * You may obtain a copy of the License at
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
 import java.util.List;

An implementation of QuerySplitter for the Datastore. Runs a query on the scatter property to get a random sampling of entities.
 final class QuerySplitterImpl implements QuerySplitter {

The number of keys to sample for each split.
   private static final int KEYS_PER_SPLIT = 32;
   static final QuerySplitter INSTANCE = new QuerySplitterImpl();
     // No initialization required.

   public List<QuerygetSplits(Query queryint numSplitsDatastore datastore)
     List<Querysplits = new ArrayList<Query>();
     List<KeyscatterKeys = getScatterKeys(numSplitsquerydatastore);
     Key lastKey = null;
     for (Key nextKey : getSplitKey(scatterKeysnumSplits)) {
       lastKey = nextKey;
     return splits;

Helper to determine if a filter operator is an inequality.
   private boolean isInequality(Operator operator) {
     return operator == . ||
         operator == . ||
         operator == . ||
         operator == .;

Verify that the given number of splits is not out of bounds.

numSplits the number of splits.
java.lang.IllegalArgumentException if the split size is invalid.
   private void validateSplitSize(int numSplitsthrows IllegalArgumentException {
     if (numSplits < 1) {
       throw new IllegalArgumentException("The number of splits must be greater than 0.");

Validate that we only have allowable filters. Note that equality and ancestor filters are allowed, however they may result in inefficient sharding.
  private void validateFilter(Filter filterthrows IllegalArgumentException {
    if (filter.hasCompositeFilter()) {
      for (Filter subFilter : filter.getCompositeFilter().getFilterList()) {
    } else if (filter.hasPropertyFilter()) {
      if (isInequality(filter.getPropertyFilter().getOperator())) {
        throw new IllegalArgumentException("Query cannot have an inequality filter.",
            new IllegalArgumentException());

Verify that the given query can be properly scattered.

query the query to verify
java.lang.IllegalArgumentException if the query is invalid.
  private void validateQuery(Query querythrows IllegalArgumentException {
    if (query.getKindCount() != 1) {
      throw new IllegalArgumentException("Query must have exactly one kind.");
    if (query.getOrderCount() != 0) {
      throw new IllegalArgumentException("Query cannot have a sort order.");
    if (query.hasFilter()) {

Create a new given the query and range.

lastKey the previous key. If null then assumed to be the beginning.
nextKey the next key. If null then assumed to be the end.
query the desired query.
  private Query createSplit(Key lastKeyKey nextKeyQuery query) {
    List<FilterkeyFilters = new ArrayList<Filter>();
    if (query.hasFilter()) {
    if (lastKey != null) {
    if (nextKey != null) {
    return Query.newBuilder(query).setFilter(

Given a number of desired splits gets a list of scatter keys with multiples at each split.

numSplits the number of desired splits.
query the user query.
datastore the datastore containing the data.
DatastoreException if there was an error when executing the datastore query.
  private List<KeygetScatterKeys(int numSplitsQuery queryDatastore datastore)
      throws DatastoreException {
    Query.Builder scatterPointQuery = createScatterQuery(querynumSplits);
    List<KeykeySplits = new ArrayList<Key>();
    QueryResultBatch batch = null;
    do {
      batch = datastore.runQuery(RunQueryRequest.newBuilder().setQuery(query).build()).getBatch();
      for (EntityResult result : batch.getEntityResultList()) {
      scatterPointQuery.setLimit(scatterPointQuery.getLimit() - batch.getEntityResultCount());
    } while (batch.getMoreResults() == .);
    Collections.sort(keySplits, DatastoreHelper.getKeyComparator());
    return keySplits;

Creates a scatter query from the given user query

query the user's query.
numSplits the number of splits to create.
  private Query.Builder createScatterQuery(Query queryint numSplits) {
    // TODO(user): We can potentially support better splits with equality filters in our query
    // if there exists a composite index on property, __scatter__, __key__. Until an API for
    // metadata exists, this isn't possible. Note that ancestor and inequality queries fall into
    // the same category.
    Query.Builder scatterPointQuery = Query.newBuilder();
    scatterPointQuery.setLimit(numSplits *  - 1);
    return scatterPointQuery;

Given a list of keys and a number of splits find the keys to split on.

keys the list of keys.
numSplits the number of splits.
  private Iterable<KeygetSplitKey(List<Keykeysint numSplits) {
    // If the number of keys is less than the number of splits, we are limited in the number of
    // splits we can make.
    if (keys.size() < numSplits - 1) {
      return keys;
    // Calculate the number of keys per split. This should be {@link KEYS_PER_SPLIT}, but may
    // be less if there are not KEYS_PER_SPLIT * numSplits scatter properties.
    double numKeysPerSplit = Math.max(1.0, (keys.size() + 1) / numSplits);
    List<KeykeysList = new ArrayList<Key>(numSplits - 1);
    // Grab the last sample for each split, otherwise the first split will be too small.
    for (int i = 1; i < numSplitsi++) {
      int splitIndex = ((int) (i * numKeysPerSplit)) - 1;
    return keysList;
New to GrepCode? Check out our FAQ X