Is there a way to query multiple hash keys in DynamoDB?

StackOverflow https://stackoverflow.com/questions/17663695

Domanda

Is there a way to query multiple hash keys using a single query in Amazon's AWS SDK for Java?

Here's my issue; I have a DB table for project statuses. The Hash Key is the status of a project (ie: new, assigned, processing, or complete). The range key is a set of project IDs. Currently, I've got a query setup to simply find all the projects listed as a status(hash) of "assigned" and another query set to look for a status of "processing". Is there a way to do this using a single query rather than sending multiple queries for each status I need to find? Code below:

    DynamoDBMapper mapper = new DynamoDBMapper(new AmazonDynamoDBClient(credentials));
    PStatus assignedStatus = new PStatus();
    assignedStatus.setStatus("assigned");
    PStatus processStatus = new PStatus();
    processStatus.setStatus("processing");

    DynamoDBQueryExpression<PStatus> queryAssigned = new DynamoDBQueryExpression<PStatus>().withHashKeyValues(assignedStatus);
    DynamoDBQueryExpression<PStatus> queryProcessing = new DynamoDBQueryExpression<PStatus>().withHashKeyValues(processStatus);

    List<PStatus> assigned = mapper.query(PStatus.class, queryAssigned);
    List<PStatus> process = mapper.query(PStatus.class, queryProcessing);

So basically, I'd like to know if it's possible to eliminate the queryAssigned and assigned variables and handle both assignedStatus and processStatus via the same query, process, to find projects that are not new or complete.

È stato utile?

Soluzione

No, as of today there is no way to send multiple queries in the same request. If you're concerned about latency, you could make multiple requests simultaneously in different threads. This would require the same amount of network bandwidth as a "dual query" would if Dynamo offered it (assuming you're making 2, not hundreds).

Altri suggerimenti

There is no way to query by multiple hash keys, but, as of April 2014, you can use QueryFilter so you can filter by non key fields in addition to hash key fields.

In a blog post on 24 April 2014, AWS announced the release of the "QueryFilter" option:

With today's release, we are extending this model with support for query filtering on non-key attributes. You can now include a QueryFilter as part of a call to the Query function. The filter is applied after the key-based retrieval and before the results are returned to you. Filtering in this manner can reduce the amount of data returned to your application while also simplifying and streamlining your code

Check this out there http://aws.amazon.com/blogs/aws/improved-queries-and-updates-for-dynamodb/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+AmazonWebServicesBlog+%28Amazon+Web+Services+Blog%29

Sharing my working answer for posterity. As of Oct 2020, there is a way to query multiple hash keys using a single query using aws-java-sdk-dynamodb-1.11.813.jar. I had the same requirement where I had to select items based on multiple hash keys(partition keys), and you can relate the requirement with the RDMS scenario, similar to the query select * from photo where id in ('id1','id2','id3'), here id is the primary key of the photo table.

Code Snippet

  • DynamoDB entity
package com.test.demo.dynamodb.entity;

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBHashKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTable;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.NoArgsConstructor;

import java.io.Serializable;

@NoArgsConstructor
@AllArgsConstructor
@lombok.Data
@DynamoDBTable(tableName = "test_photos")
@Builder
public class Photo implements Serializable {
    @DynamoDBHashKey
    private String id;
    private String title;
    private String url;
    private String thumbnailUrl;
}
  • DynamoDB Repository Class
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;
import com.amazonaws.services.dynamodbv2.datamodeling.KeyPair;
import com.test.demo.dynamodb.entity.Photo;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;

import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

@Repository
public class PhotoRepository {

    @Autowired
    private DynamoDBMapper dynamoDBMapper = null;

    public List<Photo> findByIds(Collection<String> photoIds) {
        //Constructing `KeyPair` instance and setting the HashKey,
        // in this example I have only hash key,
        // if you have RangeKey(Sort) you can set that also here using KeyPair#withRangeKey

        List<KeyPair> keyPairs = photoIds.stream()
                                         .map(id -> new KeyPair().withHashKey(id))
                                         .collect(Collectors.toList());

        //Creating Map where Key as Class<?> and value as a list of created keyPairs 
        //you can also directly use batchLoad(List<Photo> itemsToGet), the only constraint 
        //is if you didn't specify the Type as key and simply using the 
        //DynamoDBMapper#batchLoad(Iterable<? extends Object> itemsToGet)
        //then the Type of Iterable should have annotated with @DynamoDBTable


        Map<Class<?>, List<KeyPair>> keyPairForTable = new HashMap<>();
        keyPairForTable.put(Photo.class, keyPairs);
        Map<String, List<Object>> listMap = dynamoDBMapper.batchLoad(keyPairForTable);

        //result map contains key as dynamoDBtable name of Photo.class
        //entity(test_photo) and values as matching results of given ids

        String tableName = dynamoDBMapper.generateCreateTableRequest(Photo.class)
                                         .getTableName();
        return listMap.get(tableName).stream()
                                     .map(e -> (Photo) e)
                                     .collect(Collectors.toList());
    }
}
  • Test Class

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;
import com.amazonaws.services.dynamodbv2.document.DynamoDB;
import com.amazonaws.services.dynamodbv2.document.Table;
import com.amazonaws.services.dynamodbv2.document.TableCollection;
import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
import com.amazonaws.services.dynamodbv2.model.ListTablesRequest;
import com.amazonaws.services.dynamodbv2.model.ListTablesResult;
import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
import com.test.demo.dynamodb.Application;
import com.test.demo.dynamodb.entity.Photo;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.context.junit.jupiter.SpringExtension;

import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.UUID;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

@ActiveProfiles("test")
@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = Application.class,
        webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class DynamoDBFindByIdsITest {

    @Autowired
    private DynamoDBMapper dynamoDBMapper = null;

    @Autowired
    private DynamoDB dynamoDB = null;

    @Autowired
    private PhotoRepository photoRepository = null;


    @Test
    void findByIdsTest() throws InterruptedException {
        //Creating dynamodb table if not already exists
        createDataTableIfNotExists("test", Photo.class);
        int size = 5;
        //creating dummy entries for test and persisting and collecting it to
        //validate with results
        List<Photo> photos = IntStream.range(0, size)
                .mapToObj(e -> UUID.randomUUID().toString())
                .map(id ->
                        Photo.builder()
                                .id(id)
                                .title("Dummy title")
                                .url("http://photos.info/" + id)
                                .thumbnailUrl("http://photos.info/thumbnails/" + id)
                                .build()
                ).peek(dynamoDBMapper::save)
                .collect(Collectors.toList());

        //calling findByIds with the Collection of HashKey ids (Partition Key Ids)
        Set<String> photoIds = photos.stream()
                .map(Photo::getId)
                .collect(Collectors.toSet());
        List<Photo> photosResultSet = photoRepository.findByIds(photoIds);

        Assertions.assertEquals(size, photosResultSet.size());

        //validating returned photoIds with the created Ids
        Set<String> resultedPhotoIds = photosResultSet.stream()
                .map(Photo::getId)
                .collect(Collectors.toSet());
        Assertions.assertTrue(photoIds.containsAll(resultedPhotoIds));
    }

    public <T> void createDataTableIfNotExists(String tablePrefix, Class<T> clazz)
            throws InterruptedException {
        ListTablesRequest listTablesRequest = new ListTablesRequest();
        listTablesRequest.setExclusiveStartTableName(tablePrefix);
        TableCollection<ListTablesResult> tables = dynamoDB.listTables();
        List<String> tablesList = new ArrayList<>();
        tables.forEach((tableResult) -> {
            tablesList.add(tableResult.getTableName());
        });
        String tableName = dynamoDBMapper.generateCreateTableRequest(clazz).getTableName();
        if (!tablesList.contains(tableName)) {
            CreateTableRequest tableRequest = dynamoDBMapper.generateCreateTableRequest(clazz);
            tableRequest.withProvisionedThroughput(new ProvisionedThroughput(5L, 5L));
            Table table = dynamoDB.createTable(tableRequest);
            table.waitForActive();
        }
    }
}

Try this in C#. I think it's similar in Java. UserId it`s the hask key.

        var table = Table.LoadTable(DynamoClient, "YourTableName");
        var batchGet = table.CreateBatchGet();
        batchGet.AddKey(new Dictionary<string, DynamoDBEntry>() { { "UserId", 123 } });
        batchGet.AddKey(new Dictionary<string, DynamoDBEntry>() { { "UserId", 456 } });
        batchGet.Execute();
        var results = batchGet.Results;

Sharing my openings as of today.

GetItem, Query, Scan

Using normal DynamoDB operations you're allowed to query either only one hash key per request (using GetItem or Query operations) or all hash keys at once (using the Scan operation).

BatchGetItem

You can utilize the BatchGetItem operation, but it requires you to specify a full primary key (including the range key if you have one).

PartiQL

Since recently you can also use PartiQL - a query language supported by AWS to query DynamoDB tables. Using it, you can query several hash keys, using, for example, the IN operator:

SELECT * FROM  "table_name" WHERE "status" IN ['assigned', 'processing'];

I used PartiQL in my Python code, not Java, so that I cannot provide implementation details. But it should be quite easy to find since you know that you need yo use PartiQL. I'll leave here a reference for Python, just in case.

You might have a look at BatchGetItem operation or the batchLoad() method of the DynamoDBMapper. Although a little different than a query in that it's not a query with an OR condition on the hash key, it will allow you to accomplish (generally) the same thing. Here is the language agnostic documentation and here is the Javadoc.

Amazon API doesn't support multiple hashkey filter but you can use HASH KEY + RANGE KEY filter to get the results using batchGetItem method ..

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/batch-operation-lowlevel-java.html#LowLevelJavaBatchGet

Autorizzato sotto: CC-BY-SA insieme a attribuzione
Non affiliato a StackOverflow
scroll top