No, as of today there is no way to send multiple queries in the same request. If you're concerned about latency, you could make multiple requests simultaneously in different threads. This would require the same amount of network bandwidth as a "dual query" would if Dynamo offered it (assuming you're making 2, not hundreds).
Is there a way to query multiple hash keys in DynamoDB?
-
03-06-2022 - |
سؤال
Is there a way to query multiple hash keys using a single query in Amazon's AWS SDK for Java?
Here's my issue; I have a DB table for project statuses. The Hash Key is the status of a project (ie: new, assigned, processing, or complete). The range key is a set of project IDs. Currently, I've got a query setup to simply find all the projects listed as a status(hash) of "assigned" and another query set to look for a status of "processing". Is there a way to do this using a single query rather than sending multiple queries for each status I need to find? Code below:
DynamoDBMapper mapper = new DynamoDBMapper(new AmazonDynamoDBClient(credentials));
PStatus assignedStatus = new PStatus();
assignedStatus.setStatus("assigned");
PStatus processStatus = new PStatus();
processStatus.setStatus("processing");
DynamoDBQueryExpression<PStatus> queryAssigned = new DynamoDBQueryExpression<PStatus>().withHashKeyValues(assignedStatus);
DynamoDBQueryExpression<PStatus> queryProcessing = new DynamoDBQueryExpression<PStatus>().withHashKeyValues(processStatus);
List<PStatus> assigned = mapper.query(PStatus.class, queryAssigned);
List<PStatus> process = mapper.query(PStatus.class, queryProcessing);
So basically, I'd like to know if it's possible to eliminate the queryAssigned
and assigned
variables and handle both assignedStatus
and processStatus
via the same query, process
, to find projects that are not new or complete.
المحلول
نصائح أخرى
There is no way to query by multiple hash keys, but, as of April 2014, you can use QueryFilter so you can filter by non key fields in addition to hash key fields.
In a blog post on 24 April 2014, AWS announced the release of the "QueryFilter" option:
With today's release, we are extending this model with support for query filtering on non-key attributes. You can now include a QueryFilter as part of a call to the Query function. The filter is applied after the key-based retrieval and before the results are returned to you. Filtering in this manner can reduce the amount of data returned to your application while also simplifying and streamlining your code
Check this out there http://aws.amazon.com/blogs/aws/improved-queries-and-updates-for-dynamodb/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+AmazonWebServicesBlog+%28Amazon+Web+Services+Blog%29
Sharing my working answer for posterity. As of Oct 2020, there is a way to query multiple hash keys using a single query using aws-java-sdk-dynamodb-1.11.813.jar
. I had the same requirement where I had to select items based on multiple hash keys(partition keys), and you can relate the requirement with the RDMS scenario, similar to the query select * from photo where id in ('id1','id2','id3')
, here id is the primary key of the photo
table.
Code Snippet
- DynamoDB entity
package com.test.demo.dynamodb.entity;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBHashKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTable;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.NoArgsConstructor;
import java.io.Serializable;
@NoArgsConstructor
@AllArgsConstructor
@lombok.Data
@DynamoDBTable(tableName = "test_photos")
@Builder
public class Photo implements Serializable {
@DynamoDBHashKey
private String id;
private String title;
private String url;
private String thumbnailUrl;
}
- DynamoDB Repository Class
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;
import com.amazonaws.services.dynamodbv2.datamodeling.KeyPair;
import com.test.demo.dynamodb.entity.Photo;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Repository;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
@Repository
public class PhotoRepository {
@Autowired
private DynamoDBMapper dynamoDBMapper = null;
public List<Photo> findByIds(Collection<String> photoIds) {
//Constructing `KeyPair` instance and setting the HashKey,
// in this example I have only hash key,
// if you have RangeKey(Sort) you can set that also here using KeyPair#withRangeKey
List<KeyPair> keyPairs = photoIds.stream()
.map(id -> new KeyPair().withHashKey(id))
.collect(Collectors.toList());
//Creating Map where Key as Class<?> and value as a list of created keyPairs
//you can also directly use batchLoad(List<Photo> itemsToGet), the only constraint
//is if you didn't specify the Type as key and simply using the
//DynamoDBMapper#batchLoad(Iterable<? extends Object> itemsToGet)
//then the Type of Iterable should have annotated with @DynamoDBTable
Map<Class<?>, List<KeyPair>> keyPairForTable = new HashMap<>();
keyPairForTable.put(Photo.class, keyPairs);
Map<String, List<Object>> listMap = dynamoDBMapper.batchLoad(keyPairForTable);
//result map contains key as dynamoDBtable name of Photo.class
//entity(test_photo) and values as matching results of given ids
String tableName = dynamoDBMapper.generateCreateTableRequest(Photo.class)
.getTableName();
return listMap.get(tableName).stream()
.map(e -> (Photo) e)
.collect(Collectors.toList());
}
}
- Test Class
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;
import com.amazonaws.services.dynamodbv2.document.DynamoDB;
import com.amazonaws.services.dynamodbv2.document.Table;
import com.amazonaws.services.dynamodbv2.document.TableCollection;
import com.amazonaws.services.dynamodbv2.model.CreateTableRequest;
import com.amazonaws.services.dynamodbv2.model.ListTablesRequest;
import com.amazonaws.services.dynamodbv2.model.ListTablesResult;
import com.amazonaws.services.dynamodbv2.model.ProvisionedThroughput;
import com.test.demo.dynamodb.Application;
import com.test.demo.dynamodb.entity.Photo;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.extension.ExtendWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.context.ActiveProfiles;
import org.springframework.test.context.junit.jupiter.SpringExtension;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;
import java.util.UUID;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
@ActiveProfiles("test")
@ExtendWith(SpringExtension.class)
@SpringBootTest(classes = Application.class,
webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
public class DynamoDBFindByIdsITest {
@Autowired
private DynamoDBMapper dynamoDBMapper = null;
@Autowired
private DynamoDB dynamoDB = null;
@Autowired
private PhotoRepository photoRepository = null;
@Test
void findByIdsTest() throws InterruptedException {
//Creating dynamodb table if not already exists
createDataTableIfNotExists("test", Photo.class);
int size = 5;
//creating dummy entries for test and persisting and collecting it to
//validate with results
List<Photo> photos = IntStream.range(0, size)
.mapToObj(e -> UUID.randomUUID().toString())
.map(id ->
Photo.builder()
.id(id)
.title("Dummy title")
.url("http://photos.info/" + id)
.thumbnailUrl("http://photos.info/thumbnails/" + id)
.build()
).peek(dynamoDBMapper::save)
.collect(Collectors.toList());
//calling findByIds with the Collection of HashKey ids (Partition Key Ids)
Set<String> photoIds = photos.stream()
.map(Photo::getId)
.collect(Collectors.toSet());
List<Photo> photosResultSet = photoRepository.findByIds(photoIds);
Assertions.assertEquals(size, photosResultSet.size());
//validating returned photoIds with the created Ids
Set<String> resultedPhotoIds = photosResultSet.stream()
.map(Photo::getId)
.collect(Collectors.toSet());
Assertions.assertTrue(photoIds.containsAll(resultedPhotoIds));
}
public <T> void createDataTableIfNotExists(String tablePrefix, Class<T> clazz)
throws InterruptedException {
ListTablesRequest listTablesRequest = new ListTablesRequest();
listTablesRequest.setExclusiveStartTableName(tablePrefix);
TableCollection<ListTablesResult> tables = dynamoDB.listTables();
List<String> tablesList = new ArrayList<>();
tables.forEach((tableResult) -> {
tablesList.add(tableResult.getTableName());
});
String tableName = dynamoDBMapper.generateCreateTableRequest(clazz).getTableName();
if (!tablesList.contains(tableName)) {
CreateTableRequest tableRequest = dynamoDBMapper.generateCreateTableRequest(clazz);
tableRequest.withProvisionedThroughput(new ProvisionedThroughput(5L, 5L));
Table table = dynamoDB.createTable(tableRequest);
table.waitForActive();
}
}
}
Try this in C#. I think it's similar in Java. UserId it`s the hask key.
var table = Table.LoadTable(DynamoClient, "YourTableName");
var batchGet = table.CreateBatchGet();
batchGet.AddKey(new Dictionary<string, DynamoDBEntry>() { { "UserId", 123 } });
batchGet.AddKey(new Dictionary<string, DynamoDBEntry>() { { "UserId", 456 } });
batchGet.Execute();
var results = batchGet.Results;
Sharing my openings as of today.
GetItem, Query, Scan
Using normal DynamoDB operations you're allowed to query either only one hash key per request (using GetItem
or Query
operations) or all hash keys at once (using the Scan
operation).
BatchGetItem
You can utilize the BatchGetItem
operation, but it requires you to specify a full primary key (including the range key if you have one).
PartiQL
Since recently you can also use PartiQL - a query language supported by AWS to query DynamoDB tables. Using it, you can query several hash keys, using, for example, the IN
operator:
SELECT * FROM "table_name" WHERE "status" IN ['assigned', 'processing'];
I used PartiQL in my Python code, not Java, so that I cannot provide implementation details. But it should be quite easy to find since you know that you need yo use PartiQL. I'll leave here a reference for Python, just in case.
You might have a look at BatchGetItem
operation or the batchLoad()
method of the DynamoDBMapper
. Although a little different than a query in that it's not a query with an OR
condition on the hash key, it will allow you to accomplish (generally) the same thing. Here is the language agnostic documentation and here is the Javadoc.
Amazon API doesn't support multiple hashkey filter but you can use HASH KEY + RANGE KEY filter to get the results using batchGetItem method ..