Java - Google App Engine - modelling graph structures in Google Datastore

Question

Google Apps Engine offers the Google Datastore as the only NoSQL database (I think it is based on BigTable).

In my application I have a social-like data structure and I want to model it as I would do in a graph database. My application must save heterogeneous objects (users,files,...) and relationships among them (such as user1 OWNS file2, user2 FOLLOWS user3, and so on).

I'm looking for a good way to model this typical situation, and I thought to two families of solutions:

1. List-based solutions: Any object contains a list of other related objects and the object presence in the list is itself the relationship (as Google said in the JDO part https://developers.google.com/appengine/docs/java/datastore/jdo/relationships).

2. Graph-based solution: Both nodes and relationships are objects. The objects exist independently from the relationships while each relationship contain a reference to the two (or more) connected objects.

What are strong and weak points of these two approaches?

About approach 1: This is the simpler approach one can think of, and it is also presented in the official documentation but:

• Each directed relationship make the object record grow: are there any limitations on the number of the possible relationships given for instance by the object dimension limit?
• Is that a JDO feature or also the datastore structure allows that approach to be naturally implemented?
• The relationship search time will increase with the list, is this solution suitable for large (million) of relationships?

About approach 2: Each relationship can have a higher level of characterization (it is an object and it can have properties). And I think memory size is not a Google problem, but:

• Each relationship requires its own record, so the search time for each related couple will increase as the total number of relationships increase. Is this suitable for large amount of relationships(millions, billions)? I.e. does Google have good tricks to search among records if they are well structured? Or I will be soon in a situation in which if I want to search a friend of User1 called User4 I have to wait seconds?
• On the other side each object doesn't increase in dimension as new relationships are added.

Could you help me to find other important points on the two approaches in such a way to chose the best model?

Answer 1

First, the search time in the Datastore does not depend on the number of entities that you store, only on the number of entities that you retrieve. Therefore, if you need to find one relationship object out of a billion, it will take the same time as if you had just one object.

Second, the list approach has a serious limitation called "exploding indexes". You will have to index the property that contains a list to make it searchable. If you ever use a query that references more than just this property, you will run into this issue - google it to understand the implications.

Third, the list approach is much more expensive. Every time you add a new relationship, you will rewrite the entire entity at considerable writing cost. The reading costs will be higher too if you cannot use keys-only queries. With the object approach you can use keys-only queries to find relationships, and such queries are now free.

UPDATE:

If your relationships are directed, you may consider making Relationship entities children of User entities, and using an Object id as an id for a Relationship entity as well. Then your Relationship entity will have no properties at all, which is probably the most cost-efficient solution. You will be able to retrieve all objects owned by a user using keys-only ancestor queries.

Answer 2

I have an AppEngine application and I use both approaches. Which is better depends on two things: the practical limits of how many relationships there can be and how often the relationships change.

NOTE 1: My answer is based on experience with Objectify and heavy use of caching. Mileage may vary with other approaches.

NOTE 2: I've used the term 'id' instead of the proper DataStore term 'name' here. Name would have been confusing and id matches objectify terms better.

Consider users linked to the schools they've attended and vice versa. In this case, you would do both. Link the users to schools with a variation of the 'List' method. Store the list of school ids the user attended as a UserSchoolLinks entity with a different type/kind but with the same id as the user. For example, if the user's id = '6h30n' store a UserSchoolLinks object with id '6h30n'. Load this single entity by key lookup any time you need to get the list of schools for a user.

However, do not do the reverse for the users that attended a school. For that relationship, insert a link entity. Use a combination of the school's id and the user's id for the id of the link entity. Store both id's in the entity as separate properties. For example, the SchoolUserLink for user '6h30n' attending school 'g3g0a3' gets id 'g3g0a3~6h30n' and contains the fields: school=g3g0a3 and user=6h30n. Use a query on the school property to get all the SchoolUserLinks for a school.

Here's why:

1. Users will see their schools frequently but change them rarely. Using this approach, the user's schools will be cached and won't have to be fetched every time they hit their profile.

2. Since you will be getting the user's schools via a key lookup, you won't be using a query. Therefore, you won't have to deal with eventual consistency for the user's schools.

3. Schools may have many users that attended them. By storing this relationship as link entities, we avoid creating a huge single object.

4. The users that attended a school will change a lot. This way we don't have to write a single, large entity frequently.

5. By using the id of the User entity as the id for the UserSchoolLinks entity we can fetch the links knowing just the id of the user.

6. By combining the school id and the user id as the id for the SchoolUser link. We can do a key lookup to see if a user and school are linked. Once again, no need to worry about eventual consistency for that.

7. By including the user id as a property of the SchoolUserLink we don't need to parse the SchoolUserLink object to get the id of the user. We can also use this field to check consistency between both directions and have a fallback in case somehow people are attending hundreds of schools.

Downsides: 1. This approach violates the DRY principle. Seems like the least of evils here. 2. We still have to use a query to get the users who attended a school. That means dealing with eventual consistency.

Don't forget Update the UserSchoolLinks entity and add/remove the SchoolUserLink entity in a transaction.

Answer 3

You question is too complex but I try explain the best solution (I will answer in Python but same can be done in Java).

class User(db.User):
  followers = db.StringListProperty()

Simple add follower.

user = User.get(key)
user.followers.append(str(followerKey))

This allow fast query who is followed and followers

User.all().filter('followers', followerKey) # -> followed

This query i/o costly so you can make it faster but more complicated and costly in i/o writes:

class User(db.User):
  followers = db.StringListProperty()
  follows = db.StringListProperty()

Whatever this is complicated during changes since delete of Users need update follows so you need 2 writes.

You can also store relationships but it is the worse scenario since it is more complex than second example with followers and follows ... - keep in mind than entity can have 1Mb it is not limit but can be.