Need help building the best performing SQL query

Question

I'm making a Listener, so that I'll be able to post gamedata form multiple games to my server. I will use that to keep clients in sync, no matter what device they play on! This part I'm done with, Using Entity and Linq for inserts and loading playerdata when they are desync'ed.

Every webpost to the server takes about 2-10 ms, depending on the data beeing sent. (which i hope is fine!). This is mostly FirstOrDefault selects, modyfing some data and SaveChanges.

But the server also has to feed the clients with Scoreboards. both "Today/All Time" and "Everyone/Friends only".

I tried with some Linq SQL queries. but when i look at the output I'm really scared of how its gonna preform (with hopfully thousands of users, and millions of score records).

Here's an example:

using (BertEntities BGE = new BertEntities())
{
    var query =
       (from score in BGE.ScoreTable
        join game in BGE.GameSession on score.GameSessionID equals game.ID
        join user in BGE.User on game.UserID equals user.ID
        where
                game.PublicHighScore == true &&
                game.Banned == false &&
                game.GameID == GameID &&
                score.LevelID == level
        orderby score.Score descending

        select new ScoreResult
        {
            ID = user.ID,
            Name = user.FacebookName,
            Score = score.Score,
            Streak = score.Streak,
            Time = score.Time
        }).Skip(skip).Take(count);


    return JsonConvert.SerializeObject(query.ToList());
}

Which gets executed like this

SELECT TOP (100)
    [Project1].[Score] AS [Score],
    [Project1].[ID] AS [ID],
    [Project1].[FacebookName] AS [FacebookName],
    [Project1].[Streak] AS [Streak],
    [Project1].[Time] AS [Time]
    FROM ( SELECT [Project1].[Score] AS [Score], [Project1].[Streak] AS [Streak], [Project1].[Time] AS [Time], [Project1].[ID] AS [ID], [Project1].[FacebookName] AS [FacebookName], row_number() OVER (ORDER BY [Project1].[Score] ASC) AS [row_number]
        FROM ( SELECT
        [Filter1].[Score] AS [Score],
        [Filter1].[Streak] AS [Streak],
        [Filter1].[Time] AS [Time],
        [Extent3].[ID] AS [ID],
        [Extent3].[FacebookName] AS [FacebookName]
        FROM   (SELECT [Extent1].[LevelID] AS [LevelID], [Extent1].[Score] AS [Score], [Extent1].[Streak] AS [Streak], [Extent1].[Time] AS [Time], [Extent2].[UserID] AS [UserID], [Extent2].[GameID] AS [GameID]
            FROM  [dbo].[ScoreTable] AS [Extent1]
            INNER JOIN [dbo].[GameSession] AS [Extent2] ON [Extent1].[GameSessionID] = [Extent2].[ID]
            WHERE (1 = [Extent2].[PublicHighScore]) AND (0 = [Extent2].[Banned]) ) AS [Filter1]
        INNER JOIN [dbo].[User] AS [Extent3] ON [Filter1].[UserID] = [Extent3].[ID]
        WHERE ([Filter1].[GameID] = @p__linq__0) AND ([Filter1].[LevelID] = @p__linq__1)
    )  AS [Project1]
)  AS [Project1]
WHERE [Project1].[row_number] > 0
ORDER BY [Project1].[Score] DESC

I'm no SQL expert, but that looks pretty tough to me. (Taking into account that I want this to scale, even with millions of records and thousands of requests without killing the server.)

This is the database layout: www.invokergame.com/db.png

I'm thinking of storing all the data I need in the ScoreTable table, so I'd avoid making the joins. but that would result in a lot of duplicate data, and I'm pretty sure that's the wrong approach.

Maybe I should forget about Entity Linq, and write it in ADO.NET SQL (which I'd need help with)?

Sorry about the wall of text, I'm looking forward for some wise words!

Edit:

Okay TomTom, Is this what you ment by faking it? Dragging all the data I need into one dataset. And then use LINQ to update or remove and select items from this list, when a user makes a new highscore, or disable PublicHighScore?

And how will this perform?

public class ScoreCacheItem
{
    public Int64 ScoreTableID { get; set; }
    public Guid GameID { get; set; }
    public int LevelID { get; set; }
    public Int64 UserID { get; set; }
    public string Name { get; set; }
    public int Score { get; set; }
    public int Streak { get; set; }
    public double Time { get; set; }
    public DateTime Today { get; set; }
    public int Today_Score { get; set; }
    public int Today_Streak { get; set; }
    public double Today_Time { get; set; }
}


public sealed class ScoreCacheSystem
{
    private List<ScoreCacheItem> FScoreCache = new List<ScoreCacheItem>();
    private object FLock = new object();

    private List<ScoreCacheItem> LoadAllScores()
    {
        using (BertEntities BGE = new BertEntities())
        {
            var query =
               (from score in BGE.ScoreTable
                join game in BGE.GameSession on score.GameSessionID equals game.ID
                join user in BGE.User on game.UserID equals user.ID
                where
                        game.PublicHighScore == true &&
                        game.Banned == false 
                orderby score.Score descending

                select new ScoreCacheItem
                {
                    ScoreTableID = score.ID,
                    GameID = game.GameID,
                    LevelID = score.LevelID,
                    UserID = user.ID,
                    Name = user.FacebookName,
                    Score = score.Score,
                    Streak = score.Streak,
                    Time = score.Time,
                    Today = score.Today,
                    Today_Score = score.Today_Score,
                    Today_Streak = score.Today_Streak,
                    Today_Time = score.Today_Time
                });
            return query.ToList();
        }
    }

    public List<ScoreCacheItem> HighScores {
        get
        {
            lock (FLock)
            {
                if (FScoreCache == null)
                {
                    FScoreCache = LoadAllScores();
                }
                return FScoreCache;
            }
        }
    }
}

Answer 1

You have a serious problem in the SQL side. EF does quite good SQL - but here I would argue that the use of a SQL Server in general is bad.

If you need to have this type of query answered super fast, then do that from memory - not from a database server. In-memory lookups are typical for simple queries that are repetitive and time sensitive for data distribution (financial trading - no one would write prices into a database then have the client pull them; you distribute them while writing them to a store).

Otherwise you just can fake it - caching comes to my mind. There is not a lot you can do to optimize the requirement for a significant database server with this architecture. Make sure your indices are in place. Work from the query plans to make sure you do not miss anything.

But really, try to avoid hitting SQL Server at all - cache output at all levels. And the SQL is quite trivial, BTW. Things get interesting when you hit 2-3 screen long analysis queries ;) Millions of records are not the issue, the main issue will be concurrent requests. The connection better have read-committed isolation to make sure you do not set locks. And then get a proper server and scale out unless you change the architecture.

Im thinking of storing all the data I need in the ScoreTable table, so I'd avoid making the joins. but that would result in alot of duplicate data, and im pretty sure thats the wrong approach.

Absolutely not - like not the wrong approach. Denormalization is typical in a read-heavy reporting database. You can use triggers to maintain this table automatically. Whether it makes sense in your situation must be tested with a good data set. It is not generally bad but standard practice in data warehousing, although those normally deal with real amounts of data (billions of rows, not something tiny like a million or two). In my last larger data warehousing project we were loading around 450 million rows a day - and had to archive them for 10 years for reporting. Look up "Star Schema" on Google. Rules are different for basically read only / data warehousing scenarios.