Search engine too slow

Question

In my database there is a table Question with Content and Answer. I implemented a search engine which uses this:

private Expression<Func<QuestionModel, bool>> CreatePredicate(string query, bool negate = false)
        {
            query = query.ToUpper();
            if (IsATag(ref query))
            {
                return question => negate == question.Tags.Any(tag => tag.NamesCollection
                                                                .Any(translation =>
                                                                     translation.Language == currentLanguage &&
                                                                     translation.Translation.ToUpper().Equals(query)));
            }
            else if(IsSpecial(ref query))
            {
                return question => true;
            }
            else
            {
                return question => negate == question.Content.ToUpper().Contains(query) ||
                                   question.Answer.ToUpper().Contains(query) ||
                                   question.Specialization.NamesCollection.Any(trans =>
                                        trans.Language == currentLanguage &&
                                        trans.Translation.ToUpper().Contains(query));
            }
        } 

Actually it supports multiple queries but this is nicely handled by PrebicateBuilder

I created some performance tests and it doesn't run very fast for 10K records (up to 30 seconds if I use 5 queries. Is there a smart way to enhance the process? Honestly I've never seen a website running so slow, so there must be something.

This is the sql generated by my query:

SELECT TOP (100) 
[top].[Id] AS [Id], 
[top].[Content] AS [Content], 
[top].[Answer] AS [Answer], 
[top].[Assessment] AS [Assessment], 
[top].[State] AS [State], 
[top].[AuthorId] AS [AuthorId], 
[top].[AttachmentFileName] AS [AttachmentFileName], 
[top].[SubmissionDate] AS [SubmissionDate], 
[top].[Specialization_Id] AS [Specialization_Id], 
[top].[Doctor_Id] AS [Doctor_Id]
FROM ( SELECT [Project16].[Id] AS [Id], [Project16].[Content] AS [Content], [Project16].[Answer] AS [Answer], [Project16].[Assessment] AS [Assessment], [Project16].[State] AS [State], [Project16].[AuthorId] AS [AuthorId], [Project16].[AttachmentFileName] AS [AttachmentFileName], [Project16].[SubmissionDate] AS [SubmissionDate], [Project16].[Specialization_Id] AS [Specialization_Id], [Project16].[Doctor_Id] AS [Doctor_Id]
    FROM ( SELECT 
        [Extent1].[Id] AS [Id], 
        [Extent1].[Content] AS [Content], 
        [Extent1].[Answer] AS [Answer], 
        [Extent1].[Assessment] AS [Assessment], 
        [Extent1].[State] AS [State], 
        [Extent1].[AuthorId] AS [AuthorId], 
        [Extent1].[AttachmentFileName] AS [AttachmentFileName], 
        [Extent1].[SubmissionDate] AS [SubmissionDate], 
        [Extent1].[Specialization_Id] AS [Specialization_Id], 
        [Extent1].[Doctor_Id] AS [Doctor_Id]
        FROM [QuestionModels] AS [Extent1]
        WHERE ((@p__linq__0 = (CASE WHEN ((CHARINDEX(@p__linq__1, UPPER([Extent1].[Content]))) > 0) THEN cast(1 as bit) WHEN ( NOT ((CHARINDEX(@p__linq__1, UPPER([Extent1].[Content]))) > 0)) THEN cast(0 as bit) END)) OR ((CHARINDEX(@p__linq__2, UPPER([Extent1].[Answer]))) > 0) OR ( EXISTS (SELECT 
            1 AS [C1]
            FROM [TranslationModels] AS [Extent2]
            WHERE ([Extent2].[SpecializationModel_Id] IS NOT NULL) AND ([Extent1].[Specialization_Id] = [Extent2].[SpecializationModel_Id]) AND ([Extent2].[Language] = @p__linq__3) AND ((CHARINDEX(@p__linq__4, UPPER([Extent2].[Translation]))) > 0)
        )) OR (@p__linq__5 = (CASE WHEN ((CHARINDEX(@p__linq__6, UPPER([Extent1].[Content]))) > 0) THEN cast(1 as bit) WHEN ( NOT ((CHARINDEX(@p__linq__6, UPPER([Extent1].[Content]))) > 0)) THEN cast(0 as bit) END)) OR ((CHARINDEX(@p__linq__7, UPPER([Extent1].[Answer]))) > 0) OR ( EXISTS (SELECT 
            1 AS [C1]
            FROM [TranslationModels] AS [Extent3]
            WHERE ([Extent3].[SpecializationModel_Id] IS NOT NULL) AND ([Extent1].[Specialization_Id] = [Extent3].[SpecializationModel_Id]) AND ([Extent3].[Language] = @p__linq__8) AND ((CHARINDEX(@p__linq__9, UPPER([Extent3].[Translation]))) > 0)
        )) OR (@p__linq__10 = (CASE WHEN ((CHARINDEX(@p__linq__11, UPPER([Extent1].[Content]))) > 0) THEN cast(1 as bit) WHEN ( NOT ((CHARINDEX(@p__linq__11, UPPER([Extent1].[Content]))) > 0)) THEN cast(0 as bit) END)) OR ((CHARINDEX(@p__linq__12, UPPER([Extent1].[Answer]))) > 0) OR ( EXISTS (SELECT 
            1 AS [C1]
            FROM [TranslationModels] AS [Extent4]
            WHERE ([Extent4].[SpecializationModel_Id] IS NOT NULL) AND ([Extent1].[Specialization_Id] = [Extent4].[SpecializationModel_Id]) AND ([Extent4].[Language] = @p__linq__13) AND ((CHARINDEX(@p__linq__14, UPPER([Extent4].[Translation]))) > 0)
        )) OR (@p__linq__15 = (CASE WHEN ( EXISTS (SELECT 
            1 AS [C1]
            FROM ( SELECT 
                [Extent5].[TagModel_Id] AS [TagModel_Id]
                FROM [TagModelQuestionModels] AS [Extent5]
                WHERE [Extent1].[Id] = [Extent5].[QuestionModel_Id]
            )  AS [Project4]
            WHERE  EXISTS (SELECT 
                1 AS [C1]
                FROM [TranslationModels] AS [Extent6]
                WHERE ([Project4].[TagModel_Id] = [Extent6].[TagModel_Id]) AND ([Extent6].[Language] = @p__linq__16) AND ((UPPER([Extent6].[Translation])) = @p__linq__17)
            )
        )) THEN cast(1 as bit) WHEN ( NOT EXISTS (SELECT 
            1 AS [C1]
            FROM ( SELECT 
                [Extent7].[TagModel_Id] AS [TagModel_Id]
                FROM [TagModelQuestionModels] AS [Extent7]
                WHERE [Extent1].[Id] = [Extent7].[QuestionModel_Id]
            )  AS [Project7]
            WHERE  EXISTS (SELECT 
                1 AS [C1]
                FROM [TranslationModels] AS [Extent8]
                WHERE ([Project7].[TagModel_Id] = [Extent8].[TagModel_Id]) AND ([Extent8].[Language] = @p__linq__16) AND ((UPPER([Extent8].[Translation])) = @p__linq__17)
            )
        )) THEN cast(0 as bit) END))) AND (@p__linq__18 = (CASE WHEN ( EXISTS (SELECT 
            1 AS [C1]
            FROM ( SELECT 
                [Extent9].[TagModel_Id] AS [TagModel_Id]
                FROM [TagModelQuestionModels] AS [Extent9]
                WHERE [Extent1].[Id] = [Extent9].[QuestionModel_Id]
            )  AS [Project10]
            WHERE  EXISTS (SELECT 
                1 AS [C1]
                FROM [TranslationModels] AS [Extent10]
                WHERE ([Project10].[TagModel_Id] = [Extent10].[TagModel_Id]) AND ([Extent10].[Language] = @p__linq__19) AND ((UPPER([Extent10].[Translation])) = @p__linq__20)
            )
        )) THEN cast(1 as bit) WHEN ( NOT EXISTS (SELECT 
            1 AS [C1]
            FROM ( SELECT 
                [Extent11].[TagModel_Id] AS [TagModel_Id]
                FROM [TagModelQuestionModels] AS [Extent11]
                WHERE [Extent1].[Id] = [Extent11].[QuestionModel_Id]
            )  AS [Project13]
            WHERE  EXISTS (SELECT 
                1 AS [C1]
                FROM [TranslationModels] AS [Extent12]
                WHERE ([Project13].[TagModel_Id] = [Extent12].[TagModel_Id]) AND ([Extent12].[Language] = @p__linq__19) AND ((UPPER([Extent12].[Translation])) = @p__linq__20)
            )
        )) THEN cast(0 as bit) END))
    )  AS [Project16]
    ORDER BY [Project16].[SubmissionDate] ASC
    OFFSET 1000 ROWS 
)  AS [top]

These db relations don't seem to affect performance very much. If I use a query which doesn't involve them the speed is pretty much the same.

Answer 1

Table full scans and twice-nested per-row subqueries will be slow. A faster approach is to arrange your database such that a query need involve only a single index seek or index range scan.

Many people who want to build a search feature use a search index of some kind. Some relational databases support a limited form of full-text search, and this may suffice for your case.

Many people prefer to use a specialized search index. Lucene is a popular engine, with Solr and Elasticsearch exposing Lucene over the network.