FetchXml System.OutOfMemory Exception

Question

As I've learnt from my google searches, MSCRM 2011 retrieves maximum 5000 entities but I want all my entities from a marketing list. As written on the web creating "TurnOffFetchThrottling" field on HKLM\Software\Microsoft\MSCRM and set a value as 1 can solve this 5000 limit problem (In addition that I added MaxRowsPerPage field in the registry and set its values as more than 5000, but it also doesnt work). I tried it and I got System.OutOfMemory Exception error. By the way, when I delete "" and just get id attribute code works perfectly, but I need all attributes. Here my fetchxml code:

enter code here
string fetchXml = "<fetch mapping='logical' >"
                  + "<entity name='" + entityType.LogicalName + "'>"
                    + "<all-attributes />"
                    + "<link-entity name='listmember' to='" + entityType.LogicalName + "id" + "' from='entityid'>"
                        + "<filter>"
                          + "<condition attribute='listid' operator='eq' value='" + chosenMarketingListGuid + "'/>"
                        + "</filter>"
                    + "</link-entity>"
                  + "</entity>"
                + "</fetch>";

I tried one more thing, I changed the fetchxml as:

enter code here
string fetchXml = "<fetch mapping='logical' >"
                  + "<entity name='listmember'>"
                    + "<all-attributes />"
                        + "<filter>"
                          + "<condition attribute='listid' operator='eq' value='" + chosenMarketingListGuid + "'/>"
                        + "</filter>"
                  + "</entity>"
                + "</fetch>";

As it is seen, I tried to retrieve just memberlists instead of contact/lead/account enttiy types, and it works! Howver I need contact/lead/account entity types not memberlist. I'll be very grateful, if anyone helps me get out of this dark MSCRM tunnel!

here, full stack trace:

[OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.] System.ServiceModel.Security.SecurityUtils.ReadContentAsBase64(XmlDictionaryReader reader, Int64 maxBufferSize) +197 System.ServiceModel.Security.EncryptedData.ReadCipherData(XmlDictionaryReader reader, Int64 maxBufferSize) +17 System.ServiceModel.Security.EncryptedType.ReadFrom(XmlDictionaryReader reader, Int64 maxBufferSize) +858 System.ServiceModel.Security.WSSecurityOneDotZeroReceiveSecurityHeader.DecryptBody(XmlDictionaryReader bodyContentReader, SecurityToken token) +80 System.ServiceModel.Security.WSSecurityOneDotZeroReceiveSecurityHeader.ExecuteMessageProtectionPass(Boolean hasAtLeastOneSupportingTokenExpectedToBeSigned) +1611 System.ServiceModel.Security.ReceiveSecurityHeader.Process(TimeSpan timeout, ChannelBinding channelBinding, ExtendedProtectionPolicy extendedProtectionPolicy) +1576 System.ServiceModel.Security.MessageSecurityProtocol.ProcessSecurityHeader(ReceiveSecurityHeader securityHeader, Message& message, SecurityToken requiredSigningToken, TimeSpan timeout, SecurityProtocolCorrelationState[] correlationStates) +205 System.ServiceModel.Security.SymmetricSecurityProtocol.VerifyIncomingMessageCore(Message& message, String actor, TimeSpan timeout, SecurityProtocolCorrelationState[] correlationStates) +637 System.ServiceModel.Security.MessageSecurityProtocol.VerifyIncomingMessage(Message& message, TimeSpan timeout, SecurityProtocolCorrelationState[] correlationStates) +371 System.ServiceModel.Channels.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout) +471 System.ServiceModel.Channels.SecurityRequestChannel.Request(Message message, TimeSpan timeout) +175 System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout) +22 System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) +517 System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) +88 System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) +453 System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) +237 Microsoft.Xrm.Sdk.IOrganizationService.RetrieveMultiple(QueryBase query) +0 Microsoft.Xrm.Sdk.Client.OrganizationServiceProxy.RetrieveMultipleCore(QueryBase query) +626 Microsoft.Xrm.Sdk.Client.OrganizationServiceProxy.RetrieveMultiple(QueryBase query) +39 IMPlugin.MarketingListHelper.getMembersAndCountOfChosenMarketingList(OrganizationServiceProxy service, Guid chosenMarketingListGuid, Entity entityType) in C:\Users\Zafer\Documents\Visual Studio 2010\Projects\IMPlugin\MarketingListHelper.cs:130 IMPlugin.IM_SMS.fillMainPanel(Double mainPanelHeight) in C:\Users\Zafer\Documents\Visual Studio 2010\Projects\IMPlugin\IM_SMS.aspx.cs:96 IMPlugin.IM_SMS.Page_Load(Object sender, EventArgs e) in C:\Users\Zafer\Documents\Visual Studio 2010\Projects\IMPlugin\IM_SMS.aspx.cs:42 System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e) +14 System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e) +35 System.Web.UI.Control.OnLoad(EventArgs e) +91 System.Web.UI.Control.LoadRecursive() +74 System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +2207

Answer 1

The reason the 5000 limit exists is as a crude barrier to prevent exactly the issue you are encountering. You've by-passed the software limitation, you now need to overcome the hardware limitation (out of memory)...

You don't state how you are writing your code or what you are using your results for but I presume that there is additional processing that will following execution of your query. On this basis, my recommendation is that you introduce paging into your query expression and process records in batches of 5000.

A crude example:

// get records to process
string pagingCookie = null;
bool moreRecords = false;
int pageNum = 1;
DataCollection<Entity> myBatchOfRecords;    

bool continueProcessing = true;
while (continueProcessing)
{
    myBatchOfRecords = GetRecords(pageNum, ref pagingCookie, ref moreRecords);

    // process those records
    ProcessRecords(myBatchOfRecords);
    pageNum++;    
    continueProcessing = moreRecords;
}

function DataCollection<Entity> GetRecordsList(int pageNumber, ref string pagingCookie, ref bool moreRecords){
    var query = new QueryExpression(blahblahblah...);
    query.PageInfo = new PagingInfo { 
        Count = 5000, 
        PageNumber = pageNumber, 
        PagingCookie = pagingCookie };
    var results =  myOrgService.RetrieveMultiple(query);
    pagingCookie = matchedContacts.PagingCookie;
    moreRecords = matchedContacts.MoreRecords;
    return results;
}