Pagination logic in Mainframe CICS

Question

Here is my requirement.

Front (Client) end will do a search based on predefined conditions (for instance: customer id, account number, first name, last name, etc). I need to get the data corresponding to this request from a db2 database and send it back to them (Server). We use CICS channels and containers to pass requests and responses between the Client and Server.

Front end needs the data ordered by: Receive date descending, Customer id Ascending, Account number Ascending. Data are fetched in pages of 500 records. For example, if for a search request from front end would retrieve 50,000 records from the db2 database, we need to return this data in 500 record "pages". For pagination concept, we use the field security deposit number which is primary key to our database but the sorting order is not based on this field.

I would like to know whether we can use scrollable cursor logic in CICS to implement pagination.

Please note that I do not prefer to go for internal array bubble sort to send the data in response as it would degrade performance. I like to do it via query logic.any thoughts?

Example (Initial Front end input request):

• Customer id : A
• First time request (To identify whether it is first time or next or previous request for pagination)
• First security deposit number : 0
• Last security deposit number : 0

Since this is first time request, both this field will be having zero from front end and we need to retrieve records from database based on condition of security deposit > 0

Db2 database:

• There are 700 records for this criteria
• Mainframe response for first time: We will send the first 500 records

Front end will then send request for getting next set of records which will contain:

• Customer id: A
• Next request
• First security deposit numbr: 0
• Last security deposit number : 17980

So for this detail, if I query my datbase based on security deposit number > 17980, it may result in duplicate records listing in the screen once again since our sorting order in database is not based on security deposit number

How to impelement this logic??

Answer 1

Many Client/Server applications in an IBM Mainframe environment involve psuedo conversational CICS transactions. If you are using CICS in psueudo conversational mode it is not possible for the Server to hold cusors when it RETURNs to the Client. Therefore scrollable cusors are of little use in this environment. So to answer your basic question: No scrollable cursors cannot be used here.

The "trick" here is to create an SQL predicate in the Server that is restartable. It will then pick up rows in the correct order from any given stating point. When the Client calls your Server it must pass all of the positioning information to your Server.

Typically, on a first call from a Client all of the positioning values are set to cause the cursor to position itself starting with what must be the the first row. The Server then pulls in a "page" worth of data and returns it to the Client. On the next page forward request the Client sets these positioning values to the last row it displayed and calls the Server for the next "page" of data.

In your situation I would assume that the page forward cursor would look something like this, all the variables prefixed with RESTART... are what the Client must provide to the Server to start the cursor in the correct position.

 DECLARE CURSOR Page-forward FOR
    SELECT Receive_Date, Customer_id, Account_Nbr, Security_Dep_Id
    FROM Table_Name
    WHERE (   (Receive_Date    < :RESTART-RCV-DT)
           OR (Receive_Date    = :RESTART-RCV-DT AND
               Customer_Id     > :RESTART-CUSTOMER-ID)
           OR (Receive_Date    = :RESTART-RCV-DT AND
               Customer_Id     = :RESTART-CUSTOMER-ID AND
               Account_Nbr     > :RESTART-ACCT-NBR)
           OR (Receive_Date    = :RESTART-RCV-DT AND
               Customer_Id     = :RESTART-CUSTOMER-ID AND
               Account_Nbr     = :RESTART-ACCT-NBR AND
               Security_Dep_Id > :RESTART-SEC-DEP-ID))
    ORDER BY 1 DESC, 2 ASC , 3 ASC, 4 ASC

For the initial call the Client would have passed something like '9999-12-31' as the RESTART-RCV-DT, zero for the RESTART-CUSTOMER-ID, RESTART-ACCT-NBR and SEC-DEP-ID (assuming these are all numeric). If you look at the cursor predicate carefully you can verify that there cannot be any rows prior to these values - therefore this will return the first page of data. If the Client needs to page forward after this, it must tell the Server to start with the next row after the last one it received. To do this it would populate the RESTART... variables with the values from the last row on the page it just displayed. This process will drive the cursor selects forward one page at a time.

When paging up, the process is reversed (you will need a second cursor to support this, and the Client needs to tell you which direction to page: Forward or Back). The Client will need to populate the RESTART variables with the first row it recieved from the Server. The trick for the Server on a page up request is to return the data to the Client in reverse order. You may have to populate the data page passed back to the Client in reverse order (ie. put the first row retrieved into the last row of the paging area shared between the Client and the Server). The page backward cursor would look something like:

 DECLARE CURSOR Page-backward FOR
    SELECT Receive_Date, Customer_id, Account_Nbr, Security_Dep_Id
    FROM Table_Name
    WHERE (   (Receive_Date    > :RESTART-RCV-DT)
           OR (Receive_Date    = :RESTART-RCV-DT AND
               Customer_Id     < :RESTART-CUSTOMER-ID)
           OR (Receive_Date    = :RESTART-RCV-DT AND
               Customer_Id     = :RESTART-CUSTOMER-ID AND
               Account_Nbr     < :RESTART-ACCT-NBR)
           OR (Receive_Date    = :RESTART-RCV-DT AND
               Customer_Id     = :RESTART-CUSTOMER-ID AND
               Account_Nbr     = :RESTART-ACCT-NBR AND
               Security_Dep_Id < :RESTART-SEC-DEP-ID))
    ORDER BY 1 ASC, 2 DESC , 3 DESC, 4 DESC

As has been pointed out in other answers, this type of paging process does not manage or detect concurrent updates to the database that may occur duing paging transactions. That is another topic for another day...

Developing Restartable Cursors

The the key to building a paging Server is to develop a cursor that is restartable from a set of values received from a Client transaction. This leaves control of cursor positioning and direction with the Client. It also means the Client must receive all critical positioning data from the Server even though the Client might not actually use these data for any other purpose (e.g. From your question I got the impression that the Client may not require the Security Deposit Id except to supply as a positioning parameter for your Server)

To build a paging Server you need to know what the required sorting order of the data are (e.g. Receive Date Descending then Customer Id Ascending then Account Number Ascending). You also need know the set of data that uniquely identify a row returned by the cursor. In your case that would be the Security Deposit Id (this is the primary key for the table you are selecting from so it must be unique for each and every row in that table). Knowing this you then build a cursor predicate (the stuff in the WHERE clause) that will return data needed by the Client in the required sort order that also includes the full positioning key (i.e. Security Deposit Id). In the event that two or more returned rows may contain identical data if the final positioning key were elimiminated makes it important that the positioning key be included as a sort condition. It doesn't matter if it is ascending or descending, but it needs to be included on the sort to ensure consistent order of data retrieval.

A fairly simple formula may be followed to build the predicate for a restartable cusor needed to support paging Servers. Basically this is a cascade of "OR" clauses connecting a series of "AND" clauses that become progressively more selective following the sort order required by the Client and end up with the positioning key.

To see how this works consider how the query for your Server might be developed...

Start with the column from the sort order that changes least often...

SELECT ... 
FROM ... 
WHERE Receive_Date < restart value

This will retrieve all rows prior to the specified restart Receieve date regardless of what the other column restart values are (e.g. Customer ID's can range from minimum to maximum values, as long as the Receive Date is less than any Receive Date "seen" so far). Since this column only changes value after all subortinate sort columns values have been exausted you can be sure that this does not pick up any rows prior to the full restart key. But what about those rows that occur on the same date as the restart request but have a larger Customer Id? These can be picked up with....

SELECT ... 
FROM ... 
WHERE Receive_Date = restart value AND
      Customer_id > restart value

What about those where the Receive Date and Customer Id are the same as the restart key but have a larger Account Number? These can be picked up with...

SELECT ... 
FROM ... 
WHERE Receive_Date = restart value AND
      Customer_Id = restart value AND
      Account_Nbr > restart value

Continue this pattern until the full restart key has been processed. Notice that the inequality signs are determined by the sort order. Use < when the column is sorted Descending and > when Ascending. Also notice that the SELECT and FROM clauses are exactly the same for each query - which means you can put them all together using OR conjuctions...

SELECT Receive_Date, Customer_id, Account_Nbr, Security_Dep_Id
FROM Table_Name
WHERE (   (Receive_Date    < :RESTART-RCV-DT)
       OR (Receive_Date    = :RESTART-RCV-DT AND
           Customer_Id     > :RESTART-CUSTOMER-ID)
       OR (Receive_Date    = :RESTART-RCV-DT AND
           Customer_Id     = :RESTART-CUSTOMER-ID AND
           Account_Nbr     > :RESTART-ACCT-NBR)
       OR (Receive_Date    = :RESTART-RCV-DT AND
           Customer_Id     = :RESTART-CUSTOMER-ID AND
           Account_Nbr     = :RESTART-ACCT-NBR AND
           Security_Dep_Id > :RESTART-SEC-DEP-ID))
ORDER BY 1 DESC, 2 ASC , 3 ASC, 4 ASC

There you go... a restartable cursor for forward paging. Construction of the cursor for backward paging follows a similar pattern, just flip the sort orders and repeat.

Answer 2

A simplistic approach: Write your SQL to retrieve data according to your criteria, in the sort order you specify. Then only retrieve the keys to the rows you want. Save the keys somewhere you will have access to upon subsequent invocations of your transaction. Look into multi-row select in DB2. Also understand pseudo-conversational programming techniques in CICS.

And now we get to the design implications Bill Woodger mentions, that you do not specify in your question, and which are the reason I'm just hitting the high points of a simplistic approach.

If changes to your result set occur between one invocation and the next, your results will not reflect those changes. You must decide if this is important.

You mention a "front end" but do not specify what it is. If it is a BMS application, you may be able to save the keys in your commarea or in a container. If your front end is a distributed application invoking your transactions via CICS Web Services or CICS Web Support or MQ or raw sockets or whatever, you must design a mechanism to store those keys such that you can uniquely retrieve them — perhaps by sending a contrived key back to the distributed application which it must supply upon subsequent invocations. Then you must have some process to clean up your key store.

Creating a solution to your problem that is unique in your IT shop is not something to be done in isolation. You must involve others who will be tasked with maintaining your application, there may be a group external to your project tasked with making such decisions, there may be infrastructure issues with your solution.

So this isn't so much as an answer to your question as it is an elaboration upon why you may not get an answer, or at least the answer you seem to desire.