Process efficiently from TIBCO via SPRING

Question

I have a question regarding processing from a message queue in spring - to which i am very new. I am trying to increase throughput of an application which reads from a queeue and writes to a database live financial information. The problem is messages have to remain ordered, ie first in first out. This means my original approach of increasing the number of concurrent consumers is not viable as there is potentially the case where ordering is lost.

Changing the concurrent consumers for 1 to 5 meant i could process 10,000 and save a good amount of time. (approx 20 mins)

Being fairly new to Spring and being a graduate developer (in my first year) i am unsure what alternatives there are. Spring batch is something that has cropped up but, because it is finiancial information trades need to be processed from the queue as soon as possible so I cant wait for a batch of 500 to fill up for example.

Please could someone suggest what approaches are viable for this type of scenario using spring?

Thanks

Answer 1

below is an example of how to do this type of consumption with Spring Integration using the Message Driven Channel Adapter and the Jdbc Outbound Adapter. a couple of key things that will affect your performance and through put are the following;

• transactions - if it's going across the queue to the database that'll add some overhead
• transformation - how much processing goes into converting the message mapping to the table

here's the Spring Integration example;

<int-jms:message-driven-channel-adapter channel="trade.input.channel"
    concurrent-consumers="1" connection-factory="connectionFactory"
    destination="issue.queue"/>

<int:channel id="trade.input.channel"/>

<int-jdbc:outbound-channel-adapter 
    channel="trade.input.channel"
    data-source="dataSource" query="insert into target_table (issue_code,issue_price,transaction_timestamp) values (:issue_code,:issue_price,:issue_timestamp)"
    sql-parameter-source-factory="spelFactory"/>    

<bean id="spelFactory" class="org.springframework.integration.jdbc.ExpressionEvaluatingSqlParameterSourceFactory">
    <property name="parameterExpressions">
        <map>
            <entry key="issue_code" value="payload.toString().split(',')[0]"/>
            <entry key="issue_price" value="payload.toString().split(',')[1]"/>
            <entry key="issue_timestamp" value="payload.toString().split(',')[2]"/>
        </map>
    </property>
</bean>

and here's the sample message

MSFT,100.00,1373761697932

it's very rough, but could be a starting point. the performance on simple unit test was about an average of 200 messages per second but that's a bit hardware dependent.

Answer 2

If the requirement is to write to the database in the same order as the requests then you are completely single threaded.

However, if you want to be able to determine the order when reading the data back out of the database then you only need to ensure that a order field is maintained correctly (note a timestamp is not generally enough as you might get multiple requests within the minimum precision) so a timestamp plus an additional order value will allow you to determine the exact order.

So single threaded input, add a timestamp and a order value then pass the request to a pool of database writers.

The order value can reset every minute or whatever period is appropriate.