Some questions regarding architecture/design to this usecase?

Question

My application needs to work as middleware where it has got orders(in form of xml) from various customers which contains the supplier id. Once it get the order, it needs to send order request to different suppliers in the form of xml.i am double minded about three aspects of it. Here they are:-

Questions:

1. What i am planning at high level is as soon as request come, put it on jms queue.(Now i am not sure should i create queue for each supplier or one queue should be sufficient. I think one queue will be sufficient. as maintaining large number of queues will be overhead.). Advantage of maintaining separate queue per supplier is message can be processed faster as there will be separate producer on each queue.

2. Before putting the object on queue i need to do some business validations. Also the structure of input xml i am receiving and output xml i need to send to supplier is different. For this i am planning to convert the input xml to java object then put on queue so that validation can be done with ease at consumer side. Another thought is dont convert the xml into java object, just get all elements value thru xpath/xstream api and validate them and put xml string as it is on queue because. Then at consumer side convert xml to java object then to different xml format. Is there a way of doing it?

3. Now my requirement is consumer on queue process the messages on queue after every 5 hours and send the xml request to suppliers. I am planning to use quartz scheduler here which will pick the job one by one and send to corresponding supplier based on supplierId. Here is my question is if my job pick the message one by one and then send it to supplier. it will be too slow . I am planning to handle it where quartz job create ThreadPool with size of say ten threads at time which concurrently process the messages from queue(So here will be multiple consumers on queue. I think thats not valid for queue. Do i need topic here instead of queue?). Is second approach is better or there is some better than this?

i am expecting a load of 50k request per hour which mean around 15 request per second

Answer 1

Your basic requirement is ,

1. Get order from customer as XML ( you have not told how you are receiving)
2. Do basic Business validation .
3. Send the Orders to Suppliers

And you would be excepting 50k Request ( You haven't provided the approximate an Order size). Assuming your Average order size 10K, it would be around 500 MB required just to hold it in Queue ( irrespective of number of queues) . i am not sure which environment you are running.

For Point #1 I would choose single Queue instead of multiple Queue - Choose the appropriate persistent store. I am assuming you would be using Distributed Queue , so that it can be easily scale while adding clusters.

For Point #2 I would be converting in POJO (Your own format ) and perform business validation. So that later if you want to extend the business validation to ruler or any other conversion it would be easy to maintain. - basically get the input in any form ( XML / POJO / JSON ...) and convert into Middle format ( you can write custome validator / conversion utility on top of Middle fomart) . And have Keep Mappings between the Common format to input as well output. So that you can write formatters and use them. which will not impact in future while changing format for any specific supplier. Try to externalize the format mapping.

For Point # 3 in your case, A Order needs to be processed by only once. So i would go with Queue. and you can have multiple Message Listeners . Message listeners deliver order in asynchronous. So you can have multiple Listeners for an Queue. And each listeners would run separate thread. Is there a problem to send the orders as soon as it received ? It would be good for you as well as the supplier to avoid heavy load at particular time.

Answer 2

Since you are the middleware, you should handle data quick at the point of contact, to get your hands free for more incoming requests. Therefore you must find a way to distinquish the incoming data as quick and memory low as possible. Leave the processing of the data to modules more specific to the problem. A receptionist just directs the guests in the right spot.

If you really have to read and understand the received data in your specialized worker later on, use a threadpool. This way you can process the data parallelly without too much worry about outofmem. Just choose your pool size smartly and use only one. You could use a listener pattern to signal new incoming data to the worker multiton. You should avoid jaxb or better the complete deserialization of the data if possible. It eats up memory like hell.

I would not use jmx because you "messages" are relevant for only one listener.

If it is possible send the mail as soon as the worker is done with its work. If not, use a storage. This way you can later proove you processed the data and if something went wrong or you have to update your software, you do not have to worry about volatile data.