Major SQLException (XA related) in Weblogic application Server

Question

We're developing a software with Weblogic application server (12.1.1.0); We have one domain with two applications.

We just moved up from development DB to pre-production DB (similar to production), and oh boy we got a major issue going...
DBA says the DBs are running the same version - Oracle 11 (I don't recall the exact version). The only difference we can see is that the dvlp DB uses SID for connection, and the other two uses service-name.

Now, in our domain we have two data sources X and Y. Both are connected to the same DB. We use XA driver on both. Both our applications uses the same 'persistence.xml' (and entities) which has two PUs (persistence units), each using a different data source (X and Y).

The problem is this:

1. An MDB starts handling a request.
2. It uses both PUs with EntityManagers and the Y data source with DataSource interface, which is used to get a connection (we have some JDBC code).
3. It calls a bean from the other application.
4. The other bean tries to use one of the PUs (the one connected to X data source).

5. SQLException is thrown:

   XA error: XAResource.XAER_NOTA start() failed on resource 'x_my_domain': XAER_NOTA : The XID is not valid

We've searched the error online and we found out that we should change the data sources' configuration to
XASetTransactionTimeout=true
XATransactionTimeout=0
but that didn't worked.

We've also tried (a lot) to tweak the data sources' configuration, eventually removing one of them so we only need to tweak with only one of them, but nothing has worked.

In addition, while repeatedly tweaking the configuration, a different SQLException has slipped under our radar and started popping out instead of the previous:

Internal error: Cannot obtain XAConnection weblogic.common.resourcepool.ResourceDisabledException: Pool X is Suspended, cannot allocate resources to applications

Now this one is even more frustrating, because we tried everything; reset the data source; delete and re-create; delete and re-create with different name; delete the domain and re-create with a different data source name; go back using the dvlp DB; but nothing, the exception seems to persist.

We really don't have any idea of how to solve this and we can't go any further without fixing this problem.

Answer 1

We finally found how to fix the problem, but to answer the question I will have to explain more about our environment; We actually have two managed servers, and each application runs on another server. Both the admin-server and one of the MS runs on one machine, and the other MS runs on another machine (total of two machines).

The thing was that when we modified one of the data sources (X), which was used also by the other MS, we didn't restarted the other MS, but only the data-source. We guess that this caused the data-source not to update with the new modifications on the other MS, which caused the errors we encountered.

The problem was caused by a rather foolish reason, but it was hard to spot because it recurred on any environment setup of the admin-server + first MS part, even on our own PCs with a setup of only an admin-server with no definition of machines. The reason it still happened is because the address of the machine containing the other application was written in the code (read from XML configuration, but that doesn't change much).

Well, glad that's over with.