"A lock could not be obtained" when installing WL application

Question

I encounter the following problem when installing an application:
Using the Worklight Console I'm uploading a .wlapp file, removing the deployed application and trying to upload the same .wlapp again.
The Worklight Console is then stuck and even the usual server stop won't complete.
After brutally killing the server process and restarting it the .wlapp can be uploaded.

Installed Worklight Server is v5.0.6 / v5.0.6.1 CE with Derby/Liberty.

Here is the error log from the server:

[4/8/13 14:43:02:586 IDT] 00000031 com.ibm.ws.logging.internal.impl.Incident I FFDC1015I: An FFDC Incident has been created: "java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : ROW, GADGET_APPLICATIONS, (2,7) Waiting XID : {429, S} , WORKLIGHT, SELECT t0.ID, t0.ENFORCE_AUTHENTICITY, t0.ENVIRONMENT_ID, t1.ID, t1.DISP_IN_GALLERY, t1.LAST_UPDATE_TIME, t1.NAME, t0.SERVER_VERSION, t0.VERSION, t0.VERSION_LOCKED FROM GADGET_APPLICATIONS t0 INNER JOIN GADGETS t1 ON t0.GADGET_ID = t1.ID Granted XID : {422, S} Lock : ROW, GADGET_APPLICATIONS, (2,6) Waiting XID : {422, X} , WORKLIGHT, DELETE FROM GADGET_APPLICATIONS WHERE ID = ? Granted XID : {422, S} , {429, S} . The selected victim is XID : 429. com.ibm.ws.rsadapter.jdbc.WSJdbcResultSet.next 2624" at ffdc_13.04.08_14.43.02.0.log [4/8/13 14:43:02:657 IDT] 00000031 com.worklight.gadgets.serving.InvokeProcServlet E FWLSE0081E: Failed synchronizing application from database. org.springframework.orm.jpa.JpaOptimisticLockingFailureException: Unable to obtain an object lock on "A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : ROW, GADGET_APPLICATIONS, (2,7) Waiting XID : {429, S} , WORKLIGHT, SELECT t0.ID, t0.ENFORCE_AUTHENTICITY, t0.ENVIRONMENT_ID, t1.ID, t1.DISP_IN_GALLERY, t1.LAST_UPDATE_TIME, t1.NAME, t0.SERVER_VERSION, t0.VERSION, t0.VERSION_LOCKED FROM GADGET_APPLICATIONS t0 INNER JOIN GADGETS t1 ON t0.GADGET_ID = t1.ID Granted XID : {422, S} Lock : ROW, GADGET_APPLICATIONS, (2,6) Waiting XID : {422, X} , WORKLIGHT, DELETE FROM GADGET_APPLICATIONS WHERE ID = ? Granted XID : {422, S} , {429, S} . The selected victim is XID : 429. [java.lang.String]".; nested exception is org.apache.openjpa.persistence.OptimisticLockException: Unable to obtain an object lock on "A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : ROW, GADGET_APPLICATIONS, (2,7) Waiting XID : {429, S} , WORKLIGHT, SELECT t0.ID, t0.ENFORCE_AUTHENTICITY, t0.ENVIRONMENT_ID, t1.ID, t1.DISP_IN_GALLERY, t1.LAST_UPDATE_TIME, t1.NAME, t0.SERVER_VERSION, t0.VERSION, t0.VERSION_LOCKED FROM GADGET_APPLICATIONS t0 INNER JOIN GADGETS t1 ON t0.GADGET_ID = t1.ID Granted XID : {422, S} Lock : ROW, GADGET_APPLICATIONS, (2,6) Waiting XID : {422, X} , WORKLIGHT, DELETE FROM GADGET_APPLICATIONS WHERE ID = ? Granted XID : {422, S} , {429, S} . The selected victim is XID : 429. [java.lang.String]". at org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:285) at org.springframework.orm.jpa.DefaultJpaDialect.translateExceptionIfPossible(DefaultJpaDialect.java:120) at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:212) at org.springframework.orm.jpa.JpaAccessor.translateIfNecessary(JpaAccessor.java:152) at org.springframework.orm.jpa.JpaTemplate.execute(JpaTemplate.java:189) at org.springframework.orm.jpa.JpaTemplate.executeFind(JpaTemplate.java:151) at org.springframework.orm.jpa.JpaTemplate.findByNamedQuery(JpaTemplate.java:343) at com.worklight.server.database.api.JpaDaoHelper.executeQuery(JpaDaoHelper.java:36) at com.worklight.gadgets.bean.GadgetDAO.getAllApplications(GadgetDAO.java:103) at com.worklight.gadgets.bean.WidgetServiceBean$ApplicationTransaction.getAppsToRemove(WidgetServiceBean.java:370) at com.worklight.gadgets.bean.WidgetServiceBean$ApplicationTransaction.syncApplicationsStateFromDB(WidgetServiceBean.java:286) at com.worklight.gadgets.bean.WidgetServiceBean$ClusterSyncApplicationTransaction.run(WidgetServiceBean.java:475) at com.worklight.gadgets.bean.WidgetServiceBean$ClusterSyncApplicationTransaction.run(WidgetServiceBean.java:470) at com.worklight.core.util.RssBrokerUtils.doInTransaction(RssBrokerUtils.java:119) at com.worklight.gadgets.bean.WidgetServiceBean.syncApplicationsFromDBTransaction(WidgetServiceBean.java:467) at com.worklight.gadgets.bean.ClusterDataSynchronizationTask.step(ClusterDataSynchronizationTask.java:33) at com.worklight.core.tasks.TaskThread.run(TaskThread.java:98) Caused by: org.apache.openjpa.persistence.OptimisticLockException: Unable to obtain an object lock on "A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : ROW, GADGET_APPLICATIONS, (2,7) Waiting XID : {429, S} , WORKLIGHT, SELECT t0.ID, t0.ENFORCE_AUTHENTICITY, t0.ENVIRONMENT_ID, t1.ID, t1.DISP_IN_GALLERY, t1.LAST_UPDATE_TIME, t1.NAME, t0.SERVER_VERSION, t0.VERSION, t0.VERSION_LOCKED FROM GADGET_APPLICATIONS t0 INNER JOIN GADGETS t1 ON t0.GADGET_ID = t1.ID Granted XID : {422, S} Lock : ROW, GADGET_APPLICATIONS, (2,6) Waiting XID : {422, X} , WORKLIGHT, DELETE FROM GADGET_APPLICATIONS WHERE ID = ? Granted XID : {422, S} , {429, S} . The selected victim is XID : 429. [java.lang.String]". at org.apache.openjpa.jdbc.sql.DBDictionary.narrow(DBDictionary.java:4236) at org.apache.openjpa.jdbc.sql.DBDictionary.newStoreException(DBDictionary.java:4211) at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:102) at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:88) at org.apache.openjpa.jdbc.sql.SQLExceptions.getStore(SQLExceptions.java:64) at org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.handleCheckedException(SelectResultObjectProvider.java:155) at org.apache.openjpa.lib.rop.EagerResultList.(EagerResultList.java:40) at org.apache.openjpa.kernel.QueryImpl.toResult(QueryImpl.java:1228) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:990) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:805) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:775) at org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:771) at org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:517) at org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:255) at org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:294) at org.springframework.orm.jpa.JpaTemplate$11.doInJpa(JpaTemplate.java:351) at org.springframework.orm.jpa.JpaTemplate.execute(JpaTemplate.java:184) ... 12 more Caused by: java.sql.SQLTransactionRollbackException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : ROW, GADGET_APPLICATIONS, (2,7) Waiting XID : {429, S} , WORKLIGHT, SELECT t0.ID, t0.ENFORCE_AUTHENTICITY, t0.ENVIRONMENT_ID, t1.ID, t1.DISP_IN_GALLERY, t1.LAST_UPDATE_TIME, t1.NAME, t0.SERVER_VERSION, t0.VERSION, t0.VERSION_LOCKED FROM GADGET_APPLICATIONS t0 INNER JOIN GADGETS t1 ON t0.GADGET_ID = t1.ID Granted XID : {422, S} Lock : ROW, GADGET_APPLICATIONS, (2,6) Waiting XID : {422, X} , WORKLIGHT, DELETE FROM GADGET_APPLICATIONS WHERE ID = ? Granted XID : {422, S} , {429, S} . The selected victim is XID : 429. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source) at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source) at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedResultSet.closeOnTransactionError(Unknown Source) at org.apache.derby.impl.jdbc.EmbedResultSet.movePosition(Unknown Source) at org.apache.derby.impl.jdbc.EmbedResultSet.next(Unknown Source) at com.ibm.ws.rsadapter.jdbc.WSJdbcResultSet.next(WSJdbcResultSet.java:3108) at org.apache.openjpa.lib.jdbc.DelegatingResultSet.next(DelegatingResultSet.java:106) at org.apache.openjpa.jdbc.sql.ResultSetResult.nextInternal(ResultSetResult.java:222) at org.apache.openjpa.jdbc.sql.SelectImpl$SelectResult.nextInternal(SelectImpl.java:2470) at org.apache.openjpa.jdbc.sql.AbstractResult.next(AbstractResult.java:173) at org.apache.openjpa.jdbc.kernel.SelectResultObjectProvider.next(SelectResultObjectProvider.java:99) at org.apache.openjpa.lib.rop.EagerResultList.(EagerResultList.java:35) ... 22 more Caused by: java.sql.SQLException: A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : ROW, GADGET_APPLICATIONS, (2,7) Waiting XID : {429, S} , WORKLIGHT, SELECT t0.ID, t0.ENFORCE_AUTHENTICITY, t0.ENVIRONMENT_ID, t1.ID, t1.DISP_IN_GALLERY, t1.LAST_UPDATE_TIME, t1.NAME, t0.SERVER_VERSION, t0.VERSION, t0.VERSION_LOCKED FROM GADGET_APPLICATIONS t0 INNER JOIN GADGETS t1 ON t0.GADGET_ID = t1.ID Granted XID : {422, S} Lock : ROW, GADGET_APPLICATIONS, (2,6) Waiting XID : {422, X} , WORKLIGHT, DELETE FROM GADGET_APPLICATIONS WHERE ID = ? Granted XID : {422, S} , {429, S} . The selected victim is XID : 429. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 38 more Caused by: ERROR 40001: A lock could not be obtained due to a deadlock, cycle of locks and waiters is: Lock : ROW, GADGET_APPLICATIONS, (2,7) Waiting XID : {429, S} , WORKLIGHT, SELECT t0.ID, t0.ENFORCE_AUTHENTICITY, t0.ENVIRONMENT_ID, t1.ID, t1.DISP_IN_GALLERY, t1.LAST_UPDATE_TIME, t1.NAME, t0.SERVER_VERSION, t0.VERSION, t0.VERSION_LOCKED FROM GADGET_APPLICATIONS t0 INNER JOIN GADGETS t1 ON t0.GADGET_ID = t1.ID Granted XID : {422, S} Lock : ROW, GADGET_APPLICATIONS, (2,6) Waiting XID : {422, X} , WORKLIGHT, DELETE FROM GADGET_APPLICATIONS WHERE ID = ? Granted XID : {422, S} , {429, S} . The selected victim is XID : 429. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.services.locks.Deadlock.buildException(Unknown Source) at org.apache.derby.impl.services.locks.ConcurrentLockSet.lockObject(Unknown Source) at org.apache.derby.impl.services.locks.AbstractPool.lockObject(Unknown Source) at org.apache.derby.impl.services.locks.ConcurrentPool.lockObject(Unknown Source) at org.apache.derby.impl.store.raw.xact.RowLocking3.lockRecordForRead(Unknown Source) at org.apache.derby.impl.store.access.heap.HeapController.lockRow(Unknown Source) at org.apache.derby.impl.store.access.heap.HeapController.lockRow(Unknown Source) at org.apache.derby.impl.store.access.btree.index.B2IRowLocking3.lockRowOnPage(Unknown Source) at org.apache.derby.impl.store.access.btree.index.B2IRowLocking3._lockScanRow(Unknown Source) at org.apache.derby.impl.store.access.btree.index.B2IRowLockingRR.lockScanRow(Unknown Source) at org.apache.derby.impl.store.access.btree.BTreeForwardScan.fetchRows(Unknown Source) at org.apache.derby.impl.store.access.btree.BTreeScan.fetchNextGroup(Unknown Source) at org.apache.derby.impl.sql.execute.BulkTableScanResultSet.reloadArray(Unknown Source) at org.apache.derby.impl.sql.execute.BulkTableScanResultSet.getNextRowCore(Unknown Source) at org.apache.derby.impl.sql.execute.IndexRowToBaseRowResultSet.getNextRowCore(Unknown Source) at org.apache.derby.impl.sql.execute.NestedLoopJoinResultSet.getNextRowCore(Unknown Source) at org.apache.derby.impl.sql.execute.ProjectRestrictResultSet.getNextRowCore(Unknown Source) at org.apache.derby.impl.sql.execute.BasicNoPutResultSetImpl.getNextRow(Unknown Source) ... 31 more

Answer 1

This can be solved by adding the following attribute to the WorklightDS dataSource element within the server.xml:

isolationLevel="TRANSACTION_READ_COMMITTED"