OData4j exceptions - “Odd number of characters” and “bad valueString part of keyString”

Question

EDIT:

The solution was to make a view that mirrored the table in question and converted the date to varchar and then convert it back to date with a matching collation

END OF EDIT

Can anyone tell me why OData4j reads datetime values fine from one of my WCF Data Service server but runs into an illegal argument exception (bad valueString as part of keyString) when reading exact same datetime type with same format from another WCF Data Service?

java.lang.IllegalArgumentException: bad valueString [datetime'2012-01-24T14%3A57%3A22.243'] as part of keyString

The other problem is that when I ask for a JSON response from the service from which OData4j had no problem reading datetime types I get another illegal argument exception and the error message is - Odd number of characters.

java.lang.IllegalArgumentException: org.odata4j.repack.org.apache.commons.codec.DecoderException: Odd number of characters.

Because WCF Data Services can't have multiple sources, I've made 2 projects with each it's own Entity Data Model source (from existing database). And like I mentioned above, I'm getting these annoying errors.

To conclude...

Example 1: bad valueString as part of keyString - when reading datetime. Also happens with FormatType.JSON.

ODataConsumer customerInfoServices = ODataConsumer
                                     .newBuilder("http://10.0.2.2:41664/CustomerInfoWCFDataServices.svc/")
                                     .setFormatType(FormatType.ATOM)
                                     .build();

customer = customerInfoServices
           .getEntities("Customers")
           .select("name, id")
           .filter("id eq " + 5)
           .execute()
           .firstOrNull();

Example 2: Odd number of characters. Only happends with FormatType.JSON and no problem reading datetime.

ODataConsumer businessServices = ODataConsumer
                                 .newBuilder("http://10.0.2.2:35932/BusinessWCFDataServices.svc/")
                                 .setFormatType(FormatType.JSON)
                                 .build();

Enumerable<?> ordrer = businessServices
                       .getEntities("Orders")
                       .filter("custId eq " + customer.getProperty("id").getValue())
                       .execute();

What I want is to receive JSON responses (ATOM is still too bloaded for android) and no problems reading datetime properties.

---

No body able to help me?

I've been wearing my fingers down trying to find a solution on Google, without any luck.

The collation on the database without datetime problems is "Danish_Norwegian_CI_AS" and on the database with reading errors is "SQL_Danish_Pref_CP1_CI_AS". I don't know if this has any meaning but I have a suspescion it has something to do with it.

Answer 1

The solution was to make a view that mirrored the table in question and converted the date to varchar and then convert it back to date with a matching collation. :-)

Answer 2

I ran into this issue today, and after running around for about three hours Googling and looking at code I managed to figure out what is going on. Here's my setup/situation and what I found:

The Setup

1. (OData Service) Microsoft IIS 8.0 on Windows Server 2012 using default application pool.
2. (OData Producer) Microsoft WCF middle-tier using Entity Framework and Web Data Services.
3. (OData Consumer) Android client using OData4J v0.8 SNAPSHOT.

The Problem

In my Middle-tier (the OData Producer) I am using Entity Framework 5.0 to define a simple table with an Edm.DateTime column. My MessageTable.edmx file generates a simple table:

CREATE TABLE [dbo].[MessageTable] (
    [Id] int IDENTITY(1,1) NOT NULL,
    [Date1] datetime NULL
);

In my middle-tier's WCF data service (OData Producer) I intercept the OData POST from my Android client application. I manually set the Date1 column in the middle-tier using some C# code:

    private static void ProcessMessage(ODataMessage message)
    {
        .
        .
        .
        conn.Open();
        cmd.ExecuteNonQuery();
        int returnCode = (int)cmd.Parameters["@result"].Value;
        if (returnCode == 0)
        {
            message.Processed = true;
            message.Date1 = DateTime.Now;
        }
        .
        .
        .
    }

Notice that I used C#'s DateTime.Now static property. The MSDN documentation states that DateTime.Now:

Gets a DateTime object that is set to the current date and time on this computer, expressed as the local time.

The key thing to realize is that local time means the C# date-time structure now includes local time zone information.

So, after the middle-tier code finishes, the WCF service inserts the OData POST into my table. Microsoft then sends the [Date1] column back to my Android client (the OData Consumer). It appears that Microsoft encodes the date as an OData DateTimeOffset because it includes local time-zone information. I.e. as 2013-08-26T17:30:00.0000000-7:00

The Code

In OData4j there is a package org.odata.internal that handles parsing OData date-time strings. In version 0.6 I found the following comment on lines 40-44 about the DATETIME_PATTERN regex pattern used to parse date-time strings:

40   // Since not everybody seems to adhere to the spec, we are trying to be
41   // tolerant against different formats
42   // spec says:
43   // Edm.DateTime: yyyy-mm-ddThh:mm[:ss[.fffffff]]
44   // Edm.DateTimeOffset: yyyy-mm-ddThh:mm[:ss[.fffffff]](('+'|'-')hh':'mm)|'Z'
45   private static final Pattern DATETIME_PATTERN =
46       Pattern.compile("(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2})(:\\d{2})?(\\.\\d{1,7})?((?:(?:\\+|\\-)\\d{2}:\\d{2})|Z)?");
47 
48

In OData4j v0.7 the DATETIME_PATTERN has become DATETIME_XML_PATTERN:

40 
41   private static final Pattern DATETIME_XML_PATTERN = Pattern.compile("" +
42       "^" +
43       "(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2})" + // group 1 (datetime)
44       "(:\\d{2})?" + // group 2 (seconds)
45       "(\\.\\d{1,7})?" + // group 3 (nanoseconds)
46       "(Z)?" + // group 4 (tz, ignored - handles bad services)
47       "$");
48 
49   private static final Pattern DATETIMEOFFSET_XML_PATTERN = Pattern.compile("" +
50       "^" +
51       "(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2})" + // group 1 (datetime)
52       "(\\.\\d{1,7})?" + // group 2 (nanoSeconds)
53       "(((\\+|-)\\d{2}:\\d{2})|(Z))" + // group 3 (offset) / group 6 (utc)
54       "$");

I think the comment on line 46 explains everything:

... // group 4 (tz, ignored - handles bad services)

I interpret this as saying 'Any time zone information is going to be ignored - this handles bad services (i.e. Microsoft) that send time zone info'

It appears to me that the OData4j authors have decided to stick to their guns and only accept the correct Edm.DateTime string format.

The Solution

The fix is very simple if you have access to the OData Producer code:

    private static void ProcessMessage(ODataMessage message)
    {
        .
        .
        .
        conn.Open();
        cmd.ExecuteNonQuery();
        int returnCode = (int)cmd.Parameters["@result"].Value;
        if (returnCode == 0)
        {
            message.Processed = true;
            message.Date1 = DateTime.UtcNow;
        }
        .
        .
        .
    }

Use the DateTime.UtcNow property instead. The MSDN documentation states:

Gets a DateTime object that is set to the current date and time on this computer, expressed as the Coordinated Universal Time (UTC).

If you do not have access to the OData producer, the only solution I can see is to change the OData4j code. Alter the regex pattern for DATETIME_XML_PATTERN:

41   private static final Pattern DATETIME_XML_PATTERN = Pattern.compile("" +
42       "^" +
43       "(\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2})" + // group 1 (datetime)
44       "(:\\d{2})?" + // group 2 (seconds)
45       "(\\.\\d{1,7})?" + // group 3 (nanoseconds)
46       "((?:(?:\\+|\\-)\\d{2}:\\d{2})|Z)?" + 
47       "$");

Conclusion

I think this is actually a mistake on Microsoft's part. I'm sure they have some involved justification for sending an Edm.DateTimeOffset but my MessageTable.edmx file specifies [Date1] as an Edm.DateTime, so I think that the Microsoft code should either do the conversion to UTC for me, or throw an exception. An exception would have warned me that I was using an incorrect DateTime structure and helped me see the DateTime.UtcNow solution much quicker.