FSharp.Data — How to Handle Null JSON Values?

Question

I'm trying to convert the FSharp.Data examples to solutions for my problem i'm dealing with, but i'm just not getting very far.

Problem

Given an endpoint that returns json similar to:

{
  Products:[{
    Id:43,
    Name:"hi"
  },
  {
    Id:45,
    Name:"other prod"
  }
  ]
}

How can i load the data and then only get the Ids out of real, existing data?

I dont understand how to "pattern match out" the possibilities that:

• it could return nothing
• that root.Products could be not existing/empty
• that Id might not exist

Attempt Via Null Matching

namespace Printio

open System 
open FSharp.Data
open FSharp.Data.JsonExtensions

module PrintioApi =
    type ApiProducts = JsonProvider<"https://api.print.io/api/v/1/source/widget/products?recipeId=f255af6f-9614-4fe2-aa8b-1b77b936d9d6&countryCode=US">

let getProductIds url =
    async {
        let! json = ApiProducts.AsyncLoad url
        let ids = match json with
            | null  -> [||]
            | _ -> 
                match json.Products with
                    | null -> [||]
                    | _ -> Array.map (fun (x:ApiProducts.Product)-> x.Id) json.Products

        return ids
        }

Answer 1

You probably don't need pattern matching for checking whether it's an empty array of not if you have some level of confidence in the source data. Something like this might just work fine: -

let getProductIds url =
    async {
        let! json = ApiProducts.AsyncLoad url
        return json.Products |> Seq.map(fun p -> p.Id) |> Seq.cache
    }

Note you shouldn't use Async.RunSynchronously when in an async { } block - you can do a let! binding which will await the result asynchronously.

Answer 2

Edit: When I wrote this answer, I didn't fully understand the capabilities of the JSON type provider. It turns out that you can populate it with a list of sample JSON documents, which enables you to handle all sorts of scenarios where data may or may not be present. I use it quite a lot these days, so I no longer believe in what I originally wrote. I'll leave the original answer here, in case anyone can derive any value from it.

See my other answer here on the page for a demonstration of how I'd do it today.

---

While type providers are nice, I believe that it's conceptually wrong to attempt to treat something like JSON, which has no schema, and no type safety, as strongly typed data. Instead of using type providers, I use HttpClient, Json.NET, and FSharp.Interop.Dynamic to write queries like this:

let response = client.GetAsync("").Result
let json = response.Content.ReadAsJsonAsync().Result
let links = json?links :> obj seq
let address =
    links
    |> Seq.filter (fun l -> l?rel <> null && l?href <> null)
    |> Seq.filter (fun l -> l?rel.ToString() = rel)
    |> Seq.map (fun l -> Uri(l?href.ToString()))
    |> Seq.exactlyOne

where client is an instance of HttpClient, and ReadAsJsonAsync is a little helper method defined like this:

type HttpContent with
    member this.ReadAsJsonAsync() =
        let readJson (t : Task<string>) =
            JsonConvert.DeserializeObject t.Result
        this.ReadAsStringAsync().ContinueWith(fun t -> readJson t)

Answer 3

Give the type provider enough examples to infer those cases. Example:

[<Literal>]
let sample = """
{
  Products:[{
    Id:null,
    Name:"hi"
  },
  {
    Id:45,
    Name:"other prod"
  }
  ]
}
"""

type MyJsonType = JsonProvider<sample>

But do note it will never be 100% safe if the json is not regular enough

Answer 4

If you suspect that your data source may contain some missing values, you can set SampleIsList = true in the JsonProvider, and give it a list of samples, instead of a single example:

open FSharp.Data

type ApiProducts = JsonProvider<"""
[
    {
        "Products": [{
            "Id": 43,
            "Name": "hi"
        }, {
            "Name": "other prod"
        }]
    },
    {}
]
""", SampleIsList = true>

As Gustavo Guerra also hints in his answer, Products is already a list, so you can supply one example of a product that has an Id (the first one), and one example that doesn't have an Id (the second one).

Likewise, you can give an example where Products is entirely missing. Since the root object contains no other data, this is simply the empty object: {}.

The JsonProvider is intelligent enough to interpret a missing Products property as an empty array.

Since a product may or may not have an Id, this property is inferred to have the type int option.

You can now write a function that takes a JSON string as input and gives you all the IDs it can find:

let getProductIds json =
    let root = ApiProducts.Parse json
    root.Products |> Array.choose (fun p -> p.Id)

Notice that it uses Array.choose instead of Array.map, since Array.choose automatically chooses only those Id values that are Some.

You can now test with various values to see that it works:

> getProductIds """{ "Products": [{ "Id": 43, "Name": "hi" }, { "Id": 45, "Name": "other prod"  }] }""";;
> val it : int [] = [|43; 45|]

> getProductIds """{ "Products": [{ "Id": 43, "Name": "hi" }, { "Name": "other prod" }] }""";;
> val it : int [] = [|43|]

> getProductIds "{}";;
> val it : int [] = [||]

It still crashes on empty input, though; if there's a TryParse function or similar for JsonProvider, I haven't found it yet...