Trouble using Aeson to decode string

Question

I am attempting to use aeson to parse the json returned by an api. A response is fetched from the api endpoint using wreq. When running eitherDecode on this string I get:

Left "Failed reading: Cannot decode byte '\\xa3': Data.Text.Encoding.decodeUtf8: Invalid    UTF-8 stream"

This is probably due to my not properly understanding text encoding, but it seems to me that the string the api is returning isn't actually valid json. If so, how should I proceed? Either way, I would appreciate someone pointing me in the right direction!

I have included an example that reproduces my problem below.

{-# LANGUAGE DeriveGeneric #-}
{-# LANGUAGE OverloadedStrings #-}

import Data.Aeson
import Data.Text
import GHC.Generics
import Data.ByteString.Lazy


data Test = Test { priceDescription ∷ Text } deriving (Show, Generic)

instance FromJSON Test

main = do
  let a = "\r\n{\r\n\"priceDescription\": \"\163\&5.98 each\"\r\n}" :: ByteString

  let result = eitherDecode a :: Either String Test
  print result

Answer 1

It might be that your API endpoint is not returning UTF-8, but ISO-8859-1. Then the \163 would translate to a £ (pound) symbol, which would make sense given the context. So I'd recommend using the text-icu package to convert your incoming ByteString to Text and then feed this to Aeson. If you're speaking HTTP to get the data, maybe the response headers indicate the encoding?