Handling json data with mixed structure

[Nikolis_Galerakis]

Hello all

I am developing an app that is expected to update it’s data continiously using websocket connections (I am only bringing that up to calrify that several messages with data could be comming per second.)

I am getting the data need to update my model from Kraken (bitcoin trading plarform) but the problem I am currently wrestling with right now is that the data are actually coming in a structure as shown below:

[
  34,
  {
    "a": [
      "0.042062",
      21000,
      "21000.00000000"
    ],
    "b": [
      "0.041960",
      63306,
      "63306.95000000"
    ],
    "c": [
      "0.041960",
      "0.00000011"
    ],
    "v": [
      "2453051.00336514",
      "8208496.19778969"
    ],
    "p": [
      "0.041922",
      "0.042844"
    ],
    "t": [
      606,
      1557
    ],
    "l": [
      "0.041383",
      "0.041383"
    ],
    "h": [
      "0.042414",
      "0.044987"
    ],
    "o": [
      "0.042107",
      "0.044318"
    ]
  },
  "ticker",
  "ADA\/EUR"
]

As you can probably guess the mostof the field are float but somehow serialized as strings
and in some cases there is an list of string and inbitween the string there is an int as for example in the “a” field.

My question was how would you handle such a case when it comes to decode into elm values.
Would it be a good idea to create some type alias to immitate this kind of structure (is it even possible given the mixed lists ?) or would you just go with regex and just find the values of interest and if so what would be the performance implications in your experience ?

thanks in advance

[eike]

In general, the best way to proceed is to first design an Elm data type that describes the data in a way that is appropriate for your application. Then you would write a Decoder to turn this JSON monstrosity into your nice type. I would not try to imitate the structure of this JSON.

Unfortunately, I think the JSON example is to confusing for me to be able to give more specific advice. Maybe you can explain what the different parts of the JSON represent?

[rupert]

Elm decoders won’t treat strings and numbers leniently, strings must be quoted and numbers not.
Elm also requires that the elements in a List all have the same type.

I think you could use oneOf on the items in the list?

Something like:

Decode.oneOf 
    [ Decode.float
    , Decode.map (\val -> String.toFloat val |> Maybe.withDefault 0.0) Decode.string
    ]

Would that help?

[Nikolis_Galerakis]

Thanks for taking the time to respond and thanks for calling it monstrosity because for a moment I thought something is wrong with me for considering that this api is making trivial things complex.

well as you can see the the last string on the list “ADA\ / EUR” indicates a pair of currencies.
the rest of the structure are float and int values representing “o” opening values “a” best bid and so own.
but in general it’s just list of numbers (despite serialized as strings).
I am not sure if this covers your question.

[Nikolis_Galerakis]

Yes that is really helpfull and most probably the way to go for me, thank you
The documentation for the method is a bless as well

Why would someone generate JSON like this? Questions like this are not good for your health. The point is that you can use oneOf to handle situations like this!

If you can direct me to an example that would be even better, but you helped me enough already.

[eike]

Well, for example I don’t know if there are always precicely the fields “a”, “b”, “c”, “v”, “p”, “t”, “l”, “h”, “o” or if every result has different fields. In the former case, you might want to create an Elm record with precicely those fields (maybe with more descriptive names). In the latter case you’d want to decode to a Dict. For each of these fields, does every number indicate the same kind of value (e.g. the development of the price) or does the list represent a tuple (e.g. (number_of_shares, minimum_bid, maximum_bid)). Depending on what it is you simply want to decode to a list (using @rupert’s snippet, for example) or you might want to post-process it to a tuple or record on the Elm side.

Also, the complete thing seems to be a list of very different kinds of objects. You almost certainly want to handle that part in the decoder so that the rest of your app doesn’t have to handle this inhomogeneous list. That is probably the most annoying part and I don’t see a particularly elegant way of doing it. I think Decode.index is the way to go. For example, if you introduce an Elm type

type alias KrakenData =
    { id : Int
    , prices : KrakenPrices
    , type_ : String
    , currencies : String
    }

(I don’t know if these names make sense) you can decode to this type with

decodeKrakenData : Decoder KrakenData
decodeKrakenData = Decode.map4 KrakenData
    (Decode.index 0 Decode.int)
    (Decode.index 1 decodePrices)
    (Decode.index 2 Decode.string)
    (Decode.index 3 Decode.string)

where decodePrices is your Decoder for the prices object. You might also want to replace the Decode.Strings by custom Decoders that check that the type is actually "ticker" and that already split the string of two currencies into a tuple (or similar).

[Nikolis_Galerakis]

I end up going the way you indicated because it looks more straightforward-elegant but although the json is as follows

[34,{"a":["0.033967",5683,"5683.06014663"],"b":["0.033910",93443,"93443.50000000"],"c":["0.033928","1594.05655185"],"v":["846219.74717121","2737731.85431621"],"p":["0.033530","0.033756"],"t":[244,709],"l":["0.033000","0.033000"],"h":["0.034091","0.034500"],"o":["0.034072","0.034459"]},"ticker","ADA/EUR"]

I get a complain by elm:

"Failure \"Expecting an ARRAY\" <internals>"

my decoder looks like this:

dataDecoder : Decoder DataUpdate
dataDecoder =
    map4 DataUpdate
        (index 0 int)
        (index 1 decoderAssetMetaInfoUp)
        (index 2 string)
        (index 3 string)

any idea where this might be coming from ?

[eike]

Hm. I don’t see a mistake in this code. Maybe I’m overlooking something or the failure is in decoderAssetMetaInfoUp? If you post that as well, I will take a look.

[Nikolis_Galerakis]

This is my modeling so far ignoring the a,b field for the shake of simplicity

type alias DataUpdate =        
    { numb : Int               
    , metaInf : AssetMetaInfoUp
    , apiStr : String
    , pair : String            
    }

type alias AssetMetaInfoUp =
    { c : List String
    , v : List String
    , p : List String
    , t : List Int
    , l : List String
    , h : List String
    , o : List String
    }

decoderAssetMetaInfoUp : Decoder AssetMetaInfoUp
decoderAssetMetaInfoUp =
    Json.Decode.succeed AssetMetaInfoUp
    {-- |> required "a" (list string) 
    |> required "b" (list string) --}
    |> required "c" (list string)
    |> required "v" (list string)
    |> required "p" (list string)
    |> required "t" (list int)
    |> required "l" (list string)
    |> required "h" (list string)
    |> required "o" (list string)

dataDecoder : Decoder DataUpdate
dataDecoder =
    map4 DataUpdate
        (index 0 int)
        (index 1 decoderAssetMetaInfoUp)
        (index 2 string)
        (index 3 string)

this is the object pretty printed from the javascript side

[
  139,
  {
    "a": [
      "0.00028250",
      1292,
      "1292.28197076"
    ],
    "b": [
      "0.00028170",
      165,
      "165.49382420"
    ],
    "c": [
      "0.00028250",
      "51.05802924"
    ],
    "v": [
      "4307.99605359",
      "21535.20012158"
    ],
    "p": [
      "0.00027948",
      "0.00027057"
    ],
    "t": [
      93,
      214
    ],
    "l": [
      "0.00027260",
      "0.00026280"
    ],
    "h": [
      "0.00028370",
      "0.00028370"
    ],
    "o": [
      "0.00027280",
      "0.00027100"
    ]
  },
  "ticker",
  "ATOM/XBT"
]

This is what it looks like when I log it through Debug.toString in elm side :

"\"[139,{\\\"a\\\":[\\\"0.00028180\\\",747,\\\"747.17000000\\\"],\\\"b\\\":[\\\"0.00028080\\\",157,\\\"157.49382420\\\"],\\\"c\\\":[\\\"0.00028080\\\",\\\"0.00000010\\\"],\\\"v\\\":[\\\"4315.99605359\\\",\\\"21543.20012158\\\"],\\\"p\\\":[\\\"0.00027948\\\",\\\"0.00027057\\\"],\\\"t\\\":[97,218],\\\"l\\\":[\\\"0.00027260\\\",\\\"0.00026280\\\"],\\\"h\\\":[\\\"0.00028370\\\",\\\"0.00028370\\\"],\\\"o\\\":[\\\"0.00027280\\\",\\\"0.00027100\\\"]},\\\"ticker\\\",\\\"ATOM/XBT\\\"]\""

and this is the error:

"Failure \"Expecting an ARRAY\" <internals>"

thank you very much in advance for being so eager to help !

[eike]

The decoder looks fine, but the string you’re using does not. It seems to have gone through an extra layer of string escaping. My guess is that you pass in the JSON string on the JavaScript side but your Elm app expects to receive an object. It’s probably easiest to insert a JSON.parse call on the JavaScript side before passing the object to your Elm app.

(To verify that that’s the problem, try inserting the pretty-printed JSON in on the Elm side as string (use """a triple quoted string like this""" to allow for line-breaks in the string and to avoid having to escape the single quotes). Then call your decoder on that.)

[Nikolis_Galerakis]

You were absolutely right just using JSON.parse did the Job.

I was getting the value through a onMessage call back which is reasonably a json string but somehow I had in mind that it’s coming from an API(websockets this time) so javascript automatically parses it to a Json object.

thank you