JSON Referencing Schemes
9 October 2007I have been involved in some lively discussions around referencing in JSON, in particular Arthur Blake, the rest of JSON-RPC discussion group, and I have been discussing circular referencing as a construct for JSON-RPC. However, I believe that the topic of referencing should be discussed in terms of more general JSON usage, rather than simply as a particular strategy just for JSON-RPC. It would be great if there was a general strategy for referencing that wasn’t specific to JSON-RPC, but could be used in a consistent manner across different applications of JSON. Currently the JSPON specification provides a mechanism for referencing. Arthur Blake has proposed another approach to handle circular and multiple referencing that uses post processing fixups. Another proposal for handling circular referencing can be found here. Arthur and I have been having numerous discussions about these differing models, and we have been contemplating another approach that is somewhere between JSPON and his fixups scheme that utilizes path referencing. I believe that working towards a convergent model for referencing would be beneficial for interoperability rather than having widely varying forms of referencing, and I want to describe the different referencing schemes here and invite feedback on what form of referencing is best for us to follow as a community. Being the author of JSPON, I certainly have my preferences, but am hoping for unbiased third party comments. For these specs we will compare the object formed from setting up a circular reference like this:
obj = {name:"foo", child: {"name" : "bar"}};
obj.child.parent = obj;
Also, before comparing referencing schemes, it is important to consider the different forms of referencing:
- Local intramessage references - These are references within a single JSON message.
- Circular references - This seems to receive the most attention, and so is topic we are using as a starting point for discussion.
- Many to one references - This is when multiple objects reference a single object. In my experience this is a vastly more common situation.
- Local intermesssage references - These are references to target objects from different JSON messages.
- Remote references - These are references to objects from other domains.
Now let us look at the different approaches to handling references:
- fixups scheme - This is a proposal for an addition to the JSON-RPC 1.1 spec. This works by using property paths to do assignments on a JSON object to create references that are not otherwise depictable by JSON. These post assignments are done by a fixups property that has a list of lists of pairs of lists of property names to use for the assignments. The example object would be serialized within an RPC like this:
{"result":{"name":"foo", "child": {"name" : "bar"}},
"fixups":[[["child","parent"],[]]]}
There is no definition for how it would work outside of JSON-RPC. - id referencing - JSPON and Sundararajan’s approach uses id referencing to do referencing. Id referencing works by assigning ids to objects that may be referenced, and using an referencing objects to reference the target object. The example object could be serialized like this with JSPON:
{"name":"foo", "id":"1", "child": {"name" : "bar","parent":{"id":"1"}}}
or with Sundararajan’s scheme:
{"name":"foo", "$id":"1", "child": {"name" : "bar","parent":{"$idref":"1"}}} - Path referencing - This approach works by referencing objects with a reference object that using a property path to identify the target object. There are several different libraries that use this approach including Thomas Frank’s Stringify, and Project Zero. An intuitive way to define path referencing would be to use utilize the JSONPath specification. The example object could be serialized like this (in JSONPath $ indicates the root, but deeper paths use familiar dot syntax property referencing):
{"name":"foo", "child": {"name" : "bar","parent":{"id":"$"}}}
One could reference non-root objects using paths like $.child.someProperty. I am not sure if it is possible to reference the root object with Thomas Frank’s approach. With Project Zero’s approach we could write:
{"name":"foo", "child": {"name" : "bar","parent":"$jref:this"}}
Lets look at the pros and cons of these techniques.
One of the advantages of the fixups scheme is that no properties have to be inserted into the target JSON object. Referencing can be done without affecting the original object. The fixups scheme also does not require id generation. However, disadvantages include difficulty in finding a place to put the fixups in more general JSON objects, and readability. I will invite Arthur to give more pros for this technique.
JSPON uses id referencing, which solves a number of challenges beyond just intermessage/circular referencing. In particular, with id referencing, we can reference objects outside of the current message (intramessage referencing). Objects from past and future messages can be referenced by id. This can be important for performance. In situations where clients are id referencing aware, it is possible for a message to define a large object once, and future messages can refer to this object rather than transferring it each time. Conversely, a message can refer to an object that has not been transferred or defined yet, and a client can request the object if needed, i.e. lazy loading. JSPON further utilizes id referencing to provide a mechanism to perform updates on objects (simply doing a PUT to the url implied by the id using the standard relative URI scheme), by using the concepts of URIs, ids can even facilitate cross-domain object referencing (remote referencing). Now to be sure, these things are beyond the scope of just doing to simple circular references, however it provides a more general referencing solution with a single construct that is valuable to solving a large number of challenges, including circular referencing. Even if some applications may only use a id referencing for circular referencing, it surely advantageous that this same construct can be used for a number of other capabilities.
Sundararajan’s scheme is similiar to JSPON, although it uses different constructs for non-local references ($url). To me it, it seems this only adds unnecessary complexity, however conceptually it is very similiar to JSPON.
Path referencing is another approach between id referencing and the fixups scheme. We can use JSONPath (or use Frank Thomas or Project Zero’s referencing schemes) to identify target objects and therefore there is no requirement for referenced objects to have id’s added to them. Some of the advantages of this approach:
- Agents that were not referencing aware would have a closer approximation to the correct graph (vs the fixups scheme) because the property that back references would still be there, it would just contain an object with an unresolved reference, which when debugging would actually probably be a pretty good clue about what needs to be done to make it work right. Id referencing exhibits this same behavior. Path referencing may arguably provide the most informative readable clues about how references should be resolved.
- It is possible to utilize the existing JSONPath specification.
- More conceptually similiar and compatible with JSPON/id referencing, it is quite reasonable to create client code that could handle path and id references.
- No need for id generation.
Cons:
- One of the advantages to JSPON style id referencing is that it is encapsulatable, that is an JSON object can have internal references that does not require contextual information about where the root of the object graph is, it is self-encapsulated. Standard JSON objects can stand on their own, just as with normal JSON. Both path referencing and the fixups scheme do not preserve this property.
- “In object” (id or path) referencing requires walking the graph.
Anyway, I welcome your comments on this subject. I am hoping that we can move towards a consistent approach to referencing. Interoperability is not always easy. People have made investments in different approaches. Sometimes it takes some effort to change to a common convention, to move towards standards, but I think the open web is worth the effort. I for one, have put signficant effort into JSPON, but would be glad to change the JSPON spec and my projects that uses JSPON if we as a community could all agree on a convention for referencing (assuming it’s a reasonably adequate model). Perhaps we might need to support two models (like maybe path and ID referencing), but even if that is the case, I believe we as a community (I have invited the authors of all these proposals) should still try to converge on a consistent referencing convention. So questions include, which referencing models do we need? What semantics should we use for referencing, options include, but are not limited too: {”id”:”target”}, {”$idref”:”target”}, “$jref:target”, or maybe something different like {”$ref”:”target”}? If we use path referencing, what should be used to denote the root object ($ or this or something else)?
Update: You can see my proposed convention for referencing that combines the best ideas from these approaches and tries to reach a compromise.
15 Responses to “JSON Referencing Schemes”
October 10th, 2007 at 1:12 am
Personally I like ID referencing. It’s simple yet fairly powerful, easy to read and write, and IMHO holds true to the simplicity of JSON. I haven’t seen Sundararajan’s proposal before now, but after reading it I like his a bit better over JSPON for only one reason. Using ‘id’ as a property to reference other schemas or objects seems too generic and ripe for causing errors in the wrong hands, or even the right hands during a code crunch. With Sundararajan’s method, the $ instantly signifies to the reader/agent/parser that the property is something out of the ordinary and its value needs special care or processing. I also really like the concept of using the property as a hint of what type of object is going to be referenced. Makes it easier to parse. I also think the $class is interesting and could be useful to things like DWR for java.
As for fixups schemes, seems like its probably pretty slick when you get your head wrapped around it, but just looking at a simple example makes my brain hurt and doesn’t seem very simple at all.
Path references seem like a fairly good solution. I honestly don’t see much wrong with it and would probably be a good way to go in certain situations where you have complex data structures to deal with. Still, I probably like ID style referencing more just because it a bit like what I already do. Most of what we work with right now is just large chunks of data that we have to iterate over row by row, but no serious in depth data structures, so using path referencing seems a bit silly.
There are my thoughts. Interesting stuff by the way!
Scott
October 10th, 2007 at 10:50 am
Actually, the whole fixups idea was designed to be simple to understand and read from the beginning.
The primary advantage is that no id’s whatsoever are required.
Here is the prototypical example of the fixups scheme:
obj = {name:”foo”, child: {”name” : “bar”}};
obj.child.parent = obj;
Look familiar? That’s the same example Kris gave to construct a circular reference at the beginning of his discussion.
Notice the first line constructs an object, and the second line “fixes up” the object to contain a circular reference.
That’s because the way fixups work is the same way we constructed the circular reference in the first place.
In fact, the first proposal for fixups even suggested that the fixups be in this format! (executable javascript…) for ultimate simplicity (the fixups can just be eval’ed if desired.) It morphed a bit to represent the fixs ups as arrays of arrays (I agree that’s not so readable anymore) because it is much easier to write an algorithm to parse the fix ups in other languages… but if we changed this to a notation similar to JSONPATH, we’d be back in the realm of that simplicity and better readability.
October 10th, 2007 at 11:13 am
Scott - When I wrote JSPON, originally I had used an ID field that was more unique (like “$id”), but felt that readability was more important than name clashing, and it many situations, “id” naturally implies uniqueness and this can be leveraged to utilize preexisting object ids. However, if others feel that a more unique ID field is preferable, I would be glad to change.
October 10th, 2007 at 12:35 pm
One other quick comment:
There is no reason at all why the method for handling circular references we’ve proposed (the fixups method) can’t be used in any other framework. Although it was designed for JSON-RPC– there is nothing within it that precludes it from being used in any other framework that deals with json. The only difference is that there is a separate data field that has to be accounted for to hold the fixup information.
I guess the only real difference that is a point of contention which we keep coming back to, is the id’s required in the other schemes proposed.
They are a benefit on the one hand, because they can combine nicely and interact with other referencing schemes. But they are also a drawback in that they aren’t strictly required to resolve circular refs and could potentially get in the way.
The fixups method is more narrowly focused on solving the circular references / duplicates problem without depending on any kind of id scheme.
October 10th, 2007 at 12:57 pm
The path referencing schemes don’t require ids. However, I think that it is entirely possible that both path referencing and ID referencing might be important forms of referencing to retain, as each is more applicable for different situations. ID referencing is really the only way I know of to work with intermessage and remote referencing, but as you have pointed, sometimes it might be preferable to avoid id generation, and therefore path referencing is preferable in some situations. However, I believe we can still use a common convention for both path and ID referencing. For example if we decided the a reference was an object with a “ref” property, we could use it for both forms of referencing:
Id ref - {”ref”:”someId”}
or path ref - {”ref”:”$.somePath”}
October 11th, 2007 at 11:03 pm
maybe
normal - {”ref”:”someId”}
ref - {”ref”:’someId’}
but JS say no..
write in JS
October 11th, 2007 at 11:19 pm
Even though both of the problems we are solving have JSON and circular references at the core– at the end of the day, we really are trying to solve two different problems.
Also, there is nothing at all stopping anyone from using either of our schemes, or even both at the same time together — they would interoperate without any problems.
I agree that it makes sense to combine our resources and try to unify on some kind of agreed specification– but I really think that there is some large non-overlap in the problem sets we are trying to solve which may not make a complete combining desirable.
I don’t think I am willing to concede the id requirement within the json– I really like our approach that doesn’t require any of that.
However, our discussions have brought more into central focus the neat ideas behind JSONPATH, and I would consider possibly using that for the fixup reference notation instead of the very ugly looking (but very easy to process) thing that I came up with.
However, JSONPath itself is new, and I don’t know how useful that whole idea is really going to be (I also though XPATH was extremely cool when it came out, but to this day have never used it)– although it seems like a good idea. Also, my only other issue with that is that there doesn’t appear to be a Java library for dealing with JSONPath yet (let me know if I’m wrong.) in which case I (or someone else) would have to write one.
I’m going to go ahead and release a jabsorb 1.2 test version which has the circular reference scheme we have come up with, because people are waiting for it and want to use it– but there is no reason why we can’t switch to some other scheme later on– I don’t see any problem with that… especially since the idea wasn’t accepted into JSON-RPC outright (as most don’t seem to be very keen on the idea of adding that additional complexity to the spec.– maybe they’ll come around to some standard for referencing at some point if enough people start seeing the benefit… and if there is a standard.. we can adjust appropriately.)
I still think it does belong in some kind of standardized spec as circular references is something that can be represented independent of the language being used on one end of the JSON-RPC call and it does seem to have a lot of potential application when sending complex or large structures with a lot of duplicates.
I do agree in that I think there really is a lot of wisdom in keeping things simpler rather than more complex and I will admit that I myself have never even had a need for circular references in JSON-RPC although I know many others have been needing it and asking for it for a long time– when we started jabsorb, I took that on (and also because it was a challenging and interesting problem for me.)
When I have time, I’ll try and write an extension spec for JSON-RPC circular references support that is more formal looking, and I will invite Kris (and any other interested parties) to collaborate on that if they would like…
Special thanks to Kris– I’m so glad that you had the foresight to try and work together on this and come up with some kind of standard– and please do keep pushing on that, and we will combine our efforts when it does make sense.
October 16th, 2007 at 11:46 am
Thanks for that interesting article and giving some insight in the importance of handling circular references with JSON.
I am glad, that you also mention JSONPath in this context.
I am currently discussing some aspects of JSONPath with Tatu Solaranta (http://www.cowtowncoder.com), who intends to implement JSONPath in Java.
The discussion is focusing towards script expressions in JSONPath, which are very powerful but only usable in Javascript without extra parsing. So one proposal is, to define a small set of language independent expressions.
Hopefully this is of some help. I always welcome comments on that.
October 21st, 2007 at 2:14 pm
I’m having trouble seeing the utility of circular references. One of the advantages of a serialization format like JSON is that it neutralizes any circular references that might have started out in the source object. The only time I’ve really wanted .parent accessors are when my Javascript objects contain functions, and they need to reach up the tree. When it’s all just data, whatever is reading it should know how without having to insert circularity.
That said, I much prefer the executable Javascript fixup method to any other, precisely because it is executable. JSON wins because it is light and simple, and other languages are more suited to the complexity of learning to eval the simple fixup Javascript, moreso than trying to teach Javascript to interpret a complex data structure or $id reuse pattern and apply it as a circularity fixup.
IMHO, JSON should never be taken to mean more than you would get by evaling it. At least your fixup evals, even it if isn’t pure JSON.
October 21st, 2007 at 7:39 pm
I don’t use circular references much either, and so I don’t see why one would use it as the basis for an argument. The other forms of referencing are much more important for me, which is precisely why fixups are inadequate.
JSON basically _always_ means more than you get by evaling it. As you don’t eval and throw the object away, you do stuff with it. And sometimes that stuff has a concept of multiple objects referring to a single object.
JSON should be light, simple, and _readable_. There are schemes that are lighter, but readability combined with these other forms to make JSON what it is. Conventions for referencing should share this same priority for readability.
November 2nd, 2007 at 12:10 pm
Any help with this site would be appreciated.
http://www.workathomewindow.com
I am just learning web programming.
January 10th, 2008 at 6:51 am
My preference is JSPON/Sundararajan.
About the “id” vs. “$id” debate: yes, id usually signifies something unique. But JSON structures would often be constructed from objects coming from various database tables. Id fields could easily exist in every object to link it the its persistent source, but those id’s would only be unique within each table, not universally (for the entire JSON structure).
For that reason alone, choosing “id” as a special name in JSPON is not a very good idea, and I think you should consider adopting “$id” (which still has all the same symbolism and is equally easy to read).
January 14th, 2008 at 10:01 pm
Jakob, I must admit their are situations where “id” is not perfect. I think it is advantageous when possible to be able to specify your identity property. I think this could ideally be done with JSON Schema which has a “unique” attribute that can be used to signify an identity property.
May 27th, 2009 at 8:40 pm
I watch this guy for year, yea he do a lot of crazy stuff, but I know he is a really good and nice person. My boyfriend got his all best fights and we probably going to pray today and watch his in ring - so sad love you Mike.
June 25th, 2009 at 10:41 am
{
“links”:
[
{
"url":"www.google.com",
"color":$.colors.red
},
{
"url":"www.yahoo.com",
"color":$.colors.pink
},
{
"url":"www.domaine.com",
"color":$.link[0].color
}
],
“colors”:
{
“red”:[255,0,0],
“green”:[0,255,0],
“black”:[0,0,0],
“pink”:[255,127,127]
}
}