Joining SitePen
January 2nd, 2008
I am pleased to announce that I am now officially a SitePen employee. Today is my first day with SitePen, and I am very excited to be surrounded (in a virtual sense) by such a brilliant group of people. I will be a Research and Development Associate under Alex Russell. SitePen is extremely forward thinking in evolving web technology and pursuing the Open Web, and therefore I will continue to have opportunities to work towards open formats and standards for JSON interchange and interoperability, as well as other new arenas of development. SitePen has a strong sense that what is good for the web is good for SitePen, and in doing some has made incredible contributions to the open source community in Dojo, Dojo Offline (and it’s influence on Google Gears), and more, and I am exciting to join with SitePen in the pursuit of a better web for everyone.
JSONP Header Transfer Proposal
December 24th, 2007
JSONP is a proposal for performing cross site JSON data interchange by using script tag insertion with a callback to deliver the data payload. JSONP can be used in place of XmlHttpRequest in cross site situations. However, when using XmlHttpRequest, there are situations where the JSON data in content body is not the only relevant information, but the response headers, as well other aspects of the HTTP request and response may also contain important information. I propose that JSONP can extended in a small and simple manner to accommodate the transfer of header information in suitably interoperable manner.
I propose when custom headers need to be included in the request, that headers simply be included as parameters. When servers that are responding to JSONP requests want to include response header information in the response, they may optionally respond with a second argument that contains header information in the form of an JSON object/map. The object should contain properties with the names corresponding to header names, and values corresponding to header values. The request format can remain the same:
http://myserver/requestJSON?callback=mycallback&Accept=application%2Fjson
And the response would look like:
mycallback({"name":"foo"},{"Content-Type":"application/json","Expires":"Mon, 24 Dec 2007 17:09:04 GMT"})
This addition to JSONP should generally not break existing JSONP clients, as normally only one parameters is used. This allows JSONP to emulate more semantically correct manners of transferring information that conceptually belongs in headers.
Usages could include transferring the “Last-Modified” header, so clients could determine the freshness of the data, sending If-Modified-Since header in requests and many more.
In addition to transferring request and response headers, I propose that additional HTTP information could also be transferred in pseudo headers. The following pseudo headers may be used:
- request
- httpMethod - Indicates the HTTP method used (GET, POST, PUT, DELETE, etc).
- httpContent - Indicates the body of the content of the HTTP request (such as the POST body)
- httpNoCache - With non-idempotent methods, you should generally include an additional parameter with a random unique value to ensure that the request is not cached.
- response
- httpStatusCode - Indicates the HTTP status code.
- httpStatusText - Indicates the HTTP status text.
A more sophisticated example request could be:
http://myserver/requestJSON?httpMethod=POST&httpNoCache=23n9zs92l&httpContent={"name":"bar"}&callback=call1
And the response could be:
call1([{"name":"foo"},{"name":"bar"}],{”httpStatusCode”:200,”httpStatusText”:”OK”,”Date”:”Mon, 24 Dec 2007 17:09:04 GMT”})
Of course, this enables a greater level of interaction, and therefore the prerequisite security warnings must be heeded. You can get yourself in a lot of trouble with XSS if you are not using proper explicit authentication schemes.
December 21st, 2007
Below is a presentation on JSON I gave a few months ago for some colleagues. The first three videos cover different emerging JSON extensions and standards, and the next three videos cover tools that I have built for working with these JSON extensions.
These three videos cover tools I have built for JSON. The primary tool that is discussed is Persevere (and Jsponic which has been renamed to Persevere server):
Non-Relational Web DB Interoperability
December 17th, 2007
Alex Russell recently wrote about the advantages of non-relational (dynamic object) databases for application development. The flexibility of using dynamic persisted objects that behave like dynamic OOP objects provides a great level of agility in progressive iterative development. Objects can automatically be persisted, virtually eliminating the overhead of developing around a database. I certainly am a proponent for this type of application development, it is exactly what my project Persevere is built for.
Alex also talked about a number of other offerings including SimpleDB and CouchDB that can provide database services through web/HTTP interaction. While of course, I think Persevere is further along and more powerful than these other systems (with full client and server automated orthogonal persistence, function storage, JSONP, JSONPath, JSON Schema support, and more), I think that even more important than trying persuade the superiority of one system, is to espouse the benefits of interoperability of these systems. It is highly unlikely that one web service DB is going to be a monopoly anytime soon. With the availability of multiple web services, sites will certainly want to take advantage of varied web services and mashups will quickly grow in popularity. The cost and difficulty of these mashups will hinge largely on what degree of interoperability exists between these web services.
When using databases that are particularly suited for persisting dynamic object/map style data, JSON is certainly an obvious choice for interoperability, simplicity, and adoption. In addition, we have one of the most widely adopted standards on the planet available to us in the REST HTTP vocabulary (GET, POST, PUT, and DELETE). These HTTP methods in combination with JSON can provide a very simple intuitive interface for many basic database operations. While SimpleDB uses REST HTTP methods, unfortunately it does not seem to be based on JSON. However, both Persevere and CouchDB indepedently came to use this JSON REST approach for database interaction, and conseqently have a very high level of interoperability opportunity.
The basic HTTP verbs provide a broadly accepted standard for basic database operations, but generally queries entail more power and sophistication than simple id based GET requests can provide. SimpleDB defines it’s own query language. CouchDb does not have a query language (as far as I can tell), but rather relies on server side JavaScript mapping functions. Persevere uses JSONPath for it’s query languages and also supports server side JavaScript mapping functions. JSONPath seems to be an emerging standard, and as mentioned in the last post, will probably even be included in Dojo in the future.
I believe it is important that these different web storage engines both continue to innovate and evolve, but also work hard to find opportunities to work together for interoperability as well.
dojox.jsonPath
November 29th, 2007
Dustin Machi of SitePen has been working on adding a JSONPath module to DojoX. He is using also utilizing this module for a dojox.data store that extensively uses JSONPath. Of course this is not available in the general Dojo release yet, but it is in the Dojo SVN/nightlies. It is great to hear that JSONPath is continuing to gain traction.
Comet Daily
November 26th, 2007
I pleased to announce I have recently joined the team of Comet Daily contributors. I will be writing articles and musings on a regular basis for this new site. I am exciting about this opportunity to participate in the discussion and dissemination of the advance of Comet technology alongside some brilliant people in this area. You can see my first article on Comet discussing Event and REST approaches to Comet. I will probably be focusing a lot on REST approaches to Comet with my blog posts.
W3C - Enabling Read Access for Web Resources
November 16th, 2007
W3C has recently created a proposal for adding cross domain capabilities to the XMLHttpRequest (XHR) object. The proposal specifies how XHR can communicate with servers so that servers can explicitly opt-in to provide controlled access to their resources from other sites. I believe this proposal has great potential for providing a solid robust mechanism for secure mashups that isn’t built on various hacks. A number of things to understand about this proposal:
- It does not create any new vulnerabilities with existing servers. Cross domain XHR will always fail with existing servers until they have specifically added headers to define the access control. In other words it doesn’t add new vulnerabilities to the web, rather it allows those who want to add cross site access the ability to due it in a secure manner without hacks like JSONP or fragment identifier messaging.
- Both GET and POST can currently be executed cross site with scripts tags or form submission, so many threats such as CSRF and DOS already exist, the proposal does not introduce them.
- The proposal states advices agents not to expose any trusted data, such as cookies, HTTP header data, inappropriately, but it is unclear how that is determined.. This will hopefully reduce the incident of cross site request forgery, and forces developers to use more secure explicit forms of authentication maintanence.
- Developers that allow cross site access still must ensure that they are not providing privileged information to sites that should not be accessing the information. Developers that allow POST and other modifying operations should take similiar precautions.
- This provides a fine-grained access control level. When servers define access control headers that allow cross site access, they can specify which web page domains are allowed to access their resources.
This proposal seeks to adds similiar functionality as JSONRequest in terms of adding secure cross site access. Here is a short contrast between JSONRequest and this new proposal (W3C/AC):
| JSONRequest | W3C/AC | |
| Format constraint | JSON data only | Any format (provided as text through XHR) |
| Opt-in method | Add header Content-Type:application/json-request | Add header Access-Control: allow <*.example.org> |
| Domain Control | Only programmatically based off request source information | Fine grained control determined by Access-Control information |
| Performance | JSON can be parsed natively for improved performance | non-GET operations required a previous GET operation which can take extra time |
| Headers | Cookies removed, no user defined headers | Cookies and headers may still be sent. Spec advices agents not to expose any trusted data, such as cookies, HTTP header data, inappropriately, but it is unclear how that is determined. |
| HTTP Methods | GET, POST | Any (non-GET require explicit permission) |
| POST access | No prior authentication required (but cookies are still stripped) | Must be explicitly allowed in headers prior to the POST |
The W3C/AC in combination with getters/setters and a possible sandboxing mechanism in ES4 would also potentially allow sites to load and sandbox cross site advertisement scripts (one of the greatest XSS vulnerabilities on the web). I am all for this proposal. I know that new proposals that could open new vulnerabilities naturally cause fear and uncertainty, but this is a well thought out proposal that provides a solid mechanism for providiong cross site data, and is vastly better than the hacks we are currently working with. Both W3C/AC and JSONRequest would be both be fantastic additions to the browsers and open up huge opportunities for improving security of cross-domain mashups.
JSON Schema Service Descriptor
November 6th, 2007
By including method definitions, JSON Schema can easily be extended to be a service descriptor for the RPC (remote procedure call) service JSON-RPC. JSON Schema provides robust type definition which can be utilized for defining types in method call contracts. In addition, using JSON Schema allows methods to be defined in the same manner as property definitions, so REST and RPC services can be defined in a uniform manner, and services can even support both REST and RPC services. JSON Schema Service Descriptor could be considered a WSDL format for JSON.
A JSON Schema service descriptor is simply a JSON Schema with the additional definition for methods. A JSON Schema service descriptor is an JSON object with properties that correspond to available methods or procedures that can be called, and the values of the properties in the schema should contain a method definitions that provide the definition for how the methods should be called. A method definition can define various attributes of the method that define it’s usage and valid values. A method definition follows the same format as a JSON Schema property definition except, that the type value must be “method”, the property attributes describe the return value, and there are additional attributes “returns” which describes the type of the return value, and “params” which is an array of property definitions (or possibly an object if 1.1 supports objects as params). Here are the properties that are in addition to the standard JSON Schema property definition:
| Property | Meaning |
| type | Must be “method” for a method definition. |
| returns | Must be a type definition. If it is not included, the return value is not needed, the method should be considered a notification. |
| params | Must be an array of property definitions. Each property definition may also have a name property to define the name of parameters. A property definition with the name “*” can be used to define the type for additional parameters (in the case of variable parameters). Note: If objects as params is accepted as a valid value for JSON-RPC, this can be an object/schema as well, with property definitions for each property in the object. |
In addition a service descriptor should contain the id and description properties, and may contain the version property as defined here:
| Property | Meaning |
| id | A unique identifier for the service. When possible, the id should represent the URL from the which service descriptor can be accessed (relative URLs are acceptable). |
| description | This a description of the service. |
| version | This is the version of the service. |
A service descriptor could look like (each top level property defines an available method/procedure):
{"divide": {
"description":"Divide one number by another",
"type":"method",
"returns":"number",
"params":[{"type":"number","name":"dividend","required":true},
{"type":"number","name":"divisor","required":true}]
},
“sqrt”: {
“description”:”Find the square root of a number”,
“type”:”method”,
“returns”:”number”,
“params”:[{"type":"number","name":"square","required":true,
"minimum":0,"description":"Number to find the square root of"}]
},
“sqrtComplex”: {
“description”:”Find the square root of a number including negative numbers”,
“type”:”method”,
“returns”:["number", {"realPart":{"type":"number"},
"imaginaryPart":{"type":"number"}}],
"params":[{"type":"number","name":"square","required":true,
"description":"Number to find the square root of"}]
},
"sum": {
"description":"Find the sum of the parameters",
"type":"method",
"returns":"number",
"params":[{"type":"number","name":"*",
"description":"Number to include in the sum"}]
},
"description":"Some math methods. Strings as property/method definitions are ignored by JSON Schema validators, but this property can be used for a description."
}
Here you can see that we are utilizing various JSON Schema definitions such as union types (allowing for a “number” or a complex number object to be returned), and object definitions (complex number definition) in the type definitions.
Mixing RPCs and Object Data
By utilizing JSON Schema, a schema can provide information about a JSON source without any assumptions about whether it provides RPCs or direct object access (REST when using HTTP) or both. For example if there was a schema for the url http://www.myfriends.com/jim:
{"firstName":{"type":"string"},
"lastName":{"type":"string"},
"friends":{"type":{"items":{"firstName":{"type":"string"},
"lastName":{"type":"string"}}},
"addFriend":
{"type":"method",
"description":"Adds a friend to this person"},
"params":[{"firstName":{"type":"string"},
"lastName":{"type":"string"}]}
}
In this example, the schema defines methods that can be called (using JSON-RPC) on this object (http://www.myfriends.com/kris) as well as properties that should be expected from this object/url. That is the object properties might look something like this (on HTTP what would be returned by GET, but no protocol needs to be assumed):
{"firstName":"Kris","lastName":"Zyp","friends":[]}
Note that this example could be more DRY by using a JSON Referencing scheme to define a person definition, and reusing that definition for the friends array and the parameter definition, but this is not necessary.
JSON and ES4
November 5th, 2007
To say that ES4 is a hot topic is probably quite an understatement. Controversy over the proposed successor to ES3/JavaScript 1 has been intense. I don’t want to get into the general arguments about ES4, however, there are a number of aspects of ES4 that may affect JSON that I wanted to take a look at.
- toJSONString(), parseJSON() - ES4 proposes to add toJSONString as a method on all objects, and parseJSON as a method on all strings. These method will provide native support for serialization and deserialization of JSON data. This approach has met with some resistance. With Douglas Crockford’s recent change in his advised API for JSON, this approach may lose some of it’s forward compatibility as well.
- Namespaces - ES4 has introduced the concept of namespaces. Any property on a object may have a namespace in addition to the string identifier. For example:
namespace myNameSpace.
obj={foo:"bar"};
obj.myNameSpace::foo = "another value";
The “obj” object now has two properties with the name of “foo”, but with different namespaces. This presents some issues for serialization of objects to JSON. How should namespaced properties be represented in JSON? This appears to still be an open question for the ES4 working group. The current reference implementation simply fails when it encounters a namespaced property. - ES4 will also add more objects and mechanisms that will not be JSON serializable as well.
Now, I believe that ES4 is a great language, and I am excited to use it, but I just thought it would be interesting to examine how it will impact JSON usage in JavaScript.
MicroMedia IS JSON
November 1st, 2007
JSON is a BETTER syndication format than RSS.
It is the MicroMedia format of the future
To find out more, just look for micromedia110107
« Previous Page — Next Page »