The Future Ajax Wrapper
April 8th, 2008
With IE8’s new XDomainRequest feature, a new API is added for cross-site requests, instead of using the W3C cross-site access proposal. Just for fun, I thought I would provide a little glimpse of what the classic Ajax request wrapper function may look like for the next era of web developers. Just a few simple calls to XMLHttpRequest would be way too easy, so instead we get do this:
function doRequest(method,url,async,onLoad,onProgress) {
var xhr;
if ((onProgress || isXDomainUrl(url)) && window.XDomainRequest) {
// if it is x-domain or streaming/incremental updates are needed we will use IE's XDomainRequest for IE
// streaming/interactive mode is broken in IE's XHR, but for some reason works in XDR (with onprogress), so we will
// need to use XDR if incremental updates are necessary
// see bug https://connect.microsoft.com/IE/feedback/ViewFeedback.aspx?FeedbackID=334813
if (url.match(/^https:/) && !onProgress) {
// XDR doesn’t work for secure https communication
// see bug https://connect.microsoft.com/IE/feedback/ViewFeedback.aspx?FeedbackID=333380
loadUsingScriptTag(url); // script tag insertion can be more secure than XDR
// in some situations because it supports https
return;
}
xhr = new XDomainRequest;
// relative paths don’t work in XDomainRequest, see bug https://connect.microsoft.com/IE/feedback/ViewFeedback.aspx?FeedbackID=333275
if (!url.match(/^http:/)) { // test to see if it is an absolute url
url = absoluteUrl(location.href,url); // must have a function to turn it into an absolute url
}
if (!(method == “GET” || method == “POST”)) {
// XDomainRequest does not support methods besides GET and POST
// see bug https://connect.microsoft.com/IE/feedback/ViewFeedback.aspx?FeedbackID=334809
// We will try to add the method as a parameter and hope the server will understand… good luck :/
url += “&method=” + method;
method = “POST”;
}
function xdrLoad() {
if (xhr.contentType.match(/\/xml/)){
// there is no responseXML in XDomainRequest, so we have to create it manually
var dom = new ActiveXObject(”Microsoft.XMLDOM”);
dom.async = false;
dom.loadXML(xhr.responseText,200);
onLoad(dom);
}
else {
onLoad(xhr.responseText,200); // we will assume that the status code is 200, XDomainRequest rejects all other successful status codes
// see bug https://connect.microsoft.com/IE/feedback/ViewFeedback.aspx?FeedbackID=334804
}
}
if (async === false) {
// XDomainRequest does not support synchronous requests
// see bug https://connect.microsoft.com/IE/feedback/ViewFeedback.aspx?FeedbackID=336031
// so we will try to block execution on our own (which is not really possible in any reasonable manner)
var loaded;
xhr.onload = function() {
loaded = true;
xdrLoad();
}
xhr.open(method,url);
xhr.send(null);
while(!loaded) { // try to block until the response is received
// I am sure the user won’t mind just clicking OK so we can block execution
alert(”Waiting for the response, please click OK because it probably is here now”);
}
return;
}
else { // do an asynchronous request with XDomainRequest
xhr.onload = xdrLoad;
xhr.open(method,url);
xhr.onprogress = onProgress;
}
}
// we will mercifully skip all the branches for ActiveXObject(”Microsoft.XMLHTTP”) to accomodate IE6 and lower
else {
xhr = new XMLHttpRequest; // use the standard XHR for same origin and browsers that implement cross-site
// W3C requests and streaming
xhr.open(method,url,async);
xhr.onreadystatechange = function() {
if (xhr.readyState == 3) // interactive mode
onProgress(xhr.responseText);
if (xhr.readyState == 4) // finished
onLoad(xhr.responseText,xhr.status);
}
}
xhr.send(null); // finally send the request whether it be XDR or XHR
// and supporting functions
function absoluteUrl : function(baseUrl, relativeUrl) {
// This takes a base url and a relative url and resolves the target url.
// For example:
// resolveUrl(”http://www.domain.com/path1/path2″,”../path3″) ->”http://www.domain.com/path1/path3″
//
if (relativeUrl.match(/\w+:\/\//))
return relativeUrl;
if (relativeUrl.charAt(0)==’/') {
baseUrl = baseUrl.match(/.*\/\/[^\/]+/)
return (baseUrl ? baseUrl[0] : ”) + relativeUrl;
}
//TODO: handle protocol relative urls: ://www.domain.com
baseUrl = baseUrl.substring(0,baseUrl.length - baseUrl.match(/[^\/]*$/)[0].length);// clean off the trailing path
if (relativeUrl == ‘.’)
return baseUrl;
while (relativeUrl.substring(0,3) == ‘../’) {
baseUrl = baseUrl.substring(0,baseUrl.length - baseUrl.match(/[^\/]*\/$/)[0].length);
relativeUrl = relativeUrl.substring(3);
}
return baseUrl + relativeUrl;
}
function loadUsingScriptTag(url) {
… do JSONP here if we want
}
}
Disclaimer: This is completely untested code, just a glimpse of the issues and divergences I am already aware of. I am sure the working code with be even more complicated. I didn’t get into the code branches for errors (XDomainRequest uses a new “onerror” handler), and or discuss the server side branching code for two different sets of headers, or the lack of support for basic HTTP headers.
I do appreciate IE’s effort to provide some mechanism for cross-site requests. But, do we really need a new API? Also, don’t get me wrong, I think the W3C cross-site XHR has it’s warts as well, for example I think XML processing instructions do not belong in an XHR specification.
Static JSON with Callback
April 1st, 2008
JSONP has become a well known technique for loading JSON data from other sites. JSONP defines that a parameter should name a callback function to be called from the script returned from the web service provider. Yahoo, Flickr, and many other public web services provide some form of JSONP with their web services. However, there a couple significant difficulties with JSONP:
- It is impossible to make a static file available as a JSONP service
- There is no ability to attach metadata to the results
I have previously discussed a way to attach metadata to JSONP results. However, JSONP precludes the use of a static file because the client provides a dynamic name for the callback, and therefore the callback must be created dynamically based on the provided parameter. In addition, usually any given JSON file may be accessed directly (through XHR) or through JSONP, so in some situations, there should be no callback. Therefore, I propose an alternate technique for cross-site loading of JSON data.
Rather than using a client provided callback name, the callback name can simply be the full URL of the resource that is being accessed. By using the full URL, conflicts can be avoided with other resources being loading with JSONP simultaneously. Full URLs are also rarely used as global variables, and therefore make a good choice for the callback. In addition, a simple OR operator can be used to call the callback when available, or return the JSON data otherwise. The syntax for creating static JSON with a callback is this:
(window["<full url>"]||function(val){return val}) (<json data>)
So if you start with the following JSON data:
{foo:"bar"}
And if you want to make this JSON data available to XHR and cross-site script loading, you can simply add the following prefix and parenthesis around the JSON data:
(window["http://www.somesite.com/foobar"]||function(val){return val})
({foo:"bar"})
XHR can load this resource and perform an eval to parse the JSON without any modification. If the callback is not available, the pass-through function will return the JSON data and the JSON parsing eval will function operate exactly as before. If you want to load this JSON data using cross-site loading. Simply define your callback as the name of URL you are loading:
window["http://www.somesite.com/foobar"] = function(jsonData) {
handleJson(jsonData);
}
And then load the script:
var scriptTag = document.createElement("script");
scriptTag.setAttribute("src", url);
document.body.appendChild(scriptTag);
When the script is loaded the callback will be called with the JSON data passed in as the parameter.
By using this technique we can create a single static file that can provide JSON data to XHR-based JSON loaders as well dynamic script tag loaders in cross-site situations.
Update:
After discussions with Dustin Machi, I think it is worth noting that this technique is not appropriate for non-cacheable, non-idempotent requests. However, such requests would not be used against a static resource, they are inherently dynamic resources, and therefore the standard JSONP technique would not be problematic in these cases. For static resources, my proposal should still be an effective way to create a static JSON resource that is cross-site accessible.
JSON Web Services Support in Dojo
March 25th, 2008
Dojo 1.1 has expanded support for pluggable web services including JSON-RPC, REST, JSONP, and HTTP methods. This services can be defined with the new SMD specification.
SMD will provide a foundation to facilitate a future of pluggable interoperable SAO-based web services, where client and server components can be decoupled. Existing client/server frameworks tend to have tightly coupled communication, and applications built with standalone JavaScript libraries like the Dojo Toolkit generally must build their own client and server communication with the only assistance in the form of Ajax wrappers. With SMD, web services can be described with an open format, and client libraries can utilize these SMD descriptions to build high-level friendly communication wrappers.
JSONPath in Dojo
March 18th, 2008
JSONPath has landed in the Dojo Toolkit 1.1, and I have recently added some bug fixes and improvements to the Dojo implementation of JSONPath. You can read more details here, but two substantial features were added:
- Result-Based Evaluation - Result-based evaluation uses a more intuitive form of evaluation that follows JavaScript (and other languages) evaluation more closely. Peforming a JSONPath query that references a property will return that property, rather than returning an array with a single item with the value of the property.
- Safe-Evaluation - The original JSONPath specification does not provide any limitations on the sub-expressions, rather allowing any JavaScript expression. This means arbitrary JavaScript can be executed, and it also prevents portability due to it’s inherent reliance on a JavaScript engine. Safe-evaluation limits the operators in sub-expressions to facilitate portability and prevent unsafe, arbitrary code execution. This feature will probably be added in Dojo 1.2.
JSONPath queries in Dojo can be executed like this:
dojo.require("dojox.jsonPath");...dojox.jsonPath.query(object,query,options)
JSON Schema Progress
February 29th, 2008
JSON Schema continues to progress with steady refinements and lively discussion on improvements resulting in a second draft. I have been working the OpenAjax alliance on collaboration and synchronizing the work of JSON Schema with the OpenAjax Metadata specification.
I have updated my JavaScript implementation of JSON Schema with the latest modifications to the JSON Schema proposal. I also updated the common schemas (future home page of JSON Schema) for the most recent changes as well (This includes the schema for a schema). I have also done some corrections/updates to the SMD definition (mostly typos and such). The SMD definition will probably soon be hosted by Dojo as well.
HTTP as the Basis for Comet
February 15th, 2008
Michael Carter wrote:
“In short, it makes strategic sense for the Comet standard to closely resemble and interoperate with current standards. We want to be saying, “But Comet is basically HTTP, not some new, strange standard.” Then we’ll have a much easier time gaining widespread adoption.”
The Next Great Protocol: HTTP
February 12th, 2008
I suppose this post would be more prophetic a decade or two ago. It was in the 90’s that the HTTP protocol really became the Great protocol. It is foundation of the World Wide Web, and is language on which browsers were able to really open the doorway to the Internet for us. So am I little behind the times to suggest that HTTP now has an emerging future relevance? Is HTTP a relic of the past or does it have something to contribute to the future?
One of the distinctives of current Internet technological advance is in the growing realm of open sharing and utilizing data from disparate sources. Facilitating this progress is one of the principle goals of my work and this site. I want the Open Web to be more than just a bunch of pages that are developed without proprietary constraints, but for the Open Web to be the environment for open flow of information with intelligible interconnection of data that can give participants unprecedented leverage and permutations of capabilities. Mashups are a buzzword to describe this process. However, in order for information to flow rapidly, there must be commonly understood communication. JSON has enormous potential because it is so simple, expressive, and pervasive that it forms a excellent syntax for expressing data. However, JSON is not a transport. Two agents that wish to dialogue may understand JSON data, but they still need a mechanism to communicate and transfer that information. HTTP is almost ubiquitously the right choice for the transport. The incredible adoption of HTTP is main reason for this. No transport is more widely understood
What is wrong with HTTP?
Before going into the benefits of HTTP, let us look at the problems with the HTTP, or more specifically, THE problem with HTTP. The most fundamental problem with HTTP is that it requires that every single response must be preceding by one corresponding request. The specification describes HTTP as request/response protocol. This constraint has an enormous impact on the capabilities of HTTP. The first major impact is in hindering performance optimizations. In order to load a web page, every resource must be requested before the server can send the resources. This creates a signficant latency problems. There can be large gaps in transfers while servers are waiting to receive requests. However, a server could easily determine the most likely resources that a user agent will need and send them before the request if this constraint was not in place.
This constraint is also the fundamental cause of consternation with Comet development. Comet consists of efforts to allow servers to send messages to clients asynchronously instead of in immediate response to a request. Doing this requires creating an outstanding HTTP request that the server can respond to when it wants to. Comet push capabilities could easily be achieved if servers could simply send messages to the client without requiring a preceding request.
Throwing away the entire protocol because of a single issue is absurd, rather let’s fix or enhance the protocol. In recent articles I have discussed how non-request/response-bound HTTP messages can be sent within HTTP messages in order to deal with this problem using existing infrastructure.
So why is HTTP the right choice for future information transport?
- It is so pervasive - HTTP is everywhere. It is how all browsers communicate with web servers. HTTP is understood by an overwhelming amount of software. Attempts to reinvent the functionality and capabilities of HTTP is essentially asking for this broadly understood language to be ignored in lieu of new one. Without very significant advantageous to new semantics and vocabulary, such attempts are generally either doomed to obscurity or worse, a cause in division in multiple semantics for the same thing, causing increased code complexity and costs.
- It has the constructs for tomorrow - With ever increasing interchange of data in the future, more sophisticated and robust techniques for communicating data are needed. Many of these techniques already exist in HTTP, but have simply not yet been needed with yesterday’s technology. The future of high performance, intelligent data interchange will hinge on capabilities that already exist in HTTP including:
- Content negotiation - HTTP includes vocabulary for negotiating between different formats.
- Partial data transfers - HTTP has an extensible mechanism for sending a range of information.
- Robust error handling - HTTP includes a comprehensive set of errors.
- Parallel scalability - Perhaps one of the most impressive features of HTTP is how carefully designed such that demand can be easily scaled across numerous machines as HTTP proxy servers.
- REST/CRUD semantics - HTTP provides semantics for basic create, read, update, and delete operations.
- Performance improvements - HTTP pipelining has only begun to be utilized (only Opera has it turned on by default). Substantial performance improvements can be realized through pipelining.
- Emerging technologies give us new leverage with HTTP - With traditional web application development, much of the workings of HTTP were hidden away by the browser and the server. However, the Ajax revolution gives developers far greater control of HTTP messaging. Most Ajax developers have simply used XMLHttpRequest as a means for communicating simple payloads of data back and forth to server, but XHR has given developers new access to the HTTP capabilities through their header metadata, and leverage to utilize the full semantics of HTTP for more meaningful communication.
Furthermore, with new XHR capabilities coming soon (FF3 will have cross site XHR support), XHR communication will involve much more than simply communicating with your own server. Communicating with your own server does not necessarily require widely understood vocabulary, but when communicating with other servers, the cost and efficiency of integration will be directly related to how much shared vocabulary can be utilized to provide jointly understood communication. New ways to utilize, extend, and leverage HTTP are being developed as well like the Atom Publishing Protocol.
Does it really matter what is on the wire? Can’t we simply distribute API based communication handlers? Consider this, is it easier to setup a TV to tune into the available TV stations, or is it easier to setup a printer to work with your new operating system? TV stations have standardized on a single format for broadcasting content. Connecting a TV to a station is as simple as turning it on choosing your station. On other hand, printers have no standardized communication with servers. Devices like printers use API based communication handlers (AKA drivers). With a huge number of different protocols for each printer, and different operating systems to interact with, there are an enormous amount of permutations of different drivers that must be developed, very prone to incompatibilities. While the situation has improved, many have experienced the frustrating effort that can go into trying to find the right driver for your OS/printer combination.
But in the realm of browser communication does it matter since we are all using JavaScript? Absolutely. There may be a single language on the browser, but there are different JavaScript libraries, with may prefer different APIs. In addition the browser environment can be very bandwidth constrained. Requiring another library for each data endpoint that you want to connect to, does not scale well. And as we move towards more service oriented architectures, browsers will not be the only consumers of information. Many other clients must be considered as well.
We need to be examining the HTTP specifications and learning how to leverage the power that is available, to maximize our potential in coming world of extensive data interchange and mashups. Next time you need to create Ajax communication to trigger CRUD operations, consider using the RESTful HTTP methods (PUT, POST, and DELETE). Do you need to deliver multiple formats of the resource? Consider using HTTP content negotiation. Do you want Comet capabilities in your application? Consider using an HTTP standards based approach. The more we can utilize what is there, the more widely we can be understood, and the more efficiently we can utilize the infrastructure of the web. The full utilization of HTTP can provide a solid foundation for the future of data interchange.
What Makes a Good Web Framework
February 7th, 2008
A friend recently asked me if there are any server side or web frameworks I like besides the one I developed, Persevere. I have been critical of a number of frameworks, but I think it worth articulating what I don’t like about many frameworks, because there are many out there that I believe are good.
One of the principle issues I have against many web frameworks is the reliance on server session state in the web framework. Many web frameworks rely on the session data in order to achieve basic functionality. While developers should be able to use session data, in particular authentication is usually held in session, but it is advisable to avoid unnecessary use of sessions to store application state information. It is frameworks that rely on sessions themselves for basic application state storage that I do not like.
Why not use sessions to store application state? Let us consider the construct of a session. Is there anywhere else in the in computing world, where it would be acceptable to use a component that will fail if it is not used for a given amount time (of 30 minutes), and until this time of inactivity has passed, the component will consume resources? This is simply a terrible construct to build applications. It consumes unnecessary resources, and forces users to essentially restart their application if they leave it idle for too long. When application state is stored on a client, instead of the server, there is an inherent scalability advantage as well. The more clients that connect to your server, the more room there is to store application state. If application state information is stored on the server, the room for application state stays constant even though the number of clients (and required need for application state) grows.
Component-based frameworks often rely on sessions. An object or component is used as the controller for a certain part of an application, and the framework will keep this component in the session so the object can stay alive to handle and respond to different actions from the client. A particular heinous use of sessions is when frameworks keep an active copy of the UI on the server. This copy of the UI may be a various different forms, where as a DOM tree, or in set of objects that correspondence to the server’s representation of the UI. Of course maintain a copy of the UI layout on the server is particularly burdensome. Offenders include Java Server Faces and ZK.
So what are frameworks that I like? I did a project using Spring MVC, and that framework does an excellent job of avoiding the pitfalls of session reliance. However, I believe Spring is more suited for traditional page-based applications than Ajax-based. In the Ajax world, I think DWR is excellent framework. DWR does not attempt to handle UI work, but simply provides a easy framework for JavaScript and Java communication, with minimal server load.
Another important aspect of a future web frameworks is use of standards, in such a way that different frameworks can work with each other. One of the most important aspects of this in the client/server communication. Most web frameworks do not make any attempt to use a standard based client/server communication, perhaps because there has not historically been very many standards for this communication (other than HTTP, but that is a given). However, with the increasing popularity JSON and conventions like JSON-RPC (and hopefully SMD, JSON Schema, and others), it will be possible for client server frameworks to use more open standard based communication. This will allow other client and server technologies to plug in to the communication. With this mind, I also like frameworks like jabsorb. Client side libraries like Dojo also must utilize the standards, if they want to take advantage of higher levels of communication with the server, and I appreciate this.
OpenAjax Concerns
January 31st, 2008
Ric Johnson posted some concerns about OpenAjax. Ric is the donor for multiple important domains like this one, and I certainly appreciate his efforts to further the open web, and believe that it does indeed behoove us to recognize the donors of sites like OpenAjax. Thanks, Ric!
Update:
I recently heard from Jon Ferraiolo of OpenAjax, and he working on getting the next revision of the web site finished, which will include a link to their domain donor, OpenDomain. Thanks Jon, for moving this forward.
Defining JSON Referencing Convention with JSON Schema
January 22nd, 2008
I have been involved in lengthy discussions on the best approach to JSON referencing. I have long wished for a single convention for JSON referencing, but the numerous approaches currently in use have frustrated my attempts at consensus. In some situations path-based referencing can be a very clean approach for intra-message referencing. On the otherhand, id-based referencing can be preferential, especially when objects already have unique identifiers and/or when referencing outside the local JSON message is necessary, but settling on a convention for referencing is difficult. With id-based referencing one of the sticky points is defining the identity property. I have long argued for always using “id” as the identity property, but others have felt that this may result in the id property may not necessarily be globally unique. Others use an identifier property in the root of objects that defines the identity property for the rest of the object. This allows for flexibility of defining alternate identity properties, but unfortunately mixes meta data right into the data payload.
Perhaps rather than requiring a single agreed upon identity property for interoperable JSON referencing, it may be more reasonable to allow JSON providers to define their own identity and referencing property with a JSON Schema. JSON Schema currently includes a “unique” attribute that is used to define a property as being unique. A JSON Schema can easily declare the identity property by defining a single unique property in the schema. Ids can also be declared as URLs with relative resolution that has the broadly powerful capabilities of not only intra-message referencing, but also cross-site referencing.
Agreeing upon a single referencing property (I have proposed “$ref”) is less problematic, the “$ref” is unlikely to clash with existing properties, and there is little chance for property reuse like with ids. Nonetheless, it would be possible to add a special attribute to JSON Schema to denote a referencing property. This attribute could also signal what forms of referencing to expect (path-based, id-based, or both). An example JSON Schema that defines referencing capabilities might look like:
PersonSchema=
{"type":
{"person_id":{"unique":"true","type":"string","format":"url"},
"name":{"type":"string"}},
"referenceProperty":"$ref",
"referencingForm":"both"}
While I still like the ease and efficiency of a single JSON referencing convention, the flexibility of defining the id and referencing properties with JSON Schema may be reasonable approach to the future of JSON referencing. What do you think?
Next Page »