Fred Brack
Raleigh, NC

Using AJAX to Access Another Web Page Using the DOM
by Fred Brack
Updated

This is not a full tutorial on AJAX but rather an introduction and example of how to access a second web page using the DOM from your current web page. I created this web page to help me document my understanding of how to use AJAX for this purpose ... and because I like to document stuff for others! I hope you find it useful.

What Is AJAX?
The Challenge: Apply the DOM to a Second Page
The Trick: Accessing a Secondary Page of HTML With the DOM
Copying Secondary Page HTML to the Current Page
The Complete JavaScript File for AJAX
Bonus: Fred's Dynamic Content Loader
Index of My JavaScript Articles

What is AJAX?

Rather than repeat it all here, I would refer you to my original page on AJAX entitled: Using AJAX, the XMLHttpRequest Object API, to Read a Text File. But basically, AJAX is a special API for JavaScript which allows you to read any file within your own domain. It is designed to work with XML files, but it also returns plain text, which includes HTML. The methodology is explained in the referenced article, where I process a TEXT file to extract a value I want to use in my current web page. Here, we want to process that separate file as HTML using Document Object Model (DOM) methods.

The Challenge: Apply the DOM to a Second Page

As I wrote this series of articles, it occurred to me that every time I added an article to the series, I had to change the "Index" at the bottom of each old page to include the new page. I could use the technique I already documented for this purpose (processing HTML as a text file); but when I wrote about the Document Object Model (DOM) and its ability to read or write HTML, I figured I could use AJAX and DOM methods to read the Index from the bottom of the Intro page and insert that series of links in every new page via JavaScript.

What could YOU do with this technique?
As you look at this example, think about
the idea of copying ANY HTML from one
document into another -- common code,
a form of server-based Include. You could,
for example, build an otherwise unreferenced
HTML file with numerous sections of common
code to be picked up as-required in other files.
(I have just recently done this and have added it
below as Fred's Dynamic Content Loader!)
Read on to learn how to do this in general ...

The challenge, however, was how to get access to the secondary page via the DOM to use its methods. Normally you write something like document.method() to access the current page. I wanted to do something like secondPage.method(). It is not at all obvious how to take the "flat file" of HTML ultimately returned by AJAX (a string) and turn it into an object readable by the DOM methods.

So the technique described here is: how to read a second page of HTML from your current page and use the power of the DOM on that second page, as well as the first.

The Trick: Accessing a Secondary Page of HTML With the DOM

I must give credit for a tricky JavaScript sequence to David Walsh who runs a blog and wrote an article in 2017 that did exactly what I want: Convert String to DOM Nodes. I knew I could read my javascriptintro.html file using AJAX as a string (call it variable html), but I didn't know how to get it into an object (call it variable htmlObject) that the DOM would treat like an additional HTML file (which, of course, is exactly what it is). David figured it out in one line of JavaScript:

let htmlObject = document.createRange().createContextualFragment(html);

Breaking David's statement apart:

The createRange method returns a new Range object.
According to Mozilla, "The Range interface represents a fragment of a document that can contain nodes and parts of text nodes."
One of the Range methods is called createContextualFragment.
Again according to Mozilla, it "Returns a DocumentFragment created from a given string of code." (Note the word string, as that's what html is.)
What's a document fragment? Again Mozilla (italics and bolding mine): "The DocumentFragment interface represents a minimal document object that has no parent. It is used as a lightweight version of Document that stores a segment of a document structure comprised of nodes just like a standard document. The key difference is due to the fact that the document fragment isn't part of the active document tree structure." Perfect: exactly what I needed! While they speak in terms of a fragment of HTML, I happen to have the entire document, and that's OK.
The potentially bad news: Mozilla has marked the createContextualFragment method as "Experimental. Expect behavior to change in the future." Sigh. However, it is supported by all current browsers, and David was using it in 2017, so I feel reasonably comfortable using it.

Once the createRange method was executed, I could apply a DOM method to the resultant object (on the secondary page). For example (and actually what I used), to find the Index (identified with id="index") in javascriptintro.html:

const node = htmlObject.querySelector("#index");

The code being referenced in javascriptintro.html is a DIV containing my list of links. It starts out like this:

<div id="index">
...
</div>

Now that I have the pointer to the area where my Index is located, I can extract all the HTML associated with the display of the index using the outerHTML method:

let indexHTML = node.outerHTML

Now what do I do with it?

Copying Secondary Page HTML to the Current Page

So I have an object (indexHTML) with all the HTML I want to include on the current page (the Index to my articles), and I need to insert that somewhere on my current page. While there are several ways to do this, I decided on using the appendChild method. First I need to create a named (via an id) location on the current page (every page except javascriptintro.html) after which I will insert my index. The easiest way to do this seemed to be to use an empty <span> as follows:

<span id="indexhdr"></span>

The appendChild method will insert my indexHTML into the empty span (the HTML would be the child of the span).

Now I need to form a proper node of the indexHTML HTML tags to get ready to insert them in my current file. So I create (createElement) a new DIV filled with (innerHTML) the Index HTML gathered from javascriptintro.html (yes, it was outerHTML there and innerHTML here!):

let newHTML = document.createElement("div");
newHTML.innerHTML = indexHTML;

Then I locate (querySelector) the place to insert the new HTML in the current document (via the id indexhdr) and use the appendChild method to make the div a child of the existing span:

const nodeOld = document.querySelector("#indexhdr");
nodeOld.appendChild(newHTML);

You can see the results at the bottom of this page.

The Complete JavaScript File for AJAX

Here is the complete file which invokes AJAX to read my javascriptintro.html file and copies its Index into the current file of HTML on display.

// JavaScript Index: Read the javascriptintro.html file and copy its Index section
// The getData function issues the XHR request against the URL
function getData(url) {
  let request = new XMLHttpRequest();
  request.onerror = function () { alert("Unable to retrieve "+url) } // if no-can-do
  request.onload = function ()  { // do this when the load is completed
    if (this.status!==200) { alert("Unable to retrieve "+url+"; status = "+this.status) }
    else { processData(this) } // Call processData with all the data from the file at url
  }
  request.open('GET',url);
  request.send();
}

// processData does whatever I want when the URL data is successfully read in
function processData(filedata) {
  let html = filedata.response; // ALL of the input file's HTML as a string
  // credit for the following technique: https://davidwalsh.name/convert-html-stings-dom-nodes
  let htmlObject = document.createRange().createContextualFragment(html);
  // pickup the index from the referenced URL and insert it in the current doc
  const node = htmlObject.querySelector("#index");
  let indexHTML = node.outerHTML
  let newHTML = document.createElement("div");
  newHTML.innerHTML = indexHTML;
  const nodeOld = document.querySelector("#indexhdr"); // an empty <span>
  nodeOld.appendChild(newHTML);
};

// Execute the above functions to modify the web page
const url = "javascriptintro.html"; // the reference URL
getData(url);
// END

Bonus: Fred's Dynamic Content Loader

So ... after finishing the entire page above, I decided to implement the idea I proposed at the very beginning: creating a form of server-based include function which would serve to dynamically populate portions of your web page. I called my routine DCL: Dynamic Content Loader. It picks up on the above technique but allows the caller to specify which section of HTML is to be replaced with what.

We start with the initialization routine (I called it DCLinit.js) which defines the two functions we have already seen, with name changes and a few added features. This is the code:

// DCLinit: Dynamic Content Loader Initialization, by Fred Brack, Sept 2020
// Save in your JS library as DCLinit.js
// Then invoke as: DCLget(dclurl,baseid[,hereid]);
// where dclurl is the full or relative url of the base HTML file in your domain
//       baseid is the id of the HTML in the base file you want copied here
//       hereid is the id of the HTML in the current document you want replaced
//       If hereid is absent, then baseid will be used for both purposes
// Example: DCLget("mybase.html","#myid") ... based on <p id="myid">Replace me</p>
// Note1: The ID name in your documents MUST be preceded by a # when calling:
// Note2: You can use a class instead (as ".classname) if you prefer, or a tag name
// Note3: The ENTIRE line of HTML which is tagged with an ID will be replaced
// Note4: For a generic replacement, you can use <span id="myid"></span>

// The DCLget function issues the XHR request against the URL
// It simply passes along the 2nd or 3rd parameter to DCLprocess
  function DCLget(url,idBase,idHere) {
    let request = new XMLHttpRequest(); // this is known as XHR
    request.onerror = function () { alert("Unable to retrieve "+url) }
    request.onload  = function () {
      if (this.status!==200) { alert("Unable to retrieve "+url+"; status = "+this.status) }
      else { DCLprocess(this,idBase,idHere) } // Call DCLprocess with all the HTML from the URL
  };
  request.open('GET',url); // this configures the call for the send
  request.responseType = "text"; // default, and correct for HTML
  request.send();
}

// DCLprocess takes the HTML Object from XHR and assumes 'idBase' has been set for what to locate and return
// If 'idHere' is not specified, then it is assumed to be the same as 'idBase'
function DCLprocess(dclBaseHTML,idBase,idHere) {
  if (typeof(idBase)=="undefined" || idBase=="") {
    alert("ERROR: the base ID passed to function DCLprocess was null or undefined");
    return;
  }
  if (typeof(idHere)=="undefined" || idHere=="") { idHere = idBase } // when no 3rd arg is passed
  else if (typeof(idHere)=="undefined" || idHere=="") {
    alert("ERROR: the ID for THIS document passed to function DCLprocess was null or undefined");
    return;
  }
  let html = dclBaseHTML.response; // ALL of the url's HTML as a string
  let htmlObject = document.createRange().createContextualFragment(html);
  // Pickup the requested HTML by ID from the referenced URL and insert it in the current doc
  const baseNode = htmlObject.querySelector(idBase); // the node in the base document
  if (String(baseNode)=="null") {
    alert("ERROR: the base ID passed to function DCLprocess ("+idBase+") was not found in "+dclurl);
    return
  }
  const currentNode = document.querySelector(idHere); // the node in the current document
  if (String(currentNode)=="null") {
    alert("ERROR: the ID passed to function DCLprocess ("+idHere+") was not found in the current document");
    return
  }
  currentNode.replaceWith(baseNode);
};
// END

Now we need to create our current file of HTML, the one we want modified dynamically. Every place we want something modified, we identify an HTML tag with an ID. It could be a paragraph, an empty UL or OL list, or even just an empty SPAN with an ID. All the IDs have to be unique, of course. Examples:

<p id="para1">This paragraph will be replaced.</p>
<ul id="list1"></ul>
<span id="span1></span>  [this content may or may not be replaced and doesn't currently take up space]

Then we create what I call the base file of HTML which contains all the dynamic content to be picked up and loaded into other HTML pages. The idea is that you can update this one file as often as you want, and any page referencing it will reflect the changes instantly whenever they are loaded. So by itself, the page is meaningless: it could very well look like an odd assortment of words, paragraphs, lists, images, tables, or whatever. Each outer (container) HTML element must have a unique ID. So, for example, you could put an ID on a paragraph, image, table, or a DIV with lots of other tags inside. Examples:

<p id="para1">New content for para1.</p>
<ul id="list1>
  <li>Item 1
  <li>Item 2
</ul>
<img id="fred" src="images/myavatar.jpg" width=100 height=100 alt="Fred's avatar">

Back in our current file, we call up the DCLinit.js file shown above to make our two functions available (one you call, the other internal); then we call the DCLget function (defined in DCLinit) as many times as necessary to dynamically load content to one or more places on the current page. IMPORTANT: Because this routine is generalized to accept any identification of what you want replaced (ID, like we recommend, or class, or even the first occurrence of a tag), you must precede the ID by a pound sign (#), assuming that's what you are using. (A class would be preceded by a period, and a tag, nothing.) Example, utilizing the two examples above:

<script src="js/DCLinit.js"></script>
<script>
url = "mybase.html";
DCLget(url,"#para1);
DCLget(url,"#list1");
DCLget(url,"#fred","#span1");
</script>

In the last example above, a different ID was specified in the base file (#fred) from the current file. If the IDs are the same, you only need to specify the ID once. The way you read the function call is: you want to get from the base (url) the element identified with the first ID and use it to replace the element identified with the second ID (if specified, else the first one again).

To see this concept in action, run my dcltest.html file.

It may occur to you that reading the base file each time you want to make a replacement in the current file is redundant (though with the speed of modern computers and the Internet hardly a concern). I have not found a way around this because JavaScript executes asynchronously and has no way to pause execution waiting for the completion of a function (in this case, the reading of the file) before moving on to the next statement outside the function. (In other words, all the DCLgets are executed one after the other, but their final actions of modifying the HTML are done later, asynchronously, when the XHR function completes.) The JavaScript Promise Object and its associated features like async, then, and callbacks work inside a function, but not for the main flow. I have not been able to find a way to pause execution between the first and second calls to DCLget. If I could, then we could capture the pointer to the base HTML and just call the internal function DCLprocess from then on. If YOU find a way to do this, please let me know!

I hope you have found this example interesting and potentially useful.

Want more detail or someone else's explanation?
I found this site after I wrote all of the above,
but I thought it was very well written:
XMLHttpRequest at JavaScript.info.

# # # # #

Fred