Week 6 - External Data

Working with external data

Most contemporary websites are not static but make use of text, imagery and data brought in from external sources. THis means that the exact content of a page may not be known at the point that it is being designed and developed. For example a developer of a page which includes a weather forecast may be aware that it will include expected minimum and maximum temperatures,an icon summarising the weather conditions, and a short description but will not know exactly what is going to be displayed on any given day. In this case the actual data which will be displayed will be incorporated into the page from an external API at the point at which a request is made of the page to be displayed. It is possible to incorporate external data using either client or server side technologies. This presentation concentrates on client side technologies.

Many user interactions only manipulate small portions of the page but traditional techniques require that the entire page must be reloaded. Server responses contain the entire page content rather than just the portion being updated. Loading full pages typically results in several additional HTTP requests for images, style sheets, scripts, and any other content that may be on the page and so these can be slow. Currently Ajax is the most commonly used client side technique for loading data into a page from a server.

Ajax allows for HTTP communication without requiring an entire page refresh. Data is loaded into an object within a client side script (typically written in javaScript) and then integrated with the page content using DOM scripting techniques. Thus the web page can communicate with the server without refreshing the whole page.

An AJAX XMLHttpRequest is useless by itself, requires DOM scripting components to embed the received data into the current document.
AJAX

The term Ajax originated as an acronym standing for Asynchronous JavaScript and XML. It has come to encompass a group of related capabilities and techniques. At its most basic level, an Ajax solution includes

JavaScript, to capture interactions with the user or other browser-related events

XMLHttpRequest object, which allows requests to be made to the server without interrupting other browser tasks

an external data source presenting information in a data format such as JSON, HTML, XML

JavaScript, to interpret the data from the server and present it on the page

Ajax has been credited with the move from static web pages to interactive web applications. Basically it is a means of loading data from the server to the web browser without a visible page refresh. This data can take many forms, and there are different options for what to do with it when it arrives. Inconsistencies in the way that different browsers have implemented the XMLHttpRequest object have made working with AJAX quite tricky.

Many libraries and frameworks have developed functionality to facilitate working with Ajax and the XMLHttpRequest ,jQuery among them. jQuery provides some very useful methods which simplify the process of interacting with external data.

Asynchronous loading

Ajax works in an asynchronous way. Asynchronous loading means that once the HTTP request to retrieve external data has been issued, script execution immediately resumes without waiting for the data to arrive. Sometime later, the browser receives the response from the server and handles it. This is usually desirable as freezing the web browser while waiting for data to be retrieved does not provide a good user experience. If particular actions need to be delayed until the load has been completed, jQuery provides callbacks to handle the situation. Ajax callbacks execute after data arrives from the server.
jQuery Callback

javaScript statements are executed line by line but visual effects take a finite amount of time to run. If the next line of code runs before the effect is finished it can cause unintended effects A similar situation arises when data is being loaded from an external source. In this case data can arrive at any time after it is requested and if the code designed to process it runs before the data arrives errors occur.

In both cases the issue is caused by a line or lines of code running before it is intended. To prevent this use a callback function.

A callback function is executed after the current effect is finished.

The typical syntax for a visual effect is $(selector).hide(speed,callback);

Code example 1 :no callback – alert box shows too soon
```
$("button").click(function(){
    $("p").hide(1000);
    alert("The paragraph is now hidden");
});
```
Code example 2 : callback used and the alert box displays when intended
```
$("button").click(function(){
    $("p").hide("slow",function(){
         alert("The paragraph is now hidden");
    });
});
```
A simple example of a callback being used to load external data can be seen in the following example which we will consider in more detail later.
```
$(document).ready(function() {
    $('#test').click(function() {
        $.getJSON('example.json', function(data) {
            // actions to be taken
        });
        return false;
    });
});
```
Format of external data

Generally it is desirable to store information in a compact form and to transfer as little data as possible. It is useful to be able to retrieve the data in a structure that can be traversed using JavaScript. Data can be stored in html format . In this case we can traverse the HTML with jQuery selectors, and manipulate it, but this is an inefficient way of storing data. Data can also be stored using XML. XML(eXtensible Markup Language) is a superset of html .It was designed to store and transport data and to be self-descriptive. There are a variety of techniques available for traversing and manipulating XML and many API make use of the format. However currently one of the most popular formats for storing data to be included in web pages is JavaScript Object Notation (JSON). JSON is a native javaScript data format which is very compact and so involves less data to transfer than some other formats. AS it is a native javaScript format it also requires less code to process it that other formats. JavaScript objects are sets of key-value pairs, and can be defined using curly brackets {}. JavaScript arrays can be defined using square brackets [ ] and use integer indexes to access individual elements. Combining these two concepts, we can easily express some very complex and rich data structures. JavaScript Object Notation (JSON) uses this simple syntax to provide a notation to describe data. We will look at file formats in more details in a future presentation.
Importing external data
Writing AJAX code can be time consuming and fiddly as different browsers have different syntax for AJAX implementation. This means that it is necessary to write extra code to test for each browsers and then ensure that the AJAX request works in each of them. However jQuery provides a number of utilities which can simplify things and provide AJAX functionality with very few lines of code.One of the simplest ways of harnessing AJAX techniques to import external data is to use the jQuery specialised shortcut method $.getJSON(). This gives access to external JSON data while hiding much of the complexity of the basic AJAX XMLHttpRequest To use it with a local data file it is possible to pass it the filename as shown below:
```
$(document).ready(function() {
    $('#test').click(function() {
        $.getJSON('example.json', function(data) {
            // actions to be taken
        });
        return false;
    });
});
```
The function call loads the file using an anonymous function. It then uses a second argument, a callback function to specify what action to take when the load is complete.

The second callback function also takes an argument - “data”. This is filled with the external data once it has been received from the external file. Once information have been loaded into the data variable t we can use standard javaScript methods to traverse the JSON structure as necessary.

Note :in this example an anonymous function expression is used for the callback. This is common practice in jQuery code and makes it easy to see which code will execute after external data is received but it is also possible to make reference to a standard function to handle the situation.

Before starting to handle the external data we need to be aware of how that data is structured. If a file is stored locally it is easy to open the file in Notepad++ or a similar editor and inspect the file. If, on the other hand, you are accessing data from a remote source it may not be so easy. Most API will give some indication of data format but the quality of documentation varies. It is often a good idea to read the data and then display it in the browser’s console window before proceeding.
```
$(document).ready(function() {
    $('#test').click(function() {
         $.getJSON('example.json', function(data) {
             console.log(data);
         });
         return false;
    });
});
```
In the example below the data in file example.json is an object containing an array. Each element of the array consists of another object with two name, value pairs. The data is information about how many times each page of a website has been visited.
Changing the external data file will obviously change the data but can also result in a completely different data structure and so necessitate a different way of processing the information within the webpage. In the second example the file example1.json the data is an array of objects. Each object contains an array and two name value pairs. Each object contains information about a multiple choice quiz. One of the name, value pairs contains the question, the array contains possible answers and the other name, value pair contains the information about the correct answer In this case the code
```
$(document).ready(function() {
    $('#test').click(function() {
         $.getJSON('example1.json', function(data) {
             console.log(data);
         });
         return false;
    });
});
```
results in the following output:
The next example, file example2.json, the data is an array of objects. Each object contains five name value pairs. Each object contains information about a type of job.
```
$(document).ready(function() {
    $('#test').click(function() {
        $.getJSON('example2.json', function(data) {
            console.log(data);
        });
        return false;
    });
});
```
gives the output

In order to display the data on a webpage we need to move through the data building the HTML for each item.

For the array based data we could do this with a standard for loop but jQuery also provides the very useful each method. This specifies a function to run for each matched element and can be used with both objects and arrays. Instead of operating on a jQuery collection of DOM elements, this function takes an array or map as its first parameter and a callback function as its second. The method iterates through all the data items in turn, for each item the current index and the current item are passed as parameters to the callback function, we can thus use $.each() to examine each item in turn, building an HTML structure Once the entire HTML has been built for each entry it can be inserted onto the page.
```
$(document).ready(function() {
    $('#test').click(function() {
         $.getJSON('example2.json', function(data) {
             console.log(data);
             var html = '';
             $.each(data, function(index, entry) {
                 html += '<div>';
                 html += '<h3>' + entry.title + '</h3>';
                 html += '<div class="descr">' + entry.description + '</div>';
                 html += '<div class="qualifications">';
                 html += entry.qualifications;
                 html += '</div>';
                 html += '</div>';
            });
            $('#jobInfo').html(html);
        });
        return false;
 	});
});
```
Fetching data from local xml files and files with other formats

If data is not stored in json format we cannot use the $getJSON() method. JQuery does, however, provide some other methods for handling data: $load() , $.get(),$.post() and $.ajax()

$ load is possibly the simplest to use but is also the most limited, it loads data from the server and places the returned data into the specified element on the webpage. Unlike other jQuery ajax methods which return XHR (XmlHttpRequest instance) so you can treat them as if you were working with a native XmlHttpRequest, load() returns jQuery which means that you can write
```
$("#objectID").load("test.php", { 'choices[]': ["Jon", "Susan"] } )
```
without having to worry about using a callback function
```
$("#feeds").load("feeds.php", {limit: 25}, function(){ alert("The last 25 entries in the feed have been loaded"); });
```
However due to browser security restrictions, the request can not successfully retrieve data from a different domain, subdomain, port, or protocol.
$.get() and $.post() are just helpers for $.ajax() They remove some of the advanced configuration options for the users control, setting default values for what is hidden. They are quicker and simpler to use and are usually sufficient for simple data structures.

$.get() only allows GET-requests and $.post() only allows POST (remember working with get and post when setting the action attribute for forms)

You can use $.get() and $.post() to request a simple HTML page but normally the reason for using a GET or a POST request is that you wish to pass in some parameters, which are then processed by the server, for instance with a piece of PHP code, and the result returned to the page.

In the following example we use the $.get() method to get data from an xml document.

The xml document is as follows ( Note: we will be looking at the xml format in more details in later weeks of the module)
```
<?xml version="1.0" encoding="UTF-8"?>
<breakfast_menu>
  <food>
    <name>Belgian Waffles</name>
    <price>$5.95</price>
    <description>Two of our famous Belgian Waffles with plenty of real maple syrup</description>
    <calories>650</calories>
  </food>
  <food>
    <name>Strawberry Belgian Waffles</name>
    <price>$7.95</price>
    <description>Light Belgian waffles covered with strawberries and whipped cream</description>
    <calories>900</calories>
  </food>
  <food>
    <name>Berry-Berry Belgian Waffles</name>
    <price>$8.95</price>
    <description>Light Belgian waffles covered with an assortment of fresh berries and whipped cream</description>
    <calories>900</calories>
  </food>
  <food>
    <name>French Toast</name>
    <price>$4.50</price>
    <description>Thick slices made from our homemade sourdough bread</description>
    <calories>600</calories>
  </food>
  <food>
    <name>Homestyle Breakfast</name>
    <price>$6.95</price>
    <description>Two eggs, bacon or sausage, toast, and our ever-popular hash browns</description>
    <calories>950</calories>
  </food>
</breakfast_menu>
```
( this is a sample file provided by w3schools)

In this example we use the $.get() method to get data from an xml document. The method simply fetches the file at the supplied URL and provides plain text to the callback. However, if the response is known to be XML (because of its server-supplied MIME type) the callback will be passed the XML DOM tree. JavaScript provides ways of traversing XML but jQuery provides substantial additional capabilities which can significantly simplify the process of formatting XML data
```
$(document).ready(function() {
     $('#test').click(function() { 
          $.get('simple.xml', function(data) {
               $('#info').empty();
               $(data).find('food').each(function() {
                    var $entry = $(this);
                    var html = '<div>';
                    html += '<h3>' + $entry.find('name').text();
                    html += '</h3>';
                    html += '<p>';
                    html +=  $entry.find('description').text();
                    html += '</p>';
                    html += '</div>';
                    $('#info').append($(html));
                });
          });
    });
});
```
The code example above gets the xml file when the page loads and displays the name and description tags for each of the food elements in the file. It does this by utilising jQuery’s find() method. The .find() method allows us to search through the descendants of elements in the DOM tree and construct a new jQuery object from the matching elements. Notice that we have chosen not to display all of the information from the file – just selected elements. jQuery can use arbitrary XML tag names, such as food and description, just as readily as standard HTML.
$.ajax() provides most options giving over HTTP headers and direct access to the XHR-object. $.ajax() provides powerful functionality but it can be more complicated to work with and is often unnecessary. An example $.ajax() call is shown to give you an idea of the additional options it provides. This method gives access to all of the configuration options that you would get if you were to use the basic AJAX XMLHttpRequest but is still simpler to use as it does not require you write code to cope with the different versions of XMLHttpRequest implemented by different browsers. Details of all the different parameter that can be set can be found at 🔗 https://www.w3schools.com/jquery/ajax_ajax.asp and i If you want more detail on this method you should look at the jQuery documentation at 🔗 https://www.w3schools.com/jquery/ajax_ajax.asp
```
$.ajax({
  type: 'GET',
  url: filename,
  data: data,
  async: false,
  beforeSend: function (xhr) {
    if (xhr && xhr.overrideMimeType) {
      xhr.overrideMimeType('application/json;charset=utf-8');
    }
  },
  dataType: 'json',
  success: function (data) {
    //Do stuff with the JSON data
  }
});
```
We have seen several methods that all initiate Ajax transactions. Internally, jQuery maps each of these methods onto variants of the $.ajax() global function. Rather than presuming one particular type of Ajax activity, t. $.ajax() takes a single map of over 30 settings, akes a map of options that can be used to customize its behavior A few of the special capabilities that come with using a low-level $.ajax() call include:

Preventing the browser from caching responses from the server. This can be useful if the server produces its data dynamically.

Registering separate callback functions for when the request completes successfully, with an error, or in all cases

Suppressing the global handlers (such as ones registered with $.ajaxStart()) that are normally triggered by all Ajax interactions.

Providing a user name and password for authentication with the remote host.

jQuery only used to have the callback functions for success and error and complete.

It now also support promises with the jqXHR object . This allows additional methods to be supported. These methods include .done(), .fail(), .always(), etc.... The new methods serve much the same purpose as the older callbacks. To use promises rather than callbacks to import data t you use the following syntax
```
var req = $.ajax({
    "url": "/someUrl" 
});
req.done(function() {
    //do something 
});
```
the done function takes three arguments .done(response, status, jqXHR);

response is the response from the server; typically, the JSON the server has responded to

status is a string denoting the status it is nearly always “success”.

jqXHR returns the jqXHR object.

You can currently use callbacks or promises depending on which suits your own programming style.

Note : a discussion of the difference between callbacks and promises is not part of this module but the two concepts should be familiar from other programming modules. In terms of this module it is not important which of the two styles you choose to use.
Remote data sources
Web pages use the XMLHttpRequest object to send and receive data from remote servers, however they are restricted in what they can do by the Same-origin Policy. This policy prevents malicious website from obtaining access to sensitive data on another web page through that page's Document Object Model.

Mozilla Developer Network explains (🔗 https://developer.mozilla.org/en/docs/HTTP_access_control)

Cross-site HTTP requests initiated from within scripts have been subject to well-known restrictions, for well-understood security reasons. For example, HTTP Requests made using the XMLHttpRequest object were subject to the same-origin policy. In particular, this meant that a web application using XMLHttpRequest could only make HTTP requests to the domain it was loaded from, and not to other domains. This is not practical when it comes to using third-party APIs and many workarounds exist.

One of the newer workarounds is Cross-Origin Resource Sharing (CORS), which allows the server to include a header in its response, stating that a cross-origin request is valid.

Another solution, the one commonly used today, is to use JSONP, which stands for JSON Padded.

JSONP exploits the fact that itis possible to load scripts from a different domains into your page e.g. including jQuery from a CDN. One can create a <script> tag, set the src attribute to that of a file and inject it into the page but there is no way of getting at the data it contains.

JSONP (which stands for JSON with Padding) builds on this technique and provides a way to access the returned data. It does this by getting the server to return JSON data wrapped in a function call (the “padding”) which can then be interpreted by the browser. This function is defined in the page evaluating the JSONP response.
```
$(document).ready(function(){
    var req = $.ajax({
         url: "http://api.lmiforall.org.uk/api/v1/soc/search?q=computer",
         dataType: "jsonp"
});
req.done(function(data) {
    console.log(data);
}); 
});
```
The $.getJSON() function can make use of a special placeholder character, ? as shown in the following code:
```
$(document).ready(function() {
     var url = 'http://examples.learningjquery.com/jsonp/g.php';
     $('#test').click(function() {
          $.getJSON(url + '?callback=?', function(data) {
               console.log(data);
               var html = '';
               $.each(data, function(entryIndex, entry) {
                    html += '<div class="entry">';
                    html += '<h3 class="term">' + entry.term + '</h3>';
                    html += '<div class="part">' + entry.part + '</div>';
                    html += '<div class="definition">';
                    html += entry.definition;
                    if (entry.quote) {
                        html += '<div class="quote">';
                        $.each(entry.quote, function(lineIndex, line) {
                             html += '<div class="quote-line">' + line  + '</div>';
                        });
                        if (entry.author) {
                             html += '<div class="quote-author">' + entry.author + '</div>';
                        }
                        html += '</div>';
                }
                   html += '</div>';
                   html += '</div>';
                   });
                   $('#dictionary').html(html);
            });
            return false;
       });
});
```
We normally would not be allowed to fetch JSON from a remote server (examples. learningjquery.com in this case). However, as this file is set up to provide its data in the JSONP format, we can obtain the data by appending a query string to our URL, using ? as a placeholder for the value of the callback argument. jQuery automatically registers the callback function which it calls when the request returns.

Credits : much of the material for this presentation is sourced from the book Learning jQuery, Third Edition by Jonathan Chaffer, Karl Swedberg