-
Introduction
- The literature review is a vital part of any research project, and the ability to carry out a literature review is a key academic writing skill. This week we will try to answer the following questions:
- What is the purpose of your literature review?
- What consequences does this have for your reading process?
- What consequences does this have for the structure of your literature review?
- What is the process of compiling a literature review?
Note that there is audio commentary throughout this week’s presentation, which you should listen to as go read through it
Additional Audio (00:00:07)
-
Purpose of the literature review
Converting your idea into a practical project involves developing the idea into a project proposal which is driven by a statement of the problem to be solved or question to be answered. You may start off with several ideas for projects which may address different topic areas, in which case you need to develop your understanding of these areas in order to select which one is the most promising as a basis for a project.
In any case you need to refine an initial embryonic statement of an idea into an appropriate problem statement and to consider possible approaches to solving that problem or answering questions associated with it. An initial literature review is an important tool that will help you achieve this.
Then, at a later stage, while undertaking the project itself, you will need to conduct a more in-depth literature review that will underpin the full implementation of the project. In this module you will conduct only the initial literature review which will contribute to the project proposal.
How can the literature review help you define your topic further?
- The literature review will help you to:
- Gain a better understanding of the ‘ingredients of your topic’
- See what has been done before
- Identify possible questions
This has implications for how you read information
Additional Audio (00:01:29)
-
What is "the literature"?
"The Literature" refers to the collection of scholarly writings that represent current thinking or research on a topic. In academic research, this refers primarily to articles published in peer-reviewed scholarly journals. However, there can be a range of other sources which can contribute to the foundations for research. The nature of these can vary quite widely between disciplines. Here we will look at the types of source that are likely to be useful in research in Computing.
"The Literature" refers to the collection of scholarly writings that represent current thinking or research on a topic. In academic research, this refers primarily to articles published in peer-reviewed scholarly journals. However, there can be a range of other sources which can contribute to the foundations for research. The nature of these can vary quite widely between disciplines. Here we will look at the types of source that are likely to be useful in research in Computing.
What is peer review
Sources that are peer-reviewed are generally held to be more authoritative than other sources, so what does this mean, and how do you know if a source is peer-reviewed?
Most scholarly journals use peer-review. Before an article, or paper, is accepted for publication, it must be reviewed by a number of experts in the subject matter. These experts are independent of the journal and are usually academics who volunteer to review as an important contribution to the academic process. Most academics engage in the peer-review process both as authors and as reviewers.
Articles are usually "blind-reviewed", which means the reviewer doesn't know who authored the article. This avoids reviews being influenced by bias towards (or against) particular colleagues.
Reviewers look for markers of academic quality: has the research been carried out rigorously; does the paper describe this clearly and in sufficient detail; are the conclusions justified, and so on. If the consensus of the reviewers is that an article is of an acceptable standard then it may be accepted for publication. On the other hand, an article may be rejected on the basis of the reviews, or require substantial changes.
-
Types of source: peer reviewed
- Journals
- Publish scholarly articles on current research.
- Almost always peer-reviewed.
- Usually focus on a specific area within the discipline, e.g. ACM Transactions on Software Engineering and Methodology, IEEE Computer Graphics and Applications
- Written for researchers, can be difficult to read if you are not familiar with the subject area
- Some journals specialised in high-quality surveys of the literature in specific areas, e.g. ACM Computing Surveys, which can be a useful introduction to that area
- Conferences/Workshops
- Researchers often present their work initially at conferences, then go on to write journal articles
- May be published as papers in the conference proceedings
- Research is often more up-to-date than journals as the process is quicker
- May not be peer-reviewed, or less rigorously peer-reviewed than journals (but see comment below about conferences in Computing)
If you find an interesting conference paper, and want to know if it is peer-reviewed (to help you judge its quality) you can often go to the website of an upcoming edition of the same conference and look at the instructions provided for authors and reviewers to get an indication of the process that is used. You might also be able to find information on a conference's acceptance rate – a low rate, based on peer-review, can indicate a high quality threshold
Conferences are very important in Computing research, and conference papers are often peer-reviewed and considered as high quality sources, much more so than in many disciplines. Many ACM and IEEE conferences, for example, carry out a very rigorous peer-review process before accepting papers, and reject a high proportion of submissions.
-
Types of source: non-peer reviewed
- Books
- Can be useful introductory sources, usually written to educate the reader
- Explain a field and the relevant approaches and theories
- In Computing, often explain a particular technology
- Can be of high quality and are usually professionally proof-read and reviewed for accuracy (which is not the same as peer-review)
- May take a long time to write and publish, so can become out of date quickly
- Some books (monogrammes) are essentially collections of research articles written for academics, which are considered to be of similar standing to journals
- Dissertations
- Some universities make their students Masters and PhD dissertations freely available. A PhD in particular represents an in-depth study of a research topic, which has been assessed in order to award a degree.
- This is not the same as peer-review, however. Generally the work presented in a PhD dissertation will be published also in one or more journal or conference papers, and it is generally better to cite the peer-reviewed work than the dissertation. Having said that, one of the most famous and highly-cited works in Computing is a PhD 🔗 dissertation by Roy Fielding, on RESTful software architectures.
- Manuals or online documentation
- Usually essential in conducting development projects, but not generally cited in a literature review
- Can be relevant to a literature review, though, if they contain information about the capabilities of technologies that would help you judge whether they form a basis for implementing your project.
- Company websites, white papers
- Usually created to promote products or technologies
- A technology white paper is an authoritative guide or report that explains the benefits of a particular technology or product or policy
- Clearly biased in favour of those products, so should be treated with caution
- Clearly biased in favour of those products, so should be treated with caution
- Reports
- May be produced by consultants or government
- e.g. market surveys
- Professionally produced but not subject to peer-review, so may be written to promote a point of view
- May contain useful information but should be treated with caution
- May be hard to obtain
- Industry magazines, websites, blogs
- There are many magazines, often online, which carry articles relating to specific areas or technologies, or to a broad range of areas which are organised into specific topics
- e.g. Database Trends, MSDN Magazine
- Typically written to inform and educate industry professionals
- Articles may be written to promote specific technologies
- Usually easy to obtain and can be useful, but usually as a starting point for more in-depth investigation
- Blogs can be useful and potentially citeable, or may not be useful at all, in the context of a literature review at least. It depends on the author – the musings of well-regarded industry figures can be important, even though they are expressing personal views with no external scrutiny
Oates also describes a range of more general sources of information that you might consider using
-
Finding and selecting sources to cite
Use the GCU Library!
It is important to take advantage of the benefits that the Library provides for students. It provides full-text access free of charge to papers from a wide range of publishers. You can independently search databases or Google Scholar, and find useful-looking sources, but you will probably only be able to access the abstracts of these, unless you pay for access to the full text. The Library has subscriptions to many sources which allow you to access the full text of papers without needing to pay. You should complete the Week 1 & 2 lab exercises, which guide you through the GCU Library resources for searching online databases and organising your sources.
What should I cite?
Your literature reviews, in the proposal you will write in this module in in your Honours project, are research-focused academic work. The findings of your reviews should have a firm foundation in current thinking related to your topic as evidenced by reference to peer-reviewed research, so you should aim to include a substantial body of journal papers and high quality conference papers.
However, many important developments take place in the industry rather than in academic research. For example, the technologies now widely used for Big Data emerged from the needs of large internet companies such as Google, Amazon and Facebook (although sometimes companies publish papers in journals, such as Google's 🔗 famous paper on their Big Table system, which was published at a conference and in the journal ACM Transactions on Computer Systems). There are also a number of industry-based consultants who are influential within the discipline. Depending on your topic, you may need to consider a range of different types of source.
- When you consider whether to use a source, you should evaluate its quality. If a source is not peer-reviewed research, you should ask yourself questions such as:
- What contribution does it make to the findings of my review?
- Who is the author (or authors) and what is their standing within the discipline?
- What was the author's purpose in writing the material?
- Is the source promoting a point of view or product?
What if I can't find peer-reviewed sources?
- For some topics, particularly when they relate to the use of specific technologies, it can be quite difficult to find peer-reviewed research that seems really relevant. Sometimes you need to think more widely about your topic. Instead of looking for literature that is about exactly your topic, you can search for specific aspects of your topic. For example:
- Rather than focusing on the technology, search for literature on the problem you are considering solving with the technology – this might tell you about other approaches that have been used to solve this problem or similar ones, and should help you understand the problem area
- Rather than focusing on a specific technology product that you are thinking about using, search for literature on the general type of technology, e.g. search for "NoSQL database" rather than a specific database product. You can also focus other technologies that might be used in similar applications. This might help you make and justify choices of technology for your project
-
Reading academic literature – looking for the main points
There are two short extracts from academic papers below, which you can read as practice in beginning to read academically. These extracts are from papers about Usability, but the way you should read them is the same for papers on any topic.
Listen to the first audio clip before reading the extracts. Note that these extracts are referred to as “slide 5” and “slide 6”s in the audio.
Additional Audio (00:01:12)
Extract 1 (Slide 5)
Abstract: Usability is critical to the development of a user-friendly digital textbook platform interface, yet thorough research on interface development based on usability principles is in short supply. This study addresses that need by looking at usability attributes and corresponding design elements from a learning perspective. The researchers used a student survey, log files, and an expert heuristic evaluation to analyse needs for revision in the user interface of the existing Korea Education Research Information Service digital textbook platform. After using suggestions derived from this analysis to develop a new platform prototype, they tested its user interface for usability through a cognitive walk through and a formative evaluation. The results show that the usability design elements identified through the use of this iterative design and evaluation model were essential to improving the usability of the user interface and thus facilitating users’ actions and learning processes.
(Lim, Song and Lee. 2012)Extract 2 (Slide 6)
Usability is the extent to which a computer system enables users, in a given context of use, to achieve specified goals effectively and efficiently while promoting feelings of satisfaction.1 Usability evaluation (UE) consists of methodologies for measuring the usability aspects of a system’s user interface (UI) and identifying specific problems [Dix et al. 1998; Nielsen 1993]. Usability evaluation is an important part of the overall user interface design process, which consists of iterative cycles of designing, prototyping, and evaluating [Dix et al. 1998; Nielsen 1993]. 1Adapted from ISO9241 [International Standards Organization 1999].
(from: Ivory and Hurst. 2001)Now listen to the next audio clip
Additional Audio (00:00:45)
Good Reading Habits
- Don’t (always) read for detail – at least not immediately. Try to understand the main idea before you progress.
- Select relevant information; discard the rest
- Use things you have read to identify other relevant texts.
- Think about topics, not individual papers. This is also important for note-taking!!
Additional Audio (00:02:05)
The references for these extracts are:
IVORY, Melody I. and HEARST, Mary A. 2001. The State of the Art in Automating Usability Evaluation of User Interfaces. ACM Computing Services. 33 (4): 470-516
LIM, Cheolil, SONG, Hae-Deok, LEE, Yekyung. 2012. Improving the usability of the user interface for a digital textbook platform for elementary-school students. Education Research Tech Dev. 60: 159-73
-
Structure
A literature review is organised by topics. Unlike an annotated bibliography, which is another, less common, way of presenting findings from reading the literature, it is NOT organised around specific papers. Note that a literature review relates the information from your reading to your topic. Simply describing what you read is NOT enough.
Additional Audio (00:00:27)
Annotated Bibliography
Here is an example of an annotated bibliography. Read through this and listen to the audio clip.
A. Avizienis and J.-C. Laprie, Dependable Computing: From Concepts to Design Diversity, Proc. of the IEEE 74,5 (May 1986), 629-638.
An invited paper in a special issue on fault tolerance in VLSI. The paper defines the basic terms of fault tolerance and provides preliminary,high-level taxonomy of dependability. Several appraoches of the problem of fault tolerance are considered, with the emphasis placed, as would be expected from Avizienis, on design diversity as a means of tolerating design faults.
F. Christian, Understanding Fault-Tolerant Distributed Systems, Comm. ACM 34,2 (February 1991), 56-78.
A comprehensive paper containing both concepts and case studies in fault-tolerant distributed computing. The paper presents a unified discussion of the problems to be solved at the hardware and software architecture levels as well as approaches to their solution based on both avoidance (masking) and tolerance (recovery). It is suggested in the paper that the key to dependability is balanced failure detection, recovery, and masking redundancy at various levels of system abstraction, so that the provision of dependability at lower levels will make the higher levels easier to design.
GANTENBEIN, Rex E. 1992. An annotated bibliography of dependable distributed computing. SIGOPS Oper. Syst. Rev. 26 (2): 60-81
Additional Audio (00:00:16)
-
Literature Review
As a contrast to the annotated bibliography, here you will look at two extracts from a literature review, from a paper on distributed systems, although the topic is not important for our purposes here.
Read the extracts, guided by the audio clips. Note that the extracts are referred to as “slides” in the audio. Each extract appears twice, with key points, as described in the audio, highlighted the second time.
Additional Audio (00:00:28)
An important problem in distributed system design and analysis is to define and evaluate various reliability measures, as well as, to efficiently estimate the effect of program and resource distributions on the reliability of a system. This analysis is crucially important for building a reliable distributed computing system.
Many researchers have studied distributed program reliability (DPR) and distributed system reliability (DSR). Kumar et al. (1986) appears to be the first to present a definition for DPR and DSR. They constructed a distributed model including edges, nodes, and resource files and proposed the Minimal File Spanning Trees (MFST)-based algorithm to evaluate the DPR and DSR. Later, based on MFST, Raghavendra et al. (1988) addressed two measures, distributed program-user reliability and distributed system-user reliability, and proposed an algorithm for their evaluations.
An important problem in distributed system design and analysis is to define and evaluate various reliability measures, as well as, to efficiently estimate the effect of program and resource distributions on the reliability of a system. This analysis is crucially important for building a reliable distributed computing system.
Many researchers have studied distributed program reliability (DPR) and distributed system reliability (DSR). Kumar et al. (1986) appears to be the first to present a definition for DPR and DSR. They constructed a distributed model including edges, nodes, and resource files and proposed the Minimal File Spanning Trees (MFST)-based algorithm to evaluate the DPR and DSR. Later, based on MFST, Raghavendra et al. (1988) addressed two measures, distributed program-user reliability and distributed system-user reliability, and proposed an algorithm for their evaluations.
Additional Audio (00:00:55)
Now go on to read the second extract, again guided by the audio clips.
Additional Audio (00:00:19)
A fast algorithm to evaluate the DPR and DSR was developed by Kumar et al. (1988). These methods are 2-step algorithms. The first step is finding all of the MFSTs. The second step is converting these MFSTs into a symbolic reliability expression using an existing reliability evaluation algorithm like SYREL (Hariri and Ragavendra (1987)to compute the disjoint probability. The major drawback with these methods is that finding all of the MFSTs creates high computational complexity. Prior knowledge of multi- terminal connections is required in order to compute the reliability expression, thereby making these methods inapplicable for large systems. To overcome these problems, Kumar et al. (1991) proposed a 1-step algorithm, GEAR, which can avoid computing the redundant MFSTs and reduce computational time.
From: Chang, Huang and Kuo. 2010
A fast algorithm to evaluate the DPR and DSR was developed by Kumar et al. (1988). These methods are 2-step algorithms. The first step is finding all of the MFSTs. The second step is converting these MFSTs into a symbolic reliability expression using an existing reliability evaluation algorithm like SYREL (Hariri and Ragavendra (1987) to compute the disjoint probability. The major drawback with these methods is that finding all of the MFSTs creates high computational complexity. Prior knowledge of multi- terminal connections is required in order to compute the reliability expression, thereby making these methods inapplicable for large systems. To overcome these problems, Kumar et al. (1991) proposed a 1-step algorithm, GEAR, which can avoid computing the redundant MFSTs and reduce computational time.
From: Chang, Huang and Kuo. 2010
Additional Audio (00:00:38)
-
Process of Writing a Literature Review
- Make a list of the things you need to know about for your project idea.
- Add these as headings to an empty file
- Search for appropriate sources of information and formulate questions before you read them: what do you want to find out from them
- Take notes on a source, then go over them and add relevant information into the right section of your document (ref!!)
- Go over your notes and revise your project idea on a regular basis
Additional Audio (00:01:29)
Useful questions
- What “questions” is ILR trying to answer?
- What is the wider project topic context?
- Why is that topic worth investigating?
- Why is it “important”?
- Are there any key “problems” in this project topic area?
- What different solution approaches have been proposed/suggested?
- Which particular aspect of the project topic are you going to investigate, how does that relate to this context, and why is your particular project useful?
Why do we reference?
- Respect for another person’s work (similar to copyright): you acknowledge that ideas, words, diagrams, photos etc. belong to them
- Transparency: you allow your reader to look up where you found some of the ideas you use
How do we reference?
We can illustrate this by comparing references to information in hotel guides, as explained in the audio clip.
Additional Audio (00:03:07)
In-Text Citation
Cotterell (2001) suggests that teaching learning support should combine form and content.
Bibliography
Cottrell, S. (2001) Teaching study skills and supporting learning. London; Palgrave
Additional Audio (00:01:18)
Useful link for Refworks: 🔗 http://www.gcu.ac.uk/library/onlineresources/refworks/
Source -Europe Cities. 2008. Hotels in Glasgow. [online] Available from: 🔗 http://www.europe-cities.com/en/790/uk_scotland/glasgow/ 14787_city_centre/hotels/. Accessed 10.10.08