2.1 Operating Systems

2.2 Operating System Kernel

2.2.1 Overview

The central core of the operating system is the kernel which is always memory resident after booting. The primary function of the kernel is resource management. This is a trusted program (or large chunk of code) which controls and manages the hardware of the computer.

The relationship between the kernel, as a component of the overall operating system, is illustrated in Figure 1, and is itself illustrated in Figure 2.

The kernel contains, amongst other things:

the bodies of the system calls

code to implement resource management and any supporting kernel data-structures

low-level device driver code to control hardware

provided with the hardware

Processes are loaded at run time and may occupy any memory area allocated by the kernel.

2.2.2 System Calls

Providing a system call interface offers a distinct advantage to the alternative of programming the control of the hardware directly:

user programs and system utilities are smaller and simpler e.g. a program to copy files opens, reads, writes and closes files through system calls the bodies of which are in kernel device drivers; it does not have to know how the files are implemented or how the disk hardware works.

it also enhances secure access to the device in a controlled manner as only the kernel device driver can directly access the device

your program cannot execute I/O instructions directly

Application programs do not usually contain direct system calls.

instead they use library calls – the C Standard Library (implemented on Linux as the GNU C library) - and this library calls system calls

not only is this easier for the application programmer but it allows additional utility functionality to be placed around the system call

For example for file I/O the standard library adds buffering.

the file I/O system calls are unbuffered but are used in the implementation of buffered I/O code in the library

See also the discussion of user-mode and kernel mode (see below).

What’s the current version of the kernel

uname -r

What are the system calls available in this kernel?

man 2 syscalls

Tell me about the fork system call

man 2 fork

Table 1: System calls (from Wikipedia)

Process Control. load execute create process terminate process get/set process attributes wait for time, wait event, signal event allocate, free memory	Device management. request device, release device read, write, reposition get/set file attributes logically attach or detach devices
File management. create file, delete file open, close read, write, reposition get/set file attributes	Information maintenance get/set time or date get/set system data get/set file or device attributes
Communication create, delete communication connection. send receive messages. transfer status information. attach, detach remote devices.

2.3 Kernel code

The kernel will be coded largely in C.

a specialised task

The reasons for this are several:-

speed

memory efficiency

explicit control of memory allocation and de-allocation

many low-level features for manipulating memory cell contents, bits and bytes

However some of the kernel is coded in assembler.

very low level control

basically mnemonic operations which map directly to machine instructions and operands

so as low-level as you can get...

very fast as hand crafted

2.4 Other kernel-relevant concepts

2.4.1 Interrupts

An electrical signal which grabs the attention of the processor to run some other code – an interrupt handler.

An interrupt suspends the normal sequence of execution.

When the interrupt processing is completed, execution resumes at the next instruction

Note when an interrupt happens the executing program continues the instruction cycle currently executing and then runs the interrupt handler.

instructions are therefore atomic

Interrupts happen asynchronously.

programs cannot predict when they will happen at run time.

An example of interrupts in practice:-

a program asks for some data from a disk

the processor does other processing

the disk announces eventually via an interrupt that the I/O has completed

the interrupt handler deals with the data just read in

2.4.2 Timer Interrupts

Clock pulses drive processor instruction cycle.

Also need a clock at a lower frequency which continuously generates timer interrupts.

typically every 10ms (but can be set to less)

internal clock drives timer interrupts

otherwise work in same way as other interrupts

Uses:-

wake up operating system to do things e.g. scheduling another process onto the CPU

timers in application programs

updating internal clock

etc.

2.5 The System Call Interface

There are three fundamental considerations to consider in operating system operation:

How does the kernel regain the CPU in order to execute essential housekeeping operations (e.g. to switch execution between processes)?

The protection problem: how to stop one process accessing another process's address space (or the kernel address space) thereby corrupting it or breaching security.

How to control access to I/O devices or other sensitive instructions or registers in the CPU (e.g. used to implement 2. above).

2.6 User Mode and Kernel Mode

The process executes in one of two modes: user-mode or kernel-mode.

when a process executes its own code it operates in user-mode;

when it executes kernel code it operates in kernel-mode

When executing in kernel-mode the kernel is able to execute privileged instructions and has access to privileged data.

a process executing in user-mode is prohibited from executing these instructions, in particular I/O instructions, and can access data only within its own address space

therefore in kernel-mode the process runs with elevated security

All modern OSs use this concept.

and it requires additional hardware functionality which is supported on modern mainstream processors e.g. x86/x64

A process is switched to kernel-mode when one of the following happens:-

it calls on the services of the operating system via the system call interface

an interrupt (timer or I/O completion interrupts or any other type) is generated during user-mode execution

The supporting CPU is switched into an analogous privileged state (or “ring”).

only in this privileged state can the CPU execute I/O instructions and access and update certain registers

once in kernel-mode the appropriate code is executed and the kernel also has an opportunity to do housekeeping (e.g. to schedule another process onto the CPU).

In Intel x86/x64 user mode code runs in Ring 3.

In Intel x86/x64 kernel mode code runs in Ring 0.

The process and CPU now switch back to user-mode when:-

the operating system has finished servicing the system call made by the user

the interrupt which interrupted user-mode execution has been serviced

In summary I/O instructions are only available in kernel-mode and the code that uses them is in the kernel.

kernel data-structures and device-drivers are only accessed through system calls. Their accessibility is secure.

kernel code periodically regains control of system through system calls and interrupts and can undertake process scheduling, memory management...

2.7 Multitasking or Process Scheduling

A CPU core can only execute a single process or thread at any instant in time.

but there are many processes/threads running that look like they’re all running at the same time.

but this is merely an illusion achieved by rapidly switching between them

Recall the kernel can regain control through system calls and interrupts (not exclusively timer interrupts).

this allows the kernel scheduler to wake up and schedule another process/thread onto the CPU.

Each process gets a fixed length time slice (typically 10ms) on the CPU if it does not make a system call in the interim.

a timer interrupt then occurs and the process is kicked off the CPU for another to take its place

this is called a pre-emptive scheduling policy

saving one process’s register values and placing another’s into the CPU for execution to continue is called a context switch and is an overhead

I/O is slow so executing a system call initiating I/O means another process/thread is scheduled when the first waits for I/O completion

2.8 Virtual Memory

Recall to each process it looks like it has the entire computer to itself.

Virtual memory is a concept in which each process has an address space the same size as that addressable by the CPU.

64 bits => 264 bytes => max process size ~1.8 * 1019 bytes!!

obviously not possible to accommodate in main memory (and there are 100s of processes at any one time)...

Split main memory and process into equal sized pages (4KB typical).

only a subset of pages necessary for immediate execution

rest can be kept on disk

However we should accommodate all of all processes in main memory if possible.

2.9 Inter-Process Communication (IPC)

Processes are sandboxed and cannot access memory in other processes. IPC mechanisms exist to allow processes to communicate. Include:-

pipes: streamed, unidirectional, large volume communication with synchronisation

sockets: TCP/UDP/IP communication between process across network or on same computer

shared memory: connect a designated memory area to multiple processes on same computer with no synchronised access provided to the shared memory

message passing: structured and persistent small messages including one-to-many communication, maybe across a network, decoupled, asynchronous

etc. etc.

2.10 Concurrency Control

Process scheduling is pre-emptive and driven by interrupts.

when interrupts occur in terms of exact execution point in the code cannot be predicted

that is they happen asynchronously

If you have variables shared between processes/threads then you can get the wrong results caused by executions interleaving unpredictably.

this is called a race condition

this is out with your control

this can easily happen with threads or with processes using shared memory

the solution is to ensure that only one process or thread can read and write the same shared variables at the same time

The kernel will provide a locking mechanism to enforce mutually exclusive access to shared variables:-

in C programming this is usually semaphores but other locking options may be available

supported by special system calls

you have to code this and it is advanced programming – be warned!

2.11 Buffers

A buffer is a an area of main memory used for the temporary storage of data when a program or I/O device needs an uninterrupted flow of data.

used for temporary storage as data is moved to-from I/O devices e.g. a disk

this is needed because the processor and main memory generate data faster than the disk can process

when writing: the processor fills a buffer with a block, the disk writes it while a second buffer is filled with another block

when reading: the disk fills a buffer with a block, the processor uses it while a second buffer is filled with another block

so in general buffers are used when connecting slow and high speed devices

but also have many other uses...

2.12 Blocks

Memory, disks and the network all share a common concept: no matter what the data’s called it’s split into equal sized chunks.

this allows both buffer sizes to be predictable but also storage to be used more efficiently (as the chunks are smaller and equal sized).

Files are split into equal sized blocks which are randomly scattered round the disk.

Process images are split into pages which are randomly scattered round main memory.

Internet TCP messages are split into packets which can be independently routed to the destination across the network.

in fact a lot more to this and various names for the networked chunks are used...

2.13 Portable Operating System Interface (POSIX)

some UNIX flavours are POSIX certified

Linux is mostly POSIX compliant...

There is a long history of shell implementations:-

the Bourne shell was extended in functionality to become the Korn Shell

POSIX used the Korn Shell to define the POSIX Shell standard

Linux implements the POSIX shell standard (nearly) as dash (linked to sh)

bash (“Bourne again shell”) is basically a superset of the POSIX shell adding some better and best-of features from a few other shells developed over the years.

this is why there are multiple syntaxes for the same features

good example are boolean conditions:-

[ ] is in POSIX but [[ ]] is bash only

both work in bash and have subtly different features

for you- use bash and any supported syntax to get the job done is OK

2.14 X Windows and Desktops

The graphical system on UNIX/Linux is called the X Window System

this is optional and may not be installed or operational

in which case you will only have a text-only terminal available

often called X11 R7 as this is the now established current version

low-level graphical toolkit

X Windows has a distributed architecture.

XProtocol communicates over TCP/IP

this allows graphical output to be viewed on a remote computer

on the Linux box the application is called a XClient

on the remote computer the display is called a XServer

on MS Windows you need a separate XServer application (e.g. Xming) to display the output

we are not doing this....

alternatively MS Remote Desktop (RDP) is sometimes supported

The X.Org project provides an open source standardised implementation of the X Window System.

🔗 http://x.org/wiki/

Standard desktops provided today usually implemented on top of X Windows.

offering user utilities and a standard look and feel

some are built on the GTK+ widget toolkit which is itself built on X11 R7

Xfce (which we are using as it is lightweight), Gnome, KDE, Cinnamon, Ubuntu Unity...

2.15 Secure Shell (SSH)

There have been various programs/protocols over the years to provide remote login – the current common one is SSH.

SSH provides an authenticated and encrypted connection over TCP/IP.

need sshd installed and running at server

need a SSH client e.g. PuTTY on Windows, on client

secure password transmission

after logging in can optionally use an XServer on the client for graphical output

but often no graphics, just pseudo terminal and shell

this is what Linux VMs in the cloud usually offer

2.16 Text Files and Binary Files

A text file is a file which contains printable text data only and can be viewed/edited directly in a text editor.

when saved to a file C source code, shell scripts, Java source code, HTML and SQL queries are all examples of text files

gedit, vi and Eclipse are all text editors (or contain text editors as part of their constituent tools)

the text file character set may be a choice of one from many

file –i <file> will indicate if it’s a text or binary file and its character set

ASCII and UTF-8 give maximum compatibility between architectures

Text files are portable.

between Windows and Linux

but in Windows newline is <CR> + <LF> (0x0D0A)

on Linux it’s <LF> (0x0A)

another thing to handle as the <CR>s need stripped out when transferred across, where <CR> = ‘\r’

A binary file is a file which contains non-text data and cannot be viewed sensibly (or at all) in a text editor.

many different types…

a text editor would display rubbish as it would try to display the data as text data which would give meaningless character codes

if we were to write a floating point number example to a file, 01010000 00000011, i.e 0x50 0x03 and read it in a text editor, you would get “P?”, which is obviously meaningless and not what was intended…

a specialist editor can be used to edit and display binary files dependent on what they are e.g. gimp for “.jpg” files

binary data is not normally intended for direct human usage

executable code generated by a compiler is another example

or a “.jpg” or “.mp3” file…

od and hexdump will display a binary file in a readable (?) form

2.17 Comparing

To compare text files:

diff file1.txt file2.txt

diff -e file1.txt file2.txt

produces output which can be run through sed to recreate the other file

also used by some software versioning systems

Stream editor:

sed is a command which applies defined editor commands to a file without starting an interactive editor

these commands are based on an old (but well known) editor called ed and are also accessible from the vi editor

To compare binary files:

cmp executable1 executable2

although you could use cmp on text files as well...

2.18 Numbers as Text - Bloat

“123353.3352”

this is 11 text characters (UTF-8 characters)

11 bytes (and this will vary with the length of the number)

a computer cannot do arithmetic with it (at least directly)

it could be in a text file

you could send it over the Internet and anything will understand it

123353.3352

this is a float

4 bytes (and this will be fixed but float has a maximum range and precision)

a computer can do arithmetic with it

what is in the 4 bytes bears no resemblance to the text representation at the top of the slide

in this form a text editor would display it as rubbish

you could send it over the Internet but in this form if the receiver had a different architecture it wouldn’t understand it

2.19 Set up scripts

Start with a . in your home directory ~

won’t see these unless you do: ls -a

Important ones:-

.bashrc: user’s own set up file

run every time a new terminal window is opened

set what you want e.g. alias, PATH etc.

sample file (very simple) provided for you in lab image

.bash_profile: user’s own setup file on login

our environment is not set up to run this

same format as .bashrc

.vimrc: run when vi, vim and gvim are started setting editor preferences

sample file (very simple one liner) provided for you in lab image switching on line numbers