Transcript 1.01 - Fordham University

File System Implementation
CISC3595, Fall 09
1
Outline
File system introduction
File system implementation
Disk space allocation and management
Efficiency and Performance
Recovery
NFS






2
Objectives for a File Management
System

Meet the data management needs of the user






Provide I/O support for a variety of storage device types
Provide a standardized set of I/O interface routines to
user processes
Provide I/O support for multiple users (if needed)
Guarantee that the data in the file are valid
Minimize lost or destroyed data
Optimize performance
Requirements for a
general purpose system
1.
2.
3.
4.
5.
6.
7.
user should be able to create, delete, read, write and
modify files
user may have controlled access to other users’ files
user may control what type of accesses are allowed to
his/her files
user should be able to restructure his/her files
user should be able to move data between files
user should be able to back up and recover files in case
of damage
user should be able to access files using symbolic names
File-System Structure
File: logical storage unit, collection
of related information
File system resides on secondary
storage (disks), or tertiary storage
File system organized into layers



5
File-System Structure



Logical file system: manage metadata
information, all file system structures
except actual data (i.e., file content)
 directory structure, file-control block,
protection and security
File-Organization module:
 File allocation: map logic block addr.(2nd
block of a file) to physical block addr.
 Free space management
Basic file system


Issue commands to device driver
Manage memory buffers and caches

6

Buffer allocated before data transfer
Cache for storing meta-data
File-System Structure

7
I/O control: device drivers and
interrupt handlers
 Responsible for starting I/O
operations on a device
 Processes completion of an I/O
request
 Translate high-level requests to
low-level, hardware-specific
instructions (sent to disk
controller)
Virtual File Systems
Virtual File Systems (VFS):


same system call interface
(API) used for different types
of concrete file systems
Support numerous file
system types




8
ext2, ufs, fat, vfat, hpfs, minix,
isofs, sysv, hfs, affs, NTFS
NFS, CoDA, AFS ncpfs
Procfs, umsdos, userfs
Mount
Various file systems are mounted at different directories (mounting
points) in the files system name space

9
$ mount
$ /dev/sda3 on / type ext3 (rw,relatime,errors=remount-ro)
tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
varrun on /var/run type tmpfs (rw,nosuid,mode=0755)
varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777) udev on
/dev type tmpfs (rw,mode=0755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) fusectl on
/sys/fs/fuse/connections type fusectl (rw) lrm on /lib/modules/2.6.28-11generic/volatile type tmpfs (rw,mode=755) securityfs on /sys/kernel/security type
securityfs (rw) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc
(rw,noexec,nosuid,nodev) gvfs-fuse-daemon on /home/zhang/.gvfs type fuse.gvfsfuse-daemon (rw,nosuid,nodev,user=zhang)
Outline






File system introduction
File system implementation
Disk space allocation and management
Efficiency and Performance
Recovery
NFS
10
File system structures (metadata) on disk


Boot control block: contain info. needed for boot an OS
from the volume
Volume control block



contains volume (partition) detials: number of blocks, size of
blocks, free-block count and list, free FCB count and FCB
pointers
Directory structure (per file system): organize files
Per-file FCB: information about a file
11
File system structures in memory


Mount table: contains info. about each mounted volume
In-memory directory-structure cache: contains recent
accessed directory info.




For a directory that is a mounting point, contains flag indicating
it’s a mount point, and a pointer to an entry in mount table
System-wide open-file table: a copy of PCB for each open
file
Per-process open-file table: contains pointer to
appropriate entry in system-wide open-file table
Buffer: hold file system blocks being read from disk or
written to disk
12
Supporting file system interface:
file creation


directory I/O
Program issues system call, open(),
create(), fopen(), …
Logical files system


disk block #
allocates a new FCB
reads directory into memory, update it
with new file and its FCB, write it back

13
Call file-organization module to map
directory I/O to disk-block number
Supporting file system interface:
open a file




Program issues system call, open(), passing a file name
Logic file system (handler of open()) searches systemwide open-file table for the file, if not found, search
directory structure for the file, cache directory info, copy
file’s PCB into system-wide open-file table
In per-process open-file table, creates an entry for the file,
to store pointer to system-wide open-file table entry,
current location pointer, access mode info.
Return a pointer to the per-process open-file table, i.e.,
file descriptor in Unix, or file handler in Windows
14
Open/Read a file
15
Outline






File system introduction
File system implementation
Disk space allocation and management
Efficiency and Performance
Recovery
NFS
16
Directory:

Contains information about files



File Name
File type
File Organisation



For systems that support different organizations
Attributes, ownership
Location:




Volume: Indicates device on which file is stored
Starting Address
Size Used : Current size of the file in bytes, words, or blocks
Size Allocated : The maximum size of the file
Operations Performed
on a Directory

A directory system should support a number of
operations including:





Search
Create files
Deleting files
Listing directory
Updating directory
Directory Implementation


Linear list of file names with pointer to the data blocks.
 simple to program
 time-consuming to search
 Sorted list? Tree structure?
Hash Table – linear list with a hash table
 hash table takes a value computed from file name and
returns a pointer to the file name in a linear list
 decreases directory search time
 collisions – situations where two file names hash to the
same location
19
Outline






File system introduction
File system implementation
Disk space allocation and management
Efficiency and Performance
Recovery
NFS
20
Allocation Methods

Allocate disk space to files

Contiguous allocation

Linked allocation

Indexed allocation
21
Contiguous Allocation


Each file occupies a set of contiguous
blocks on the disk
Pros:



Simple – only starting location (block #)
and length (number of blocks) are
required
Random access
Cons:



22
Wasteful of space: dynamic storageallocation problem: how to satisfy
request from list of non-contiguous free
holes
External fragmentation
Files cannot grow
Linked Allocation



Each file is a linked list of disk
blocks: blocks may be scattered
anywhere on the disk.
Directory contains pointer to the
first and last blocks of the file.
Pros:


Solve problems with contiguous
allocation
Cons:


Inefficient direct access
Reliability issues
23
File-Allocation Table

File-allocation table (FAT) – disk-space allocation
used by MS-DOS and OS/2.
FAT: for each volume
 one entry for each disk block
Pros?

Cons?


24
Indexed Allocation


Brings all pointers (block #s) together into index
block.
Logical view.
index table

25
Each file has its own index block
Indexed Allocation
26
Indexed Allocation: Multi-level indexing

outer-index
index table
27
file
Combined Scheme: UNIX inode (4K bytes per block)
28
Free-Space Management

Bit vector (n blocks)
0 1
2
n-1
bit[i] =


…
0  block[i] free
1  block[i] occupied
First free Block number:

(number of bits per word) *(number of 0-value words) +
offset of first 1 bit
Easy to get contiguous free blocks

Bit map requires extra space


29

E.g., block size = 212 bytes, disk size = 230 bytes (1 gigabyte)
n = 230/212 = 218 bits (or 32K bytes)
Free-Space Management (Cont.)

Linked list (free list)



Grouping



Cannot get contiguous space easily
No waste of space
First free block contains address of n free
blocks
The n-th block therein contains address
of another n free blocks, …
Counting


30
Free blocks might be contiguous
Keep starting block # and length
Outline






File system introduction
File system implementation
Disk space allocation and management
Efficiency and Performance
Recovery
NFS
31
Efficiency and Performance

Efficiency dependent on:



disk allocation and directory algorithms
types of data kept in file’s directory entry
Performance



32
disk cache – separate section of main memory for frequently
used blocks
free-behind and read-ahead – techniques to optimize
sequential access
improve PC performance by dedicating section of memory as
virtual disk, or RAM disk
Page Cache

A page cache caches pages rather than disk blocks using
virtual memory techniques

Memory-mapped I/O uses a page cache

Routine I/O through the file system uses the buffer (disk)
cache

This leads to the following figure
33
I/O Without a Unified Buffer Cache
34
Outline






File system introduction
File system implementation
Disk space allocation and management
Efficiency and Performance
Recovery
NFS
35
Recovery

Consistency checking – compares data in directory
structure with data blocks on disk, and tries to fix
inconsistencies

Use system programs to back up data from disk to
another storage device (floppy disk, magnetic tape, other
magnetic disk, optical)

Recover lost file or disk by restoring data from backup
36
Log Structured File Systems


Log structured (or journaling) file systems record
each update to file system as a transaction
All transactions are written to a log



Transactions in the log are asynchronously written to
file system


37
A transaction is considered committed once it is
written to log
However, file system may not yet be updated
When file system is modified, the transaction is removed
from log
If file system crashes, all remaining transactions in log
must still be performed
Outline






File system introduction
File system implementation
Disk space allocation and management
Efficiency and Performance
Recovery
NFS
38
The Sun Network File System (NFS)

An implementation and a specification of a software
system for accessing remote files across LANs (or
WANs)

The implementation is part of the Solaris and SunOS
operating systems running on Sun workstations using an
unreliable datagram protocol (UDP/IP protocol and
Ethernet
39
NFS (Cont.)


Interconnected workstations viewed as a set of independent
machines with independent file systems, which allows sharing
among these file systems in a transparent manner
A remote directory is mounted over a local file system
directory


Mounted directory looks like an integral subtree of local file system,
replacing the subtree descending from the local directory
Specification of remote directory for mount operation is
nontransparent: host name of remote directory has to be provided


Files in the remote directory can then be accessed in a transparent
manner
Subject to access-rights accreditation, potentially any file
system (or directory within a file system), can be mounted
remotely on top of any local directory
40
NFS (Cont.)



NFS is designed to operate in a heterogeneous
environment of different machines, operating systems, and
network architectures; the NFS specifications
independent of these media
This independence is achieved through the use of RPC
primitives built on top of an External Data
Representation (XDR) protocol used between two
implementation-independent interfaces
NFS specification distinguishes between the services
provided by a mount mechanism and the actual remotefile-access services
41
NFS Protocol

Provides a set of remote procedure calls for remote file
operations. The procedures support the following operations:








searching for a file within a directory
reading a set of directory entries
manipulating links and directories
accessing file attributes
reading and writing files
NFS servers are stateless; each request has to provide a full
set of arguments
(NFS V4 is just coming available – very different, stateful)
Modified data must be committed to the server’s disk before
results are returned to the client (lose advantages of caching)
The NFS protocol does not provide concurrency-control
mechanisms
42
Three Major Layers of NFS Architecture


UNIX file-system interface (based on the open, read,
write, and close calls, and file descriptors)
Virtual File System (VFS) layer – distinguishes local files
from remote ones, and local files are further distinguished
according to their file-system types



The VFS activates file-system-specific operations to handle local
requests according to their file-system types
Calls the NFS protocol procedures for remote requests
NFS service layer – bottom layer of the architecture

43
Implements the NFS protocol
Schematic View of NFS Architecture
44
NFS Path-Name Translation

Performed by breaking the path into component names
and performing a separate NFS lookup call for every pair
of component name and directory vnode

To make lookup faster, a directory name lookup cache on
the client’s side holds the vnodes for remote directory
names
45
NFS Remote Operations



Nearly one-to-one correspondence between regular UNIX
system calls and the NFS protocol RPCs (except opening and
closing files)
NFS adheres to the remote-service paradigm, but employs
buffering and caching techniques for the sake of performance
File-blocks cache – when a file is opened, the kernel checks
with the remote server whether to fetch or revalidate the
cached attributes



Cached file blocks are used only if the corresponding cached
attributes are up to date
File-attribute cache – the attribute cache is updated whenever
new attributes arrive from the server
Clients do not free delayed-write blocks until the server
confirms that the data have been written to disk
46