Transcript High level design of the DeFiler

DeFiler interfaces
create, destroy, read, write a dfile
list dfiles, sync()
DFS
read(), write()
startFetch(), startPush()
waitValid(), waitClean()
isBusy()
DBuffer dbuf = getBlock(blockID)
releaseBlock(dbuf)
sync()
DBufferCache
DBuffer
startRequest(dbuf, r/w)
VirtualDisk
ioComplete()
blockID = getBlockID()
buf[] = getBuffer()
DeFiler interfaces: overview
create, destroy, read, write a dfile
list dfiles
DFS
DBuffer dbuf = getBlock(blockID)
releaseBlock(dbuf)
DBufferCache
read(), write()
startFetch(), startPush()
waitValid(), waitClean()
DBuffer
ioComplete()
startRequest(dbuf, r/w)
VirtualDisk
DFS
/* creates a new dfile and returns the DFileID */
public DFileID createDFile();
/* destroys the dfile named by the DFileID */
public void destroyDFile(DFileID dFID);
/* reads contents of the dfile named by DFileID into the ubuffer
* starting from ubuffer offset startOffset; at most count bytes are transferred
*/
public int read(DFileID dFID, byte[] ubuffer, int startOffset, int count);
/* writes to the file named by DFileID from the ubuffer
* starting from ubuffer offset startOffset; at most count bytes are transferred
*/
public int write(DFileID dFID, byte[] ubuffer, int startOffset, int count);
/* List DFileIDs for all existing dfiles in the volume
*/
public List<DFileID> listAllDFiles();
Small details: low-level I/O
DBuffer
VirtualDisk needs basic info about a
DBuffer for I/O: the dbuf’s blockID
and a reference to its byte buffer.
VirtualDisk has private methods to “format”
the “disk” (VDF file), and fetch/push blocks at
specified blockIDs. We give you sample
code for these functions.
VirtualDisk
required interfaces
blockID = getBlockID()
buf[] = getBuffer()
Small details: initializing
new DFS(format) Initialization starts with a call from the test program.
DFS
If DFS constructor call
has format == true,
then all data in VDF is
discarded and zeroed.
constructor
(boolean format)
constructor
(int numblocks)
DBufferCache
DBuffer
Create numblocks DBuffer objects (dbufs) and memory
buffer regions (of size blocksize) for those dbufs.
VirtualDisk
Create/truncate VDF (w/ optional name), or open existing VDF if format == false.
Small details: exiting
sync
DFS
A test program should call DFS sync
before exit(), to force any dirty blocks in
the I/O cache out to disk.
sync
DBufferCache
VirtualDisk
Sync is implemented in DBufferCache,
and is synchronous: don’t return until all
writes complete.
Big details: DBuffer
DFS
A DBuffer dbuf returned by
getBlock is always associated
with exactly one block in the
disk volume. But it might or
might not be “in sync” with the
underlying disk contents.
read(…)
write(...)
startFetch(), startPush()
waitValid(), waitClean()
DBuffer
A dbuf is valid iff it has the “correct” copy of the data. A dbuf is
dirty iff it is valid and has an update (a write) that has not yet been
written to disk. A valid dbuf is clean if it is not dirty.
Your DeFiler should return only valid data to a client. That may require
you to zero the dbuf or fetch data from the disk. Your DeFiler should
ensure that all dirty data is eventually pushed to disk.
DBufferCache
/* Get buffer for block specified by blockID
The buffer is “held” until the caller releases it.
A “held” buffer cannot be evicted: its block ID cannot change.
*/
public DBuffer getBlock(int blockID);
/* Release the buffer so that it may be eligible for eviction.
*/
public void releaseBlock(DBuffer dbuf);
/* Write back all dirty blocks to the volume, and wait for completion.
*/
public void sync();
DBuffer
/* Start an asynchronous fetch of associated block from the volume */
public void startFetch();
/* Start an asynchronous write of buffer contents to block on volume */
public void startPush();
/* Check whether the buffer has valid data*/
public boolean checkValid();
/* Wait until the buffer has valid data (i.e., wait for fetch to complete) */
public boolean waitValid();
/* Check whether the buffer is dirty, i.e., has modified data to be written back */
public boolean checkClean();
/* Wait until the buffer is clean (i.e., wait for push to complete) */
public boolean waitClean();
/* Check if buffer is evictable: not evictable if I/O in progress, or buffer is held. */
public boolean isBusy();
DBuffer
/* Reads into the ubuffer[ ] from the contents of this Dbuffer dbuf.
* Check first that dbuf has a valid copy of the data!
* startOffset is for the ubuffer, not for dbuf.
* Reads begin at offset 0 in dbuf and move at most count bytes.
*/
public int read(byte[] ubuffer, int startOffset, int count);
/* Writes into this Dbuffer dbuf from the contents of ubuffer[ ].
* Mark dbuf dirty! startOffset is for the ubuffer, not for dbuf.
* Writes begin at offset 0 in dbuf and move at most count bytes.
*/
public int write(byte[] ubuffer, int startOffset, int count);
These calls are for use by the DFS layer to read/write user data between
client ubuffers and Dbuffer dbufs. DFS may read/write only on a held dbuf.
VirtualDisk
/*
* Start an asynchronous I/O request to the device/disk.
* The blockID and buffer array are given by the DBuffer dbuf.
* The operation is either READ or WRITE (DiskOperationType).
*/
public void startRequest(DBuffer dbuf, DiskOperationType rw) throws…;
Big issues: caching
DeFiler uses an I/O cache in memory to stage transfers to/from disk and to
reduce the need for I/O. The cache has a set of DBuffer dbuf buffer objects.
DFS
Each dbuf is either free or it is associated with exactly
one disk block blockID. I/O to/from a block is staged
from its dbuf. Each block has at most one dbuf.
The dbuf for a block is kept in cache after access. If a
requested block is already resident in the cache, then
dbuf = getBlock(blockID) getBlock finds its dbuf and returns it. Else it allocates a
releaseBlock(dbuf)
free dbuf for the block.
The system discards (evicts) a cached block if it has a
better use for the memory. It frees the evicted block’s
dbuf and soon reuses the dbuf for some other block.
DBufferCache
DBuffer
Big issues: eviction
The I/O cache system has a replacement policy to
select candidate blocks for eviction. It keeps an evict
pool of dbufs ordered by some measure of their
suitability for eviction, e.g., Least Recently Used (LRU).
DFS
dbuf = getBlock(blockID)
releaseBlock(dbuf)
The evict pool data structure may require more state in
dbufs, or interactions between DBufferCache and
DBuffer. This is up to you: no formats or interfaces are
specified.
DBufferCache
DBuffer
startRequest(dbuf, r/w);
VirtualDisk
ioComplete()
Buffer states
DFS
DBufferCache must not evict a block when its dbuf is in
use by the layer above or below. You must think
carefully about DBuffer (dbuf) states and how to
synchronize access to dbufs. This is up to you.
Suggestion. A dbuf is pinned if I/O is in progress, i.e., a
dbuf = getBlock(blockID) VDF request has started but not yet completed. A dbuf
releaseBlock(dbuf)
is held if DFS obtained a reference to the dbuf from
getBlock but has not yet released the dbuf. Don’t evict
a dbuf that is pinned or held: pick another candidate.
DBufferCache
DBuffer
startRequest(dbuf, r/w);
VirtualDisk
ioComplete()