2014-EAKC-YFS-Deep-Dive
Download
Report
Transcript 2014-EAKC-YFS-Deep-Dive
YFS: An Introduction to the next /afs®
Jeffrey Altman, Daria Brashear, Marc Dionne, & Simon Wilkinson
Your File System Inc. and Your File System Ltd.
2014 European AFS and Kerberos Conference
Your File System Inc. (YFSI) is a New York State
Corporation with HQ in Manhattan and
registered as a business entity in Canada
Your File System Ltd is a wholly owned
subsidiary of YFSI with HQ in London
YFSI is privately owned and operated
YFSI is a Red Hat Partner ISV
Location Transparency: one name space
User Mobility: access from any device
Security: Flexible model for authentication, privacy, data
protection and access control
Availability: Temporary loss to small groups for short time
periods
Integrity: No user initiated backups
Heterogeneity: Multiplatform
Self service: Low Help Desk costs
Atomic Publishing: Software, documentation, web sites, ..
Real time collaboration: Distributed File Locking
Distributed Administration
The vision was decades ahead of its time in 1983
The implementation is decades behind in 2014
Limited network throughput
Increased call processing latency
Decreased service reliability and availability
Elevated risk of distributed deadlocks
Inability to use full capability of available hardware
Failure to keep up with competing technologies
That /afs is still in use today is a credit to its vision and
the strength of its architecture.
Major system rewrites are few and far
between
“Contractor Model of Support” leads to many
small and localized changes
A lack of consistent vision and quality control
Few incentives to invest in the next 30 years
Application Transparency
•
Be a Tier One file system on all major OSes
Embrace multi-producer, multi-consumer
work flows
Extended Integrity: Disaster Recovery
Be performance competitive
•
Lustre, GPFS, Panasas, …
Best of breed data security
Improved Ease of Use
Designed for the long term
Improved
performance with existing hardware
Cost
reductions due to hardware consolidation
Zero
data loss as part of a transition
No
flag day required
• Mixed deployments are encouraged
Performance issues restrict the jobs that
sites are willing to run in /afs
Deploying excessive hardware to solve load
distribution and fairness problems is
expensive
Support for multiple file systems costs
money, requires additional staff, can result in
data duplication and out of sync issues
Reduced contention in the listener thread
•
255 packet window size (per call) without
degradation
•
10 gbit network interface saturation
Order of magnitude faster on high latency links
Dynamic Thread Pools
•
Thread Count limited by OS resources
64-bit
volume IDs
96-bit (79 octillion) vnode IDs
64-bit,100ns granular timestamps
• 2038 ready
Ubik
databases extensible up to 16 exabytes
Partitions, volumes and quotas tracked up to
16 zettabytes
Optimized Cache Manager handshakes
Volume Status Information
Reduces number of GetVolumeStatus RPCs
• Permits RW / RO data cache sharing
• Improved caching of RO volume per user permissions
•
Fewer FetchStatus RPCs for RO volumes
Host and callback package rewritten
•
Significantly faster callback breaks
Vnode lock contention dramatically reduced
Distributed writing to shared data sets now
possible
Open mode supported on some OSes
Bypasses VFS cache and AFS cache for both
read and write
No file threshold to tune
Data is copied directly to the caller, or directly
from the caller to the file server
Data breaches and exposures are followed by
a high cost
•
•
•
•
•
Public Relations Nightmare
Costs of Identity Theft Detection Services (in U.S.)
Loss of employment for key staff members
Organizational reorganization
Disruption of core mission when forced to address
security concerns in crisis mode
Multi-layered policies
Flexibility for self service end users
• System administrator controls
•
Network Security
Reduced Information Exposure
Minimal Privilege Services
Self Service Group Management
Per-Object ACLs
•
Cross directory hard links now permitted
Volume ACLs
•
Limits the permissions that end users can grant
Volume Security Policy
•
Per-Volume minimum acceptable rx connection
security properties
File Server Security Policy
Per-server minimum acceptable rx connection
security properties
• Only volumes with weaker or equivalent security
policy can be attached, moved to, or restored to.
•
YFS RXGK Security Class
GSS Kerberos 5 authentication
• AES-256 wire privacy and integrity protection
•
Cell wide key for DB servers
Individual keys for file servers
Per-host keys for BOS Overseer Service
YFS protects the callback channel with AES256 privacy and integrity protection
when rxgk is used for the incoming connection
• Avoids leaking information about volume and file ids
accessed by a client
• Prevents forged messages from invalidate callback
state
•
Server Processes execute under a daemon
account
•
Not Root
Cache Managers can be issued
a Kerberos keytab
• a Protection DB Machine ID
•
Keyed Cache Managers can use privacy for all
connections
Machines IDs are similar to User IDs
Can be placed on ACLs and added to Groups
• But are not included in system:authuser
•
File System Extensions
Per
File ACLs
Cross directory hard links
Extensions for Microsoft Windows
Mandatory Locks
• Advisory locks are not enforced by the file server
Symlink
Updates
• Reparse Points can be updated without FileID change
CreateFile with Lock
• Avoids races when simulating OpLock semantics
Command Output Clarity
• Modifications to human readable and machine
readable output
• vos examine, listvol, rxdebug, xstat_fs, …
Consolidate output
Introduce consistency across command options
• Machine readable output –format is not human
formatted
All fields are now separated by single tabs
Easy to import into spreadsheets and databases
Library Cleanup
• All libraries are thread safe
• Built using libtool
• Intended for use implementing language bindings
libacquire
• A library to obtain tokens
rxkad
yfs-rxgk
• aklog is a wrapper
• Can be linked to pam modules
Automated Windows Domain Token
Acquisition
Triggered
by access denied errors
Automatic Token acquisition using Logon
Session Kerberos Credentials
Works with all applications that use
• WNet API: Network Providers
• Shell API: Explorer, Office, anything with an Open
dialog
Simplify Server Configuration
Provide Extensibility for New Features
BOS command lines are limited in length
Permit the construction of flexible test suites
Greatly improved configuration flexibility and
convenience
Custom file layouts are possible
All settings centralized in a single configuration area,
single file or directory
A configuration directory can ease distribution of
custom options
All command line options can be set in configuration
Goal
Provide a test for every new feature, library function,
RPC
Provide a test with every bug fix, if possible
Requirements
Ability to spin up the various servers and provide a test
configuration
All tests must be able to run as a regular user
Must be able to serve test data not necessarily under
/vicep*
Extensive test suite coverage
A complete test cell can spin up in a few
seconds
Many tests spin up a cell and destroy it when
done, maintaining test independence
Client testing through libafscp and fuse client
All new features require tests before merging
Sample systemd yfs-server.service
[Unit]
Description=YFS Server Service
After=syslog.target network.target
[Service]
EnvironmentFile=-/etc/sysconfig/yfs
ExecStart=/usr/local/sbin/bosserver -config
/s/yfs/server/yfs-server.conf -nofork
ExecStop=/usr/local/bin/bos -config
/s/yfs/server/yfs-server.conf shutdown
hurricane.marcdionne.net -wait -localauth
User=yfs
Group=yfs
[Install]
WantedBy=multi-user.target
Sample file layout
[marco@hurricane /s/yfs/server ]$ ls -l
total 60
drwxr-xr-x. 2 yfs yfs 4096 Mar 23 04:00
-rw-r--r--. 1 yfs yfs
526 Jul 15 2013
-rwxr-xr-x. 1 yfs yfs
26 Jul 15 2013
drwxrwxr-x. 6 yfs yfs 4096 Jan 11 15:36
drwxrwx---. 2 yfs yfs 4096 Oct 25 10:47
-rw-r--r--. 1 yfs yfs
4 Jan 6 09:52
-rw-r--r--. 1 yfs yfs
144 Jan 6 09:52
drwxrwx---. 2 yfs yfs 4096 Mar 25 10:28
drwxrwxrwx. 2 yfs yfs 12288 Mar 25 10:29
-rw-r--r--. 1 yfs yfs
15 Sep 12 2013
-rw-r--r--. 1 yfs yfs
114 Dec 19 16:56
-rw-r-----. 1 yfs yfs 2000 Aug 5 2013
drwxrwxr-x. 2 yfs yfs 4096 Mar 26 18:25
bos
bos.keytab
cacheinfo
data
db
KeyFile
KeyFileExt
local
logs
ThisCell
UserList
vl.keytab
yfs-server.conf
[marco@hurricane /s/yfs/server ]$ ls -l yfs-server.conf/
total 8
-rw-r--r--. 1 yfs yfs 645 Mar 26 18:25 cellservdb.conf
-rw-rw-r--. 1 yfs yfs 792 Mar 4 15:48 yfs-server.conf
Sample cellservdb.conf
[cells]
grand.central.org = {
example.com = {
description = "GCO Public CellServDB 23 Apr 2008"
description = "Test cell"
servers = {
servers = {
penn.central.org = {
blizzard.marcdionne.net = {
addr = 128.2.203.61
addr = 192.168.0.113
}
}
grand.mit.edu = {
}
addr = 18.9.48.14
}
}
marcdionne.net = {
andrew.e.kth.se = {
description = "Marc's cell"
addr = 130.237.48.87
servers = {
}
hurricane.marcdionne.net = {
}
addr = 192.168.0.107
}
}
}
}
Sample server configuration
[dirpath]
SERVER_ETC_DIR = /s/yfs/server
SERVER_DB_DIR = /s/yfs/server/db
SERVER_LOGS_DIR = /s/yfs/server/logs
SERVER_BOSCONFIG_DIR = /s/yfs/server/bos
SERVER_LOCAL_DIR = /s/yfs/server/local
SERVER_PART_PREFIX_DIR = /s/yfs/server/data
[vlserver]
keytab = /s/yfs/server/vl.keytab
auditlog = /s/yfs/server/logs/audVl
[fileserver]
d = 125
p = 200
nojumbo =
auditlog = /s/yfs/server/logs/audFile
security = yfs-rxgk:crypt rxkad:clear rxnull rxkad:crypt
[bosserver]
auditlog = /s/yfs/server/logs/audBos
[volserver]
d = 125
auditlog = /s/yfs/server/logs/audVol
[ptserver]
auditlog = /s/yfs/server/logs/audPt
[salvager]
auditlog = /s/yfs/server/logs/audSalv
[salvageserver]
auditlog =
/s/yfs/server/logs/audSalvserv
Installation is the initial experience an end
user has with the product
If the installation process is frustrating, the
end user is likely to be unhappy with the
product
Lack of digital signatures can block the
installation of a package or trigger a
frightening dialog
New installation packages
Windows
• OSX
• Linux
•
Debian
Fedora
RHEL6 and RHEL7
Single installer
64-bit and 32-bit components
• Heimdal Side by Side Assembly
• Heimdal Command Line tools
• Automatic Cache Sizing
•
All components digitally signed
•
Microsoft Cross Signing of Drivers
Flat package
Integral packages for client, server and
development
Digital signatures on the package, the kext
and the binaries using Apple-issued
certificate
New packaging for Debian, Fedora and RHEL
Integral packages for client, db services, and
file service
Digital signatures on installation packages
Dual Protocol Stacks
Allows
advanced features while maintaining
backwards compatibility with AFS®
AFS protocol suite has all of the capabilities
and limitations of OpenAFS
YFS features only available on YFS protocol
suite
• rxgk, file server, vol server, vl server, pt server
enhancements
Transparent
negotiation of protocol suite
Mixed Mode Cells
Two
cell types are defined:
• AFS cell deploys OpenAFS or IBM AFS vlservers
• YFS cell deploys the YFS location server
OpenAFS
and YFS File Servers can be joined to
either cell
YFS Client in AFS Cell
No
support for RXGK, AES-256
No support for file server security policies
YFS Server in AFS Cell
Improved
No
RX Performance for writes
Rxgk
Volume IDs restricted to AFS limits
Security Policies cannot be enforced
Only AFS compatible capabilities can be
registered
IPv6 addresses cannot be registered
YFS File Server in AFS cell
vos
AFS
vlserver
AFS volume
format
AFS
fileserver
YFS
fileserver
YFS Server in AFS Cell
Improved
No
RX Performance for writes
Rxgk
Volume IDs restricted to AFS limits
Security Policies cannot be enforced
Only AFS compatible capabilities can be
registered
IPv6 addresses cannot be registered
AFS Client in YFS Cell
No
support for RXGK, AES-256
No support for file server security policies
Volumes with ID above 232-1 inaccessible
Mandatory locks cannot be requested but will
be enforced
Volume sizes and quotas >231KB will be faked
Other restrictions as required to enforce
security policies
AFS File Server in YFS cell
YFS
location
server
vos
AFS volume
format
AFS
fileserver
YFS
fileserver
YFS volume
format
5
9
AFS and YFS Volserver Compatibility
RW
volumes on YFS server cannot be
replicated to AFS server
Volumes containing YFS tags cannot be moved
to an AFS server
• ACL Data
• Volume Attributes (ACL or Security Policy)
Data
transfers protected with Rxkad and Fcrypt
RX performance improved in YFS to AFS
direction
YFS POSIX attribute backend store
YFS protocol suite
AFS protocol suite
6
1
64 bit Ubik database
YFS protocol suite
AFS protocol suite
6
2
64 bit Ubik database
YFS protocol suite
AFS protocol suite
rxgk
keyserver
6
3
Documentation
Updated
man pages
New Quick Start Guides
Updated Administrator’s Guide
Export Licenses
The
U.S. Government has classified YFS 1.x as
a mass market product
Worldwide Export permitted with a few
exceptions
No export restrictions on distribution by
customers
YFS 1.0 Binary License
A
full suite of clients and servers
• Windows
• OSX
• iOS
• RHEL5, RHEL6, RHEL7
• Fedora
• Debian
• Solaris
• AIX
Support
Free
updates to new releases (one year)
• Every four month release cadence
Free
security updates (two years)
Unlimited e-mail / web support (one year)
Cell performance evaluation (once per year)
Remote monitoring service (one year)
Products
Cell (no replication)
1 Server (DB and File)
Base cell (replication)
4 DB Servers
4 File Servers
1000 User or Machine IDs
Unlimited Client devices
Additional Servers
Additional IDs
Annual purchases continue support
Non-redistribution Source code license available
Training (on-site or web)
Availability
General
First
Availability End of May 2014
update, September 2014
2014 Road Map
Feature Priorities
• IPv6 enhancements
• Rapid Partition Relocation
• Extended Volume Names
• New Directory Format
Unlimited Directory Sizes
Extended Attributes
Alternate Data Streams
• Read/write Replication
• Extended Callbacks