AD replication and database internals
Download
Report
Transcript AD replication and database internals
Ondřej Ševeček | GOPAS a.s. |
MCM: Directory Services | MVP: Enterprise Security |
[email protected] | www.sevecek.com |
REPLICATION
Active Directory Replication
INTRO
Central Database
LDAP – Lightweight Directory Access Protocol
database query language
similar to SQL
TCP 389, SSL TCP 636, GC TCP 3268, GC SSL TCP 3269
Windows NT 4.0 SAM
SMB/CIFS TCP 445 (or NetBIOS)
password resets, SAM queries
Kerberos
UDP/TCP 88
Design Considerations
Distributed system
DCs disconnected for very long times
several months
Multimaster replication
with some FSMO roles
Design Considerations
Example: Caribean cruises, DC/IS/Exchange
on board with tens of workstations and users,
some staff hired during journey. No or bad
satelite connectivity only. DCs synced after
ship is berthed at main office.
Challenge: Must work independently for long
time periods. Different independent cruiseliners/DCs can accomodate changes to user
accounts, email addresses, Exchange
settings. Cannot afford lost of any one.
Database
Microsoft JET engine
JET Blue
common with Microsoft Exchange
used by DHCP, WINS, COM+, WMI, CA, CS, RDS
Broker
%WINDIR%\NTDS\NTDS.DIT
ESENTUTL
Opened by LSASS.EXE
Installed services
LSASS
Security Accounts Manager
TCP 445
SMB + NamedPipes
Kerberos Key Distribution Center
UDP, TCP 88
Kerberos
Active Directory Domain Services
UDP, TCP 389
LDAP
NTDS.DIT
Installed services
NT4.0
LSASS
SAM
TCP 445
SMB + NamedPipes
NTLM Pass-through
Connect to Domain
KDC
UDP, TCP 88
Kerberos
Windows 2000+
NTDS
UDP, TCP 389
LDAP
LDAP/ADSI Client
Restartable AD DS
Windows Server 2008
Active Directory Domain Services service
LSASS.EXE
Can log on DS Restore Mode Admin
HKLM\System\CurrentControlSet\Control\LSA
DsrmAdminLogonBehavior = 1
DNS Best Practice
DC1
DC2
AD
AD
DNS
DNS
Active Directory Replication
LOGICAL STRUCTURE
Logical Structure
Partitions
separate “subdatabases”
replication domains
RootDSE (Root Directory Services Enterprise)
Schema
Configuration
Domain
can contain user accounts
Application
can contain user accounts
Global Catalogue
Replication domains
RootDSE1
RootDSE2
Domain A
Domain A
RootDSE4
RootDSE5
Config
Config
Domain B
Domain B
Config
Config
Schema
Schema
RootDSE3
App1
Domain A
Config
Schema
Schema
RootDSE6
App1
Domain B
Schema
Config
App1
Schema
App2
Global Catalogue
[email protected]
[email protected]
MBX1
MBX2
Paris
Dublin
SMTP
Exchange
HUB
[email protected]
MBX1
London
Prague
[email protected]
MBX3
Global Catalogue
[email protected]
[email protected]
Paris
Dublin
SMTP
Exchange
HUB
[email protected]
GC
London
[email protected]
MBX1
[email protected]
MBX2
[email protected]
MBX3
[email protected]
MBX1
Prague
[email protected]
Global Catalogue (DC data)
DC
object
GUID DN
Display
name
Tel.
user
#1
CN=Kamil...
Kamil
555-666
Sevecek
C 915
CN=Sales...
CN=People...
user
#2
CN=Judith...
Judith
Hava
D 308
CN=Sales...
CN=People...
global
group
#3
CN=Sales...
Sales
CN=Kamil...
CN=Judith...
universal #4
group
CN=People...
People
CN=Kamil...
CN=Judith...
ou
#5
OU=London...
GPO
share
#6
CN=share...
\\srv8\doc
dns
record
#7
CN=pc31...
10.10.0.71
777-888
Office Member
Custom
data
Global Catalogue (GC data)
DC
object
GUID DN
Display
name
Tel.
user
#1
CN=Kamil...
Kamil
555-666
Sevecek
user
#2
CN=Judith...
Judith
Hava
global
group
#3
CN=Sales...
Sales
universal #4
group
CN=People...
People
ou
#5
OU=London...
share
#6
CN=share...
dns
record
#7
CN=pc31...
Office Member
777-888
CN=Kamil...
CN=Judith...
Custom
data
GC and Logon
U
GC
U
C
Kamil
SID #1
U
D
Kamil
SID #2
U
E
Kamil
SID #3
U
E
Judith
SID #3
DL
A
G
B
D
DC3
Paris
Paris
DC2
C
London
Kamil
E
DC4
DL
DC1
U
U
Prague
SRV
E
GC and Logon
U
GC
U
C
Kamil
SID #1
U
D
Kamil
SID #2
U
E
Kamil
SID #3
U
E
Judith
SID #3
DL
A
G
B
D
DC3
Paris
Paris
DC2
C
London
Kamil
E
DC4
DL
DC1
U
U
Prague
SRV
E
GC and Logon
U
DL
A
G
B
U
D
E
DC4
DC3
Paris
DC1
U
Paris
C
London
Kamil
U
C
Kamil
SID #1
U
D
Kamil
SID #2
U
E
Kamil
SID #3
G
B
Kamil
SID #4
Ticket
DL
DC2
Prague
DL
E
Kamil
SID #5
SRV
E
Active Directory Replication
ATTRIBUTE NOTES
Attribute Types
string, integer, datetime, boolean, binary
DN reference
multivalue
up to 5000 items
linked multivalue
unlimited, requires 2003 Forest Level
backlink
memberOf
computed
primaryGroupToken, tokenGroups, lastLogonTimestamp
write/only attributes
unicodePwd
Group membership
Sales
member
CN=Kamil,OU=London,DC=...
member
CN=Judith,OU=Paris,DC=...
Link
member CN=Victor,OU=London,DC=...
member
CN=Stan,OU=London,DC=...
Judith
Backlink
memberOf
CN=Sales,OU=Groups,DC=...
memberOf
CN=IS Access,OU=Groups,DC=...
(Not)replicated attributes
Not replicated
logonCount
badPasswordCount
badPasswordTime
lastLogon
lastLogoff
Replicated
pwdLastSet
lockoutTime
lastLogonTimestamp (since 2003)
Logon timestamps (2003 DFL)
lastLogon
lastLogonTimestamp
11:00
DC
lastLogon
lastLogonTimestamp
11:00
11:00
11:38
DC
lastLogon
lastLogonTimestamp
9:00
DC
Client
-
lastLogonTimestamp
Requires 2003 domain level
Updated only once per 14-random(5) days
DC=idtt,DC=local
msDS-LogonTimeSyncInterval
1+ – minimum without randomization
5+ – randomization starts
14 – the default
...
Password changes
Normal replication
Client
hash
PDC
Immediate
Replication
Password
Change
password
hash
DC
Normal replication
hash
Password changes
pwdLastSet
PDC
pwdLastSet
DC
pwdLastSet
DC
Client
pwdLastSet
DC
Authentication failures
pwd1
DC
pwd1
PDC
pwd1
DC
Client
Authentication failures
pwd1
DC
pwd2
PDC
pwd2
pwd2
DC
Client
Authentication failures
pwd1
pwd2
DC
pwd2
PDC
pwd2
DC
Client
Authentication failures
badPasswordCount
7
badPasswordCount
PDC
2
DC
lockoutTime
badPasswordCount
3
DC
badPasswordCount
DC
Client
2
Security Principals
Users
login, password, SID + SID history
Computers
user + computer attributes
Service Accounts
computer + specific attributes
Groups
login, SID + SID history
Computer Password Age
Active Directory Replication
TOPOLOGY
Intrasite Replication
Topology
DC1
DC4
DC2
DC3
Originating Updates and
Notifications
DC1
15 sec
DC4
DC2
3 sec
3 sec
DC3
Notification and Replication
I have got some
changes
DCOM
TCP
Random
Kerberos Authenticated
DC1
DC2
Give me your replica
DCOM
TCP
Random
Kerberos Authenticated
Intrasite Replication – 3
Hops max.
DC1
DC4
DC2
DC3
DC5
DC7
DC6
Intersite Replication (no
Bridgeheads)
DC1
DC5
DC2
DC3
DC4
DC6
DC7
Intersite Replication (no
Bridgeheads)
15 sec
DC1
DC5
DC2
DC3
3 sec
schedule
3 sec
DC4
3 sec
DC7
DC6
3 sec
Intersite Replication with a
Bridgehead
15 sec
DC1
DC5
schedule
DC2
3 sec
DC3
3 sec
DC4
3 sec
DC7
DC6
3 sec
Intrasite Replication
Uses notifications by default
(originating/received)
300/30 sec on Windows 2000
15/3 sec on Windows 2003
Occurs every hour as scheduled
nTDSSiteSettings
At this frequency KCC detects unavailable partners
HKLM\System\CCS\Services\NTDS\Parameters
Replicator notify pause after modify (secs)
Replicator notify pause between DSAs (secs)
Intrasite Replication
notification
15 sec
random TCP
download
changes
DC1
DC2
random TCP
download
changes
schedule
random TCP
Intersite Replication
DC1
DC2
download
changes
schedule
random TCP
Intersite Replication
Does not use notifications by default
siteLink: options = USE_NOTIFY (1)
Compression used
siteLink: options = DISABLE_COMPRESSION (4)
Bridge all site links
Static TCP for Replication
HKLM\System\CurrentControlSet\Services
NTDS\Parameters
TCP/IP Port = DWORD
Replication
Netlogon\Parameters
DCTcpipPort = DWORD
LSASS (Pass-through)
NTFRS\Parameters
RPC TCP/IP Port Assignment = DWORD
DFSRDIAG StaticRPC /port:xxx /Member:dc1
Urgent Replication
(Notification)
Intrasite only
intersite also if notification enabled
Do not wait for delay (15/3 sec)
In the case of
account lockout
password and lockout policy
RID FSMO owner change
DC password or trust account password change
Immediate Replication
(Notification)
Password changes
from DCs to PDC
Regardless of site boundaries
PDC downloads only the single user object
all changed attributes but only single object
From DC/PDC further with normal replication
Example Replication Traffic
Atomic replication of a single object with a
one byte attribute change
Notification + replication
intersite compressed
Overall 7536 B
30 packets ~10 round trips
50 ms round trip means 500 ms transfer time
consumption at 120 kbps
Useful data ~80 B
Bridge All Site Links On
A
Olomouc
A
B
A
Prague
London
A
Paris
B
Roma
site links are transitive
can be disabled on IP
Cyprus
transport
A
Bridge All Site Links Off
A
Olomouc
A
B
A
Prague
London
A
Paris
B
Roma
site links are not
transitive
Cyprus
Cyprus partition is cut off
A
GC Replication
A
GC
Olomouc
A
GC
Prague
A
London
GC
Paris
B
Roma
Cyprus
A
one-way:
from the source NC into
the nearest GC
two-way:
GCs between themselves
A
GC Replication
A
Olomouc
A
Prague
A
B
London
GC
Paris
Roma
B
Cyprus
A
one-way:
from the source NC into
the nearest GC
two-way:
GCs between themselves
A
Subnetting in AD (Apps)
DC1
DC5
Exchange
Exchange
Exchange
DC2
10.10.0.248 / 29
DC3
DC4
10.10.x.x / 16
Subnetting in AD (Recovery)
DC1
DC5
Recovery Site
10.10.0.7 / 32
DC2
DC3
DC4
10.10.x.x / 16
Active Directory Replication
MODIFICATIONS
Modification operations
Create new object
Modify attributes
change/delete value
change distinguishedName = rename
Rename container
all subobjects renamed as well
Replication Metadata
REPADMIN /ShowObjMeta
all attributes
when
originating DC
Replication conflicts
The later action wins
if no one is later then random (USN)
Attribute modified on two DCs “simultaneously”
only one change wins
Linked multivalue attribute modified
merged (on 2003+ forest level)
Object/container deleted and object modified
deleted
Object moved into a deleted container
CN=lost and found
Two objects with the same sAMAccountName, cn or
userPrincipalName created
object renamed, logins duplicit
Linked Multi-values
Replication
11:05
Kamil
10:00
Helen
11:00
DC1
DC1
9:00
DC2
Replication Basics
11:30
Kamil
10:00
Helen
11:00
DC1
DC1
11:30
Kamil
10:00
Helen
11:00
DC2
Replication Basics
12:05
Kamil
10:00
Helen
11:00
Judith
DC1
12:00
DC1
11:30
Kamil
10:00
Helen
11:00
DC2
Replication Basics
12:30
DC1
12:30
Kamil
10:00
Kamil
10:00
Helen
11:00
Helen
11:00
Judith
12:00
Judith
12:00
DC1
DC2
Replication Basics
12:30
Kamil
10:00
Helen
11:00
DC1
12:30
Kamil
10:00
DC1
Helen
11:00
DC1
Judith
12:00
DC1
DC2
Marie
Judith
12:00
DC1
DC3
11:00
Me
Replication Basics
12:30
Kamil
10:00
Helen
11:00
DC1
12:30
Kamil
10:00
DC1
Helen
11:00
DC1
Judith
12:00
DC1
DC2
Marie
Judith
11:00
12:00
DC1
Kamil
DC3
DC1
10:30
DC2
7:00
10:00
DC1
Me
Replication Basics
13:30
Kamil
10:00
Helen
11:00
DC1
12:30
Kamil
10:00
DC1
Helen
11:00
DC1
Judith
12:00
DC1
DC2
Marie
Judith
11:00
12:00
DC1
Kamil
DC3
DC1
10:30
DC2
7:00
10:00
DC1
Me
Replication Basics
13:30
Kamil
10:00
Helen
11:00
DC1
12:30
Kamil
10:00
DC1
Helen
11:00
DC1
Judith
12:00
DC1
DC2
Marie
Judith
11:00
12:00
DC1
Kamil
DC3
DC1
12:30
DC2
13:30
10:00
DC1
Me
Replication Basics
14:15
Kamil
10:00
Helen
11:00
Judith
12:00
DC1
Kamil
10:00
DC1
Helen
11:00
DC1
Judith
12:00
DC1
Marie
11:00
DC2
DC3
DC1
12:30
DC2
13:30
USN
Each object modification increments USN for
that object and for the whole DC
Each DC remembers USNs of its replication
partners
repadmin /showutdvec
USN
2
USN
5001
1
USN
1001
2
5001
3
3001
1
1001
3
3001
3
USN
3001
1
1001
2
5001
USN
Kamil
1002
John
1003
2
USN
5001
1
USN
1003
2
5001
3
3001
1
1001
3
3001
3
USN
3001
1
1001
2
5001
USN
Kamil
1002
John
1003
1
USN
1003
2
5001
3
3001
2
USN
5001
Notify
Give me
1
1001
3
3001
1002, 3
3
USN
3001
1
1001
2
5001
USN
Kamil
1002
John
1003
Kamil
5002
John
5003
1
USN
1003
2
5001
3
3001
2
USN
5003
1
1003
3
3001
3
USN
3001
1
1001
2
5001
USN
Kamil
1002
John
1003
Kamil
5002
John
5003
Maria
1
USN
1003
2
5001
3
3001
2
USN
5004
5004
1
1003
3
3001
3
USN
3001
1
1001
2
5001
USN
Kamil
1002
John
1003
Kamil
5002
John
5003
Maria
1
USN
1003
2
5001
3
3001
2
USN
5004
5004
1
1003
3
3001
3
USN
3004
1
1003
2
5004
Kamil
3002
John
3003
Maria
3004
USN
1
Kamil
1002
1
John
1003
1
Kamil
5002
1
John
5003
2
Maria
1
USN
1003
2
5001
3
3001
2
USN
5004
5004
1
1003
3
3001
3 1
USN
30041
Kamil
3002
John
3003
Maria
3004
1
2
1003
2
5004
USN
1
Kamil
1002
1
John
1003
1
Kamil
5002
1
John
5003
2
Maria
1
USN
1003
2
5001
3
3004
2
Maria
2
USN
5004
5004
1
1003
3
3001
3 1
USN
30041
Kamil
3002
John
3003
Maria
3004
1
2
1003
2
5004
Active Directory Troubleshooting
DELETE OPERATIONS
Delete operations
Delete only removes most attributes from an
object
tombstone
Replicates as normal object change/move
Deleted by individual DCs after
tombstoneLifetime
CN=Directory
Services,CN=Services,CN=Configuration,...
Reanimating objects
LDP
Options – Search
Extended
Return deleted objects
View – Tree
CN=Deleted Objects
Tombstone lifetime
Windows 2000
60 days
Windows 2003 SP1+
180 days
upgrade keeps the 60 days from previous version
Tombstone lifetime
CN=Directory Services,CN=Windows
NT,CN=Services,CN=Configuration,DC=idtt,DC=
local
tombstoneLifetime
garbageCollPeriod (12 hours by default)
Garbage collection does not delete white space
from the database
only offline defragmentation
the amount can be logged by setting
HKLM\System\CCS\Services\NTDS\Diagnostics
6 Garbage Collection = 1
AD Recycle bin
Optional feature with Windows 2008 R2
forest level
Preserves all attributes on deleted objects for
the tombstone lifetime
after that, the object becomes normal tombstone
for another lifetime
Does not preserve attribute changes
recovery site still useful
Keeps deactivated links (group membership)
Active Directory Replication
REPLICATION PROBLEMS
The Three Problems
Single DC offline for a long time
not so long as tombstone!
authentication problem
Tombstone lifetime
two separate DC zones
not a “business” consistency problem
USN rollback
restore from snapshot, image, manual backup
total inconsistency!
DC Offline for Long Time
Month 0
PWD
21
OLD PWD
-
DC2
DC1
DC2
PWD
21
DC3
PWD
31
PWD
31
OLD PWD
-
DC3
DC Offline for Long Time
Month 1
PWD
22
OLD PWD
21
DC2
DC1
DC2
PWD
21
DC3
PWD
31
PWD
32
OLD PWD
31
DC3
DC Offline for Long Time
Month 2
PWD
23
OLD PWD
22
DC2
DC1
DC2
PWD
21
DC3
PWD
31
PWD
33
OLD PWD
32
DC3
DC Offline for Long Time
Month 3
PWD
23
OLD PWD
22
PWD 21
Kerberos
KDC
DC2
Ticket
DC1
DC2
PWD
21
DC3
PWD
31
PWD
33
OLD PWD
32
DC3
DC Offline for Long Time
Month 3
PWD
23
OLD PWD
22
PWD 23
KDC
Disabled
DC2
Ticket
Kerberos
KDC
DC1
DC2
PWD
21
DC3
PWD
31
PWD
33
OLD PWD
32
DC3
Lingering Objects
When DC didn’t replicate during the
tombstoneLifetime, it halts replication
Can be restored by Allow Replication with
Divergent and Corrupt Partner
HKLM\System\CCS\Services\NTDS\Parameters
turn on, replicate, turn off
Objects and Tombstones
Frank
DC1
Stan
Tania
Frank
DC3
Stan
Tania
Frank
DC2
Stan
Tania
Frank
DC4
Stan
Tania
Objects and Tombstones
Frank
DC1
Stan
Tania
Frank
DC3
Stan
Tania
Frank
DC2
Stan
Tania
Frank
DC4
Stan
Tania
Objects and Tombstones
Frank
DC1
Stan
Tania
Frank
DC3
Stan
Tania
Frank
DC2
Stan
Tania
Frank
DC4
Stan
Tania
Objects and Tombstones
Frank
DC1
Stan
Tania
Frank
DC3
Stan
Tania
Frank
DC2
Stan
Tania
Frank
DC4
Stan
Tania
Objects and Tombstones
Frank
DC1
Frank
DC3
Tania
Tania
Frank
DC2
Frank
DC4
Tania
Tania
Lingering Objects
Frank
DC1
Stan
Tania
Frank
DC3
Stan
Tania
Frank
DC2
Stan
Tania
Frank
DC4
Stan
Tania
Lingering Objects
Frank
DC1
Stan
Tania
Frank
DC3
Stan
Tania
Frank
DC2
Stan
Tania
Frank
DC4
Stan
Tania
Lingering Objects
Frank
DC1
Frank
DC3
Tania
Stan
Tania
Frank
DC2
Frank
DC4
Tania
Stan
Tania
Lingering Objects
Frank
DC1
Frank
DC3
Tania
Stan
Tania
Frank
DC2
Frank
DC4
Tania
Stan
Tania
Lingering Objects
Lingering Objects
Strict Replication Consistency
HKLM\System\CCS\Services\NTDS\Parameters
1 – do not replicate
0 – request full copy from source
By default only on new Windows 2003+
installations
Lingering Object
found/deleted
Correct Registry Settings
Long term normal operation
Strict consistency = 1
Allow divergent partner = 0
Temporary repair operation
Strict consistency = 1
Allow divergent partner = 1
USN Rollback
May or may not be detected
Cannot be repaired
not always lingering objects!
DC must be denoted/repromoted
unplug network
DCPROMO /forceremoval
NTDSUTIL Roles
NTDSUTIL Metadata Cleanup
USN Rollback
DC1
1001
Snapshot
2
USN
5001
1
1001
3
3001
USN Rollback
DC1
1001
Kamil
1002
John
1003
Judith
1004
Helen
1005
Eva
1006
Snapshot
2
USN
5001
1
1001
3
3001
USN Rollback
DC1
1001
Snapshot
Kamil
1002
Kamil
1002
John
1003
John
1003
Judith
1004
Judith
1004
Helen
1005
Helen
1005
Eva
1006
Eva
1006
2
USN
5001
1
1006
3
3001
USN Rollback (Detectable)
DC1
1001
Restore
Kamil
1002
Frank
1002
John
1003
Stan
1003
Judith
1004
Helen
1005
Eva
1006
2
USN
5001
1
1006
3
3001
USN Rollback (Non-detect.)
DC1
1001
Restore
Kamil
1002
Frank
1002
John
1003
Stan
1003
Judith
1004
Tania
1004
Helen
1005
Mark
1005
Eva
1006
Martin
1006
Victor
1007
Leo
1008
2
USN
5001
1
1006
3
3001
USN Rollback (Non-detect.)
DC1
1001
Frank
1002
Stan
1003
Tania
1004
Mark
1005
Martin
1006
Victor
1007
Leo
1008
Restore
Kamil
1002
John
1003
Judith
1004
Helen
1005
Eva
1006
Victor
1007
Leo
1008
2
USN
5001
1
1008
3
3001
Restoring VM Snapshots
Restore offline
HKLM\System\CurrentControlSet\Services\NTDS
Database Restored from Backup = DWORD = 1
Restart NTDS service
changes InvocationID of the database instance
Active Directory Troubleshooting
DATABASE INTERNALS
Stealth
Microsoft’s internal tool
test only, not supported
Must run from writable location
C:\ etc.
creates temporary DB
Database attributes
ATTxyyy
attribute
ATTxyyy
attribute
m3
cn
k589827
objectSID
m4
sn
k589826
objectGUID
m11
ou
k589825
name
m13
description
m590045
sAMAccountName
m42
givenName
m590480
userPrincipalName
m8
state
m131218
company
m6
country
m131203
country
m19
physicalDeliveryOfficeName
m131328
address
m20
telephoneNumber
m12
jobTitle
m131213
department
m7
city
m131682
employeeNumber
m131085
displayName
m1376259
email
Database structure
DNT – Distinguished Name Tag
unique identification inside the table
NCDNT – Name Context DNT
reference to which partition the object belongs
RDNT – Parent DNT
reference to parent OU/CN object
objects do not store their whole DN, the hierarchy
is built by the references
link_table
Contains linked multivalue references
LINK_DNT
DNT of the referencing object (group)
BACKLINK_DNT
DNT of the referenced object (user, phantom)
Phantom
GUID, NC reference, name
Active Directory Troubleshooting
BACKUP AND RESTORE
Backup
Windows 2003 streaming backup
must have read the data from the database using
API and built a new database into the backup
media
Windows 2008+
Volume Shadow Copy
AD Writer available
makes the database into clean shutdown
Restore
Not supported
to restore manually only the database
to restore from disk image
to restore from virtual machine snapshot
may be partially recovered by the Database
Restored from Backup registry
Supported
restore whatever you want first
then restore System State
Recovery Site
You can separate one or more DCs into a
slowly replicating site to avoid the need for
offline restore
Authoritative Restore can be done even on
non-restored DCs
the database must still be brought offline
Authoritative Restore
Marks some objects as authoritative
Replicates all their attributes over any other
objects found on other DCs
When restoring whole OUs, consider their
relationship to GPOs linked
Authoritative Restore
Problems
old computer, user, service and trust
passwords restored
membership in remote Domain Local and
Universal groups lost
exports .LDF file containing the group
memberships
connections to GPO lost at OU level
orphaned GPOs
Ondřej Ševeček | GOPAS a.s. |
MCM: Directory Services | MVP: Enterprise Security |
[email protected] | www.sevecek.com |
THANK YOU!