Transcript Document 7152392

Current Sphinx Client/Server
Multi-threaded
Control Process
Message Interface
Message Interface
DAG Reducer
Prediction Engine
Submitter
Sphinx Client
Data Warehouse
Planner
Resource
Information Gatherer
Sphinx Server
Monitoring Service
Generic Grid Site
Sphinx Client/Server with V2 Components
Multi-threaded
Control Process
Message Interface
Message Interface
DAG Reducer
Prediction Engine
Submitter
Admission Control
Data Warehouse
Sphinx Client
Planner
Resource
Data Management
Information Gatherer
Sphinx Server
Monitoring Service
Generic Grid Site
Stand alone Sphinx Server with light Client
Multi-threaded
Control Process
Message Interface
Message Interface
DAG Reducer
Prediction Engine
Admission Control
Data Warehouse
Sphinx Client
Planner
Submitter
Resource
Data Management
Information Gatherer
Sphinx Server
Monitoring Service
Generic Grid Site
Data Warehouse
Multi-threaded
Control Process
Message Interface
•
•
•
•
Message Tables
Policy Information Tables
Entity Accounting Tables
Work (DAG, Job, etc) Tables
•
•
•
•
•
•
•
•
DAG Reducer
Prediction Engine
•
•
•
Planner
Submitter
Data Management
•
•
•
•
•
Replica Catalogue (from RLS)
Profiles (Size, Bandwidth requirements)
History (Access rate, etc)
CPU, Disk, Network Connection, middleware
configuration, etc
Grid Weather Tables
•
•
•
Sphinx Server
Transformation Catalogue
Profiles (CPU, Disk, Bandwidth requirements)
History (CPU, Disk, Bandwidth, actual use)
Resource Property Tables
•
Information Gatherer
QoS Requests
Resource Assignment for next k steps
Data Tables (in general, distributed…)
•
•
•
•
Do not start before “event”
I/O
Application Tables (in general, distributed…)
•
•
•
•
Unreduced
Unpredicted
Unaccepted (not admitted)
Unplanned (not yet feasible)
Unsent (assigned)
Unfinished (executing)
Remove (accounted or rejected)
Entity
Resource Property Requirements
Dependency Requirements
•
•
Admission Control
Data Warehouse
State
CPU View
Storage View
Bandwidth View
What about including Virtual Data Tables?
Message Interface Component
Multi-threaded
Control Process
Message Interface
DAG Reducer
Prediction Engine
Admission Control
Data Warehouse
Planner
Submitter
Data Management
Information Gatherer
Sphinx Server
• Input (incoming)
•
•
•
•
DAG
Requirements
QoS requests
Status requests
• Output (incoming)
• DAG Table
(“unreduced”)
• Job Table
(“unreduced”)
• application
• Requirements
• QoS
• Input (outgoing)
• Message
• Output (outgoing)
• Status updates
DAG Reducer (Decomposer?)
Multi-threaded
Control Process
Message Interface
DAG Reducer
Prediction Engine
Admission Control
Data Warehouse
Planner
Submitter
Data Management
Information Gatherer
Sphinx Server
• Input
• DAG (“unreduced”)
• Available (existing) data
• Action
• Remove DAG nodes
for which input data
exists
• Add Jobs to Job table
• Output
• Pruned DAG
(“unpredicted”)
• Jobs (“unpredicted”)
Prediction Engine Component
Multi-threaded
Control Process
Message Interface
DAG Reducer
Prediction Engine
Admission Control
Data Warehouse
Planner
Submitter
Data Management
Information Gatherer
Sphinx Server
• Input:
• “unpredicted” Job
• Requirements (if
available)
• CPU hours
• Storage hours
• Bandwidth
• Profile/History
• Output:
• Estimated
Resource Usage
• “unaccepted” job
Admission Control Component
Multi-threaded
Control Process
•
•
•
Message Interface
•
Prediction Engine
•
•
Predictions of Jobs
(decomposed DAG)
Resource Usages
Policy Constraints
Current schedule for next k
steps
Output
•
Submitter
•
Information Gatherer
Sphinx Server
•
Planner
Data Management
Entity Account Information
“unaccepted” DAG Request
• Requirements
• QoS
DAG Reducer
Admission Control
Data Warehouse
Input
Reject DAG Request
• “Remove” DAG
• “Remove” Jobs
Accept DAG with Estimated
QoS
• “Unplanned” DAG
• “Unplanned” Jobs
Planning Component
Multi-threaded
Control Process
•
Job Planner (independent “work” that
can be fully resolved onto resources at
time t)
•
Message Interface
•
•
•
•
•
•
•
•
Prediction Engine
•
Admission Control
Planner
Submitter
“unplanned” Job Information
•
•
DAG Reducer
Data Warehouse
Input
Job QoS Requests
Entity Account Information
Policy Constraints
Available Data
Available Applications
Available Resource Properties
Prediction of Job Resource Usage
Output
•
•
•
QoS Deliverable
“unsent” Job(s) schedule for the next
k steps
DAG Planner (intra-dependent “work”
that can not all be fully resolved onto
resources at time t)
•
Input
•
“unplanned” DAG Information
•
•
DAG QoS Requests
Current Job Schedule for next k steps
•
Data Management
Information Gatherer
Sphinx Server
Requirements
Dependencies (I/O)
•
Requirements
Output
•
•
QoS Deliverable
Modified Job Schedule for next k steps
Job Submission Component (also part of Sphinx Client)
Multi-threaded
Control Process
•
Input
•
Fully planned Job:
•
•
Message Interface
•
DAG Reducer
•
Action
•
Admission Control
Planner
•
•
•
Information Gatherer
•
environment
Local I/O data (if necessary)
Submit to Condor-G/DAGMan
•
•
Data Management
Set up environment
Import input data (if necessary)
Publish to DMC (if necessary)
Run application
Export output data (if
necessary)
Publish to DMC (if necessary)
Cleanup
•
•
Submitter
Sphinx Server
Construct Job DAG
•
•
•
•
•
Prediction Engine
Data Warehouse
Job state “unsent”
Assigned gatekeeper and
jobmanager
I/O data movement locations
Job state “unfinished”
Monitor Job statistics
Output
•
•
Exit status (if available)
Final Job statistics (resource
usage)
•
•
•
•
•
CPU
Storage
Bandwidth
Entity account update
Job state “remove”
Data Management Components
Multi-threaded
Control Process
•
“User Demanded Movement”
Module
•
•
•
Message Interface
•
DAG Reducer
•
Input
•
•
•
Admission Control
•
•
•
Planner
Data Management
•
Sphinx Server
Data Movement
DMC update
Prediction Module
•
Input:
•
Information Gatherer
Available Data
Available Resources
Prediction of Data Access
Patterns
DAG (“reduced”)
Jobs (“unsent”)
Job(s) schedule for next k steps
Output
•
•
•
DMC update
“Pre-Planned Movement” Module
(most important to Sphinx)
•
Submitter
Source(s)
Destination(s)
Output
•
Prediction Engine
Data Warehouse
Input
•
Historical Data Access Patterns
Output:
• Future Data Access Patterns
Information Gathering Module
Multi-threaded
Control Process
•
Resource Property Module
•
Input
•
Message Interface
•
DAG Reducer
•
Data Warehouse
Information Gatherer
Input
•
•
Grid Monitoring systems (such
as GEMS, MonALISA, etc)
Output
•
Grid Weather Views
•
•
•
•
CPU
Storage
Bandwidth
Grid Weather Prediction Module
•
Input:
•
•
Grid Weather History
Output:
•
Sphinx Server
Work Views
Grid Weather Module
•
Submitter
Job Monitoring information
(such as Condor-G, etc)
Output
•
Planner
Data Management
Input
•
Prediction Engine
•
Resource Property views
Job Tracking Module
•
Admission Control
Output
•
•
Grid Information Systems (such
as MDS using GLUE Schema,
etc)
Grid Weather Forcast for next k
steps