Transcript The CERN LHC central timing A vertical slice Pablo Alvarez Jean-Claude Bau

The CERN LHC central timing
A vertical slice
Pablo Alvarez
Jean-Claude Bau
Stephane Deghaye
Ioan Kozsar
Julian Lewis
Javier Serrano
Talk layout
•
The CERN central timing overview
–
–
–
•
Hardware
–
–
–
–
–
–
–
–
–
•
Accelerator complex, LHC injector chain
Cycles and Super-cycles
The LHC beam, UTC machine, Synchronization
Central timing layout
Reflective memory
Event frames and their distribution
Clocks and calibration
The Controls Timing Receiver module CTR
The Multitasking Timing generator module MTT
Safe Parameters Verification
The CTR hardware support for safety critical situations
Postmortem behavior
Software
–
–
–
–
LSA and FESA requirements
Telegrams and events
MTT tasks and event tables
The table compiler, assembler and task loader
CERN accelerator network
sequenced by central timing generator
LHC….….
….….
Experimental
area
SPS
Experimental
area
CPS
Experimental
Area
PSB
The LHC Proton Injector Chain
Strongly time coupled
R1
LHC
CNGS
TI8
Dump
TCLP
Linac
PSB
CPS
SPS
SPS
Dump
TI2
Dump
D3
Dump
R2
LHC
Beam Coordination Diagram BCD editor
The LHC Beam
The LHC timing is only
coupled by extraction
start-ramp
event
LHC Injection plateaux
Injection
Injection
LSA Beam request:
RF bucket
Ring
CPS batches
Extraction
Extraction
Extraction
Forewarning
Extraction
Forewarning
SPS injection plateaux
SPS Cycle for the LHC
CPS Batch 1
PSB1 PSB1
CPS Batch 2
PSB2 PSB2
CPS Batch 3
PSB3 PSB3
CPS Batch 4
PSB4 PSB4
RF Synchronization
SB-Frev jumps to
the next LHC TB-Frev
P
ABORT GA
Ba
tc
h
Jump
LHC TBF
LHC TBF
SPS SBF
SPS SBF
1
Filling
LHC Ring 1
SPS
Target bucket
revolution frequency
SPS RF
BA3
R1 Bucket from
telegram
Target bucket revolution
frequency
LHC RF
SR4
4
Ba
tch
1
2
Target Bucket
FREV
Batch 3
Jump
ch
Bat
RF frequency is
nearly fixed. Just
jump to the next
bucket. This can
only happen if the
SPS is empty
Ba
tc
h2
Ba
tch
CPS
PSB
Re-phase with
beam ~50ms
In
Out
Re-phase with
beam ~20ms
Batch 3
+
-
VCO
CPS Source
bucket FREV
Batch 5
Injection Kicker
CPS RF
RF
Kicker tail is to long to allow
inserting batch 4 between batch
3 and batch 5
Next SPS target
bucket FREV
Time
difference
to digital
convertor
Batch 4
Reference
frequency
Rephase with beam: The time
difference between the target
bucket FREV and the source
bucket FREV is measured by a
TDC. The TDC value then
changes the RF frequency so
that they come in line.
This takes time to complete, and
disturbs the orbit.
Batch number to Bucket number
conversion
Bucket = K + h/n
h = 35640 RF Buckets
n = 12 Batches
K = 121 Approx (Offset)
CBCM
Sequence
Manager
LSA
Sequencer
“LSA”
TI8/TI2
Dump
Master
ON/OFF
LHC
User
LHC Fill Requests:
Bucket
Ring
Batches
Request
TI8/TI2
Dump
Request
LHC User
The
Central Timing
layout
CS-CCR-CTMGW
FESA API
Normal
Spare
LHC Fill Requests:
Bucket
Ring
Batches
Reflective
Memory
Link
LIC
Sequence
Inhibits
Requests
Interlocks
SIS TI8/TI2
& SPS Dump
Inhibits
CBCM
Controlling
Injector Chain
SPS destination request
R1,R2
LHC Timing
LHC
Central Timing
LSA Master
CPS Batch Request 1,2,3,4
SEX.FW1K
SPS.COMLN.LSA_ALW
LIC Timing
CTR
HIX.FW1K
LSA changes Allowed
Reflective memory
LHC Timing generator A
Identical
except
ID event
LHC Timing generator B
2Gbit/S
Token ring
VMIACC-5595
Single Mode
Hub
Reflective memory:
Protects token ring
A and B must always be
in the same state.
If no restrictions for
switch over
64Mbyte
VMIPMC-5565
Reflective memory
LHC Gateway
Implements FESA API
Reflective Memory XMEM
Daemon
• DMA manager
• Topology management
• Memory initialization
Library
Configuration
data
Synchronization
Library calls
Driver
Token
Ring
Shared memory
segment
XMEM
segment
XMEM
segment
DMA
Shared memory
segment
XMEM
segment
Shared memory
segment
XMEM
segment
Shared memory
segment
Client RT task
Event frames
•
•
•
•
Timing frames are encoded at 500KBit/S using a 1MHz clock.
Each frame carries 32 data bits.
One frame transmitted each 128us, 8 per millisecond.
Frames are broken into bit fields
–
–
–
–
4 Bits Accelerator
4 Bits frame Type
8 Bits Code
16 Bits Payload
[A]
A
[T]
[CODE]
[PAYLOAD]
T
CODE
PAYLOAD
• Some frame are recognized by the hardware and cause special treatment
–
–
–
–
Two UTC frames carry the time of day in their payload
Millisecond frames are always sent in phase with the PPS
Telegram frames are stored in double buffers
Event frames cause counters to be loaded and triggered and may produce bus
interrupts
Millisecond
Millisecond
Clock Source &
Calibration
Symmetricom
CS4000
portable
Atomic Clock
GPS
GPS
One pulse
per Second
PLL
One pulse
per Second
Symmetricom
RS485 XLI
Timing
CERN
UTC
Time
Basic Period
1200/900/600 ms
Synchronization
module in each
timing generator
crate
Phase locked
10MHz
Advanced (100us)
One pulse
per Second
Synchronized 1KHz
(slow timing clock)
40MHz PLL
Phase locked
10MHz
Control System
Timing receiver
PPS
Phase looked
40 MHz Event
encoding clock
10 MHz
Delay
CTRP
1 KHz
25ns steps
Event
tables
Set once on startup
& on Leap Seconds
UTC time (NTP or GPS)
40MHz
External events
MTT
Multitask
Timing
Generator
MTT
RS485
Timing
CERN
UTC
Time
Controls Timing Receiver CTR
Timing
22Bit 50MHz
Counters
Content Addressed
Memory - Triggering
Bus
Interrupt
Hybrid
PLL
40MHz
Delay
Lookup
Load
Trigger
Frame
Decoder
UTC
TGM
TGM
Counter
Configurations
Loader
Output
•Start
•Previous
•External
•Event
•Clock
•Previous
•40MHz harmonic
•External
•Modes
•Once
•Multi-pulse
•Burst
•Trigger
•Event frame
•Wildcard
•Telegram
•Action
•Output
•Bus interrupt
•Output
•TTL/TTL bar
•Pulse width
CTRV – CTRI – CTRP formats
Timing receiver function
• Counting Sequence
Cycle A
Loading Event
C0
Telegram Ready
Telegram
C1200
Counting
Counter Start
Counter Load
•Delay Cycle A
•Clock Cycle A
•Enable Cycle A
Counter Output
Telegram Ready
Acquisition
Performed by
RT task
Multitask Timing Generator MTT
•
Implements general purpose CPU with memory mapped IO registers
–
–
–
•
Op-codes are triadic: AND REG,REG,REG – AND 0x7,VMEP2,TMP
–
–
–
–
–
•
32 local registers per task
218 Global registers
6 Memory mapped IO registers
Host processor access to all registers
Tasks defined from host via Task Control Block
–
–
–
•
Arithmetic and logical
Move indexed, literal, register
Wait value, relative
Conditional branch
Interrupt host
Hardware multitasking for 16 tasks based on work space pointer, all available to host
–
–
–
•
•
Event Out register
VME P2 In register
Free running millisecond register
PC
PC Offset
Processor Status Word
Command and status registers allow host access to running tasks
MTT hardware module
SYNC
PPS
40MHz
FPGA (Spartan2 600E)
V
M
E
VME BUS
External
Events
I
n
t
e
r
f
a
c
e
UTC
SERIALIZER
CPU
V
M
E
P
2
See: The LHC central timing hardware implementation
P. Alvarez, J. Lewis, J. Serrano CERN, Geneva, Switzerland
This conference
Timing
Signal
Simplified Layout of the Safe Machine Parameters Distribution
SMP @ 10Hz
16Bit 1010
(Flags, E & Int.)
Events, UTC, &
Telegrams
(including SMP)
Energy A
Flags TTL
Hw Output
EXP
BEM
@1kHz-24Bit 108
BEM
Energy B
BCT “A”
CTRx
CTRx
CTRx
CTRx
LHC
Timing
Generator
Safe Machine
Parameters
Controller
for LHC
(CTRV)
BCT “B”
B.P. 23rd Aug.07
EXP
Reads status
LHC Beam
Sequencer
main Safe Machine Parameters for LHC:
- Energy
- Safe Beam Flags
- Movable Devices Allowed Flag
- Safe Injection Flags (new)
BLM
BLM
BLM
Timing
Network
BIS
Management
Critical
Settings
Line
driver
CTRV
CTRV
CTRV
I_beam1 & 2
Reads thresholds
If length > 5m
CTRV
CTRV
CTRV
CTRV
CTRV
Kickers
Kickers
BIS
BIS
BIS
CTRV Timing receivers Safe Parameters
HX.SBF1 & Safe
Load
Out
Beam Energy is available
directly on P2 serially
encoded
Delay=1
ms
GMT
Start
Clock
Divide
Out
Delay=40
Start
Clock
40MHZ
1MHz
Pulse Stretcher
Safe Beam Flag
(Level)
Safe
Stop
Dangerous
HX.SBF1 & Safe
ms
Load
Out
Delay=10^6
Start
Clock
SBF1/2 sent at 10 Hz
Postmortem Event suppression
Two counters are used in the CTR, one per Beam-Permit-Flag (BPF)
Each counter clock is connected to one of the BPF flags
The "Disable Post-Mortem Ring 1" event disables the counter connected to BPF-1
The "Enable Post-Mortem 1" event enables the counter connected to BPF-1
When the counter is disabled and the BPF goes down nothing happens
When its enabled the counter makes an output triggering the PM event
It will be sent twice if both counters are enabled and both rings are dumped
CTR
BPF1
CLK
Delay=1
Disabled
CTG-MTT
Disabled
Warn-Inject
VME/P2
BPF2
CLK
Delay=1
Enabled
Enabled
PM-1
Suppress
Table
Loads
LSA
LHC GMT
Disable-1, Enable-1
Dump-1
PM
MTT External Events Task
strp:
cont:
tdmp1:
tdmp2:
tinj:
tpm1:
tpm2:
% Start program and interrupt survey task
movv
TskStsRUNNING
LRegTASK_STATUS
movv
ConsNOT_SET
LRegParRUN_COUNT
int
2
% Say we are running
% Run forever
% Notify survey we are running
% Wait for the VME P2 bits and send out events accordingly
worv
ConsVMEP2_BITS
VMEP2
movr
VMEP2
RegVmeP2
% Wait for VME P2 bits
% Copy reg and clear bits
% Test
andv
beq
movv
movv
for dump ring 1
ConsHX_DMPD1_BIT
tdmp2
ConsHX_DMPD1
ConsHX_ENBPM1
LRegTEMP
%
%
%
%
Test dump 1 bit
Go check for dump 2 bit
Send dump 1
Re-enable after a dump
% Test
andv
beq
movv
movv
for dump ring 2
ConsHX_DMPD2_BIT
tinj
ConsHX_DMPD2
ConsHX_ENBPM2
LRegTEMP
%
%
%
%
Test dump 2 bit
Go injection warning bit
Send dump 2
Re-enable after a dump
% Test
andv
beq
movv
for LHC injection
ConsHIX_FW_BIT
RegVmeP2
tpm1
ConsHIX_FW
EVOUT
LRegTEMP
% Test inject bit
% Go check for PM ring 1
% Send injection forewarning
% Test
andv
beq
movv
for post mortem bit 1
ConsHX_PM1_BIT
RegVmeP2
tpm2
ConsHX_PM1
EVOUT
LRegTEMP
% Test PM ring 1 bit
% Go check for PM ring 2
% Send PM-1 trigger
% Test
andv
beq
movv
for post mortem bit 2
ConsHX_PM2_BIT
RegVmeP2
cont
ConsHX_PM1
EVOUT
LRegTEMP
% Test PM ring 2 bit
% Go check for PM ring 2
% Send PM-1 trigger (not PM-2)
jmp
cont
RegVmeP2
EVOUT
EVOUT
RegVmeP2
EVOUT
EVOUT
% Go wait for next P2 interrupt
LSA and FESA
• The FESA API is implemented on the LHC timing
gateway
• Accesses timing generators across reflective memory
• Implements
–
–
–
–
–
–
–
Load or Unload event table
Get running tables list
Set event table run count and synchronization event
Stop or Abort event table
Set telegram parameters
Send event
Read status of tasks and MTT module
Telegrams and events
• The telegram is a set of parameters that
describes the machine state
• Telegrams are distributed each second
• Each telegram parameter is shadowed by an
event
• Real time tasks subscribe to events and read the
telegram to know the current machine state
MTT Telegram Task
strp:
cont:
next:
% Start program and interrupt survey task
movv
TskStsRUNNING
LRegTASK_STATUS
movv
ConsNOT_SET
LRegParRUN_COUNT
int
2
% Say we are running
% Run forever
% Notify survey we are running
% Wait
weqv
addv
movv
andv
lorv
movv
movv
%
%
%
%
%
%
%
Wait for the PPS
Next second
Reset the millisecond modulo
Basic period in telegram
Send BPNM event
Index points to LHC telegram
First group in telegram
% For each telegram group send out the frame
wrlv
10
MSFR
lsv
16
RegGrNum
RegEvFrm
lorv
ConsTgmFRAME
RegEvFrm
RegEvFrm
movir RegGrVal
andv
0xffff
RegGrVal
RegGrVal
lorr
RegGrVal
RegEvFrm
EVOUT
addv
1
RegGrNum
RegGrNum
addv
1
INDEX
INDEX
subv
ConsTgmGROUPS
RegGrNum
LRegTEMP
ble
next
%
%
%
%
%
%
%
%
%
%
Wait 10 ms to spread telegram events out
Shift left to form frame group number
Or in the telegram header frame
Get the group value
Mask off the top 16 bits
Senf frame and payload
Next group number
Next group value index
Test count against group number
Still groups to process ?
wrlv
movv
wrlv
lorv
weqv
movv
1
ConsLHC_CID
1
ConsMASTER
ConsC_MINUS1
ConsTgmREADY
%
%
%
%
%
%
Load spreader
Set the LHC cable ID
Load spreader
Send out Host ID A/B
Wait for C-1
Send ready telegram
jmp
cont
fot Sync (PPS) and increment BP and zero ms
0
TSYNC
1
GRegBASIC_PERIOD GRegBASIC_PERIOD
1
MSMR
0xFFFF
GRegBASIC_PERIOD GRegTgmBPNM
ConsFrmBPNM
GRegTgmBPNM
EVOUT
GAdrTELEGRAM
INDEX
1
RegGrNum
MSFR
EVOUT
MSFR
RegIdHst
TSYNC
EVOUT
EVOUT
% Wait for start of next basic period
Event tables
• Event tables are ASCII strings sent from LSA to the
LHC timing gateway
• Each entry in the table contains
– The name of the event to send “HX.START-RAMP”
– The relative time to wait in seconds before sending the event
“1.01”
– The payload the event will carry “0x0005”
– A carriage-return <CRTN> terminates the entry
• The table run count and synchronization event will be
filled in later when the table is to be executed
Event Table Processing
LSA
Event HX.Start-Ramp 1.01 0x5
Table HX.Start-Freq 0 0x05
FESA API
Event Table
template
Table Compiler
waitr MSFR,1001
0x14020005 EVOUT
Translation movv
movv 0x14030005 EVOUT
Translate
and Merge
Table Task
Assemble
task
Assembler
Data
Object Code
Process
Garbage Collect
Load object
Initialize TCB
Run task
Task
Loader
Reflective
Memory
Hot Standby
MTT
Program
Memory
Event Table processing
• The table-compiler output assembler code containing wait and send
operations corresponding to the table entries
– WAITR MSFR 1001
– MOVE 0x14020005 EVOUT
• The table-compile then builds a task by including the above output into an
event table task template
• The assembler takes the above code and assembles it into position
independent object code
• The table loader locates free program memory (it may need to do garbage
collection) and loads the object into it. It then initializes a TCB and starts the
task running.
• The task runs and sets its status to “Waiting-run-parameters” and waits for
LSA to set them
– Positive Run-Count (Or run forever)
– Synchronization event (Or ignore)
MTT Event Table Template
strp:
wrnc:
loop:
sevt:
% Initialize status
movv
TskStsWAITING_PARS LRegTASK_STATUS
movv
0
LRegParRUN_COUNT
movv
0
LRegParSTART_EVENT
int
1
%
%
%
%
% Wait for non zero run count
worv
ConsALL_SET
LRegParRUN_COUNT
movv
TskStsWAITING_TRIG LRegTASK_STATUS
% Wait till run count not zero
% Waiting for trigger
% Wait for trigger event and send events
movr
LRegParSTART_EVENT LRegTEMP
beq
sevt
weqr
LRegParSTART_EVENT EVOUT
% Zero start event ?
% Not set
% Wait for synchro
movv
% Say we are running
%
%
%
%
%
TskStsRUNNING
LRegTASK_STATUS
Waiting for parameters
Kill run count
Zero start event
Notify survey task we are waiting
Send the events out. Here we include the LSA tasks event list.
This is just a list of...
WAIRL <time> MSFR
MOVV <event> EVOUT
that has been generated from the LSEQ event table
#include <task_events.ass>
tcnt:
endp:
% Test the run count count
subv
ConsNOT_SET
LRegParRUN_COUNT LRegTEMP
%
beq
loop
%
subv
1
LRegParRUN_COUNT LRegParRUN_COUNT %
bgt
loop
%
% End of program
jmp
strp
Run forever ?
Yes keep sending events
Decrement run count
Not zero yet, keep looping
% Go wait for the next run