Transcript Document 7566808

ALICE Detector Data Link (DDL)
and it’s use in STAR TOF
J. Schambach
DDL
1
Presentation Outline
•
•
•
•
•
•
Hardware Overview
DDL Protocol / Transactions
RORC
DDL in TOF
Software
Demo
DDL
2
Readout system
Front-end electronics
Source
Interface
Unit
Destination
Interface
Unit
Read
Out
Receiver
Card
DDL SIU
Optical Fibre Detector
~200 meters Data
Link
DDL DIU
RORC
PC
Data Acquisition
PC
DDL
3
DDL architecture
• Source Interface Unit (SIU) (1)
– Interface to the Front-end Electronics (2)
• Destination Interface Unit (DIU) (3)
– Interface to the Readout Receiver Card (4)
• Full duplex optical link (5)
– Multimode optical cable of up to 200 m
1
3
5
2
4
DDL
4
DDL hardware
DDL
5
SIU Physical Layout
DDL
6
DDL interfaces
• SIU-FEE interface
–
–
–
–
–
3.3V (LVTTL) interface
32-bit wide half-duplex data bus (bi-directional bus)
Bi-directional flow control
User defined clock (synchronous interface)
JTAG interface
• DIU-RORC interface
–
–
–
–
3.3V (LVTTL) interface
32-bit wide full-duplex data bus
Bi-directional flow control
User defined clock (synchronous interface)
DDL
7
DDL Interface signals
DDL
8
SIU Connector Pinout
DDL
9
SIU-FEE interface
fbD(31..0)
fbTEN_N
fbCTRL_N
fiDIR
fiBEN_N
fiLF_N
foBSY_N
foCLK
TAP_TCK
TAP_TDI
TAP_TDO
TAP_TMS
TAP_TRST
- data lines
- transfer enable
- CONTROL qualifier
- bus direction
- bus enable
- link full
- front-end busy
- interface clock
- JTAG clock
- JTAG data in
- JTAG data out
- JTAG mode select
- JTAG reset
DDL
(bi-directional)
(bi-directional)
(bi-directional)
(FEE input)
(FEE input)
(FEE input)
(SIU input)
(SIU input)
(FEE input)
(FEE input)
(SIU input)
(FEE input)
(FEE input)
10
Link management
DDL
11
DDL Configurations
DDL
12
Front-end Commands
DDL
13
Front-end Status Words
Data Transmission Status Word (DTSTW), produced by SIU:
DDL
14
Front-end control
RORC
FEE
command
DIU
SIU
Online
Online
FEE
foCLK
fiBEN_N
FEE
control
fiDIR
fbD
FECTRL
fbTEN_N
Report
fbCTRL_N
DDL
15
Front-end status read
RORC
FEE
command
DIU
SIU
Online
Online
FEE
foCLK
foCLK
fiBEN_N
fiBEN_N
fiDIR
fiDIR
fbDfbD
FESTRD FESTW
HiZ
HiZ
HiZ
HiZ
HiZ
HiZ
fbCTRL_N
fbCTRL_N
HiZ
HiZ
fbTEN_N
fbTEN_N
Status and
report
HiZ
DDL
FEE status
read
16
Event read
foCLK
fiBEN_N
foCLK
foCLK
fiDIR
fiBEN_N
fiBEN_N
fbD
fbTEN_N
HiZ
Dn-1fiDIR
fiDIR
HiZ
D0
Dn
fbD
fbD
HiZ
FESTW
D1
RDYRX
D2
EOBTR
min. 16 cycles
D4
HiZ
HiZ
HiZ
HiZ
fbCTRL_N HiZ
fbTEN_N
fbTEN_N
HiZ
HiZ
fiLF_N
fbCTRL_N
fbCTRL_N HiZ
HiZ
DDL
D5
D0
17
Block write
RORC
DIU
SIU
Online
Online
FEE
FEE command
foCLK
fiBEN_N
Block data
STBWR
fiDIR
Report fbD
STBWR
Dn-1
Dn
D0
D1
D2
fbTEN_N
EOBTR
D4
D5
FEE data
Flow control
FEE command fbCTRL_N
foBSY_N
EOBTR
Report
DDL
18
Block read
RORC
DIU
SIU
Online
Online
FEE
FEE command
foCLK
foCLK
foCLK
fiBEN_N
fiDIR
Report fbD
Block data fbTEN_N
FEE command fbCTRL_N
fiLF_N
STBRD
fiBEN_N
fiBEN_N
fiDIR
fiDIR
HiZ
Dn-1
HiZ
HiZ
D0
Dn
fbD
fbD
HiZ
FESTW
D1
STBRD
D2
EOBTR
D4
HiZ
HiZ
HiZ
fbCTRL_N
fbCTRL_N HiZ
HiZ
fbTEN_N
fbTEN_N
D5
HiZ
FEE data
HiZ
Flow control
HiZ
EOBTR
Report
DDL
19
RORC features
• Interface between the DIU and PCI local bus
– pRORC: 32 bit/33 MHz PCI version, max. throughput 132 MB/s
– D-RORC: 64 bit/66 MHz PCI version, max. throughput 528 MB/s
• PCI master capability, data push architecture
– Autonomous operation with little software assistance
– Supports multi-paged memory management
• Direct data transfer to the PC memory
– No local memory on the board
– Small elasticity buffers between different clock domains
• Built-in test capability
– Internal pattern generator can produce formatted data
DDL
20
D-RORC Hardware
D-RORC with plug-in DIU
• to read out single DDL channel
• to support the tests of FEE readout sytems
D-RORC with integrated DIU ports
• to read out two DDL channels
• to support integration with the HLT system
DDL
21
RORC Roadmap
• pRORC:
32-bit, 33 MHz (PCI I/F by ASIC)
– well adapted to the prototype version of the DDL
– can be used for the new version of the DDL (adapters)
– already used by several test beams (SDD, HMPID)
• D-RORC I:
64-bit, 66 MHz (PCI I/F by IP core)
– well adapted to the new version of the DDL
– will integrate two DIU functions on-board
– will support the DAQ/HLT interface
• D-RORC II:
64-bit, 66/133 MHz (PCI-X I/F by IP core)
– to avoid compatibility issues
– to follow the evolution of the PCs
DDL
22
Hardware Architecture
Configuring
P12
Busy
I/F
APEX
FPGA
P14
Media 250 MB/s
I/F 2
P11
Media 250 MB/s
I/F 1
Conf.
Flash
CMC I/F
P13
LVDS I/F
Optical I/F
JTAG JTAG
528 MB/s
64-bit/66 MHz, PCI/PCI-X
DDL
23
Firmware Architecture
DIU or Media I/F
DIU I/F or DIU core
TAF – Transmit Address FIFO
length
address
length
Transmit
DMA
Receive
DMA
TX FIFO
Control
registers
address
RX FIFO
TAF
RAF – Receive Address FIFO
RAF
Slave I/F
DMA control
Master I/F
PCI core
(64-bit master, memory mapped)
PCI bus
DDL
24
The Free FIFO
PRORC
Free FIFO
PC memory bank
Firmware
page address
page address
page address
PC CPU
readout
Allocation of free pages
DDL
25
DDL
PRORC
Direct Memory Access
PC memory bank
Firmware
PC CPU
No involvement
DDL
26
DDL
The Ready FIFO
PRORC
PC memory bank
Firmware
Ready FIFO
page status
length
page status
page status
length
length
PC CPU
readout
Delivery of filled pages
DDL
27
Firmware
RORC
Initialize internal data structures
stop flag = ON
acknowledge reset done
Firmware
Free FIFO
start
address
block
size
Ready FIFO
Index of
Ready FIFO
data
length
transfer
status
Transfer status possible values:
ffffffff unloaded (set by sw)
00000000 loaded, no DTSTW (set by fw)
else
loaded, DTSTW (set by fw)
Y
stop flag ON?
N
size = 0
Y
Software
Free FIFO empty?
N
Load Free FIFO
Initialize Ready FIFO
Load configuration registers
Clear stop flag
N
Pop descriptor from Free FIFO
Move data into buffer until DTSTW
or up to buffer size
Update size
N
Event ready?
Y
Pull data length and transfer status
from Ready FIFO
Push pending free blocks
on Free FIFO (if any)
Release event
 block:
Reset entries of Ready FIFO
Push address & size on Free FIFO
(if space available in Free FIFO)
DTSTW?
Event ready test:
Push data length in Ready FIFO
Push transfer status = 00000000
Optional: update Ready FIFO in memory
1) SOFTWARE
start = curr;
do {
while (Ready FIFO [curr].status == 0 ) curr = NEXT(curr);
} while (Ready FIFO [curr].status == 0xffffffff ||
Ready FIFO [curr].status == 0)
end = curr;
curr = NEXT( curr );
Blocks from start to end are now available
2) HARDWARE
delivery of signal, enabling of semaphore or setting of flag
DDL
N
Y
size > max size?
Y
Throw away rest of data
do not increment block length
until DTSTW received
Mark DTSTW “OVERRUN” bit
Push data length in Ready FIFO
Push transfer status = DTSTW
Update Ready FIFO in memory
Deliver “done” interrupt
Optional: set stop flag on error
28
Test equipments
• Front-end Emulator Interface Card (FEIC)
–
–
–
–
–
Fully functional hardware to emulate the detector front-ends
Formatted data block generation
Internal (free running) or external (pulse) triggering capabilities
Adjustable parameters (using front-end control)
Operates at the nominal speed of the DDL
• Source Interface Unit Simulator (SIMU)
– Simulates the behavior of the DDL without any additional hardware
– Eases the development and the hardware debugging
– Size is similar to the real SIU
DDL
29
Bandwidth: D-RORC to D-RORC
Testing the transfer between two D-RORC cards
• rorc_send –g ... (pattern generator)
• rorc_send ...
(DMA from)
DDL
30
TOF Essential Model Level 1
L0 Trigger
TCD & Busy
Logic
Tray
Data, Clk, Ctrl
32 RPC
With 6 channels
each
On-Tray Electronics
CANbus
CANbus
30
Trays
THUB
Slave Clock
Dual Fiber
DAQ
Dual Fiber
DAQ
L0 Trigger
Clk, Ctrl
CANbus
Control PC
with
CANBus I/F
Tray
32 RPC
With 6 channels
each
Data, Clk, Ctrl
On-Tray Electronics
CANbus
THUB
Master Clock
30
Trays
Clk, Ctrl
Clk, Ctrl
CANbus
To other
THUB’s
Start Detector
with
24 Scintillators &
24 PMTs
Start Detector
Electronics
Data, Clk, Ctrl
DDL
One
Detector Side
31
TOF Essential Model Level 2
TOF Tray
DIFFERENTIAL DATA AND CLOCK
L0 Trigger
MULTIPLICITY
TRAY CAN BUS
COPPER:
DATA, SAMPLE CLOCK, RESET
TRIGGER STROBE & DATA
TDIG
COMMANDS
RHIC CLOCK
TCD
TDIG
COPPER LINKS
TO 29 TRAYS
TCPU
TINO
48
CHAN
TINO
48
CHAN
THUB
SIU
48
CHAN
TOP LEVEL CAN BUS
FIBER
DAQ
Reset & Clock
From other THUBs
4
4
Reset & Clock
To other THUBs
MRPC
48 CHAN
CAN BUS TO
29 TRAYS
Start Detector
Copper:
Data, Sample Clk, Reset
Trg Strobe & Data
MRPC
Diff. Data & Clk
PMTs
TPMT
TSTD
TCPU
Local CANbus
DDL
32
Run 5 TCPU
ICD2
header
Power
Leds,
switches,
temperature
monitor
ROM
TOP LEVEL
CAN BUS I/F
TRAY LEVEL
CAN BUS I/F
MCU
Ext clock
output
JTAG for
config
Clock select
and distribute
JTAG
multiplexer
Local osc
Ext clock input
To TDIG cables
Config eprom
TCD I/F
Ext ‘reset’
output
TDIG CABLE I/F
Leds and
switches
PLD
ALTERA
STRATIX
EP1S10
Jtag for
codetap
TDIG CABLE I/F
Test header
TDIG CABLE I/F
TDIG CABLE I/F
DDL-SIU
DDL
33
DDL Interface Implementation
DDL
34
TCPU Firmware Data Path
TCD CABLE
TCD I/F
TDIG CABLE
TDIG CABLE
TDIG CABLE
TDIG CABLE
FIFO
64 X 32
FIFO
2048 X
32
FIBER
4 TO 32
BIT
DEMUX
FIFO
256 X 32
4 TO 32
BIT
DEMUX
FIFO
4 TO 32
BIT
DEMUX
FIFO
4 TO 32
BIT
DEMUX
FIFO
MUX
5:1
MUX
FIFO
DDL
SIU
L2
Trigger
Data
FIFO
2048 X
32
DDL
CAN BUS
MCU
I/F
MCU
35
DDL Software
• PCI driver
• API routines for DATE
• Executable utility programs for test and stand-alone (= without FEE or
DATE) use of the DDL, such as
– Reset the RORC, DIU or SIU
– Display the status of the RORC, DIU or SIU
– Test the functionality and measure the performance of the whole DDL and RORC
system
DDL
36
Installation of the utilities
•
•
•
•
•
Utilities are in the directory
./rorc/Linux/
Linux kernel version: 2.4
Driver module must be inserted. As root type:
/sbin/insmod ./Linux/rorc_driver.o
or insert a similar line into /etc/rc.d/rc.local
Checked if physmem and RORC driver are loaded:
cat /proc/modules
Check if RORC card is plugged:
rorc_find
DDL
37
Identifying the RORC card
$ /sbin/lspci
00:00.0 Host bridge: Intel Corp. 82840 840 (Carmel) Chipset Host Bridge (Hub A) (rev 01)
00:01.0 PCI bridge: Intel Corp. 82840 840 (Carmel) Chipset AGP Bridge (rev 01)
00:02.0 PCI bridge: Intel Corp. 82840 840 (Carmel) Chipset PCI Bridge (Hub B) (rev 01)
00:1e.0 PCI bridge: Intel Corp. 82801AA PCI Bridge (rev 02)
00:1f.0 ISA bridge: Intel Corp. 82801AA ISA Bridge (LPC) (rev 02)
00:1f.1 IDE interface: Intel Corp. 82801AA IDE (rev 02)
00:1f.2 USB Controller: Intel Corp. 82801AA USB (rev 02)
00:1f.3 SMBus: Intel Corp. 82801AA SMBus (rev 02)
01:05.0 Multimedia audio controller: Cirrus Logic CS 4614/22/24 (rev 01)
01:06.0 Network controller: CERN/ECP/EDU: Unknown device 0033 (rev 01)
01:08.0 Ethernet controller: Accton Technology Corporation SMC2-1211TX (rev 10)
02:1f.0 PCI bridge: Intel Corp. 82806AA PCI64 Hub PCI Bridge (rev 02)
03:00.0 PIC: Intel Corp. 82806AA PCI64 Hub Advanced Programmable Interrupt Controller (rev 01)
03:04.0 Network controller: CERN/ECP/EDU: Unknown device 0033 (rev 02)
03:09.0 SCSI storage controller: Adaptec AIC-7892P U160/m (rev 02)
04:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200 AGP (rev 03)
DDL
38
Program ‘rorc_find’
List all plugged and presently not used RORC devices together with their version and serial
numbers.
rorc_find
The following device(s) found:
------------------------------------------------------------------------------Minor Channel Device type and HW identification
(RORC’s FW version)
------------------------------------------------------------------------------0
0
D-RORC
new DIU
0
1
(FW: 1v35 of July 2 2003)
DDL card 2v0 LD: 20K60E
SP: 2125 Mbps S/N: 00102
D-RORC
no DIU
1
0
pRORC
new DIU
pRORC 1v1 S/N: 00101
DDL card 2v0 LD: 20K60E
(FW: 1v75 of June 6 2003)
SP: 2125 Mbps S/N: 00118
------------------------------------------------------------------------------3 RORC channel(s) not in use was found.
RORC driver reported 2 RORC device(s).
DDL
39
rorc_reset
Initialize the RORC card and the DDL link
rorc_reset [-{M|m} <minor> | 0]
[-{D|d|B|b|S|s|F|f|O|o|E|e|C|c}]
Where
-D or –d reset the DIU
-B or –b reset both RORC and DIU
-S or -s reset SIU
-F or –f clear Free FIFO
-O or –o clear RORC’s other FIFOs
-E or –e clear RORC error bits
-C or –c clear RORC’s byte counter
No option reset RORC
DDL
40
rorc_id
Display RORC’s hardware and software identification words
rorc_id [-{M|m} <minor> | 0] [-{D|d}{S|s}]
[-{T|t} <time-out> | 100000]
Where
-D or –d display DIU’s firmware id as well
-S or –s display SIU’s firmware id ad well
time-out
time-out value for DIU or SIU reply
e.g.:
rorc_id
RORC driver version: 4.2
RORC revision id: 1
Hardware identity word of the RORC: pRORC 1v1 S/N:00103`, i.e.
Version: 1.1, S/N: 00103.
Firmware identity word of the RORC: 0x02190550, i.e.
Version: 1.72 Release date : October 16 2002
Free FIFO size: 128 entries
DDL
41
rorc_send_command
Send a DDL command and receive the reply
rorc_send_command [-{M|m} <minor> | 0] -{C|c} <command>
[-{T|t} <time-out> | 100000]
[-{V|v} <diu_version> | 2]
Where
command a hexadecimal number starting with “0x”, or
an ASCII mnemonic of a DLL or pRORC command, e.g.:
LBON
RORC loop-back on
RDYRX
ready to receive message to the SIU
EOBTR
end of block transfer message to the SIU
(see the sw manual or the program help for command codes and format)
time-out
time-out value for RORC, DIU, SIU or Front-End reply
diu_version
1 for prototype, 2 for final version.
DDL
42
rorc_receive
Send and receive data to the Front-End. The most important options:
rorc_receive [-{M|m} <minor>|0] [-{G|g}] [-{Y|y}] [-{Z|z}]
[-{X|x} <check_level>] [-{K|k} <output file>]
[-{E|e} <events>] [-{P|p} <pattern>|0]
[-{I|i} <init word>|0] [-Q <GBytes>|1]
Where
(from version 4.2: rorc_receive)
-G
send data using RORC’s data generator
-Y
do not loop-back generated data but send it via the link
-Z
do not send RDYRX and EOBTR commands
output file
dump the receives the data into this file without checking it
check_level
0: do not check the received data, 1: check only the first word,
2: not the first word, 3: check the whole event
events
# events to send
pattern event pattern to send or receive, it could be: c, a, 0, 1, I, d
init word
the first word of each event’s payload (after the event serial number)
GBytes
display the number of received bytes after each received Gbytes data
(see the sw manual for further options)
DDL
43
feic.menu
Check and set the Front-End Emulator Interface Card
feic.menu [-{M|m} <minor>|0]
This script calls the rorc_send_command program several times to check the FEIC setting or to set new
parameters. The following features of the FEIC’s data generator can be modified and displayed:
data pattern
event length
trigger mode
seed
alternating, flying 0 or 1, incrementing, decrementing data
16, 32, 64, …., 256 Kwords
push button, ext. trig., 16 or 128 clocks after each event,
every 10
or 100 ms
the seed value for random data length
See the description of rorcArmFeic routine in the sw manual for further details.
DDL
44
Other utilities
program name
program function
diu_id
siu_id
DIU hw and fw identification
SIU hw and fw identification
show RORC status
show RORC registers
ask and display DIU status
ask and display SIU satatus
download data from the PC to
the FEE
download JTAG data
rorc_status
rorc_reg
diu_status
siu_status
rorc_send
rorc_send_jtag
DDL
45