Charis Quay Huei Li

Supervisor: Mary Bishai, Fermilab

Alternate Supervisor: Jean Slaughter, Yale University

Abstract: This paper concerns the author’s internship at Fermilab in the summer of 1999. Described are several tests of the CDF (Collider Detector at Fermilab) SVX (Silicon Vertex Detector) DAQ (Data Acquisition System) and their results. Some details of, and efforts to streamline, the running of the system are also included.

August 5, 1999.

1. INTRODUCTION

The Collider Detector at Fermilab is one of two detectors at Fermilab, the other being DZero. CDF is also known as BZero. This is because there are 6 places around the ring where collisions can occur, AZero ... FZero. Figures 1 and 2 show what CDF looks like.

The Silicon Vertex Detector is the innermost layer of the CDF. It is so named because it is made primarily of silicon, and because its main job has to do with vertices. The primary and secondary vertices of a particle are where it is produced and where it decays respectively. When dealing with particles with short lifetimes – such as the infamous top quark – it is crucial, and yet difficult to distinguish these vertices because they are so close together. The SVX is able to do this well because it has excellent spatial resolution, on the order of < 10 microns.

The first SVX was added to the CDF in 1992. It was then replaced with a radiation-hard (not easily damaged by radiation) SVX’. In Run II of the accelerator, scheduled to start in early 2000, the silicon vertex detector is called the SVXII. The main features that distinguish it from the earlier SVX’s are:

1. Shorter bunch spacing, i.e. 396ns, then eventually 132ns. Bunch spacing is the time between collisions of p and p bunches.
2. Dead timeless mode. In Run I, new events could not be tagged while another event was being digitised and read out. Any potentially interesting events that occurred during this ‘dead time’ were lost. In SVXII, up to 4 events can be queued at a time, greatly reducing the possibility of dead time.

When a charged particle passes through the SVX, it induces electrical signals on strips in its path. The strips are then read out by silicon chips, which sit on hybrids at either end of the ladders. Hybrids are circuits that connect multiple chips to the silicon. A hundred and eight strips, or channels, are read out by each chip. What happens after this is covered in the next section about the DAQ (Data Acquisition System).

The DAQ handles the data that comes back from the chip. Please refer to the simplified schematic on the next page.

The Trigger Supervisor, common to all of CDF, sends triggers to all the different parts of CDF when it sees an event that it thinks might be interesting. These triggers are received in the SVX by the Silicon Readout Controller (SRC), the ‘brain’ of the DAQ. It then sends commands to the VFO’s (VRB Fan Outs) and FFO’s (FIB FanOuts) which are fanned out to the VRB’s (VME Readout Buffer) and FIB’s (Fibre Interface Boards). There are 10 FIB’s/VRB’s for each of their FanOuts.

The FIB’s pass the commands on to the chips via junction boxes and port cards. There are 2 junction box-port card pairs for every FIB. Each Port Card is responsible for one end of a wedge, reading out 5 the 5 layers in 5 channels. If you are paying attention, it should make sense that this makes for 10 channels per FIB.

Data coming back from the chips goes through the FIB’s and is passed on to the VRB’s via 4 fibre optic links (G-links) per FIB-VRB pair.

All these things – FIB’s, VRB’s and the SRC – are boards that sit in slots in crates in the detector area. Each crate has a VME processor so that the Windows NT computers that we use to run the system can ‘talk’ to it via the Ethernet. In Run II, there will only be one SRC, but right now, there are several; one for each test stand and a few extra for a total of nine. They are named after Snow White, Prince Charming (often called ‘the Prince formerly known as Charming’ when he behaves badly) and the Seven Dwarves.

To understand the next section about how we debug the DAQ, you first need to know a little about the workings of the chip. The current version is 3D. (Does not stand for 3-dimensional.) When an event occurs, the aforementioned ‘electrical signal’ is first amplified and stored as some quantity of charge on a capacitor known as the integrator, which sits at the front door of the chip. The charge is then transferred to one capacitor, or ‘cell’ in a ‘pipeline’ of 47. One of the cells is used as a reference for calculating event to event and channel to channel ‘noise’, or variations.

If the Trigger Supervisor thinks that an event might be of interest, a Level One Accept (L1A) is sent to tag the corresponding cell. A pointer is set so that that cell is bypassed in subsequent ‘writings’ (transfer of charge) to the pipeline until it has been digitised and read out. It is effectively taken out of the pipeline. Up to four cells can be tagged at once.

When a PRD1 (Pipeline Readout One) signal is sent, the analog data (i.e. charge) on the capacitor is passed to the ADC (Analog to Digital Converter) for digitisation and readout. After digitisation, the cell is returned to the pipeline upon receipt of the PRD2 (Pipeline Readout Two) signal. PRD2 also resets the reference cell.

Another important signal is PARST (pronounced Pre-Amp Reset), which resets the front-end integrator. This has to be done because the integrator accumulates charge at each event, transferring the difference to the pipeline. If not reset occasionally, it will become saturated.

Now I have to confess that we don’t work with as grand a system as described in the previous section. There are several groups of people working on debugging the DAQ. Each of them works on a test stand (DAQ1 – DAQ4) consisting of close to the minimum number of boards (usually 1 or 2 of each) necessary to simulate the DAQ.

My supervisor and I are on DAQ3. For the most part, we run single chips on one port card. Sometimes, we run 2 chips on separate port cards simultaneously. Since we are not yet connected to the rest of the detector, the SRC emulates the Trigger Supervisor, sending LIA’s and L2A’s randomly. We don’t really read out ‘events’, but rather, background charge on the chip.

We employ a very simple procedure for debugging the SVX DAQ – run the system and look for bugs. When the system refuses to initialise altogether, often the culprits are trivial things like broken bonds, blown regulators or power supplies that aren’t turned on. Sometimes it turns out to be something bigger, but most of our bugs are discovered when the system is up and running.

One of the first places we usually see bugs is the histograms output by the Java program that runs the system, but we also keep our eyes open for other signs, e.g. unusual current draw from power supplies.

We then try to figure out what is causing the bug by probing signals coming from different parts of the DAQ. We look at them on the logic analyser and oscilloscope to see if they look normal and to check to see if signals are lost anywhere along the line. Then we make ‘fixes’ that we think might solve the problem. It’s like putting together pieces of a puzzle.

When we have worked out all the bugs we can find, we change some variables and run in different modes to see if the chip is affected. For example, although in theory the chip has a pipeline of 42 cells (47 – 1 reference – 4 possibly tagged), we had the Pipe Depth – the effective pipeline depth – set to 4 for a while. In one of the tests, I ran the system at varying pipe depths. Effects were observed that suggested problems in the SRC pipecap logic, which tells the SRC which cell is tagged. These were later corrected by Petar (Maksimovic) and the other SRC people.

When we think we understand what makes the chip tick at one level, we add more complexity. For example, at the moment, among other things, we are moving from running with single chips to running with multi-chip objects.

This is the main window of the DAQ–running software. We click on things in here to make the system run.

The following section describes in greater detail one particular bug – the Pre-Amp Reset (PARST) and Cell ID Problem – and how we solved it.

1. Ancient History



Figure 8: Nice Cell ID’s Figure 9: Cell ID 0. Not good.

Figure 8 shows one of the many histograms it is possible to get the Java program that runs the system to output.

The x-axis is the Cell IDs, the ‘name tags’ of the capacitors in the chip’s pipeline. The y-axis is the number of times events have been read out from that particular cell. Since the SRC is sending L1A’s (tagging cells) randomly, we would expect a more or less uniform distribution of Cell ID’s. The histogram

on the left would be considered more or less ‘normal’ or ‘good’ (which does not by any means imply that it was typical).

Now a little aside about port cards. These things come in 2 breeds, discrete and compact - also known as DPC’s and CPC’s. The major difference is that the circuit elements on the DPC are discrete, i.e. you can see the little capacitors and what not sitting on the thing, and those on the CPC are in integrated circuits, so that all you see is a box. In addition, DPC’s can only read out one ‘SVX device’ (i.e. chip or multi-chip object), while CPC’s, as advertised earlier in this report, can potentially read out 5. The DPC is just for testing and is not going to be used in Run II.

A further tangent on the use of the acronym HDI: It stands for High Density Interface, and I am told originally meant the cables that run from port card to chip, hybrid and ladder. However, it is now used variously for the cables, the channels on the port card that the cables plug into and even the SVX devices (the stuff on the port card).

Not long after CPC’s started being tested, it was observed that chips hooked up to it sometimes exhibited behaviour shown in the Figure 9, the infamous ‘Cell ID 0 problem’.

The only signals entering into the logic that determines Cell ID are PARST, LIA, PRD1, PRD2 and FECLK (Front-End Clock). We (or rather Mary and Jean) did not see how L1A and the PRD’s could cause this problem, so we started off by looking at PARST.

 

2. D V and D t

We modified the PARST signal by putting a resistor between it and ground. This serves to lower the level of PARST. We found that in the range of about 200W - 400W the Cell ID’s were uniform. Below 200W , they start to show ‘stuck-at-zero’ behaviour and above 400W , they become ‘pitchforked’.

Then we varied the level of DVDD – the voltage supplied to the digital end of the chip – trying various combinations of resistor and DVDD values. (The normal value of DVDD is 5V.) It was observed that the chip becomes unhappy when the level of PARST is not far enough below DVDD. A difference of about 1V seemed to iron out most problems.

Next, we tried putting capacitors between PARST and ground. This lengthens the rise and fall time of the signal, i.e. the time it takes for the signal to go from 10% to 90% of the maximum value and vice versa, respectively. Capacitors of > 300pF, corresponding to rise and fall times of > about 40ns also seemed to so the trick.

We swopped chips several times to make sure that this problem was not chip-specific. It wasn’t.

At this point, there was some debate about whether we should try to understand why the resistors and capacitors seem to solve the problem, or whether we should just make the fix and move on.

 

3. It’s All about Timing

On June 22, Amy Conolly at LBL (Lawrence Berkeley National Lab) observed that the Cell ID problem goes away if PARST falls when FECLK (Front–end clock) is high. Later, the chip experts informed us that this was an unexpected design flaw in the chip.

 

Figure 10: PARST Falling in FECLK. And it was Good.

Mary moved PARST fall forward until it fell during FECLK high. This is done by messing with things called ‘FIB sequences’. When the SRC sends a command, say digitise, to the FIB to pass on to the chip, the FIB translates this single command, digitise, into a ‘FIB sequence’ of more detailed commands that tells the chip exactly how to digitise. However, the Cell ID problem was still observed.

 

 

4. Solved it!

Tom Zimmerman (chip expert) suggested that PARST must still be falling during FECLK high. Although we did not quite see how this could be, Ken (Treptow, FIB expert) got the oscilloscope to trigger on PARST fall when FECLK is low, and sure enough, we found that it was happening fairly frequently. We also observed that PARST width wasn’t constant. Sometimes it was high for a long time, and sometimes only for a short time.

At Jean’s suggestion, I looked at what the other signals were doing when PARST fell during FECLK low. I noticed that PRD1’s always appeared soon after PARST fall. Ken observed that PRD1 is part of the digitise cycle (i.e. the group of commands that tells the chip to digitise) and suggested that we look in that part of the FIB sequence. We did, and found a ‘PARST stop’ command at the beginning that everyone had forgotten about. All that had to be done was to move it to within a FECLK.

Afterwards, it was discovered that a few other seemingly unrelated problems were caused by the same thing. So we killed multiple birds with one stone, and that’s always good.

This was one of our bigger bugs. The cause now seems trivial, but it took more than a few people a good amount of time to solve. However, while hunting down this bug, quite a few improvements were made to the system. For example, the FIB sequence resolution (amount by which you can move signals around) was halved. Software was developed by Mary to look at ‘what the SRC thinks the Cell ID is vs. what the chip thinks it is’ more easily. This software later came in very useful in other tests. We also discovered minor bugs, like the fact that PRD2’s during digitise cause Cell ID’s to become ‘stuck-on-63’.

There will be no conclusion to the debugging of SVX DAQ, not until Run II of the accelerator next year, and perhaps not even then. Therefore, I offer you some philosophical thoughts in lieu of a conclusion.

This summer I was exposed to many new things – Big Science, big accelerators, the real world, physics collaboration and Windows NT. I’ve also learnt a lot – Perl, how to use logic analysers and oscilloscopes, some new and very cool UNIX commands, and more than I ever wanted to know about CDF’s SVX. It’s been a great experience.

Many thanks are due to the SVX DAQ group in general and my supervisor and alternate supervisor, Mary Bishai and Jean Slaughter in particular, for their longsuffering and guidance throughout the summer. I must not forget Mary’s lifts home on rainy days, the loan of Jean’s bike and many other small things that helped make my time at Fermilab happy, productive and educational.

The moral support of the physics department at Mount Holyoke and my family is much appreciated.

I am also grateful to Elliot McCrory, Dianne Engram, Jim Davenport and the rest of the SIST committee for making this opportunity possible for the other interns and me, and for their administrative and organisational efforts.

Last but not least, thanks to God, from whom all blessings flow.

Bishai, M. et. al. An SVX3D Chip User’s Companion. Fermilab. 1999.

CDF II Collaboration. The CDF II Detector Technical Design Report. Fermilab, 1996. (96/390-E)

Likte, A. M. and Schwarz, A.S. The Silicon Microstrip Detector. Scientific American, May 1995, pp. 76-81.