EMC
Electronics Work Summary

05/06/08: ADB 3-1-0 was masked out.

After a source calibration on May 2nd ADB 3-1-0 was seen to be essentially dead. A new pedestal calibration confirmed this and the 12 channels of ADB 3-1-0 were masked out.

04/08/08: PS 5-0 (BB 10) temp Alarm

We've received a temp alarm for PS 5-0 (BB 10, or Q1-10). First tried power cycling, which didn't help. Then we tried swapping with its neighbor (so now 5-0 and 5-1 have been swapped). The alarm remained in the slot at 5-0. Jong raised the temp alarm warning to 58.000 C, and we're going to see what happens.

02/27/08: Replaced ADB 419 with ADB 379 in 3-1-4

This went smoothly. We replaced ADB 419 with ADB 379 in IOB slot 3-1-4.

02/14/08: 3-1-4: Masking out ADB

We tried masking both diodes seperately, but ultimately found 3-1-4 to not work. We've left the 12 channels masked out.

01/19/08: PS 5-10 swapped with PS 5-9

Noisy channels were observed in 5-10 and so PS 5-10 was swapped with a neighbor (5-9). The problem did not come back although this may have been due to power-cycling (which should have been carried out first).

01/24/08: 7-2 ROM replaced

7-2 ROM: SBC (single board computer) on this board has been causing a lot of trouble (deadtime, DF errors and such) forcing reconfigure. This has been swapped with a spare

01/14/08: 8-9:14 masked out

8-9:14: becomes noisy on and off. This affected a few runs in the last couple of days. It is now masked out effective Run 78244.

01/09/08: 3-10 IOB replaced

3-10: fiber C was too weak for any good and was masked out a while back. With a new IOB, things look fine, and we now have the 24 channels back.

01/05/08: 4-9 diode B masked out

4-9: leakage current on diode B was pushing the CAEN limit. This is correlated with the weather (big storm), but one can never be sure about these things. 4-9 now has only diode A powered from Ch 10. As a side note, the labeling on Ch 10 supply board is an odd one: "A" means A is not powered.

01/03/08: 4-9 bias moved to spare channel

4-9: leakage current becomes excessively high and prevents the rest of BB Q0 from powering up. This is now moved to Ch.10 ("spare").

12/21/07: 7-3: ROM replacement

7-3 ROM: this was causing an unreasonable amount of damages (>10%-90%). When the ROM was swapped with a neighbor (7-4), the problem followed the ROM. The ROM is now replaced. The damages are now gone.

12/19/07: 3-10: Fiber C masked out

3-10-C: this fiber with low light output was causing unreasonable amount of damages, not only for 3-10, but also for other slots. It is unknown whether the damages are caused by low output (fiber kink) or a bad glink chip or the transmitter. This IOB should be replaced at the earliest convenience.

12/15/07: 2-10 & 4-9 bias cables returned to their respective home

2-10 bias cable pulled from ch10 and back to ch3
4-9 bias cable pulled from ch11 and back to ch7

11/26/07: ROD: BF - Repaired IOB 2-10.

Fiber A on 2-10 was missing bits and had a low optical power.

11/26/07: ROD: BB - Replaced IOB 5-4

The old IOB in 5-4 had problems with fiber B. Perhaps a bad g-link chip!?

11/24/07: ROD: Investigation of fibers with low optical powers

1-6-(B&C) : receiver was bad. 2-4-C : low due to a combination of ROM & IOB. Looked good after replugging. 2-5-A : receiver was bad. 2-7-C : had to be cleaned. 2-10-A : IOB is bad (~900). 3-4-B : combination of ROM & IOB. 3-7-B : combination of ROM & IOB. 3-10-C : IOB needs to be replaced. 4-2-C : needed cleaning. 4-5-C : needed cleaning. 4-9-C : receiver was bad. 5-7-A : keep an eye on. 5-8-B : IOB needs replacing. 5-10-(A&C) : looked OK after cleaning. 6-3-B : just needed cleaning. 7-1-A : combination of ROM & IOB. 7-3-B : just needed cleaning. 7-7-C : just needed cleaning. 8-1-A : keep an eye on. 8-9-B : keep an eye on.

11/24/07: ROD: BF - Re-seated fiber C belonging to 3-10.

Fiber C gave low optical powers and was missing bits. The fiber was found to be under unneccessary stress.

11/21/07: ROD: BF - Replaced IOB 3-7 and ADBs 3-7-1 & 3-7-3 & then all of 3-7s ADBs

3-7 Was having problems after the power outage after the water leak. After many attempts at fixing it it seems the problem may have been with protruding screws from the optical fiber cases on the IOB. 3-7-1 also caused problems and several ADBs have been tried there recently.

11/16/07: ROD: BF - Replaced ADBs 1-2-4 & 3-7-1.

Probably damaged when powering down the EMC after the water leak.

11/16/07: ROD: BF - Replaced IOBs 1-2 & 3-7.

Probably damaged when powering down the EMC after the water leak.

11/15/07: EW Radiation monitoring board was replaced.

The previous board suffered water damaged and was also damaged during subsequent testing. (Too high a voltage was input.)

11/14/07: ROD: BF - Replaced IOB 3-10.

IOB 3-10 was damaged after EMC had to be powered down after the water leak.

11/14/07: ROD: EC - Replaced IOB 9-9.

Due to water damage, IOB 9-9 had to be replaced due to a bad fiber 0.

11/14/07: ROD: EC - Replaced ADBs 9-2-(1,2&3) & IOB 9-2.

Due to water damage, three ADBs had to be replaced in 9-2. These were 9-2-1, 9-2-2 & 9-2-3. IOB 9-2 also had to be replaced.

10/20/07: ROM 3-0 ethernet card changed.

3-0

10/29/07: ROM 8-4 swapped with 8-5

8-4 ROM: This slot couldn't be calibrated. To investigate ODF was nice enough to swap this with 8-5. The problem did not move with the ROM, so IOB became the target.

10/30/07: Full ecal on crates 2, 4 & 8

A full ecal was performed on crates 2, 4 & 8 to see how things looked after the ROD replacements on the 29th of October (yersterday!).

10/29/07: ROD: BF - Replaced IOB 8-4

8-4

10/29/07: ROD: BF - Replaced IOB 2-10

2-10

10/29/07: ROD: BB - Replaced ADB 4-8-4

4-8-4

10/28/07: Full ecal on all but Crate 8.

A full electronics calibration was performed on the calorimeter to see if things were ok after the work carried out on the 26th of October. Crate 8 was not tested because of a module(or IOB) problem with 8-4.

10/25/07: Pedestal calibration.

A pedestal calibration was performed on the entire emc.

10/25/07: ROD: EC - Replaced ADB 9-6-0

The CARE chip was bad.

10/25/07: ROD: BB - Replaced ADB 4-4-0 & 7-3-5 And IOB 7-3

4-4-0

7-3-5

7-3

10/25/07: ROD: Replaced ADB 3-5-5

Ch 61 had x1 & x4 ranges all messed up. (Effectively no change; B-diode masked out.)

10/25/07: Swapped EC E LV PS 03 for a spare

Power supply was showing high currents on the -5V supply. Power cycling didn't help. The PS had been in a different slot earlier and there it had similar problems for the +5V supply.

10/24/07: Full ecal

A full ecal was performed on BF.

10/24/07: Pedestal calibration

A pedestal calibration was performed on the entire detector.

10/24/07: Crate modification complete

The final crates were completed adn re-fitted today. Clean filter boards were placed in BF0 and BF2 was removed, its back plate replaced, clean filter boards put in and it was put back.

10/23/07: Pedestal calibration

A pedestal calibration was performed on almost the entire EMC, BF0 was the only crate not included.

10/23/07: BF1 Modified & Ready, BF0 Modified but awaiting filter boards.

BF1 was removed, a new back plate fitted and clean filter boards were put in. BF0 was also removed and a new back plate was fitted. It was then replaced and is awaiting filter boards. (As usual other crates were effectively power cycled to allow the procedure; all BF and BB0 in this case.)

102207: EC full ecal

A full calibration (pedestal, Ecal, xtalk & AR) was performed on the EC.

102207: Full Pedestal calibration

A pedestal calibration was performed on the entire detector.

102207: BF3 Modification.

BF3 was removed, a new back plate fitted and clean filter boards were put in. (As usual other crates were effectively power cycled to allow the procedure; all BF and BB0 in this case.)

10/22/07: Reseated and power cycled EC East 03

Current reading of Endcap East LV PS 03 -5V line was occasionally jumping up to 4.3 A, giving alarms. I reseated and power cycled the supply.

101907: A full calibration was performed on BB

Serial, pedestal, ecal and auto-range tests were performed on the backward barrel.

101907: Clean filter boards were fitted to BB3, EE & EW

Three sets of clean filter boards were fitted to BB3, EE & EW.

10/18/07: Pedestal calibration.

A pedestal calibration was performed.

10/18/07: EE & EW crates modified and prepared. TRB crate modified and ready.

The TRB crate was modified. Its old back plate was swapped for the new one and its dirty filter boards were replaced with clean ones. The same procedure was carried out for the endcap too. However only 10 clean filter boards were available today and so while these crates have been modified, they join BBQ3 in that they're ready for action, but have no boards.

10/17/07: Pedestal calibration

A pedestal calibration was performed for the entire EMC.

10/17/07: BB2 Crate modification & BB3 prep.

BB2 crate was removed. Dirty filter boards were replaced by clean ones and the new back plate was fitted. While work was perfomed the following crates had to be powered off; BBQ1/2/3 and both EC east and west and the TRB crate. (BB3 was also removed and prepared for the same procedure tomorrow.)

10/17/07: Pedestal calibration

A pedestal calibration was performed for the entire EMC.

10/17/07: BB1 Crate modification

BB1 crate was removed. Dirty filter boards were replaced by clean ones and the new back plate was fitted. While work was perfomed the following crates had to be powered off; BBQ1/2/3 and both EC east and west and the TRB crate (see previous entry for more detail).

10/16/07: Pedestal calibration

A pedestal calibration was performed for the entire EMC.

10/16/07: BB0 Crate modification

BB0 crate was removed. Dirty filter boards were replaced by clean ones and the new back plate was fitted. While work was perfomed the following crates had to be powered off; All of BF. (The rest of the EMC was also effectively powercycled before work began because it was assumed that the second racks breakers controlled only the second rack. This is not true. The breakers on rack 1 (BF) control rack 1 and BBQ0 of rack two. The breakers in rack 2 control the rest of the EMC apart from the diode power.)

09/24/07: LV PS BF 35 replaced

A few hours after the electronics were turned on after the power outage, BF 35 tripped off. Reseating and power cycling didn't help so I replaced it with a spare. Mike's tests showed that the +5V supply is completely dead.

08/30/07: Pedestal Calibration

Pedestal Calibration

08/30/07: Pedestal calibration

08/23/07: PS crate for BF Q0 replaced

BF Q0: PS crate replaced. The power supplies in this quad showed signs of misbehaving +10V line. Maybe this explains all the minor troubles we've been seeing in the past few months?

07/10/07: TPCs swapped between crates 0 and 1

TPC

07/10/07: Calibrated all of EMC.

Calibrated EMC. Calibration notes are stored at the usual location.

07/10/07: TRB BF0 replaced

TRB BF0: There have been some indications (systematic problems seen in elex calib and TC damages from various slots in this crate) that there may be a problem with the TRB. The new TRB did not make any obvious difference. The next victim in this quest is the TPC.

07/10/07: 8-1 & 8-1-0 replaced

8-1-0: non-communicating ADB replaced. The connector is a suspect.
8-1: Replaced to see if a different IOB would work better with the ADBs in this minicrate. The connection between IOB & ADB is assumed to be the casuse for the dead ADB00.

07/02/07: 0-5 ROM swapped with 0-7

0-5

06/28/07: 8-1-0 died. again.

8-1-0 died overnight; power cycling did not resurrect it so it is now masked. again.

06/27/07: New pedestal Calibration

Not sure what else to say...

06/27/07: Replaced LV PSU for TRB BF0

Jong thought that TRB BF0 was starting to mis-behave, so asked for me to replace the power supply.

06/27/07: Swapped LV PSUs 0-5 and 0-6.

Crystal spectra for IOB 0-5 looking bad, so swapped power supply with neighbour.

06/25/07: Replaced PSU for EC IOB W-3

Power supply for end cap west 3 failed. Swapping with neighbour moved the problem with the PSU; replaced faulty PSU.

06/25/07: Replace power supply for TRB BB0

+5V and +10V line on PSU for TRB BB0 failed, giving red alarm and making EMC not-runnable. Power cycling did not work, replacing the PSU fixed the problem.

06/25/07: 1-6 damages

1-6

06/21/07: Calibrated all of EMC.

Calibrated EMC. Calibration notes are stored at the usual location.

06/21/07: 8-1-0 Recovered.

8-1-0

06/21/07: 2-9-0 was swapped.

2-9-0

06/21/07: 7-2 ROM swapped

CPO changed 7-2 ROM with the suspected repaired ROM.

05/25/07: 2-9-0 Masked

2-9-0: became dead after powercycling to cure a bad CARE chip.

05/14/07: 8-1-0 Masked

8-1-0: became dead shortly after magnet came back online. Powercycling and a long cool-down didn't help. The currents on the PS is low, so it's speculated that this ADB is completely dead.

05/11/07: 7-4-4, 4-9-0 replaced

7-4-4: this ADB died one last time! So it's replaced.
4-9-0: desperate attempt to see if a different ADB would quiet this position down. It didn't help!

05/09/07: 7-4-4 masked out

7-4-4: started being noisy (and sometimes too quiet) during cosmics. Powercycling stopped helping this guy.

05/01/07: 9-1-0 & 2-6-0 replaced

9-1-0: suspected of having a bad CARE chip.
2-6-0: faulty calibration connector was preventing this minicrate from being calibrated. This is now replaced. note: there must be some alignment problem with the bulkhead. It is very difficult to align the first 2 ADBs with the rest.

04/26/07: 9-1-0 (ch 8-11) masked

9-1: channels 8-11 were causing out-of-time hits; probably from a bad CARE chip. These are now masked out.

04/21/07: Swapped power supplies for 1-5 and 1-6. New peds for 0 & 1.

1-5 was being noisy, so after trying a power cylce we swapped power supplies. Re-did the pedestals for crates 0 and 1 as well.

04/19/07: 4-9 ROM replaced

4-9 has been requesting sync at unreasonable rates for a while. The IOB has been replaced with no significant improvement. cpo replaced the ROM with a spare. Things look much better now.

04/19/07: CALIB: pedestal & ecal on EC (9-10-3 recovery; 9-1-0 notes)

Pedestal done on all crates
ecal done on EC: 9-10-3 looks to have recovered by itself, so older constants are no longer required. On the other hand, some channels (8-11 namely) on 9-1-0 show slight deviations on the auto-range only. Need to keep eyes on these channels.

04/18/07: 7-9, 1-6 damages

7-9: dataflow damages at a rate of > a few percent. cpo powercycled the DAQ crate and "fixed" the problem. This problem is likely to come back.
1-6: dataflow damages at a rate of a percent or so. Initial powercycle of the minicrate resulted in fiber C not locking. Swapping fibers indicated that this might be a IOB-ROM combo problem. Subsequent powercycle (for longer off cycle) "fixed" the problem. This problem is also likely to come back.

04/10/07: Elex calib (except 2-6, 9-10)

Full elex calib for crates 0,1 & all of BB and EC.
2-6-0 still has a calibration problem, so old constants are still used. This ADB should be on the list for the next ROD.
9-10-3 is a problem. Masking one set of diodes makes the calibs worse, so the only option is to enable both diodes and use old constants. This ADB should be on the list for the next ROD.

04/10/07: 5-7-0 Unmasked

After magnet down and cooling the minicrate for a while, this ADB decided to come back to life.

04/10/07: ROD: TRB BF0 Replacement

TRB BF0: after seeing unreasonable number of slots generating dataflow damages and strange failure modes in just about all new IOBs in this crate, this TRB was a target for replacement.
The new TRB has an error with EmcFeeTest on channel 6 (counting from 0), but all indications are that it's working just fine.
This new TRB fixed the problems with dataflow damages in several slots (from about 2% per run before the replacement to none after replacement). Also, the strange odd channel problem in 0-9 has disappeared!

03/29/07: ROMs 0-2, 0-5, 0-9 swapped

Matt swapped the ROMs 0-2, 0-5, 0-9 with their neighbors to see if the syncRequests would settle down.

03/29/07: ROD Summary (03/26/07-03/28/07)

 
TRB: 
   BB1 : no fiber lock 

IOB: 
   4-9 : to fix OoS damages 0-2 : (twice) noisy hitmap and spectrum 
   0-5 : bad spectrum (recent runs look okay, but we don't get get the 
         chance too often) -> tons of DF damages -> 
         swapping fibers with 0-4 makes the damages disappear! 
   0-9 : noisy and bad hitmap -> tons of DF damages -> 
         swapping fibers with 0-10 ->? 
   3-10 : fiber A started not locking during the ROD 

ADB: 
   1-3-1: shitty AR results 
   1-3-5: bad spectrum 
   1-5-3: to save 1-5-43 (OoT hits) These may be from bad IOB, 
          but to save work, they were swapped out 
   0-2-0: jump in x256 
   0-2-1: huge jumps in all 
   0-2-2: jump in x256 
   2-9-0: (twice) noisy ADB -> replacement sucked current like crazy 
          driving up IOB temp -> 2nd replacement is okay 

calib: 
   1-2-3: even channels lost diode A -> mask A! 
   1-3-1: all channels are bad in x1 -> mask B! 
   2-6-0: even channels lost diode ? 
   0-9  : ALL ODD channels are bad in AR -> mask all B diodes! 

AR: 
   2-3:12 - smooth stepped rise 
   5-7:29 - bad 5-7:50 - bad 
   6-2:50 - smooth stepped rise 6-9:23 - bad

03/29/07: 5-7-0 Masked out

This appears to be a casualty of magnet ramp-up. Subsequent attemps to revive this has failed. This ADB needs to be replaced!

03/28/07: ROD: FW: 1-3, 1-5, 2-9

1-3: ADB05 energy spectra looked horrible. After replacing the ADB, ADB00 started causing trouble in the ecal. So ADB00 had to be replaced as well.
1-5: ADB03 was replaced to save the OoT channel 30.
2-9: ADB00 has always been horrible. We tried replacing this, but the replacement ADB caused the temp to rise too much for the IOB. The ADB had to be replaced again.

03/28/07: ROD: FE: 0-2, 0-5, 0-9, 3-10

0-2: noisy IOB and ADB04 replaced. After the replacement, ecal looked horrible for the entire slot. Replacing IOB (again) and ADBs 00, 01, 02 fixed the problem.
0-5: this IOB has been acting up from time to time. So this along with ADB05 were replaced. Since then this slot generates unreasonable amount of dataflow damages. This IOB should be replaced when possible.
0-9: this IOB along with ADB04 & 05 have been noisy on the hitmaps. After replacing these, the new IOB has a problem with even channels. Masking B diode fixes the problem for now. This IOB, therefore, needs to be replaced!
3-10: IOB gave bad pedestal after power cut during the 1st day of the ROD. Replacement has been working fine so far.

03/28/07: ROD: BE: 4-9 IOB replacement

4-9 has been generating unacceptable amounts of dataflow damages. Replacement looks fine so far.

03/28/07: ROD: TRB BB01 Replacement

TRB BB01 failed to lock the night before the scheduled 3-day ROD.
Problem is traced to either a weak Finisar transceiver or cold solder joint on g-link chips

03/28/07: ROD: TRB PS crate replacement

TRB PS crate has been showing some monitoring problems: BF00 current on 10V kept on spiking up. The new crate is not perfect either. It shows 0 current for -5V on BB01, but that's better than unpredictable rise in the current reading.

03/28/07: Masked ADB 5-7-0

Noisy channels did not disappear with new power supply (see previous entry), so we've had to mask the whole ADB.

03/28/07: masking noisy channels

Masked channels (33,39), (33,40), (35,39) as were being noisy. May be power supply related (high current on 10V line) - Andy is replacing the power supply then we'll try unmasking.

02/22/07: 0-2-4 Masked out

0-2-4 continues to be slightly noisy, and sometimes it gets noisy enough to trigger an EMT tower.

02/14/07: Masked diode A on 8-1:37

Masked diode A on 8-1:37
Updated calibrations to include new mask.
Saved new Calibration.

02/13/07: Masked diode A on 6-5:57 and 9-8:5

Masked diode A on 6-5:57
Masked diode A on 9-8:5
Updated calibrations to include new masks.
Saved new Calibrations.

02/13/07: Complete calibration of emc after rod

After the rod we did a complete calibration of the emc. The results of the calibration have been saved.

02/13/07: Replaced (6-4-1, 6-9-1, 6-10-4, 9-10-0) and swapped (2-9-0,2-9-1) ADBs during ROD

Swapped ADBs 2-9-0 & 2-9-1 - problem with channel 0 - most likely due to water damage.
Replaced ADB 6-4-1 (had a care chip which was noisy when magnet changed state).
Replaced ADB 6-9-1 to try and fix problem with channel 23 (this appears to be due to a broken pin on the front-end) - channel not rescued.
Replaced ADB 6-10-4 as was sporadically noisy.
Fiddled with 9-8-1 because of a noisy channel. Seems to be okay now.
Replaced ADB 9-10-0 (died during power outage on 02/12/07).
Masked diode B on 6-4 ch.4.

02/12/07: Masking ADB 9-10-0

9-10-0 appears to have died during the power outage. The 12 channels on it have been masked. Previously the odd channels had diode B masked.

02/09/07: Unmasked 6-10-4

6-10-4

02/09/07: Pedestal on all crates and ecal on cr 2

Apparently crate 2 was last calibrated (AR) with the wrong key, probably back on 01/17/07. So this crate is completely recalibrated (combo, xtalk, AR) today. This takes effect starting 70108.

02/09/07: Swapped LV PS BF08 for a spare

BF-08 LV PS: Reading on the I(+10V) line suddenly dropped to 0.7V when normal readings 5.2V. Nicolas powercycled old PS first but we ended up putting in a spare just in case. Epics readings look OK for now. PS In: 2172 PS Out: 2080

02/08/07: pedestal calibration

Pedestal calibrated in the process of diagnosing 6-10-4. The ADB is definitely dead. In the past, this ADB came back after being cooled for a while. It's probably a connection problem, so it'll be replaced during the next ROD.

02/08/07: 7-9 ROM swapped with 7-8

CPO swapped the ROMs 7-9 & 7-8 in hopes of finding out what's causing all the damages from 7-9.

02/08/07: Masked out 6-10-4

6-10-4

02/05/07: ROMs replaced for 7-2 and 4-9

7-2 had intermitten corrupt time stamp
4-9, fiber A in particular, has been generating a high rate of SyncRequests. Mike K.'s cap replacemenet didn't really help much with this one.

02/05/07: Masked out 9-8:17

9-8:17 shows low gain in the high energies. Historically, diode A was masked because of the low gain; but over the long shutdown, diode B has gone completely bad. So rather than depending on the low gain from diode A, the channel is now completely masked out.

01/31/07: ROM 0-3, 4-9, 7-7 repaired

Mike K. found that by replacing one of the capacitors, the ROMS behaved better. The accepted fiber strength goes from -7 db to -10 db. These ROMs were chosen as testers because they seem to generate high rate of SyncRequests.

01/31/07: calibrated cr 7 and performed pedestal on all crates

Elex calib done on cr 7 after replacing 7-9-0
Pedestal performed on all crates (9-10 really needed new peds)

01/31/07: Replaced 7-9-0

Jen replaced 7-9-0; This ADB had ch0 A diode masked, and ch2 was not responding. The new ADB looks fine now. So ch0 diode mask has been removed.

01/29/07: 4-9 Out of Sync problem

Note

was

01/29/07: masked out 1-5:43 and disabled A on 9-8:32

1-5:43 was hot at low energies and produced some out of time hits -> disabled
9-8:32 was producing garbage or very cold at high energies. ecal looks fine but historically, A diode was always masked out. After masking A, the channel looks fine.
This affects run 69726 and onward.

01/27/07: Power cycling on 6-10 and 0-2

0-2-4 showed some noisy channels this morning, power cycling calmed it down. 6-10-4 showed some noisy channels in the evening; again power cycling fixes it. Will re-do pedestal when we get the chance.

01/25/07: Pedestal & investigate 9-10-03 & 7-9-2

Pedestal on 9-10-03 has drifted...recalibrated
7-9:2 appeared dead in the latest runs. This is confirmed by ecal. We are now up to 4 dead channels

01/20/07: Calibrate cr 6 (6-4-01)

6-4-01 is now alive again (slow magnet effect?)...calibrated this crate (ecal, xtalk, AR)

01/19/07: ROM replacements (1-7, 1-10, 5-4)

cpo replaces 1-7, 1-10 & 5-4

01/19/07: Pedestal (9-10-03 acting up)

Pedestal calib on all crates (9-10-03 has been noisy for a couple of days)

01/17/07: ROD: Pedestal & ecal on crates 0 & 2

Pedestal calib over all crates
ecal, xtalk, AR on crates 0 & 2

01/17/07: ROD: 0-2, 2-6, 2-10

Replaced 0-2-01 ADB (died during startup) and 0-2 IOB (source cal indicated problems with this IOB)
Replaced 2-6-01 ADB (died during startup) and 2-6 IOB (probably a connection problem)
Replaced 2-10-00 ADB

01/08/07: Elex calib marking the end of this long recabling process

01/08/07: Recabling (Debugging...)

Masked out 0-2-1 (ch 12 causing a hot tower in EMT)
Masked out a care chip in 2-6-1
6-4-1: care chip was noisy, but after cooling the minicrate, it's alive and behaving.
Elex calib sequence done

01/04/07: Recabling (Debugging...)

Replaced 6-10-4: ADB was sitting to far out...replaced anyway since we only get one shot with Bwd.
Replaced 2-4-0: started speaking gibberish this morning. The replacement worked fine all day, but it started doing the same thing toward the night. ADB was replaced again and IOB and copper plate were reseated and adjusted.
A whole lot of work on 2-6 (see below)...which led to...
Replaced TRB for Fwd Q2.
Elex calib sequence done

01/03/07: Recabling (Debugging...)

Replaced 1-10 IOB (noisy from yesterday) and reseated the copper plate
Replaced 2-6 IOB (kept killing slot 0 ADB) and reseated the copper plate. This killed the whole minicrate (tried 2 different IOBs with the same result). Will need to play with the copper plate again as the minicrate seems to behave w/o it.

01/02/07: Recabling (Debugging...)

Swapped 2-6-0 with 2-6-1 (2-6-0 died AGAIN soon after it was replaced); noticed ADB00 (dead one) had loose connectors -> replaced
Replaced 1-10-0 (died during the long holiday break)
Replaced 1-10 IOB (after replacing 1-10-0, 1-10-1 started misbehaving); This IOB is noisy -> will need a different one.

12/15/06: Recabling (Debugging...)

Replaced 2-6-0 (died overnight)
Replaced 2-6 IOB; the old IOB may be an ADB eater.
Replaced 1-10-0 & 1-10-1 (1-10-1 behaved badly for ecal; 1-10-0 was greasy as hell); Reseated 1-10-2 (was sticking out - no bent pins found in any of the ADBs).

12/14/06: Recabling (Debugging...)

Replaced 2-6-0 (died yesterday)
Replaced 3-1-5 ADB & 3-1 IOB to diagnose bad ecal channels... the problem persists, so the problem is thought to be on the preamps.
Replaced EC West temperature GMB; all temp sensors recovered.

12/12/06: Recabling (Debugging...)

Replaced 3-7-0 (died during overnight burning-in)

12/11/06: Recabling (Debugging...)

Replaced 0-6-1 & 0-6-3 (died during burning-in)
Replaced 3-8-0 (died during burning-in)
Replaced 3-7-0 (complete garbage in LP calib)
Replaced 9-10-0 (1 really bad channel)
0-4 SyncRequests -> ROM swapped with 0-5 by cpo
Measured with light meter for low output fibers:
Bit pattern test troublemakers:

12/06/06: Recabling Day #9

FWD:
Put Al. covers back on Q0
Put cable tray covers back on
Keep an eye on 0-4: sends unreasonable amount of SyncRequests
BWD:
Watch the bias current on 4-9

12/05/06: Recabling Day #8

AM:
FWD: Continue debugging Barrel Fwd electronics (0-9, 0-4, 0-2-0, 2-4-5)
EC: Continue repairing 8-1-0, 9-4-1, 9-9-0/1 swap, 9-10
BE: Examine bias cable in 4-9
PM:
FWD: Put covers on
EC: Continue calib and repairing

12/04/06: Recabling Day #7

AM:
FWD: Continue debugging Barrel Fwd electronics (0-9, 2-4-5, 1-6-0)
EC: Continue repairing 9-4 IOB & 9-8 bias
BW: ADB replacement for 6-10-4 if manlift is available (maybe...this started behaving after a 24-hour burn-in)
BE: IOB replacement for 7-9 if manlift is available, examine bias cable in 4-9
PM:
FWD: Put covers on (tomorrow)
EC: Continue calib and repairing (continuing)

12/02/06: Recabling Day #6

Continue debugging Fwd elex (0-3: fiber A/B swapped???, 3-1, 0-9-fiber A, odd-channel calib issues)
Start repairing Endcap (8-1, 8-2, 9-4, 9-8, 9-9, 9-10)
Performed overnight high rate test -> the only trouble comes from 0-9 (SyncRequests & Dropped Contribution from fiber A)

12/01/06: Recabling Day #5

AM:
FWD: Debug Barrel Fwd electronics (ADB repl. in 0-9-1 & 0-8-0 in particular) (preliminary round of checking is done...continue with more stubborn cases tomorrow)
PM:
FWD: put back copper plates and covers (DONE with copper plates. Covers will be next week)

11/30/06: Recabling Day #4

AM:
FWD: EndCap, LP & PIN-diode cables (DONE)
BW: 6-9, 5-7 (Done...see notes below)
PM:
FW: LP reconnect & finish installing IOBs in Q2 (DONE)
FE: Finish installing IOBs (DONE)
Fwd: Test BF electronics (Tested BB to be okay...BF is now for tomorrow morning)
Fwd: Finish CANbus work (RadFet board and reterminate temp sensors in West) (DONE...but only a portion of EC West xtal temps could be recovered)

11/29/06: Recabling Day #3

BF West:
1. (AM) install TRBs for Q1 & Q2 (DONE)
2. (AM) turn on Q1 and test (in progress...)
3. (PM) install IOBs for Q2 (2 more to go)
4. (PM) turn on Q2 and test
BF East:
1. (AM) install IOBs for Q0 (9 more to go)
2. (PM) install IOBs for Q3 (1 more to go)
BB West:
(AM & PM) Repair Q1 & Q2 problems (Mostly done - will give 6-9 one more try)
Finish CANbus work (RadFet board and reterminate temp sensors in West) (acquiring parts)

11/28/06: Recabling Day #2

11/27/06: Recabling Day #1