← AI Foundry lab dashboard › Worklog
Out of service from Wed 7 Oct, 16:46 PDT, to Fri 9 Oct. Back in service since Fri 9 Oct, with a two-fan bracket at the card.
The card is out of service again. Idle, with nobody using it, it heated from 64 °C to a 138 °C mean (hottest sensor 144 °C) at 133 W, and fell off the PCIe bus at 16:46, 24.7 hours after the 6 October fan fix brought it back. It failed exactly as on 2 October (138 °C, hottest 144 °C, 134 W). It is still off the bus; the machine itself is up.
What changed was the cooling at the card, not the card. At 03:55 on Wednesday, with the host idle and its own temperatures unchanged, the card started heating and settled about 11 °C warmer. In the afternoon it got worse again and crossed the tipping point. All day the card drew exactly the power that its temperature calls for, on the same curve as on 2 October. Something at the card stopped carrying heat away: its own fan, or a fan near it, that stopped, slowed or moved would do this. Fan speeds are not visible from the machine, so someone has to look on site.
The idle card's last day
aifoundry2's card, mean die temperature, one point a minute (every reading in the last minutes), from the moment the fan fix brought it back. Hover, tap or focus and use the arrow keys for values.
What happened
| When (PDT) | The card | What happened |
|---|---|---|
| Tue 6 Oct 16:05 | 67 °C, 28.8 W | Back on the PCIe bus after the BIOS fan fix (every fan at full speed). From 16:22 to 16:35 that day's load tests held the card; nobody held it after that. |
| 17:00 to 03:50 | 63–70 °C, 26.9–30.3 W | Idle and steady through the evening and the night. |
| Wed 7 Oct 03:55 | 64 °C, then 78 °C by 04:50 | The cooling at the card gets worse, in one step. The host is idle (0.5% CPU) and its own temperatures do not move. |
| 05:00 to 11:30 | 74–78 °C | Steady again, about 11 °C warmer than in the night. |
| about 14:25 | 80 °C, then 85 °C by 14:44 | The cooling gets worse again. |
| 15:30 | 90 °C, 43 W | At the tipping point: from here the leakage grows faster than the cooling can carry it away. |
| 16:39 to 16:46 | 100 → 138 °C, 54 → 133 W | Runaway. |
| 16:46:32 | 5 corrected receiver errors on the card's PCIe link in four seconds (kernel log). | |
| 16:46:40 | 138 °C (hottest sensor 144 °C), 133.4 W | The card's last reading. There is none from 16:46:45 on. |
| 16:55 | The driver finds the card's queue unreadable: reads from the card return all ones. | |
| 17:01:20 | The card's PCIe root port is reset; by whom or what is not known. The card does not come back, and is still off the bus. |
Only the cooling at the card changed
Three things rule out everything but the card's own cooling.
- Nothing ran on the card. No process held it after the 6 October tests ended at 16:35, and at 03:55 the host's CPU was 0.5% busy.
- The card drew what its temperature calls for, and nothing more. On 2 October the card's power fitted 13.6 W plus a part that doubles every 23 °C (the chip's leakage). On Wednesday every reading sat on that same curve, 0.25 W rms over the day and within 0.4 W at every temperature the card held for a minute or more (the second chart). Power only ever followed temperature: had work started, or had the firmware changed the clock or the voltage, the readings would have left the curve.
- The rest of the machine did not warm up. Across 03:55 the host's NVMe drive read 45.9 then 46.3 °C, its network chip 54.1 then 53.8 °C, and its CPU 39 then 38 °C. The room and the case air stayed as they were.
The 2 October heat model puts a number on it: the air temperature the card's heat balance needs (the model's single "air" number, so a loss of airflow also shows up as warmer air). Overnight that was about 39 °C, as just after the fix. After 03:55 it was about 46 °C; after 14:25 about 51 °C, which is where it was before the fix. The model checks itself: with air at 39, 46 and 51 °C it says the idle card settles at 63, 76 and 91 °C, which is what it did. Above 51.5 °C it has no steady temperature at all.
Only the card's air changed
Change from each line's own level in the night (00:00–03:50, Wed), °C, nine-minute running medians. The card's line is the air temperature its heat balance needs, by the 2 October model (night level 39.4 °C), and stops at 16:30, when the die starts climbing too fast for the model's 6-minute slope. The band spans the host's NVMe drive and network chip (night levels 46.0 and 54.5 °C).
The card drew the power its temperature calls for
Board power against mean die temperature, Wednesday's readings (median power at each whole degree), with the leakage curve fitted on 2 October. Hover, tap or focus for values.
Why the card could not save itself
From 65 °C to 138 °C nothing on the card slowed it down, today as on 2 October. The card's
firmware (release 1.3.1, boot loader 0.20.0; its closest public source is
et-platform ffca4cbb4) does run its thermal check on an idle card above 65 °C.
But all the check can do is step the minion clock down the voltage table, and the idle card already sits at the bottom
of it, 600 MHz, so nothing changes. The emergency safe state changes nothing either: it moves the clock only when
looking up its voltage fails, and its voltage step is commented out
(fixed upstream on 5 September 2024). Nothing in the firmware shuts the card down. The
current upstream code does no better at idle: its check (check_power_throttle_conditions(),
836a4ab) does not run at all while the card is idle.
So the two problems meet. Hotter silicon leaks more, which heats it further; a fan keeps that loop from closing, and the firmware does nothing once it does. Until the firmware can cut an idle card's power or shut it down, the cooling at the card is the only thing that keeps it out of the loop.
What is not known: the fan
The machine cannot see its fans: no fan sensor driver is loaded for this board, and loading one needs root. Two causes can still be ruled out. The BIOS setting cannot have reverted, because the machine has not rebooted since the fix, and Linux never switched a fan (the kernel's ACPI fans show no change of state since boot). That leaves the hardware: the card's own fan, if it has one, or a case fan by it, that stopped, slowed or was knocked out of line, or something in the way of its air. The change came at 03:55 at night, all at once, while nobody was there.
On site: look at the card's own fan (follow its cable, if it has one) and at the case fans by it, then at the BIOS's Smart Fan 6 page, which shows each fan's speed (System 5 read 2,368 RPM at full speed on 6 October).
What changes for people using the lab
- Run nothing on aifoundry2's card. The lab's docs, the new-user brief and the dashboard mark it out of service; new users set up on aifoundry3 or aifoundry1. The machine stays up.
- To bring it back: first fix the cooling at the card, then the full reset as root, the cold reboot that power-cycles the slot. A warm reboot or a slot reset leaves it off the bus, as the 17:01 reset showed.
- Then watch it. After a reset the card looks healthy for about an hour while it heats. This time it ran 25 hours, and the trouble started 12 hours in.
The numbers behind the charts
Hour by hour: the card, the model's air and the host's sensors
Die and power: the range of the 5-second readings in the hour. Air: the model's mean (to 16:30). NVMe and network chip: the host's sensors, hourly means. Tue 16:00 includes the fix's load tests (16:22–16:35).
| Hour (PDT) | Die °C | Power W | Air at the card °C (model) | NVMe °C | Network chip °C |
|---|---|---|---|---|---|
| Tue 16:00 | 63–76 | 27.0–33.2 | 41.6 | 46.2 | 53.0 |
| Tue 17:00 | 68–69 | 29.0–29.7 | 41.9 | 46.9 | 54.8 |
| Tue 18:00 | 68–69 | 29.0–29.9 | 42.2 | 46.9 | 55.3 |
| Tue 19:00 | 68–70 | 29.0–30.3 | 42.3 | 47.4 | 55.6 |
| Tue 20:00 | 69–70 | 29.4–30.2 | 42.3 | 47.8 | 55.7 |
| Tue 21:00 | 68–69 | 29.1–30.1 | 42.2 | 47.1 | 55.6 |
| Tue 22:00 | 68–70 | 29.2–30.1 | 42.3 | 46.9 | 55.3 |
| Tue 23:00 | 66–69 | 28.2–29.8 | 41.2 | 46.9 | 55.1 |
| Wed 00:00 | 64–67 | 27.5–29.0 | 40.2 | 46.3 | 54.9 |
| Wed 01:00 | 64–65 | 27.3–27.9 | 39.2 | 45.9 | 54.6 |
| Wed 02:00 | 63–65 | 26.9–27.9 | 38.8 | 45.9 | 54.4 |
| Wed 03:00 | 63–69 | 27.1–29.6 | 39.9 | 45.9 | 54.0 |
| Wed 04:00 | 67–78 | 28.8–34.7 | 45.6 | 46.4 | 53.8 |
| Wed 05:00 | 77–78 | 34.1–35.0 | 46.7 | 46.9 | 53.6 |
| Wed 06:00 | 75–78 | 32.8–34.7 | 46.0 | 46.0 | 53.1 |
| Wed 07:00 | 74–76 | 32.1–33.6 | 45.3 | 45.6 | 52.5 |
| Wed 08:00 | 74–75 | 32.0–33.0 | 45.0 | 44.9 | 52.1 |
| Wed 09:00 | 74–75 | 32.2–33.0 | 45.3 | 44.9 | 51.9 |
| Wed 10:00 | 74–75 | 32.2–33.2 | 45.4 | 44.9 | 51.8 |
| Wed 11:00 | 75–76 | 32.7–33.5 | 45.8 | 44.9 | 52.1 |
| Wed 12:00 | 76–79 | 33.4–35.1 | 46.7 | 45.8 | 52.4 |
| Wed 13:00 | 78–80 | 34.7–36.2 | 47.5 | 45.9 | 53.0 |
| Wed 14:00 | 80–86 | 35.6–40.6 | 48.9 | 46.9 | 53.1 |
| Wed 15:00 | 85–90 | 39.9–43.9 | 50.6 | 47.2 | 51.4 |
| Wed 16:00 | 89–138 | 42.4–133.6 | 51.2 | 48.0 | 51.9 |
Sources and method
- Readings. The lab's live collector on aifoundry2 reads the card once a second and keeps one record every 5 s: the mean of the 34 minion-shire temperature sensors ("die"), the highest reading since the card started ("hottest"), board power, and whether anyone holds the card; and the host's own sensors. 18,047 records, 17,678 with a card reading.
- The model. The 2 October fit, recomputed from that day's records by the same method: power P(T) = 13.56 + 2.025·e0.0300·T W; heat balance C·dT/dt = P − (T − Tair)/R with C = 296 J/°C and R = 0.897 °C/W. Each minute's air is T − R·(P − C·dT/dt), with dT/dt a least-squares slope over 6 minutes. These are model numbers, not measurements, and R and the air are correlated in a one-run fit.
- The kernel log. The card's lines, 16:46:32 to 17:01:20. The 16:46 lines come from a read at 17:16: the ring buffer, flooded with AppArmor audit lines, had dropped them by 17:40.
- Earlier pages. 2 October, the first runaway and its fit; 6 October, the fan fix.
The records, the kernel lines, the analysis and this page's builder are in
docs/reports/data/2026-10-07-idle-runaway-aifoundry2
of the lab repository, with this page as docs/reports/2026-10-07-aifoundry2-idle-runaway.html.