Let me give you the ending first.
Nine months of chasing, twenty-three meetings, and what the last box says is: Unknown.
At one of the customer's wind farms, two hundred-odd units, twenty-four failed.
The report said the failure was an optocoupler on the main board.
I was in charge of purchasing at that supplier in those years. The first set of minutes I got hold of was dated 30 July 2009 — already the third meeting.
What happened in those meetings, and what we were doing on our side, I only understood later. More than half of it was not analysing the problem. It was trying to get hold of the data.
The customer reported several failures and sent not one piece of failure information — so we had to issue a service bulletin to improve reporting, dig into our own failure database, and write to the customer asking for a contact. All of it to get the parts and the data back into our hands.
Months later a correction appears in the minutes, to the effect that:
So far, neither we nor the inverter manufacturer have confirmed a single case of optocoupler failure.
We had got wrong even what had failed.
What actually failed was the IGBT. Of that batch of failures, nineteen were IGBT failures, with the cause listed as "unknown".
Nineteen out of twenty-four, roughly eight in ten, and nobody knew why from the very beginning.
Follow it up the chain. The IGBT is made by a device maker further upstream.
Us → the inverter maker → the device maker.
Send the samples, wait.
Shipped on 21 August. On 4 September, the device maker had not replied — they were on holiday.
The report came back eventually: the IGBT appeared to be fine, the problem pointed to a current-measurement stage inside the module, and it was closed as "an isolated case of a measurement card failure".
Back at the site, nineteen IGBTs with unknown causes were still sitting there.
Over that stretch we crossed out every reason we could think of, one by one.
Was it a bad batch? The failed IGBTs ran from September 2008 to April 2009 — no common batch. Crossed out.
Was it lightning? The correlation with thunderstorm statistics was very weak. Crossed out.
Was it the environment — moisture, condensation? That one left something behind: there were corrosion marks on the boards, we added a protective plate and revised the site commissioning instructions.
But the minutes add one honest line: because of the nature of the failures, it is impossible to estimate what proportion of the problem these changes will solve.
Cannot say how much it fixes.
In the end five cases remained where "the root cause is still disputed". Their fate in the minutes is one sentence: no further discussion required, closed.
Only now do we get to the thing I want to say.
Over those years I received a good number of these reports. The format was fixed, nine boxes, from "what is the problem" all the way down to "how do we prevent it happening again". The industry calls it an 8D.
The box that does the real work is the fourth — root cause.
The core of such a report is one thing: asking "why" downward, one layer at a time.
Why did it fail? Because it overheated there. Why did it overheat? Because that part was fitted off-centre. Why was it fitted off-centre? Because...
Ask until you can ask no further.
The logic of it is hard to argue with: only with the right cause is a right remedy possible. Get the cause wrong and every move afterwards is wasted effort.
But on this case the asking stopped dead.
Ask: why did the IGBT fail?
Answer: the samples went to the device maker.
Ask again: what did the device maker say?
Answer: the IGBT appears to be fine.
Ask again: so why did it fail?
Answer: Unknown.
Two layers down and you hit a wall. Because the door to the layer above is not in our hands.
I went back through the reports from those years and counted the root cause boxes. Three kinds.
The first: the cause sits on the supplier's own layer — a work instruction written wrong, incomplete manual information, the wrong mould temperature. Those are filled in plainly.
The second: the cause sits on a purchased part. One of them reads: "possibly the connector was fitted with the wrong thread during installation." The connector was bought in too; you cannot get to the bottom of it, so you write "possibly".
The third is this case: you cannot produce a report that fills all nine boxes at all.
If you cannot, you hold more meetings.
Anyone who has done these reports knows the particular feeling of helplessness.
It is not the kind you get from being shouted at by a customer, or the kind you get from overtime.
It is knowing exactly what to ask, and not being able to ask it.
Nine boxes; you have filled in every one you can. Meetings, chasing, scheduling, adding people — every step is moving, and the form shows no sign of stalling. You can produce a stack of minutes to prove how diligent you have been.
Only the fourth box is still empty.
And it is empty not because you are lazy.
But there is something stranger still: the root cause box can be empty while the corrective action box is full.
The protective plate went on, the engineering change order was raised, the site commissioning instructions were revised, the lightning-protection installation recommendation went to the customer.
Not one of those measures was derived from a root cause — because the root cause had not been found.
The lightning line had long been crossed out by the statistics (very weak correlation), and the lightning training and installation recommendations went on regardless. The environment line could only say "there are corrosion marks", and the protective plate, the production-area audit and the engineering change went on regardless too.
The minutes say so honestly themselves: because of the nature of the failures, it is impossible to estimate what proportion of the problem these changes will solve.
The root cause box can stay empty; the action box cannot — the report has to go out, the customer has to see movement, the meetings have to go on.
So everyone works on "possibly this cause".
The people doing that work are not ignorant of logic. It is that none of us can bear "there is nothing we can do".
An 8D is a tool for a two-layer relationship. A supply chain is a three-layer reality.
Over nine months this case produced twenty-three meetings and twenty-one action items.
Two of them hung on from the first meeting to the last without ever being closed.
One thing I counted out afterwards: at the third meeting there were 9 open action items; by the twenty-first there were 16.
The problem multiplied as the meetings went on.
Attendance went from five people to nineteen.
20 May 2010, the twenty-third meeting.
There are a few lines on that page I still remember.
Roughly 256 failure reports had still not been submitted.
Sixty-three units were scrapped at a subcontractor — most of them from 2008 — with no reports kept.
Possibly more than sixty-one more never reached the inverter maker.
And the status box on that item says three letters and a question mark:
Closed?
Below it, on the same page: should we change the format of these meetings?
Nine months, twenty-three meetings, and what everyone was discussing at the end was whether to keep meeting this way.
The last stroke came when the customer shipped another batch of boards back.
That was the last analysis material available. On first unpacking, it could not be analysed — the boards had been pressed against each other with no separation and no covering between them.
After nine months of chasing, even the chance to "open it up and look" had been crushed out of it in transit.
What I want to say is not "that supplier was no good".
He was not. He was chasing it too — the device maker never gave him the data, and there was nothing he could do.
What I want to say is this: that helplessness is not your problem. It is a problem of position.
You cannot ask the question not because you are unprofessional or irresponsible, but because an 8D assumes you can ask it, and your position does not let you.
So now, when a problem report lands in front of me, the first thing I look at is not the root cause box.
I go to the last page and look at the row of action items — how many are still open, how long they have been open, and whose head they are sitting on.
Whether that box gets filled is not mine to decide.
But what is hanging on my desk will not hang there forever.
Twenty-four years in, I have stepped in plenty of pits ahead of you. We fill in as many as we can.