Program failure at 80+ degrees

 Program failure at 80+ degrees

Hello,

I was hoping to hear your opinion about a serious problem I have - it is either I solve it or reduce my LPC2478 CPU speed from 72[MHz] to 64[MHz] (11% loss. The problem does not seem to be occurring at lower MHz settings). I posted about this in the past but it was a long time ago.
When I place a controller in an environmental chamber and increase the temperature to 80+ Celsius degrees, I often see data abort exceptions, and sometimes I get the impression that the PC takes a hike (even the firmware LED that blinks every 1 second becomes irregular for a while before it stops). The program is launched by a boot loader and has a lower level supporting firmware layer that handles some interrupts (not all). I also see that if RTX is not started at all (but the application hangs in a "for (;;)" loop instead, hence the bootloader and firmware layer were/are involved, but the application is idle) - the system never crashes! I have excluded, as far as I could tell, the roll of external memory or RTX in this situation. However, I still suspect RTX a little (even though my test programs never crashed).
My question: did you ever encounter such a situation? Where do I look best? can this be the result of a misbehaving peripheral? NXP have confirmed the LPC2478 is not the reason.

 RE: Program failure at 80+ degrees

NXP have confirmed the LPC2478 is not the reason.

Odd ... the maximum allowed ambient temperature for the commercial version of the chip is 85 degrees Celsius. "80+ degrees" sounds awfully close to that.

Besides the CPU, there are a few other possible suspects, like the power supply. I wouldn't exclude the external memory until after a thorough examination - at 80 °C the external memory interface of the chip might be operating close to its worst-case specs. Another candidate for investigation is the flash memory of the chip (if it is used).

 RE: Program failure at 80+ degrees

more data: if I empty all the RTX tasks - I experience no crash. How can the temperature determine the behavior, assuming all the components support these temperatures (they do) ?

 RE: Program failure at 80+ degrees

Another thing: I assume those 80 degrees are the temperature inside the environmental test chamber. Is the device in some kind of case or enclosure, so that the components might experience higher temperatures?

If so, it might be worth to stick a thermometer in there and check.

 RE: Program failure at 80+ degrees

if I empty all the RTX tasks - I experience no crash.

Hmmm... if there's nothing being done any more by any task, and the system crashes: how exactly did you expect to experience that fact?

How can the temperature determine the behavior, assuming all the components support these temperatures (they do) ?

Maybe because that assumption is wrong, or some components are actually hotter than you think they are, or this close to their thermal limits, enough components have begun to change in behaviour that your electronic design has been driven beyond at least one of its design margins.

This is pretty much guaranteed to be a hardware problem. The only way heat can affect software behaviour is by affecting hardware first.

 RE: Program failure at 80+ degrees

"The only way heat can affect software behaviour is by affecting hardware first."

Unless the OP is [once again] using undocumented RTX calls ;)

 RE: Program failure at 80+ degrees

The RTX call in question wasn't undocumented. As a matter of fact, the RTX call might actually be incorrectly - or at least misleadingly - documented.

Without the source for the function, I can't prove the correctnes of the documentation, but the documentation did miss reporting one important issue - how the delay function behaved if called at too low frequency. And the no-no part of the documentation looked more like a "you should think about ..." section that somehow had been upgraded into a no-no without a real reason.

 RE: Program failure at 80+ degrees

"... a no-no ... "

If something is stated as being a no-no, and someone decides to use it, then they are (in my book) using undocumented details.

Hmmm... the sign say's don't open this locked door. I'll break it open anyway. Whoops, they didn't tell me that there was a pride of lions on the other side.

 RE: Program failure at 80+ degrees

This is an off-topic discussion that has been run already, but a quick summary.

Missing information that just should be available for os_itv_wait() is that if you call the function with a delay longer than the timer repeat time, the missed events will be counted and you will get one or more instant returns from the os_itv_wait() when you finally calls it, until you have consumed the back-log.

The reason why Keil did say that os_itv_wait() and os_dly_wait() shouldn't be used at the same time is just that a long os_dly_wait() call will result in os_itv_wait() having multiple ticks of back-log. That is not a no-no reason. That is just a "watch out for this".

If Keil thought it important to mention os_dly_wait() and os_itv_wait() for the same thread, then they should have documented the real reason, since the real reason can be trigged by many other ways. Any other wait function will give the same result. A too high IRQ load will give the same result. A high-priority task with a lot of work to do with give the same result.

So in the end, Keil updated a "think about" into a "no-no" for the wrong reason, and while giving the user too little information to be able to catch all the other _potential_ problems of using os_itv_wait() in relation to other code.

 RE: Program failure at 80+ degrees

Just remember how the post started:

"If you violate RTX's rule ..."

Anyway ... As you say, of topic. But slightly more relevant to Keil tools than use of environmental chambers!

 RE: Program failure at 80+ degrees

Hans-Bernhard,

you are right. I am now trying to determine the upper thermal limits of the entire system. it seems a lot more stable now with the production hardware in use - the previous one did use some components whose thermal limits were below 80 degree. so far, 79.3 degrees seem do no harm!

 RE: Program failure at 80+ degrees

"...the previous one did use some components whose thermal limits were below 80 degree."

And you found it was failing at 80+ degrees. Well, there's a surprise (not).

This thread might well have been started with:

"If you violate the thermal specifications ..."

 RE: Program failure at 80+ degrees

No experience of this problem but just a random thought: how susceptible to high temperatures is the quartz crystal that you are using? A quick Google search came up with this statement from a manufacturer:

"Operating Temperature: Standard Operating Temperature ranges are generally considered as -20-+70 degrees Celsius (considered "commercial" Operating Temperature), and -40-+85 degrees Celsius (considered "Industrial" Operating Temperature)"

 RE: Program failure at 80+ degrees

thanks for your attention. there is no casing involved (yet): the product is required to operate at temperatures of up to 70 degrees, but it might exceed that if the casing is installed, thus the vigorous testing. the crystal does not seem to be the problem - at least our hardware people said that...
NXP claim that the LPC2478 was tested at their labs at up to 105 degrees, and some applications allow for us to 120 degrees...!

 RE: Program failure at 80+ degrees

the crystal does not seem to be the problem - at least our hardware people said that...

This is quite likely a hardware problem (I'd wager a few Euros ;) ), so the hardware people should be investigating it or at least be tightly in-the-loop. :)

(This means they should be poking the device with scope probes when it is misbehaving.)

At least around here, all things related to environmental testing are done by our wonderful hardware people.

 RE: Program failure at 80+ degrees

Another thing: Did you test different devices, and are they all showing the same symptoms? Nothing is more annoying than a wild goose chase that, after many dead ends, resolves to merely being a manufacturing defect.

Also, consider using cooling spray to find out which part of the circuit is sensitive to heat.

 RE: Program failure at 80+ degrees

Does it matter if the processor caches memory accesses?

Does it matter if the processor runs at 100% CPU load, or if it sleeps between interrupts?

Have the hw guys verified/probed all external signals - oscillator, power, interrupt inputs, voltage references, ...

 RE: Program failure at 80+ degrees

Christoph, Per,

I _think_ this is something in the application itself, even though LR indicates the data abort originated from RTX code (that is always the case). I tried to change the tick rate to no avail. Tried cooling spray already - I think the processor is the factor, not sure though. My test applications in the same conditions never crash. Other prints suffer from the same problem, no localized defect...
the hardware seems verified.

Per,

Could you please explain why you asked

Does it matter if the processor caches memory accesses?

Does it matter if the processor runs at 100% CPU load, or if it sleeps between interrupts?

?

 RE: Program failure at 80+ degrees

The CPU load will affect the power consumed by the processor which will affect the additional heating internally. It will add extra stress to both processor and power supply.

Caching of memory will both affect CPU load, but also memory timing. If the supply voltage isn't stable or is slightly changing, or the oscillator is jittering or slightly changing frequency, the safety margins can be reduced.

If the chip is running with DRAM, then the refresh needs will be higher at high temperatures but the RAM access patterns can hide the refresh problem. Caching of RAM will change the access pattern and hence change the amount of refresh from normal access cycles. At room temperature, the DRAM refresh may be several times too slow without any problems possible to see because the much slower self-discharge at room temperature.

 RE: Program failure at 80+ degrees

Per,

thanks. I tried refining the parameters of the DRAM according to the data sheet - it is speced to work until 85 degrees, to no avail. but the DRAM is a suspect, as it seems to make a difference if I cool only that component...

 RE: Program failure at 80+ degrees

even though LR indicates the data abort originated from RTX code (that is always the case).

Does the RTX code do anything that the regular code does not (or only infrequently)?

For example:

- Run from internal flash
- Use internal RAM

 RE: Program failure at 80+ degrees

guess what Christoph, how the failure happens somewhere else (LR indicates a non-existing "internal RAM" like address) so I cannot answer your question (I did not write down the data yesterday, darn!). I have asked Samsung to advise us about the changing timing of the DRAM at high temperature.

 RE: Program failure at 80+ degrees

I am working with the production revision of the hardware. this one seems more stable, but still fails. if I learn something new, I will post it.

 RE: Program failure at 80+ degrees

Maybe it is a solder problem.

 RE: Program failure at 80+ degrees

I doubt it. it occurs on many systems, and the hardware has been verified.

 RE: Program failure at 80+ degrees

"if I learn something new, I will post it."

Like when something is not related to Keil tools.

Egads.

 RE: Program failure at 80+ degrees

I understand that you have serious problems in finding something that invokes your intellect. do yourself (and us all, unless you have something meaningful to say) a favor, then: be gone.

 RE: Program failure at 80+ degrees

If you think that the DRAM is the problem, I would try:

- To do a small app where: write to the whole memory (at the highest bandwidth) and then verify. Make an infinite loop waiting for a fail or exception.
- Write down info like time-temp.
- Run the app for DRAM and then for IRAM or if you have many boards do it at the same time with 3-4 of them.

 RE: Program failure at 80+ degrees

Try increasing the temperature to see if the problem gets any worse.

 RE: Program failure at 80+ degrees

Leandro,

Despite "stunned Steve"'s doubt about me, I have already tried this (but thanks for the tip anyway) - all my test programs, including the one you described work well under the above conditions and worse. still searching for the smoking gun/component!

 RE: Program failure at 80+ degrees

Despite "stunned Steve"'s doubt about me

Should of course read:

Despite "stunned Steve"'s doubts about me

 RE: Program failure at 80+ degrees

Despite "stunned Steve"'s doubts about me

I guess you (unlike me, unfortunately...) have found your smoking gun. _SNIFF_ !

 RE: Program failure at 80+ degrees

Have you checked for dry joints?

 RE: Program failure at 80+ degrees

Nevill,

Yes, but not me personally. It seems that until 80 degrees (and perhaps slightly more) everything is fine, but them there is a rapid failure as the temperature increases. can it be that the coating in the print disturbs heat dissipation?

 RE: Program failure at 80+ degrees

What is the humidity of your test chamber?

 RE: Program failure at 80+ degrees

humidity: 0%.

 RE: Program failure at 80+ degrees

"humidity: 0%."

Whew, dry heat man. Doesn't sound likely to use.

Are you absolutely sure about that?

 RE: Program failure at 80+ degrees

well, the control panel of the chamber says: "current value: 0". my girlfriend could use the air inside to feed her hair dryer...do you think I should test another setting (but I think that increased humidity will hurt heat dissipation, not...?) ?

 RE: Program failure at 80+ degrees

It may say 0, but it's very unlikely that humidity in the test chamber is really 0%.

 RE: Program failure at 80+ degrees

of course, I realize that.

 RE: Program failure at 80+ degrees

can it be that the coating in the print disturbs heat dissipation?<p>

Hm, the whole print is coated (including the components)?

 RE: Program failure at 80+ degrees

Hm, the whole print is coated (including the components)?

yes.

 RE: Program failure at 80+ degrees

yes.

In that case, a look at the thermal properties of the coating material might be in order. And maybe a test with an uncoated unit, if available.

 RE: Program failure at 80+ degrees

Remember that a loop continuously accessing the DRAM will look like a super-charged RAM refresh. For problems with RAM refresh, it is often better to fill the RAM with a known pattern and then make sure that the RAM is not touched for a long time so that the only refresh there is comes from the DRAM controller performing background refreshes. Then revisit the chip one every hour and verify that the pattern is still correct.

 RE: Program failure at 80+ degrees

Per,

that's a good idea that I haven't tried yet. thanks!

 RE: Program failure at 80+ degrees

this situation seems to be directly related to the external RAM being addressed by both the application and the LCD controller. if the refresh rate of the LCD controller is reduced, or another type of software is flashed on the controller that does not benefit from the LCD controller - the issue disappears. I will have to adjust the timing parameters of my external RAM, it seems.

 RE: Program failure at 80+ degrees

For the amount of stuff (and content of the stuff) you're posting, you would be better off with twitter :o

 RE: Program failure at 80+ degrees

I don't know about you, but for me, these problems are the cream of the crop of this kind of line of work. almost every problem is a mystery, every problem can be solved (?) in different ways. did I enjoy sitting 3 days in front of an environmental chamber (I have a few burn makes!) ? no way, and the problem is not solved yet (but the probable cause known). but in the end, it is/was a lot of fun!

 RE: Program failure at 80+ degrees

I don't know about you, but for me, these problems are the cream of the crop of this kind of line of work. almost every problem is a mystery, every problem can be solved (?) in different ways. did I enjoy sitting 3 days in front of an environmental chamber (I have a few burn makes!) ? no way, and the problem is not solved yet (but the probable cause known). but in the end, it is/was a lot of fun!

I second this!!, thats a real engineer soul, we are in some way... masochist geeks :)

 RE: Program failure at 80+ degrees

"I second this!!, thats a real engineer soul, we are in some way... masochist geeks :)"

I would totally agree with that, I've been involved in plenty of projects where I've been totally engrossed for weeks/months on end, keeping a note pad by my bed for when I wake up with 'the ultimate answer' (much to the annoyance of my beloved wife).

But ... the difference is, most don't keep trying to share this random blabber out.

Have you ever been to a party and sat next to Mr. Boring?

 RE: Program failure at 80+ degrees

related to the external RAM being addressed by both the application and the LCD controller.

A conclusion you made impossible for anyone else to arrive at, by not mentioning anything about an LCD before, much less that it shared external RAM with the CPU. Is that dual-ported RAM, or how else do you organize shared access?

 RE: Program failure at 80+ degrees

The data transfers LCDController <-> Memory are done by DMA, and there is an automatic mechanism for arbitration, This should not be a problem.

But it should be taked into account if the Video Buffer is located in DRAM and you want to test the DRAM PerÂ's suggestion.

 RE: Program failure at 80+ degrees

Remember that a loop continuously accessing the DRAM will look like a super-charged RAM refresh. For problems with RAM refresh, it is often better to fill the RAM with a known pattern and then make sure that the RAM is not touched for a long time so that the only refresh there is comes from the DRAM controller performing background refreshes. Then revisit the chip one every hour and verify that the pattern is still correct.
True.

Then, for DRAM memories the test case should be: fast access and slow access.

 RE: Program failure at 80+ degrees

Note that fast/slow accesses for a DRAM would most often be the actual timing of the signals. How long time for signals to settle, or to hold. Number of wait states.

There are quite a lot of tests needed for memory. Some for prototypes. Some for factory production. Some for every boot or maybe even regularly when run.
- correct supply voltages at all temperatures and loads
- correct timing of signals
- good flanks and high/low logic levels for signals
- all unused chip-selects etc having pull-up/pull-down
- correctly wired (no shorts/breaks)
- all memory cells working
- stability at maximum load at low/high temperature
- stability at zero load at low/high temperature (refresh working)
- low-power retention (mainly SRAM with super-cap or battery)
- ...

 AHBCFGx settings

Hello,

I have learned a little more about this problem in the mean time and was wondering if you can enlighten me further. I am currently running a weekend test of a controller that utilizes the LCD controller of the LPC2478 vs. a controller that does not. The first one is reduced to 64 [MHz] while the second one still runs at 72[MHz], and they communicate via a RS485 bus. Hopefully this remains stable but either way, I have just reduced the display's processing capacity by 12%...
'Samsung' have promised me that their DRAM (K4S561632J) does not suffer from any issues and that the EMC timing settings used now should apply to the entire range of temperatures (maybe the controller was not warmed up entirely or long enough when I concluded otherwise). I am not sure about the refresh rate, but either way I did try to play with it without any positive results. I am aware that the signals to the DRAM should be measured, but that is not so simple at 80+ degrees.
The latest LPC24xx data sheet elaborates on the AHBCFGx registers which determine the arbitration of the AHB busses (my LCD, DRAM and peripheral(MCI interface uses GPDMA) hang on AHB1) . This is a very fundamental setting that I have no experience changing. Do you think this could help me out? I did a few tests with a negative result, but I feel that I have not exhausted it. Either way, can you think of another system setting that might influence this particular problem? I have, for now, ruled out bad traces and noise as another controller (without an LCD) uses the same hardware design and accesses to external RAM (MCI DMA) ) does not crash.

 RE: AHBCFGx settings

What does the manual say?

Try http://www.embeddedrelated.com/groups/lpc2000/show/35996.php

 RE: AHBCFGx settings

I have found this reference myself, but unfortunately NXP do no explain the impact of modifying these registers. It is of course exceedingly hard to solve a problem that you do not fully understand with tools you do not fully understand...
I believe this has something do to with how DMA/LCD DMA and the processor interact with the AHB bus, which changes slightly when temperature rises. I asked NXP to confirm that they have tested the LCD controller of the LPC2478 at these extreme temperatures but they have not replied yet.

 RE: AHBCFGx settings

If only you hadn't upset Master Zeusti.

 RE: AHBCFGx settings

Right now I am willing to use just about any help - Zeusti, that Steve figure from above, anything. It is either I solve this, or (assuming the system survives the weekend test!) CPU speed for the display has to go down to 64[MHz] !

 I quickread this thread and did not see ....

It is either I solve this, or (assuming the system survives the weekend test!) CPU speed for the display has to go down to 64[MHz] !

I quickread this thread and did not see it mentioned that the internal heat generated by the chip is proportional to the clock speed.

NXP claim that the LPC2478 was tested at their labs at up to 105 degrees, and some applications allow for us to 120 degrees...!
Under which operating conditions??

Erik

 RE: I quickread this thread and did not see ....

Erik,

Thanks for your comments. The answer to your questions is that I do not know: NXP did not elaborate, as far as I can tell, on the exact environmental conditions used to test the chip in any report I could get my hands on. I just don't have enough data to handle this properly...! And you are right: Going down to 64[MHz] might just mask a still existing problem. But at the moment, I don't have any other choice - product beta (thus, installation at the client site) phase is approaching.

 RE: I quickread this thread and did not see ....

OK, twittering continues. The display at 64[MHz] made it though the weekend. There are a couple of display related issues, but it is alive!

 OK, twittering continues.

"...it is alive!"

I can breathe again. (Not to be confused with a yawn.)

 RE: OK, twittering continues.

I will sure to keep you up to speed, Stunned Steve. Hang in there!

 RE: OK, twittering continues.

As promised, I have an update that might interest operators of a LPC2470/78 using the LCD controller. I have found that:
1. lowering the CPU clock speed to 64[MHz] at 80+ degrees seems to stabilize the system. There are no additional legal PLL settings between 72[MHz] and 64[MHz] that support USB, I'm afraid.
2. This code

AHBCFG1 &= ~1 ;
AHBCFG1 |= (3<<12) ;
AHBCFG1 |= (4<<16) ;
AHBCFG1 |= (2<<20) ;
AHBCFG1 |= (1<<24) ;
AHBCFG1 |= (5<<28) ;

when placed in main() {the data sheet does not specify that these fundamental settings are disallowed in application code and indeed they work}, will put the LCD in the most preferred position to access the AHB1 bus. This prevents image jitter and distortion when doing time consuming drawing on the LCD at 64[MHz].

 RE: OK, twittering continues.

"(I have a new, interesting post)"

Where?

There's a new post, but surely interesting is not a true description?

Or was that an attempt at irony?

 RE: OK, twittering continues.

Not knowing your true name, I must wonder: have you ever posted something useful on this (or any other) forum? please stop hijacking my thread. I'm trying to be informative (=helpful). if you are in a mood for child's play, I suggest you go to a kindergarten.

 I'm trying to be informative (=helpful).

Is that you being serious again?

Have you tried looking for the Keil command line option -SetMaxTemp=80

You'll not find it. It's not a Keil related problem.

 RE: I'm trying to be informative (=helpful).

this really is beyond you, ha? this is an issue that might effect Keil USERS and the safety of their product since Keil supports the LPC2478, thus very much their concern. Apart from that, I don't remember asking you when, if or what I may or may not post. Just go away, it won't help you.

 Problems, magnets.

Ok.

You're obviously happy with your tenuous links.

 RE: OK, twittering continues.

There are no additional legal PLL settings between 72[MHz] and 64[MHz] that support USB, I'm afraid.

Have you eliminated the PLL as the cause of the problem? We do some very high temperature stuff and have had problems with PLLs becoming unstable or ceasing to function altogether in devices that have worked fine with an external oscillator.

 RE: OK, twittering continues.

Jack,

As I have mentioned, I have another flavor of this product that does not use the LCD controller but is almost identical in any way (processor print, external RAM etc.). It does not crash under the same conditions. Our hardware guy promised me that the crystal is not the issue. He did not measure yet, though.

 RE: OK, twittering continues.

A small update: Our hardware guy has installed an extra resister across the DRAM clock input and it seems to do wonders! We saw that the first 2 clock pulses of a refresh cycle were far from perfect. On the other hand, this could be an issue with the particular DRAM. Soon we will try another one, and will know for sure.

 RE: OK, twittering continues.

What do you mean "across"? Having a resistor in series with high-speed signals will reduce the rise/fall times. It will reduce the amount of radiated noise but can also solve problems with ringing signals where the input side may see multiple toggles. Having too large resistor will on the other hand produce a shark tooth waveform where you either don't reach the full voltage swing fast enough or where the flanks are so slow that an input without schmitt-trigger will get false readings.

 RE: OK, twittering continues.

I meant: "in series". Sorry!

 RE: OK, twittering continues.

No problem. I did write a bit of ambiguous text too.

I wrote "will reduce the rise/fall times" but should probably have written "will slow down the rise/fall", since the rise and fall times are actually increased :)

 Sign of the times.

Whatever happened to the days where people involved in firmware knew something about the fundamentals of the hardware on which their code would run?

 RE: Sign of the times.

Whatever happended to the days when people on the net always tried to help and be productive, instead of focusing 100% of all comments on harassing people?

 RE: Sign of the times.

Stunned character,
I never used this word on this forum (I apologize to the rest of you in advance) as far as I can recall, but you really are an idiot. Really.

 QUOTE: but you really are an idiot

<sigh/>

 RE: QUOTE: but you really are an idiot

Whatever happened to the days where people involved in firmware knew something about the fundamentals of the hardware on which their code would run?

Interesting argument. FYI, the erroneous clock signal was a direct product of higher temperature. One can induce this using even something as simple as a warm air blower directed at the DRAM, but this behavior is not documented anywhere nor does it happen with other DRAMs. So you see, this is what happens when one focuses on making empty arguments for the sake of making them rather than on the facts. This is engineering man, not religion.

 RE: QUOTE: but you really are an idiot

"the erroneous clock signal was a direct product of higher temperature."

You stun me with every detail you provide.

"This is engineering man..."

LOL

 RE: QUOTE: but you really are an idiot

Next time you go the liqueur store, Steve, try to be polite and remove the hard glued sticker from your forehead: "Born to be a loser".

 QUOTE: Born to be a loser

<sigh/>