hard fault caused by scatter loader decompress routine

Does it depend how long it's been running? If you add an idle loop in ResetHandler does it change the behaviour? How about after the DDR is initialized?

Decompression most often fails or goes off in the weeds, when the data stream it is decoding is corrupted. It's designed for efficiency, not integrity.

The linker chooses to use compression, or not, depending on the static data it is trying to copy into RAM, and if compression actually makes the image smaller, or not.

Hi
Thanks for getting back to me so quickly. The decompress routine is called as part of the C library initialization long before my code starts. So it is not directly a issue with anything that my codes does because the system hard faults before it can run.

The scatter load process which decompress is part of is trying to initialize static and initialized variables before my code runs.

The DDR is successfully initialized by a script that the debugger runs before loading symbols and running any code. I have verified that is this does work correctly and memory is accessible.

Over the weekend my trick of getting out/in of the debugger was workign but then I made one small change (added a +1) to a routine in my code and recompiled and reflashed it into nvm and now that hard fault happens every single time I run the code. It always faults in the same loop in the decompress routine. It tries to load a value from undefined memory. It looks like it is a computed address from looking at the assembly code but it is pretty complicated.

This is why I was thinking there was some type of memory alignment problem or something because the only thing I did was make the code a few bytes bigger.

After this started happening and having absolutely no other ideas as to what to do, I recompiled all of my code in the project, reflashed, and was able to get to main. No hard fault. I tested and wanted to restarts so I did my trick with getting out and back into the debugger which had cleared this up before (I have no idea) but it did not work. This was exactly the same code in flash that started without a hard fault just a few moments before. What would change? power cycling does not help either.

"Decompression most often fails or goes off in the weeds, when the data stream it is decoding is corrupted. It's designed for efficiency, not integrity."

I had thought about this but the it looks like the addresses and even some of the data to be written are "calculated" using the current PC address or some odd math which I assume is done to save code space. And this code and the calculation are flashed into nvm so there should not be any corruption. Although if it uses the stack it would be vulnerable to having values corrupted. But since it is the only code running, it would be stepping on its own variables/data.

Since this is generated code, I do not have any direct control over how its logic works. That is the scary thing.

Is there any way to tell the linker not to use compression? Maybe there is a problem with the decompression logic that I can avoid by asking it not to do that.

I am at a total loss as to why the C library decompression routine would make a mistake and cause a hard fault every time now. And before it was only after clicking reset but could be cleared if I got out of the debugger and back in.

You seem to be having a lot of fun with this platform.

Code that you provide must initialize external buses and memories BEFORE they are used. Keil's C runtime code which initializes the statics is called by __main prior to it calling your main() function. The linker's understanding of memory is controlled by the target GUI, or the scatter file you supply. The processor Hard Faults in the same way an 68K would DTACK fault if the underlying memory (address) you are attempting access is NOT present, or doesn't permit writing, etc.

You can add code to the startup_arch.s file, and you are expected to initialize SDRAM/DDR via routines in, and called by, SystemInit(). But be conscious that such routines CAN'T use memory that will be subsequently initialized. As such you should explicitly initialize any variables, and not assume they will be zero, or have statically assigned values. The C runtime code has not been run at this point.

I'd start by assuming that the linker/compiler are doing their job, the code has been tested and used in millions of devices.

The linker creates tables of load regions which need zeroing, or copying from flash. There may be options to prevent compression being used, but in general you're just seeing that it's more sensitive to bus/data anomalies, and this is really a big red flag for underlying issues you need to address.

Focus on what's broken in your M3 architecture, flash and prefetching strapped to the core. Use a debugger and trace hardware to understand exactly is supposed to be read and what is actually being read by the executing code.

Review if the .MAP files suggests data is being placed outside the scope of the hardware design.

Might also suggest you add a small assembler checksum or CRC routine as the first instructions in the ResetHandler, and have it validate the executable image against data you've computed on the PC from the generated .HEX/.AXF files.

This will save you a lot of time chasing ghosts in the machine.

Thanks for your response. Yes I have been turning very grey over working with the M3 ARM. I have worked with many different microprocessors over the past 34 years and this is my first experience with a ARM and its been very frustrating.

I have verified that the correct device is selected so the CMSIS initialization should be correct, it initializes the DDR memory so doing it in the debugger script is redundant when using nvm based code. I will not use the script until I start using remapping again and see if that helps. Other than DDR I do not see anything else being setup or initialized by the CMSIS code. The reset is the scatter load code.

Outside of having the scatter file locate my variables and heap in DDR memory I do not force any memory to be in any particular place. I leave that up to the linker

Below is the part of the code that is causes the fault at address x0210, R4 contains 0x9FFFFFEE which is totally illegal and the LDRB read causes the fault. I do not know what it trying to read. I would have thought it would only be writing to memory and using the stack area if anything for reads. It looks like it computed this value at 0x1FA by doing a subtraction with R1 which contains the start address of DDR 0xA0000000. By going backwards from there instead of forwards into the DDR it uses a bad address.

                 __scatterload_copy:
0x000001C4 F8103B01  LDRB     r3,[r0],#0x01
0x000001C8 440A      ADD      r2,r2,r1
0x000001CA F0130403  ANDS     r4,r3,#0x03
0x000001CE BF08      IT       EQ
0x000001D0 F8104B01  LDRB     r4,[r0],#0x01
0x000001D4 111D      ASRS     r5,r3,#4
0x000001D6 BF08      IT       EQ
0x000001D8 F8105B01  LDRB     r5,[r0],#0x01
0x000001DC 1E64      SUBS     r4,r4,#1
0x000001DE D005      BEQ      0x000001EC
0x000001E0 F8106B01  LDRB     r6,[r0],#0x01
0x000001E4 1E64      SUBS     r4,r4,#1
0x000001E6 F8016B01  STRB     r6,[r1],#0x01
0x000001EA D1F9      BNE      0x000001E0
0x000001EC 2D00      CMP      r5,#0x00
0x000001EE D015      BEQ      0x0000021C
0x000001F0 F8104B01  LDRB     r4,[r0],#0x01
0x000001F4 F003030C  AND      r3,r3,#0x0C
0x000001F8 2B0C      CMP      r3,#0x0C
0x000001FA EBA10404  SUB      r4,r1,r4
0x000001FE BF0A      ITET     EQ
0x00000200 F8103B01  LDRB     r3,[r0],#0x01
0x00000204 EBA41483  SUB      r4,r4,r3,LSL #6
0x00000208 EBA42403  SUB      r4,r4,r3,LSL #8
0x0000020C F1050301  ADD      r3,r5,#0x01
0x00000210 F8146B01  LDRB     r6,[r4],#0x01
0x00000214 1E5B      SUBS     r3,r3,#1
0x00000216 F8016B01  STRB     r6,[r1],#0x01
0x0000021A D5F9      BPL      0x00000210
0x0000021C 4291      CMP      r1,r2
0x0000021E BF38      IT       CC
0x00000220 F8103B01  LDRB     r3,[r0],#0x01
0x00000224 D3D1      BCC      0x000001CA
0x00000226 4770      BX       lr

I have spend this evening looking at what decompress is doing before it fails and I found something interesting. When the uVision debugger is started it writes to my DDR memory a lot of values starting at the start of DDR. I found that the first thing the decompress assembly does is to read these values beginning at the start of DDR and uses them to compute addresses. I see as it loops the start of DDR being overwritten by what I assume are the initial values that to be there. It overwrites those initial value that it used to compute the addresses.

If I click on reset and restart, of course those values are not there but decompress uses them anyway and that is that causes it to come up with a bad address. the debugger did not write the value into the DDR for it when restarting.

I do not know why the debugger would write these very critical values to the DDR or why the decompress would look to what should be uninitialized RAM for any values to use. when I deploy this software the debugger will not be thee to write these values to DDR for decompress to use.

The code needs to write whatever values it needs to use for these addresses to DDR itself when starting up. That is not happening. I did a test by starting the debugger, it did its write to memory and I clear the first 5 long words hoping to see that the code would put the correct values back when it started and that the debugger was not needed. Well, it did not write anything to DDR and hard faulted.

So that is what is going on. I just need to know why this dependency on the debugger to write these critical values for decompress to work correctly exists and how to stop it.

Does this make sense?

Is there a compiler switch for debug verses deployment that may be preventing some additional initialization code from being written?

If I can get that code to put those values out there in DDR, then it will not compute a bad address and hard fault.

How does the application get into DDR on the deployed system?

How does the data in DDR when it first starts related to the data in the .HEX or .AXF representations of the memory space? Does a checksum mechanism confirm the integrity?

Does the image in DDR corrupt itself? Does it behave like a FLASH/ROM image? Or do things like the vector table get copied, modified, or remapped?

Do the ROM and RAM regions defined for the linker overlap? Or stack or heap create conflict?

Not sure what the debugger or debug script are contributing here, or if the debugger is attempting to break-point the initial run so as to "run to main"

Does the code function normally if you deliver the image to DDR and run it absent the debugger?

Hi

I have flash it into memory and let it run and it still gets a hard fault.

I have been doing some testing/investigating and I think I was off on thinking it was the compiler setting. But there is something going on interesting. I wish I could post a screen shot because that would be clear as to what I have found out.

When the debugger starts it does write to DDR. ESRAM also contains the exact same data in the same order. I believe decompress is copying the data in esram to the DDR. It writes what will be my initialized data. It writes to 0xA0000000 (mixed hex and ascii)

0x0 0xD 0x1A SD_NO_ERROR . .. .

My memory map says that at this address should be my error array which has a first value of SD_NO_ERROR (ascii). THE 0 is the offset to the current address to write, the 0xD is the length of the string and the 0x1A is ???. there are 3 hex bytes in front of every string.

I can run to main and this is what the code initialized the DDR memory to:

SD_NO_ERROR .....

This is exact correct. What the debugger wrote was ignored and over written as it should be.

Now, if I click reset and go to decompress and I see that the 3 bytes before the string are missing from ESRAM so when it loads the offset to write it gets 0x53 ("S") instead of 0 and a count of 3 bytes instead of 0xD:

D_NO_ERROR

If I let it continue with these bad values it will eventually cause hard fault because it tried to write to address 0x9FFFFB5 which would be below my DDR.

If I reload with the debugger and run to the start of decompress the esram looks good with with those 3 bytes before each string. I can reset over and over and every time it looks the same and correct.

But if I allow it to write just the first string SD_NO_ERROR to DDR and then do a reset and run to the start of decompress, those 3 bytes are missing from ESRAM and SD_NO_ERROR is at 0x2000000 instead of those 3 bytes and then the string.

Why writing the string to DDR would cause whatever initializes the esram with the string values to be off by 3 bytes is a mystery? The code is in nvm memory and so should these initial values that are being put into esram and writing them out to DDR should have not affect.

Getting out of the debugger and back in fixes this data in esram some how because after doing that and running to decompress the esram is correct again.

The debugger is doing something with getting that data into the esram correctly although that all should be done by the code itself.

This is why i thought it might be a compiler switch issue but now it looks like the act of writing the data to DDR changes what is loaded upon the next reset by 3 bytes.

I'll look back farther into the scatter loader to see how that esram data is being put into esram memory.

Ok I found how the data that was being put into esram is coming from. The scatter loader is called first and it calles scatter loader_null which believe it or not copies the values in the DDR into the esram!!!!! Holy cow. The DDR has been either just written by the debugger with the correct init values or was over written with the initialization values.

On the second (after reset) and all other runs of the code it will continue to copy the final initialized code into esram DDR and then copies the esram data BACK INTO DDR. The 3 bytes at the beginning of each string is long gone. It is only there on the first run when the debugger puts it into DDR. All runs of the code after that are picking up whatever the DDR was initialized to.

This is not offset, string length or any information to tell the decompress routine how much to copy and to where. Its just chaos at this point.

So the code is definitely depending upon the debugger to put this proper data into DDR so it can copy that to esram so that it can be decompressed back into DDR memory minus the 3 bytes of string information.

This cannot not possibility be how this ARM processor and code will run operationally. It cannot depend upon a debugger to put that information in DDR memory.

There is some code that should be created by the linker to do this and not the debugger. Obviously this code is not being generated by the linker so that the proper initialization information is made available to the decompress routine.

How does one turn that on? I need some help here because I have not idea why this missing code is not being created.

So exactly what does your scatter file look like?

Sounds a bit like some data that in a normal build should be stored in flash is in your build stored in a RAM region (downloaded by the debugger) that will be overwritten when the program starts to run - so a second run will basically have "parts of the 'nonvolatile flash' destroyed".

Exactly - show us your scatter file. The problem is there, or in your hardware.

OK that would make total sense. I am sure I have accidentally put that initial data into the DDR memory space through the use of a wildcard * .

I didn't even think about that. I assumed that the data would be a integrated part of the code and not a separate data item. I just put every RW section into DDR. I assumed that this static data would RO only. It should be RO data and not RW. I think that would be a misclassification of that type of data

Do you know what it would be called? Its the .data section right? The .bss is my read/write variables?

0xA0000000 is where I found the scatterload loading the initialization data and that is where the .data sections are a grouped together.

FLASH_LOAD 0x00000000 0x00080000      ; load region size_region
{
    ER_RO 0x00000000 0x80000    ; load address = execution address
    {
        *.o (RESET, +First)
        *(InRoot$$Sections)
;        startup_m2sxxx.o (.text)
       system_m2sxxx.o (.text)
;       sys_config.o (.text)
       low_level_init.o (.text)
        retarget.o  (.text)
         * (+RO)
    }

    ER_RW 0x20000000 UNINIT 0x10000
    {
        startup_m2sxxx.o (STACK)
    }

}

MDDR_RAM 0xA0000000 0x10000000
{
    ER_DDR 0xA0000000 UNINIT 0x10000000  ; RW data  DDR
    {
       * (+RW +ZI)
       * (HEAP)
    }

}

Base Addr    Size         Type   Attr      Idx    E Section Name        Object




    0xa0000000   0x00001c8e   Data   RW            7    .data               accumulationnode.o
    0xa0001c8e   0x00000002   Data   RW          535    .data               statemachine.o
    0xa0001c90   0x00000004   Data   RW          777    .data               processidle.o
    0xa0001c94   0x00000001   Data   RW         1008    .data               timerirq.o
    0xa0001c95   0x00000001   PAD
    0xa0001c96   0x00000002   Data   RW         1069    .data               dummy.o
    0xa0001c98   0x000001e4   Data   RW         1204    .data               diskio.o
    0xa0001e7c   0x00000006   Data   RW         1365    .data               ff.o
    0xa0001e82   0x00000002   PAD
    0xa0001e84   0x00000055   Data   RW         1435    .data               uart.o
    0xa0001ed9   0x00000003   PAD
    0xa0001edc   0x0000001c   Data   RW         1603    .data               system_m2sxxx.o
    0xa0001ef8   0x00000004   Data   RW         1672    .data               retarget.o
    0xa0001efc   0x00000008   Data   RW         1888    .data               mss_can.o
    0xa0001f04   0x00000018   Data   RW         2047    .data               mss_pdma.o
    0xa0001f1c   0x0000002c   Data   RW         2121    .data               mss_comblk.o
    0xa0001f48   0x00004d34   Zero   RW            5    .bss                accumulationnode.o
    0xa0006c7c   0x0000000c   Zero   RW         1203    .bss                diskio.o
    0xa0006c88   0x000002d0   Zero   RW         1504    .bss                fault_handler.o
    0xa0006f58   0x00000018   Zero   RW         1950    .bss                mss_hpdma.o

Moving the .data sections to nvm solved the problem. I can run to main every time now. Decompress does not even get called now. I could not see the forest because of the trees.

Thank you very much for you help in leading me to the answer.

Steve