Hex file converter

[Gunplumber]

Hi Guys
I have decided to look into a bootloader for my project. I've done a lot of reading of various threads here on the basics of bootloading and i think i have a reasonable grasp of how it works.
I am now at the point of looking at how best to handle the HEX file from Swordfish. I want to keep my boot loader as small as possible so i plan to do the decoding on my PC and then save the resulting binary file to an SD card that the PIC can read.
I would like to add some form of encryption, although i have not looked at this as yet.
So before i go headlong into a program to convert hex file format to binary, has anyone else done it? Any tips or pitfalls to be aware of?
I have studied the HEX format and i think i can deal with it, but a few questions first..
Is the addressing and data in a hex file always sequential? Ie once you know the first address can you then just decode the data bytes and store them sequentially?
Does the hex file usually have gaps between addresses that need to be recognized and filled with NOPs (FF)?


Cheers
Lee

[Jerry Messina]

Does the hex file usually have gaps between addresses that need to be recognized and filled with NOPs (FF)?

Yes. You will also need to account for addresses that don't fall on a write block address.

The CONFIG and any EEPROM initialization data will also be in the hex file, and those are located way past the "end of program memory" for a K40.

Is the addressing and data in a hex file always sequential? Ie once you know the first address can you then just decode the data bytes and store them sequentially?

There's nothing in the hex file spec that requires the blocks to be sequential, but having said that I don't recall the tools ever producing one that was out of order.

[Gunplumber]

G'day Jerry

Thanks for the reply as usual!

I've had a good dig through some hex files and have started work on my hex to binary converter tools.
One thing i have noticed and can't seem to figure out... I've found the configuration bytes at the end of the hex file as indicated by an address of 0x300000 followed by 12 bytes of data which i assumed was the configuration words. However when i open the hex file in PICKit programmer the bytes in the configuration words are different.. What am i missing here???
I've attached a screenshot to show..

Cheers
Lee

[Jerry Messina]

I only looked at a few bytes, but it looks to me that the only difference is in how the 'unimplemented bits' are displayed.
In the hex file, they're '1' (the erased, unprogrammed state), while the app seems to be showing them as '0'.

For example, the first byte @ 300000 is 0x8C = 1000 1100. The app shows it as -000 -100, or 0x04.
Same with the second byte @ 300001... 0xFF = 1111 1111, shown as --1- 1--1, or 0x29.

In any case, you should think long and hard about allowing your bootloader to reprogram the config section.
If you get it wrong, or the bootloader has a problem while reprogramming them (ie power glitch) you can brick the device.

I usually only allow rewriting the config in very special circumstances. It does mean you have to build the bootloader and app with the same config settings, so that's a consideration.

You can write-protect the config (WRTC) and boot block code areas (2KB, 0x0000-0x07FF) (WRTB, EBTRB) if it's appropriate.

[Gunplumber]

Jerry

Ahhh of course.. Now that i look at it you are correct. If i change how PicKit displays them i can get them to match.
I totally agree with you regarding the bootloader not changing the config bytes,i was just working out how they work.
My Conversion tools will ignore them.

Any thoughts on my idea of XOR Encryption? I've done some reading and i get very mixed opinions on the security of XOR encryption, however most of the examples i have seen refer to encrypting plain text, not Micro code bytes..
I suppose i could always research a full on AES encryption algorithm, but i want to try and keep things a simple and small as possible..

Cheers
Lee

[Jerry Messina]

>Any thoughts on my idea of XOR Encryption?

Sounds reasonable to me... I would think that would be enough to deter most folks.
I guess it depends on how secure you want things.

[octal]

Any thoughts on my idea of XOR Encryption? I've done some reading and i get very mixed opinions on the security of XOR encryption, however most of the examples i have seen refer to encrypting plain text, not Micro code bytes..
It appears to me like a 64 byte key should be nearly impossible to brute force, but i do know there are several mathematical and statistical shortcuts for XOR encryption.. I suppose i could always research a full on AES encryption algorithm, but i want to try and keep things a simple and small as possible..

Cheers
Lee

As for AES encryption/decryption on an 8bit chip, I would stay far from that unless you really need it. I already implemented it about 8 or 9 years ago for a customer (on Swordfish BASIC from scratch using the official AES specification description pdf file) and it worked like a charm (data was decrypted on a linux server with OpenSSL).
As of XOR encryption, almost ANY encryption scheme can be defeated these days (especially with AI powered tools).
What you should ask yourself is: if I encrypt my firmware using ANY scheme, and without giving any info about my encryption scheme (on social networks or other communication channels), how would one take my file, try all decryption algorithms, understand what you encrypted and how final firmware should look (as he must do comparison with some samples)?
if the app you give to final end customer already use encrypted firmware and send it as is to the firmware, that means end user wont even get any info from disassembling the app in order to understand the encryption scheme.
I really think that if a final end user find a way to defeat your encryption scheme, he will certainly have competence to write the firmware from scratch. And and if he has not such competences and your firmware is worth all that efforts, he would certainly spend $5000 or $6000 using the service of any chineese company and getting your device fimrware unlocked from a physical chip (they have labs very good at that).

use a good encryption scheme, and understand that the best protection for me is:
- Never talk about your encryption tools, schemes and even choices to anybody
- Never implement encryption/decryption or verification in the app that will feed the encrypted firmware to your firmware

By respecting those two rules, you'r safe for long time until your product succeeds, and then you should prepare yourself to spend parts of your earning in developing new solutions and new versions of the firmware/hardware, and keep in mind: everything is copiable these days

[Gunplumber]

Jerry, Thanks once again for all your advice.

Octal,
Thanks for the detailed reply. I really am out of me depth with encryption. I don't think my end device would warrant a cost of $5000 to crack the firmware, but i didn't want to make it trivial either. I'll take your advice and Edit my posts here.
Cheers
Lee

[Gunplumber]

Jerry
I now have some working tools to convert a hex file into a Binary file that i can read from an SD card.
I am now trying to understand how the #option ORG_Program commands works, and the intricacies of moving Swordfish generated code in memory.
At this stage my plan is a high memory boot loader, with the user program in low memory. This is not set in stone, but it does negate the need to deal with interrupt vectors etc.. I plan to give access to the bootloader via the main low memory user program. How would one go about this?
I understand how the ORG statement can locate code at specific memory locations, so if i use the following at the start of my boot loader program

#Option Org_Program=$8000

Locates the start of the program at address $8000, however the compiler also adds GOTO SBCDSTD at address $0000. I understand the reason for the goto but how would one go about jumping to SBCDSTD from another Program in low memory without knowing its address?
I suppose i could pull that address from the boot loader hex file, but is there a way to set it (or determine it) inside swordfish?
The GOTO statement in swordfish is always given with a address label. Is there a way to use GOTO to an actual literal address? Again i suppose one could do it directly in an inline assembly block, but is there a better way?

Cheers
Lee

[Jerry Messina]

If you want to move the bootloader up into high memory, use '#option ORG_RESET' instead of '#option ORG_PROGRAM'
ORG_RESET will move everything for the bootloader project.

Then, build your app program as normal. To jump to the bootloader you could use something like:

Code: Select all

#option BOOTLOADER_SIZE = $1000                 // size of the bootloader (4096)
#variable _maxrom = _maxrom - BOOTLOADER_SIZE   // limit size of app to exclude the  bootloader space
const BOOTLOADER_ADDR = _maxrom                 // addr of the bootloader

INTCON = 0	// disable all intrs
BSR = 0		// init bank select reg
STKPTR = 0	// clear stack
asm-
    goto BOOTLOADER_ADDR
end asm

Remember when the bootloader starts, the app will have already initialized many registers, so don't assume anything.
Also, there's no way to get into the bootloader without it being invoked from the app unless you ensure that the entire flash memory below the bootloader is erased (eventually you'll get to the bootloader as it executes 60K of 'nops')

[Gunplumber]

Jerry
Once again thanks for your reply..
A few more questions if i may pester you further.
I have noticed that the 16 bit data words in the Hex file are stored as Byte1-Byte2, Byte1-Byte2 etc. But when you look at the code window in the PICKit programmer it shows them in reverse order IE Byte2-Byte1..
Which order do they go when writing to the TABLAT register?

The 26K40 has a 64 byte block write cycle, but i can't find anything in the literature to state whether that is from any address, or from a multiple of 64 byes in the address?
IE can i only do a write from address $0000, then another from Address $0040 and so on?
Or can it be a 64 byte block from any address? IE write at $0012 followed by another write block at address $0052 ?

Cheers
Lee

[Jerry Messina]

Which order do they go when writing to the TABLAT register?

The PFM is byte addressable. Write them in the order they are in the hex file, which would be lowbyte, highbyte.
The programming app is displaying them as little-endian words, so you see it as highbyte/lowbyte.

Regarding the write block and alignment, supposedly you can write to non-aligned addresses as long as the write buffer contains 0xFF.
The datasheet states (DS40001816F, bottom of page 167):

The default value of the holding registers on device Resets and after write
operations is FFh. A value of FFh in a holding register will not change the PFM location
corresponding to that register when the block is programmed.

That's because it can't change a byte that's already programmed with a '0' back to a '1'... only an erase can do that.
You can't cross a page boundary, though, so if it's non-aligned you can't write 64 bytes.

Personally, to avoid any confusion I put the write data into a 64-byte buffer that I prefill with FF and always do the writes page-aligned.
In any event, when you're done coping the data to the page write buffer make sure the bottom six bits of the TBLPTR are still pointing at the page you want to program before you initiate the write. If the copy was done using the asm 'TBLWT*+' instruction (write w/autoincrement), after 64 bytes it'll be pointing to the next page.

You can copy 63 bytes using the 'TBLWT*+' and then the last 64th byte with 'TBLWT*'

[Jerry Messina]

... and for future reference note that not all chips work this way as far as writing partial pages.

For example, the Q40 datasheet states:

To modify only a portion of a previously programmed page, the contents of the entire page
must be read and saved in the buffer RAM prior to the page erase. The Read Page operation is the
easiest way to do this. The page needs to be erased so that the new data can be written into the buffer
RAM to reprogram the page of PFM. However, any unprogrammed locations can be written using the
single word Write operation without first erasing the page.