It depends on how your linker script is written, if your entry point address is not set to a static location in the linker script, then there could be the chance that other data/code is put in the object file before your entry point, effectively moving the location of your entry point and indeed causing problems.
I typically solve this by creating a new section with only 1 symbol in it, and a jump/branch instruction as follows:
.section entryPointSection
b myCodeEntryPoint
Then in your linker script put the entryPointSection
at the hard coded address that your bootloader will jump to.
The myCodeEntryPoint
label can be the name of a C function (or assembly label if necessary) that is in the normal .text section and can be linked anywhere within reach of the jmp. It will become your entry point, but you don't really care where it is because the linker should find it and link it properly.
Consider posting your linker script if you have further questions.