Your error is twofold. First of all you write template code in a .cpp file. Then, to correct that error, you include the .cpp file in a header, another bad idea.
Why?
A class template is not really a class, yet, just a template for a group of classes. The classes that can be made from it are created as needed. A .cpp file, on the other hand, is only compiled once, which is not enough and not even meaningful, since you only have a template at this point.
On the other hand, to include a .cpp file is a bad thing in itself, since it typically results in recompilation of code that may not be recompiled, hence the link errors.
EDIT: Either you can define the specializations in a .cpp file (but no pure template code, and don't include it!), or you can inline them and keep them in the header:
#if USE_NEON_UPSAMPLING2X
template<> inline Matrix<float> Matrix<float>::upsample2x() const{
... // specialized for ARM NEON float32_t
}
#endif
#if USE_NEON_THRESHOLD
template<> inline Matrix<float> Matrix<float>::threshold(float thresholdValue)const{
... // specialized for ARM NEON float32_t
}