So, so far I have ended up implementing the touchesShouldBegin:withEvent:inContentView:
method of my UIScrollView subclass but with delayContentTouches
set to YES (default) instead of NO as @nhahtdh was suggesting.
Strangely enough even only implementing the method was sufficient for my subviews to intercept the dragging, and still my scrollview is scrolling properly, while with delayContentTouches set to NO I was not able to scroll it as all the subviews were starting to move around.
Really the credit for this is @nhahtdh, so man, if you post an answer I will accept it, thank you very much for your help.