I realised there was something wrong with the directional light I had. I just tweaked it a little bit and VIOLA! It's solved.
The problem was with the global directional light, not the point, because on the directional light I had the "Enable Shadows" unticked and it was rotated awkwardly, so I set the rotation to 90/90/90 and made the color black, set the intensity and ticked enable shadows and that solved it!