The FFT solves an interpolation problem, the IFFT the corresponding multi-point evaluation problem
The result of the FFT are coefficients of a polynomial p(z) where p(wk)=xk, w is the unit root associated with the dimension D of the FFT.
Denote with ps,k(z) the polynomial formed from the subsequence of coefficients of p(z) starting in k and spaced with index distance s.
If D=2N is even, the general idea to get from the values to the coefficients is to use wN=-1 and the decomposition
p(z) = p2,0(z²) + z * p2,1(z²)
in the relations
xk = p(wk)=p2,0(w2k) + wk * p2,1(wk)
xN+k = p(wN+k) = p(-wk)
= p2,0(w2k) - wk * p2,1(w2k)
to reduce the big interpolation problem into two subproblems of half the size,
2*p2,0(w2k) = xk + xN+k
2*p2,1(w2k) = w-k * (xk - xN+k)
In the next step,
- p2,0 is reduced to p4,0 and p4,2, and
- p2,1 is reduced to p4,1 and p4,3
etc.
For the sequence of 8 inputs, one gets in the first step w=(1+i)/sqrt(2) and w2=i
- x1,0 = 2*p2,0( 1) = x0 + x4
- x1,2 = 2*p2,0( i) = x1 + x5
- x1,4 = 2*p2,0(-1) = x2 + x6
- x1,6 = 2*p2,0(-i) = x3 + x7
..
- x1,1 = 2*p2,1( 1) = (x0 - x4)*1
- x1,3 = 2*p2,1( i) = (x1 - x5)*(1-i)/sqrt(2)
- x1,5 = 2*p2,1(-1) = (x2 - x6)*(-i)
- x1,7 = 2*p2,1(-i) = (x3 - x7)*(-1-i)/sqrt(2)
In the next step, one gets w=i and w2=-1
- x2,0 = 4*p4,0( 1) = x1,0 + x1,4
- x2,4 = 4*p4,0(-1) = x1,2 + x1,6
..
- x2,2 = 4*p4,2( 1)= (x1,0 - x1,4)*1
- x2,6 = 4*p4,2(-1)= (x1,2 - x1,6) * (-i)
--
- x2,1 = 4*p4,1( 1) = x1,1 + x1,5
- x2,5 = 4*p4,1(-1) = x1,3 + x1,7
..
- x2,3 = 4*p4,3( 1)= (x1,1 - x1,5)*1
- x2,7 = 4*p4,3(-1)= (x1,3 - x1,7) * (-i)
and finally the FFT coefficients Xk=x3,k using w=-1 and w²=1
- X0 = 8*p8,0( 1) = x2,0 + x2,4
- X4 = 8*p8,4( 1) = (x2,0 - x2,4)*(-1)
--
- X2 = 8*p8,2( 1) = x2,2 + x2,6
- X6 = 8*p8,6( 1) = (x2,2 - x2,6)*(-1)
--
- X1 = 8*p8,1( 1) = x2,1 + x2,5
- X5 = 8*p8,5( 1) = (x2,1 - x2,5)*(-1)
--
- X3 = 8*p8,3( 1) = x2,3 + x2,7
- X7 = 8*p8,7( 1) = (x2,3 - x2,7)*(-1)
Please check that these steps are reflected in your code, especially, that the square root of 2 occurs at the necessary places. It is highly unlikely that the output of random integer inputs is again strictly integer.