There is not a simple formula as such but there is an algorithm you could program to determine the results. Pentadecagon is also correct in that you need to take into account n*360. You are also right in that you stop one of the planets and work on the difference of the speeds.
After d days the difference in degrees between the planets is 30 + d*5.
Since we are only interested in degrees between 0 and 360 then the difference of the angle between planets is (30 + d*5) mod 360.
In case you do not know a mod b gives the remainder when a is divided by b and most programming languages have this operation built in (as do spreadsheets).
You have spotted you want the values of d when the difference is 90 degrees or 270 degrees
So you need to find the values of d whenever
(30 + d*5) mod 360 = 90 or (30 + d*5) mod 360 = 270
pseudo code algorithm
FOR (d=0; d<11; d=d+5)
IF((30 + d*5) MOD 360 = 90 OR (30 + d*5) MOD 360 = 270)
PRINT d
NEXT