If I understand well, you want to model the two main effects (say, a and b) only together with their interaction (a:b). So how about: subset = !a | (xor(a, b) | 'a:b')
(enclose a:b in backticks (`) not straight quotes), e.g:
library(MuMIn)
data(Cement)
fm <- lm(y ~ X1 * X2, Cement, na.action = na.fail)
dredge(fm, subset = !X2 | (xor(X1, X2) | `X1:X2`))
or wrap this condition into a function to have the code more clear:
test <- function(a, b, c) !a | (xor(a, b) | c)
dredge(fm, subset = test(X1, X2, `X1:X2`))
that produces: null, X1, X2, X1*X2 (and excludes X1 + X2)