Theoretically, it is possible, but its probably going to require extraordinarily verbose code, especially in XSLT1.0.
Network masks rely heavily on bitwise logic, which this answer covers in XSLT. But not only that, you'll need to tokenize the string first, which isn't easy/short in XSLT 1.0.
Then you need to verify that each octet is correct (i.e. consecutive 1
's followed by consecutive 0
s).
In all, it might just be shorter code-wise to list the 31 cases, and check against those in its own little named template somewhere. Possibly even computationally quicker, since string tokenization would be recursive, as would the bit logic.