One way is listed by @ildjarn in the comments:
let isInt x = box x :? int
An idiomatic way would be to use pattern matching. First, define a discriminated union which defines the possible options:
type Test =
| IsAnInteger of int
| IsADouble of double
| NotANumber of Object
then use a match statement to determine which option you got. Note that when you initially create the value you wish to use with a match statement, you need to put it into the discriminated union type.
let GetValue x =
match x with
| IsAnInteger(a) -> a
| IsADouble(b) -> (int)b
| NotAnInteger(_) -> 0
Since you're probably going to use your test to determine control flow, you might as well do it idiomatically. This can also prevent you from missing cases since match statements give you warnings if you don't handle all possible cases.
>GetValue (NotAnInteger("test"));;
val it : int = 0
>GetValue (IsADouble(3.3))
val it : int = 3
>GetValue (IsAnInteger(5))
val it : int = 5