This really racked my brain for quite some time. I finally figured out you need to enter the number in decimal not in binary or hex.
C:>bitsadmin /SetSecurityFlags myJob 8
The 8 will make the "Ignore invalid certificate authority in server certificate :true"