Have a look at the example in the documentation: https://github.com/posabsolute/jQuery-Validation-Engine#selectors
Remove the quotes from your regex definition:
"onlyMAC":{
"regex":/^([A-Z0-9]){16}$/,
"alertText":"* MAC invalid"}
The issue is that you are mixing the regex literal syntax with the regex constructor:
/^([A-Z0-9]){16}$/ // RegExp literal uses the slashes
new RegExp('^([A-Z0-9]){16}$') // RegExp constructor takes a string (without the slashes).