From the documentation on date:
In order to output single quote, use two single quotes in a sequence (e.g. "h 'o''clock'").
Since a string literal must also be enclosed in single quotes, you need four single quotes in a row. And since you're using single quotes in your format, you need to enclose the entire format string with double quotes, rather than single quotes.
Thus, your formatter string should be {{model.since | date:"MMM d, ''''yy"}}