A couple of things wrong:
1) You are not closing your attribute selector brackets (as suggested already)
2) Your href value contains a full stop, because of this you need to wrap the value in quotes too
This is what you want:
$("a[href$='login.asp']").attr("rel", "lightbox")