I found a solution on the solarized bugtracker.
This config lets one assign colors to certain elements and groups:
highlight CursorLineNr cterm=none ctermfg=0 guifg=#073642
highlight NonText cterm=none ctermfg=0 guifg=#073642
highlight SpecialKey cterm=none ctermfg=0 guifg=#073642 ctermbg=8 guibg=#002b36
Notice the
cterm=none
which fixes the problem.