Another (more complex) regex to match the function name.
:hi luaCustomFunction ctermfg=yellow
:syn match luaCustomFunction "\(\<function\>\)\@<=\s\+\S\+\s*(\@="
This uses a look-behind to match the word function. Then after 1 or more spaces it will highlight any word that has a parenthesis as the first non whitespace character after it.
This only highlights the function name. It does not highlight the parenthesis.
I think if you put these commands in .vim/after/syntax/lua.vim
it should work.
The problem with putting them in your vimrc is that sometime after the vimrc is sourced the syntax highlighting file is sourced and generally the first line in there is syn clear
. (Which wipes the custom syntax highlighting you just set)