Your frame has no content, you never added anything to it.
You need to add your panel to the frame.
There are a number of ways to do this, in this case it's simplest to do:
frame.setContentPane(panel);
You should also look at using vlcj's EmbeddedMediaPlayerComponent, it's much easier (less lines of code) to set up.