After looking some more with the DOM Inspector I finally got it.
The toolbar and toolbox ids are header-view-toolbar
and header-view toolbox
and the classes are inline-toolbar
and inline-toolbox
. The CSS class for buttons is msgHeaderView-button
. The XUL for adding a button there should look like this:
<toolbarpalette id="header-view-toolbar">
<toolbarbutton id="my-button"/>
</toolbarpalette>
<toolbarbutton id="my-button"
label="This is my button!"
tooltiptext="Click on it!"
oncommand="MyButton[1]()"
class="toolbarbutton-1 msgHeaderView-button my-button"
/>