In your _config.php file you can alter the HtmlEditorConfig. In your mysite/_config.php (or modules _config)
$config = HtmlEditorConfig::get('cms');
$config->enablePlugins(array('pluginname' => 'plugin.js'));
$config->addButtonsToLine(2, 'pluginname');
A description of the plugin javascript is available at http://doc.silverstripe.org/framework/en/topics/rich-text-editing