You can use an OSGi fragment for this. In a nutshell, a fragment is added to the host bundle (i.e. the one which "hosts" the fragment) and it can overwrite anything in the host bundle. If you think in terms of a classpath, then the fragment always comes first and then the host bundle.
That means you can take the existing preferences page, copy this single source file into a fragment, change it as needed and then compile it into a fragment.