Found it!
Simply changing the root view in the layout xml file to a ScrollView
does the job.
I don't even need to specify android:windowSoftInputMode
Of course I have to have everything else in a single layout as the child of the ScrollView
Edit
your .xml file should look like this:
<?xml version="1.0" encoding="utf-8"?>
<ScrollView xmlns:android="http://schemas.android.com/apk/res/android"
android:layout_width="fill_parent"
android:layout_height="fill_parent"
... >
<LinearLayout> // could be any other layout but linear works well here
//insert all other views of the activity here
</LinearLayout>
</ScrollView>