I finally ended up with capturing at 640x480, cropping it to 480x480 while splitting to frames using ffmpeg, then finally when frames are combined back to video using ffmpeg, it is up-scaled to 640x640.
String[] ffmpegCommand = {"/sdcard/frames/ffmpeg",
"-i", "/sdcard/frames/test.3gp",
"-vf", "crop=480:480:80:0,transpose=1",
"-r", "30", "-an", "-qscale:v", "2", "-vsync", "1", "-threads", "4", "/sdcard/frames/image%03d.jpg"};
The transpose=1 is required when the app is working in portrait mode.
Note : When capturing, to show only the part of the video that will be included in the final cropped video, I added two black bars of 80 pixels on top of the preview display to hide the 80 pixels on either side of the video that later gets cropped of. So the user gets what he sees on the preview.