If it's a fixed size virtual disk you can not shrink it unless you convert it to a dynamic size. The only way to do that is to copy all the data into a new dynamic sized disk with
VBoxManage clonehd [old-VDI] [new-VDI] --variant Standard
Just note you'll need the space to store two copies of the disk so this might not work without more storage space
If it's dynamicly sized you need to remove files in the virtual OS to shrink it other wise there's no way to shrink it.