I guess you will need something like this
<?php
$string = '[img alt="" src="IMAGE LINK, NOT ENDING IN AN IMAGE FORMAT EXTENSION" style="max-width:100%;"]';
$matches = array();
print preg_replace('|\[img.*src="(.*)" style.*]|', '[img]$1[/img]', $string);
prints
[img]IMAGE LINK, NOT ENDING IN AN IMAGE FORMAT EXTENSION[/img]