I figured out the problem ... With Rails 4 strong parameters I didn't have the parameter id
included. The result is that the attributes are now:
params.require(:event).permit(:name, :logo, :description, :start_time, :end_time,speakers_attributes:[:id,:name,:short_bio,:avatar,:contact,:_destroy])
Hopefully this saves someone else some pain :)