If I'm understanding correctly, you require a single average of all the date differences. In which case it would be best to use the AVERAGE spreadsheet function outside the QUERY function.
=AVERAGE(QUERY(A1:B3;"select datediff(A,B) label datediff(A,B) ''"))
It actually isn't really necessary to put the label datediff(A,B) ''
as the AVERAGE function will ignore the text header populated by the QUERY, but I put it in for the sake of completeness.