Okay, so I've tried all the solutions and comments and it turns out that the only way to do it is to make a div that holds the other elements like @james said in a comment.
Since all elements are supposed to be draggable according to the spec, I'm guessing that the rationale for not including draggable support for inputs is because they are supposed to grab focus automatically onclick. I'd be curious to see what the real rationale was, so I may open a bug with Firefox.
No major browser that I tested supports draggable <input>
or <textarea>
elements with only the attribute (and no enclosure div) as of right now.
The spec (http://www.whatwg.org/specs/web-apps/current-work/multipage/dnd.html#the-draggable-attribute) is notably silent on this issue.