Does one need to put aria-label on a form submit button?

Question

This depends, both of your examples wouldn’t need it per se, because there is visible (and readable) text. If you want to describe the element in greater detail you’d use the title attribute before you use the aria-label attribute; as stated in the standard text.

<button title="Continue here after you’ve filled out all form elements">Submit</button>

<input title="Continue here after you’ve filled out all form elements" value="Submit" type="submit">

The thingy with the closing x is somewhat different, because if I say to you “x” what do you think? That’s why we want to help the screen readers (or other technology). But a title attribute would still be better and create a visible tooltip for all users.

<a href="/" title="Close this foo."><span aria-hidden="true">x</span></a>

ARIA navigation example that validates (see comments).

<!doctype html>
<html>
<head><meta charset="utf-8"><title>ARIA Example</title></head><body>
<!--
More HTML that makes up your document
-->
<!--
We apply the role navigation on the nav element to help older browsers, of course
the nav element already has the ARIA meaning, but it doesn't hurt anyone to make
sure that all browsers understand it
-->   
<nav role="navigation">
    <!--
    Omit title from display but not from e.g. screen readers see h5bp.com/v or
    h5bp.com/p
    -->
    <h2 class="visuallyhidden">Main Navigation</h2>
    <ul role="menu">
        <li>
            <a href="/" role="menuitem">Home</a>
        </li>
    </ul>
</nav>
<!--
More HTML that makes up your document
-->