"Efficient downloading" is a nonissue. Existing compression, concatenation, and minification techniques yield file sizes that a binary format will have a great deal of trouble beating at all, and even when it does, the savings will be no more than a few bytes at best.
"High performance" is a nonissue. This is what asm.js is for, and indeed, the existing polyfill uses asm.js to achieve its performance gains. This is a newer solution than those for the previous problems, but either way, the problem is solved.
"A standard runtime specification" is a nonissue. We already have one of those. It's called ECMAScript.