Hi mharcement,
the html5 mode can not limit the user to select what file extensions, it only work for silverlight and flash mode. Like the screenshot below.
maybe you can set the silverlight and flash upload mode in a higher priority. like the code below.
<CuteWebUI:Uploader runat="server" ID="Uploader1" UploadTypePriority="Silverlight,Flash,HTML5">
</CuteWebUI:Uploader>
Regards,
Ken