thanks Terry for idea. After changing order of HttpModules in web.config the ajax uploader works fine. The error "upload data transfer expired" and "unexpected data end" have disappeared.

The conflicting module is my custom LocalizationModule. LocalizationModule sets culture and rewrites request path for specific urls. I do not know why it conflicts with AjaxUploader requests. Non the less, moving LocalizationModule after  CuteWebUI.UploadModule uploader solved problem.