RxJs File Chunking & Uploading service

Recently I was presented with the challenge of writing some file upload components for a single page app. There are lots of existing upload libraries out there already, but due to an existing server side API that could not be changed I had trouble finding one that could do the job. The existing document processing service we were working against required the following:

• The user had to be able to select as many files as they wanted for upload
• The file had to be split into 1MB chunks
• A CRC had to be calculated for each chunk and the overall file
• Each chunk for a given file had to be uploaded sequentially, but multiple files could upload concurrently

On top of this we wanted to limit the number of chunks that could be in progress at any one time in order to prevent the maximum number of concurrent connections to one domain being maxed out. We wanted the user to be able to continue to use the app while uploads were in progress, and this becomes impossible if all the http connections are tied up (less of a problem with http2, but still on issue on poorer connections).

Given the async nature of all the steps in the process (chunking, checksumming, uploading) it seemed like RxJs would be an ideal solution. A little digging in the docs also revealed that the mergeMap operator even has a built in parameter to limit how many elements from the source stream should be processed concurrently. With this, the bulk of the work of queuing the chunks for upload is handled by 2 lines of code!

chunkQueue$
    .pipe(mergeMap((data) => data, null, maxConnections))
    .subscribe();

The other tricky part was working out how to provide a progress indicator, but luckily Axios provides a progress callback. It was reasonably straight forward to wrap the Axios promise based calls in an Observable in order to merge them in with the other RxJs streams, as below.

export function httpUpload(file) {
    return Observable.create((observer) => {
        var config = {
            onUploadProgress: (progressEvent)  => {
                var percentCompleted = Math.round((progressEvent.loaded * 100) / progressEvent.total);
                observer.next({ progress: percentCompleted });
            }
        };
        axios.post(`${appConfig.apiUrl}/upload`, file, config)
            .then((response) => {
                observer.next({ status: response.status });
                observer.complete();
            })
            .catch((error) => {
                observer.error(error);
            });
    })
};

A working example of the full chunking and uploading process  can be found at https://github.com/glendaviesnz/rxjs-file-chunker. At some point I may get around to adding some extra configuration and error handling in order to make it a more reusable module.