7 June, 2020

Problem

You’re using Emscripten and would like to cache downloaded files to IndexedDb so that they appear in the IDBFS filesystem.

Solution

Mount an IDBFS filesystem at startup

Create a JavaScript file with these contents, and name it pre.js.

Module['preRun'] = [
    function() {
        FS.mkdir('/assets');
        FS.mount(IDBFS, {}, '/assets');
    },
];

This uses the File System API to mount an IndexedDB database as an IDBFS filesystem under /assets.

To run this before your program starts, update the Emscripten command line as follows:

Add -lidbfs.js to tell Emscripten to use the IDBFS filesystem.
Use --pre-js to tell the Emscripten compiler about the pre.js file.

Here’s an example command line using the Emscripten compiler front end, emcc.

emcc main.c -s EXPORTED_FUNCTIONS="['_main', '_Run']" -lidbfs.js --pre-js pre.js -o out/index.html

Note that this also exports the main() and Run() functions to make them callable from JavaScript.

Sync to make cached files available in the filesystem

When your program starts, you must sync from IndexedDB to the filesystem to make any previously cached files available.

int main(int argc, char* argv[])
{
    // Sync from IndexedDB to the filesystem so that we can see any files that are already present.
    EM_ASM(
        FS.syncfs(true, function(err) {
            assert(!err);
            ccall('Run', 'v')
        });
    );

    return 0;
}

This uses FS.syncfs(true, ...) to sync from IndexedDB to the filesystem. This step is necessary because IDBFS needs to be synced, otherwise files in IndexedDb won’t appear in the filesystem.

Download files using emscripten_async_wget()

The ccall() function in the previous example calls Run() when FS.syncfs() completes successfully.

Here’s Run(). It tries to open a file from the IDBFS filesystem.

void Run(void)
{
    const char* filename = "/assets/book.txt";
    FILE* f = fopen(filename, "r");
    if (!f)
    {
        // Download the file because we don't have it locally.
        const char* url = "book.txt";
        printf("Downloading %s from %s\n", filename, url);
        emscripten_async_wget(url, filename, DownloadSucceeded, DownloadFailed);
        return;
    }
    
    // Print the size of the file.
    fseek(f, 0, SEEK_END);
    long where = ftell(f);
    printf("%s is %ld bytes\n", filename, where);

    // Print a small part of the file.
    fseek(f, 0, SEEK_SET);
    char buf[128 + 1];
    for (int i = 0; i < 10 && fgets(buf, sizeof(buf), f); i++)
    {
        printf("%s", buf);
    }

    fclose(f);
}

If fopen() succeeds, then the file is in the cache, so the function goes on to display its length followed by its first few lines.

If fopen() fails, then the file is not in the cache, so it calls emscripten_async_wget() to download the file from a URL.

Use FS.sync() to persist the files to IndexedDB

Once emscripten_asyc_wget() finishes downloading the file, it calls DownloadSucceeded().

Now the file must be synced from the filesystem to IndexedDB, otherwise it won’t be cached. This is done by calling FS.syncfs(false, ...), as shown here.

void DownloadSucceeded(const char* filename)
{
    // Sync from the filesystem to IndexedDB to persist the downloaded file.
    EM_ASM(
        FS.syncfs(false, function(err) {
            assert(!err);
        });
    );

    printf("Downloaded %s\n", filename);
    Run();
}

Finally, it calls Run() again. This time fopen() will succeed and the file’s information will be displayed.

The program

Here’s the example program in its entirety.

#include <emscripten.h>
#include <stdio.h>

void Run(void);

void DownloadSucceeded(const char* filename)
{
    // Sync from the filesystem to IndexedDB to persist the downloaded file.
    EM_ASM(
        FS.syncfs(false, function(err) {
            assert(!err);
        });
    );

    printf("Downloaded %s\n", filename);
    Run();
}

void DownloadFailed(const char* filename)
{
    printf("Failed to download %s\n", filename);
}

void Run(void)
{
    const char* filename = "/assets/book.txt";
    FILE* f = fopen(filename, "r");
    if (!f)
    {
        // Download the file because we don't have it locally.
        const char* url = "book.txt";
        printf("Downloading %s from %s\n", filename, url);
        emscripten_async_wget(url, filename, DownloadSucceeded, DownloadFailed);
        return;
    }
    
    // Print the size of the file.
    fseek(f, 0, SEEK_END);
    long where = ftell(f);
    printf("%s is %ld bytes\n", filename, where);

    // Print a small part of the file.
    fseek(f, 0, SEEK_SET);
    char buf[128 + 1];
    for (int i = 0; i < 10 && fgets(buf, sizeof(buf), f); i++)
    {
        printf("%s", buf);
    }

    fclose(f);
}

int main(int argc, char* argv[])
{
    // Sync from IndexedDB to the filesystem so that we can see any files that are already present.
    EM_ASM(
        FS.syncfs(true, function(err) {
            assert(!err);
            ccall('Run', 'v')
        });
    );

    return 0;
}