PouchDB-CouchDB sync via Service Workers, as goofily illustrated on the author’sSurface
At Offline Camp California in November 2016,John Kleinschmidt and I tried to tackle the seemingly clear-cut problem of how to get PouchDB replication to run correctly in a Service Worker . However, even though John and I were familiar with both PouchDB and Service Workers, and even though I work for a browser vendor that’s currently implementing Service Worker , we ran into impedance mismatches between what we thought a Service Worker was capable of and what we were trying to accomplish with offline replication.
The goal of this article is to explain where we went wrong in our understanding of Service Workers, and how we began adapting PouchDB replication to a new and unfamiliar context. I’ll also go over some of the trickier parts of the Service Worker API, which may prove surprising to the uninitiated. John wrote up his own thoughts on the subject , and in this post I’d like to expand on what we’ve learned since then.
Scoping theproblemFirst off, what was the problem we were trying to solve? Well, one issue John identified was the performance impact of IndexedDB on the main thread , which can only be alleviated by moving PouchDB either to a Web Worker or to a Service Worker. John’s project, HospitalRun.io , was already making use of Service Worker, and so it seemed to be a natural fit ― no need to allocate additional workers just to run PouchDB.
However, the first thing we realized, and which I summarized in a talk I later gave at DotJS , was that a Service Worker can’t really be treated the same as a Web Worker. They share some superficial similarities, but at a core level they have very different architectures, which lead to different strategies when designing for one versus the other.
On the surface, Service Workers look quite similar to Web Workers. They both run on separate threads from the main UI thread, they have a global self object, and they tend to support the same APIs . However, while Web Workers offer a large degree of control over their lifecycle (you can create and terminate them at will) and are able to execute long-running javascript tasks (in fact, they're designed for this), Service Workers explicitly don't allow either of these things. In fact, a Service Worker is best thought of as “fire-and-forget” ― it responds to events in an ephemeral way, and the browser is free to terminate any Service Worker that takes too long to fulfill a request or makes too much use of shared resources.
This led us to our first real hurdle with Service Worker. Our goal, as we originally conceived it, was to use PouchDB’s existing replication APIs to enable bi-directional sync between the client and the server, with the client code isolated entirely to the Service Worker. So as a first pass, we simply loaded PouchDB into our serviceWorker.js , waited for the 'activate' event, and then used the standard PouchDB “live” sync API:
This resulted in a silent error, which took quite a while to debug. The culprit? Well, PouchDB’s “live” sync depends on HTTP longpolling ― in other words, it maintains an ongoing HTTP connection with the CouchDB server, which is used to send real-time updates from the server to the client. As it turns out, this is a big no-no in Service Worker Land, and the browser will unceremoniously drop your Service Worker if it tries to maintain any ongoing HTTP connections. The same applies to Web Sockets, Server-Sent Events, WebRTC, and any other network APIs where you may be tempted to keep a constant connection with your server.
What we realized is that “the Zen of Service Worker” is all about embracing events . The Service Worker receives events, it responds to events, and it (ideally) does so in a timely manner ― if not, the browser may preemptively terminate it. And this is actually a good design decision in the spec, since it prevents malicious websites from installing rogue Service Workers that abuse the user’s battery, memory, or CPU.
So, keeping in mind that we’re trying to accomplish bi-directional replication (i.e. data flow from server to client and vice versa), let’s review what events the Service Worker can actually receive and respond to.
The ‘fetch’eventThis is the most famous feature in the Service Worker arsenal ― by capturing this event, the Service Worker can intercept network requests and respond with its own content.
Because the CouchDB API is entirely REST-based, it would be technically possible to implement a caching layer via the 'fetch' event. However, this would be working against the grain of CouchDB's bi-directional replication model, and would do little to answer the question of how to handle conflicts and merges, as the existing CouchDB replication protocol does.
Such a technique may make sense for lightweight caching of a read-only REST API, but it doesn’t make much sense for CouchDB. In the same way that it would be silly to take the Git protocol and map every HTTPS request to an object that can be independently cached and invalidated, we decided it would be silly to route our PouchDB replication system through 'fetch' events. That led us to the other Service Worker APIs.
The ‘sync’eventThis event is defined by the Background Sync API , and is probably the most confusingly-named of the Service Worker events. Rather than having anything to do with “sync,” this is merely an event that fires when the browser goes from an offline state to an online state. The goal is to allow for the Service Worker to use this “just went online” event as an opportunity to push unsynced changes from the client to the server.
In the case of PouchDB, this would be as straightforward as waiting for the 'sync' event and then firing a single-shot replication from the client to the server. There would be no need to keep a persistent HTTP connection open we can merely wait for the 'sync' event to notify us that the device has come online, and then we allow PouchDB to replicate from the last checkpoint it may have stored from any previous replications. Once that’s complete, we would just patiently wait for the next 'sync' event before doing the same single-shot replication all over again.
When thinking about this design, the first question you might ask is why the 'sync' API even exists, given that there is already navigator.onLine and similar online/offline events available in the browser. The answer is that since this event is availabl