Sync & cursors

Cursor stores

On a serverless or edge runtime, a delta sync without a persisted cursor is a full sync — every single run.

The default cursor store lives in memory. A long-lived Node host keeps that memory, so delta sync works. A request-scoped runtime (Cloudflare Workers, Vercel, Lambda) discards it between invocations, so every run starts from null and re-reads your entire source tables.

Nothing about this looks wrong from the outside. The sync reports success, the row counts are plausible, and clivly status passes. The only symptom is your database bill: one deployment on Workers ran a delta sync each minute and exhausted a 5 GB monthly Neon network allowance before launch, with two users.

Use the store the SDK ships. It works with any key-value client that has get and put — Workers KV, Upstash, node-redis:

import { createClivlySDK, kvCursorStore } from "@clivly/sdk";

const clivly = createClivlySDK({
apiKey: process.env.CLIVLY_SECRET_KEY ?? process.env.CLIVLY_API_KEY!,
entities,
source: { /* ... */ },
cursorStore: kvCursorStore(env.CLIVLY_CURSORS),
});

Writing your own

There is one trap, and it is why kvCursorStore exists: SyncCursor.time is a Date. A JSON round-trip turns it into a string, and the sync path then calls since.time.toISOString() on it and throws. Serialize to ISO on the way out and parse back to a Date on the way in:

cursorStore: {
get: async (entity) => {
  const raw = await kv.get(`clivly:cursor:${entity}`);
  if (!raw) return null;
  const { id, time } = JSON.parse(raw);
  return { id, time: new Date(time) }; // back to a Date
},
set: (entity, cursor) =>
  kv.put(
    `clivly:cursor:${entity}`,
    cursor ? JSON.stringify({ id: cursor.id, time: cursor.time.toISOString() }) : ""
  ),
}

Return null rather than throwing if a stored value is missing or malformed: the cost is one full re-read, where throwing would stop sync entirely.

Choosing a schedule

Delta sync is cheap; running it constantly is not. On a usage-billed database (Neon, PlanetScale) a per-minute cron is roughly 43,000 runs a month and will keep the database awake continuously. Hourly is a sensible default, with instant push covering latency in between. Reserve per-minute schedules for sources that genuinely need it.

Note that runScheduled() always sends a heartbeat before syncing, so a frequent sync schedule is also a frequent heartbeat.

Other symptoms

If rows sync on insert but never on edit, the cursor column does not change when a row is updated. Add an updated_at column and use it as cursorField.

If rows sync on insert but never on edit, the cursor column does not change when a row is updated. Add an updated_at column and use it as cursorField.

The sync trigger

The sync trigger is how Clivly Cloud reaches your app to ask for a sync, rather than your app dialling in.

syncTrigger: {
// `path` lets the SDK build its own public URL from the hosting platform's
// environment, so on those platforms you never set the URL by hand.
path: "/api/clivly/tick",
url: process.env.CLIVLY_SYNC_TRIGGER_URL,
secret: process.env.CLIVLY_SYNC_TRIGGER_SECRET,
},

CLIVLY_SYNC_TRIGGER_SECRET signs the cloud's requests to your tick route. With it set, unsigned requests are rejected with 401. Without a secret the route stays open to anyone who can reach it — always set one if the route is publicly routable.

CLIVLY_SYNC_TRIGGER_URL tells the cloud where the route lives. The SDK also self-reports this on each heartbeat, deriving it from your platform's env (Vercel, Cloudflare Pages, Railway, Render, Fly).

Rotating the secret requires redeploying the host — remote sync fails until both sides match.

Only POST triggers a sync

The trigger route answers three ways:

If you drive sync from a cron or uptime check, make sure it sends POST. A crawler, a link prefetcher or an <img> tag on any page can send a GET, so allowing it lets anyone start your sync.

If a caller cannot switch yet, set CLIVLY_TRIGGER_REQUIRE_POST=false. It is read at request time, so it takes effect on the next cold start with no redeploy. The SDK logs a warning while it is set, and the override will eventually be removed.

Previews need a signed request

When you build or preview a mapping, Clivly asks your tick route for a few sample rows. Those requests return data from your database, so the route only answers them when they are signed with CLIVLY_SYNC_TRIGGER_SECRET. Without a secret, a preview or "test with one record" fails with 403 signature_required, and an ordinary sync still runs.

Near-real-time sync with pushOne / insertAndPush

By default, a row reaches Clivly on the next scheduled delta sync — every few minutes, or whenever the trigger fires. For a signup flow or anything else where a row should show up in Clivly within about a second, opt into instant push:

const clivly = createClivlySDK({
apiKey: process.env.CLIVLY_SECRET_KEY!,
entities,
source: { /* ... */ },
sync: {
  // Off by default — turning this on changes a write's async failure
  // surface (a push can fail independently of the write that triggered
  // it), so an existing install upgrading its `clivly` version keeps
  // today's behavior until it opts in.
  instant: true,
},
});

For a table your own code inserts into directly, use insertAndPush from clivly/drizzle in place of a plain db.insert(...):

import { insertAndPush } from "clivly/drizzle";
import clivly from "./clivly.config";

// Created once per mapped table — mirrors `fromDrizzle`'s own convention,
// so a table two entities share (e.g. `user` for both `user` and
// `organizationOwner`) is never ambiguous about which entity a row belongs to.
const insertUser = insertAndPush(clivly, db, schema.user, { entity: "user" });

// in the signup handler:
const created = await insertUser(newUserValues, c.executionCtx.waitUntil);

c.executionCtx.waitUntil (Hono's name for it; every Workers-compatible framework exposes the same primitive under some name) extends the request's lifetime past the response, so the push survives on Cloudflare Workers and similar runtimes — which otherwise kill background work the instant a response is sent. Omit it on a long-lived process where nothing tears the request down early; insertAndPush then awaits the push inline instead.

insertAndPush translates the insert's .returning() row from Drizzle property names (fullName) to real database column names (full_name) before pushing — the same translation fromDrizzle performs for a normal sync, and for the same reason: every consumer downstream of a push addresses columns by database name, and skipping this translation silently pushes empty fields with no error anywhere.

When a library owns the write

If a table is created by a dependency rather than your own code — an auth library's own sign-up flow, for example — there's no insert call site to wrap with insertAndPush. Call clivly.pushOne(entityKey, row) directly from that library's own lifecycle hook instead, using toColumnKeyedRow (also from clivly/drizzle) to translate a row you re-query yourself:

import { toColumnKeyedRow } from "clivly/drizzle";
import clivly from "./clivly.config";

databaseHooks: {
organization: {
  create: {
    after: async (createdOrg) => {
      const [full] = await db
        .select()
        .from(schema.organization)
        .where(eq(schema.organization.id, createdOrg.id));
      if (full) {
        await clivly.pushOne("organization", toColumnKeyedRow(schema.organization, full));
      }
    },
  },
},
},

Re-querying the row (rather than trusting whatever fields the library's own callback happens to pass) means the pushed data is always complete, regardless of what that callback includes.

Instant push never replaces the scheduled sync

pushOne is fire-and-forget — it never throws, and a failed push is only logged, never surfaced to the caller. It also never advances the delta cursor, so it can never make a later delta sync skip rows. Both are deliberate: instant push is a latency optimization, not the source of correctness. Keep your scheduled sync (a cron calling runScheduled({ mode: "delta" }), or clivly.config.ts's own keep-alive loop on a long-lived host) running underneath — it's what catches anything instant push misses: a transient push failure, a raw SQL write, a bulk import, or an admin script.

On a serverless host, that cron needs a cursorStore or each run re-reads every table, and hourly is a better default than per-minute.

If instant push ever needs to be turned off quickly — say, it starts flooding Clivly's ingest endpoint — set CLIVLY_INSTANT_PUSH_DISABLED=true in your host's environment. It's checked at request time, not baked into the build, so it takes effect on the next cold start with no redeploy.