DockBlocks / Handoff / Njui

The loop that does not close

The CDP holds state correctly and governs nothing. Findings are detected and then dropped. No send remembers the last one. Four steps close it, and none of them is a new platform.

Written2026-09-07 ForNjui (KN) Verified againstorigin/main 31b6b89 StatusNot started, nothing built

Why you are reading this

Alecia asked which open source tool could run DockBlocks outbound and lifecycle marketing end to end, and specifically whether anything tracks what happened today so the next send is not treated as brand new.

Five research lanes covered marketing automation engines, workflow orchestrators, CDP and reverse ETL, and sales engagement tooling. The conclusion is to buy nothing. The platform already exists and you built most of it. What is missing is the loop: the gate has no callers, findings do not survive delivery, and nothing accumulates what our own sends did.

This document is the evidence, the four steps, and an explicit invitation to reject the order I put them in.

What already works

Verified on origin/main, not the local checkout. Everything in this table is live unless marked otherwise.

CapabilityWhereState
Identity resolution, person_mapdagster_project/assets/identity.pylive 02:30
Per-channel consentassets/consent_events.py, consent_state.pylive
Per-contact stage with dwell timeassets/person_lifecycle.pylive 03:00
Fail-closed send gatesrc/dockblocks_ops/eligibility/gate.pyzero callers
Outbound pipeline, six stages, traceablepivot-outbound/core/pipeline.pybuilt, dormant
Per-person send memorydockblocks/.logs/comms_ledger.jsonllive, 338 rows
Unsubscribe watchdogscripts/monitoring/send_watchdog.pylive, 15 min
Apollo in the warehouseno asset, no resource, no connectorabsent

person_lifecycle is the piece every vendor on the shortlist was being evaluated to provide. The five-rung Zoho ladder with person_deal_age_days, is_recent_proposal and stage_skip_pattern, computed against an explicit as_of clock rather than wall time, is dwell-time-aware per-contact state. The as_of parameter is the better design because it makes state reproducible: you can ask what a contact's stage was last Tuesday, which no live-updating journey engine can answer.

The loop, end to end

Solid lines run today. Dashed red lines are the four things that do not exist. The whole build is visible in one picture: three connections and one source.

The DockBlocks data loop and its four breaks Sources flow into bronze, then into the gold person lifecycle table, then to send paths, then to recipients. Four connections are missing: Apollo as a source, the eligibility gate governing sends, send outcomes written back into the warehouse, and a durable findings record. Apollo no asset, no connector 04 Sources Zoho, Brevo, Shopify Bronze partitioned by day person_lifecycle gold, overwritten each run Send paths Brevo, Apollo, SMS Recipients opens, bounces, replies eligible() fail-closed, zero callers 02 no send outcome is ever written back 03 runs today does not exist
Read it left to right, then look at the bottom. Everything forward works. Nothing comes back. The warehouse rebuilds each night from what Zoho and Brevo currently say, so it is honest about the present and remembers nothing we did.

The four gaps, with evidence

1. The gate governs nothing

eligible(person_ids, channel, as_of) has zero callers anywhere. Not in dockblocks, not in dockblocks-data-ops. On origin/main the only matches are its own definition and its own error strings.

Meanwhile three rival functions of the same name run their own suppression logic:

scripts/brevo/lifecycle_campaigns.py:195   def eligible(p)
scripts/brevo/lifecycle_campaigns.py:242   def eligible(p)
scripts/brevo/sms_drip.py:410              def eligible(contact, seen, now)

A gate with no callers and a gate that always returns true are indistinguishable from outside. Only the call graph separates them.

2. Findings are detected and then dropped

gate-integrity ran today, found a real breach, enforced a denominator floor correctly, and then lost the result:

BREACH  government_sector      413 rows (floor 489)
        -> 76 of 489 government leads have no sector
UNKNOWN downstream_seams       14 rows (floor 16)
[deadman] scheduled jobs: 39 healthy, 0 DEAD, 1 unchecked

ops_alert: every token failed (CLICKUP_API_TOKEN: <urlopen error
[Errno 8] nodename nor servname provided, or not known>); alert dropped
The findings path and where it fails Checkers detect problems and hand them to ops_alert, which tries to deliver to ClickUp or DM. When the network fails the alert is dropped and no person is told. There is no durable findings file to fall back to. Checkers gate-integrity, watchdogs ops_alert tries once ClickUp, DM needs network A person reads it, or does not alert dropped Append-only findings file read by the next run 01
The checker is good. The last hop is not durable. An alert that evaporates on a network blip is a finding nobody will ever read, and as of this writing that government_sector breach is still unread. Step 01 is the fallback that makes detection survive delivery.

3. Nothing remembers what our sends did

Gold assets, person_lifecycle included, write a single part-0000.parquet that is overwritten every run. No asset reads its own prior output. brevo_campaigns_bronze is a full snapshot with no history retained, so campaign trend requires diffing something that is not kept.

change_events and consent_events do accumulate, but they diff what Zoho and Brevo changed to. They are not a record of what this pipeline decided. Rep_Worklist is the single place the pipeline writes its own decision back, and by its own docstring it is a same-day snapshot with stale rows deleted each run.

In pivot-outbound, the Drop record whose docstring reads "Why one lead did not make it. Kept, never discarded" is written only when --drops-out is passed, and the file is opened in "w" mode. Nothing reads a prior run.

On the dockblocks side the same habit shows as 17 separate merge_audit_YYYY-MM-DD.jsonl files and roughly a dozen timestamp-named run files containing zero rows.

4. Apollo is absent from the warehouse

No bronze asset, no resource, no connector. Cold outbound sequence and reply state is invisible next to a person's Zoho, Brevo and Shopify state, which means half of outbound cannot be reasoned about at person grain.

The build

Four steps. The ordering claim is that 1 and 2 make the system honest about the present, 3 and 4 make it remember, and the read surface comes last. Doing the read surface first produces another console over state that nothing governs and nobody records.

01

Give findings a durable fallback

Add an append-only sink to ops_alert so a failed delivery lands in a findings file that the next run reads, instead of vanishing. Smallest change here, and a precondition for the rest, because every step below produces findings that must survive.

  • A simulated network failure still leaves the finding on disk
  • The next run reads the unsent backlog and reports it
  • Replaying the 2026-09-07 gate-integrity drop produces a retained record
  • Honours SELFTEST=1, so the suite never pages anyone
02

Give eligible() its first caller

Wire zoho_to_brevo.py and campaign_kit to the real gate and retire the three rival eligible() functions. Highest value per line changed in the whole review.

  • At least one production send path calls eligibility.gate.eligible
  • The rival functions are deleted, not left beside it
  • A known-suppressed person is refused, proven on real data
  • A gate failure blocks the send rather than defaulting to allow
03

Build the send-outcome ledger

One append-only row per send: what went out, to which segment, from which sending domain, under which gate decision, and what came back at 24 hours and 7 days. This is Stage 08 of the outbound console, the one labelled "learn: what actually happened".

It belongs in the warehouse as a partitioned asset rather than in .logs/. Bronze already retains daily partitions; .logs/ is where 17 dated files and a dozen empty ones went to be forgotten. Make --drops-out mandatory and appending in pivot-outbound so the drop reasons feed it.

  • One row per send, appended, never overwritten
  • Outcomes at 24h and 7d attach to the originating send
  • A second run of the same campaign does not destroy the first
  • The next campaign reads it before building its audience
  • Zero findings over zero input fails rather than reporting clean
04

Add Apollo as a bronze source

Sequence state, enrolment, bounces and replies at person grain, partitioned daily like brevo_engagement_bronze. pivot-outbound's six-stage spine and its trace(email) are the natural producer.

Read docs/workflows/apollo-engagement-routing-as-built.md before starting. It records several traps that cost days, including that activity_feed and contact_stages live on api.apollo.io/api/v1 and not /v1, where a wrong base path 404s exactly like a missing feature.

  • Apollo bronze partitioned by day, retained
  • Sequence and reply state joins to person_id
  • A truncated read fails loudly rather than reading as an absence

How we will know it works

One acceptance test, and it is already written down in the code you would be extending:

Trace one lead from source to enrolled, including why the ones that dropped were dropped, is not a nice-to-have, it is how you tell a working machine from a script that printed a number. pivot-outbound/core/pipeline.py

Concretely: given one email address, return the resolved identity, the current stage and how long it has been held, every send that person received and the gate decision that permitted it, what came back, and what we concluded. Today that answer does not exist in any single place.

Where the order may be wrong

Open to challenge

You know the warehouse better than I do, and this sequencing was derived from outside it. If steps 3 and 4 are cheaper together, or if the send-outcome ledger is better modelled as an extension of change_events than as a new asset, say so and we take your order instead. The same applies if you think the durable-findings sink belongs in the pipeline rather than in ops_alert.

What is not open to challenge is the goal: the next send must begin knowing what the last one did, and a finding must survive its own delivery.

Two things to know before you start

  • The local checkout is stale. Local main in dockblocks-data-ops is 507 commits behind origin/main, dated 2026-06-16 against 2026-09-06. Anything read from the working tree is three months out of date. This produced one wrong conclusion during the review, corrected only by reading the remote.
  • Two monitors exit 1 and both are behaving correctly. rep-digest-deadman and gate-integrity return non-zero because they found something, which is the intended contract. Do not "fix" them. The defect is the dropped delivery in step 1, not the exit code.

Appendix A: what was ruled out, so it is not researched again

ToolCategoryWhy not
DittofeedJourney engineBest architectural match and upstream is dormant. Last commit to main 2026-03-27, zero commits in 90 days, newest release still alpha, August community PRs unmerged.
LaudspeakerJourney engineDead. Acquired by PostHog, 1 commit in 90 days.
MauticJourney engineGenuinely live and GPLv3, but adopting it means a second marketing platform beside Brevo and a second sending reputation on an account already at strike two.
Keila, Listmonk, phpListBroadcastBroadcast only. No per-contact journey state, so they cannot hold a lifecycle gate.
n8nOrchestratorStrong fit on connectors and observability, but fair-code licensed rather than OSI, and redundant with Dagster which is already deployed.
Windmill, Temporal, Airflow, Dagster-alternativesOrchestratorEither engineer-only with no visual layer, or batch-DAG shaped and wrong for per-entity lifecycle tracing.
RudderStack, Jitsu, Snowplow, PostHogCDP / eventsAll move or report data. None decides and holds lifecycle state, which is the part already solved here. Snowplow relicensed to SLULA 2024-01-08 and is no longer OSI open source.
Airbyte, Meltano, MultiwovenEL / reverse ETLPlumbing only. Their notion of state is a sync cursor.
GrouparooReverse ETLArchived 2022-04-12.
Apollo replacementsSequencerNo credible open source sequencer exists. Every real one is closed SaaS. Keep Apollo.

The one genuinely unowned gap outside this build is a rep reply surface, where a rep can answer a prospect in thread and have the outcome written back to the CRM. Chatwoot (MIT) is the leading candidate and needs a webhook bridge either way. That is a separate decision, not part of these four steps.

Appendix B: reproducing the claims

Every assertion above came from these. Run them rather than trusting the prose.

# the checkout is stale, read the remote
git -C dockblocks-data-ops fetch origin
git -C dockblocks-data-ops rev-list --count main..origin/main
git -C dockblocks-data-ops log origin/main --since=2026-06-09 --oneline | wc -l

# the gate has no callers
git -C dockblocks-data-ops grep -n 'eligible(' origin/main -- '*.py' \
  | grep -v 'def eligible\|test\|analysis/'
grep -rn 'from dockblocks_ops\|eligible(' dockblocks/scripts/webhooks/zoho_to_brevo.py

# the rival gates
grep -rn 'def eligible' dockblocks/scripts/brevo/

# the dropped finding
tail ~/Library/Logs/pivot/gate-integrity.err

# per-run files that were never read back
ls dockblocks/.logs/merge_audit_*.jsonl | wc -l

# drops are discarded unless a flag is passed, and overwritten when it is
grep -n 'drops_out' pivot-outbound/jobs/source_and_verify.py