LeadPool

Self-hosted, enterprise-grade platform for selling lead data to B2B clients.

Language
TypeScript
Stars
0
Last updated

From the README

  • Pools — one per data type (auto warranty, medical, solar, …), each with its own admin-defined columns.
  • Imports — CSV/XLSX, millions of rows, streamed and deduplicated by phone/email identity.
  • Clients — their own portal: browse pools as products, pick an age tier (1 day … 120+ days), filter, see live availability + price, order, download CSV or Excel.
  • Mixed orders — several age tiers in one order, split by percentage of a total ("50% fresh · 30% this month · 20% aged") or by exact lead counts; each tier is filled from its own age window, priced on its own and refunded on its own if short.
  • Repeat orders — any order (one tier or a mix, with filters) placed automatically every day, on chosen weekdays or monthly, at a time in the client's own zone; paid from the balance, sent to their email / WhatsApp / Teams / cloud; skipped (never charged) when the balance is short and paused after 3 skips.
  • Lead returns — clients send back bad leads within a window after delivery, up to a share of each order; staff approve per lead, approved leads are credited at the price paid and taken off sale. One switch turns the feature on or off for everyone (Settings → Lead returns), and it can be switched off per client.
  • Sortable, filterable tables — click any column heading to sort (again to reverse, a third time to clear); the filter button next to it narrows by text, a pick-list, a number / money range or a date range (Today, 7 / 30 / 90 days, or from–to). Big lists (orders, returns, wallet history, audit log, a pool's leads) sort and filter in the database across every record, not just the page on screen; each table remembers its view while you move around.
  • Hard rule — a client never receives the same lead, or the same person (any matching phone or email), twice.
  • Suppression — clients upload their own lists; staff maintain a global DNC applied to everyone.
  • Money — prepaid balances in micro-dollars, holds at order time, automatic refunds, append-only ledger.
  • Approval flow — per client: auto-deliver instantly, or send each request to the admin queue.
  • Two-factor sign-in — authenticator-app codes (TOTP) + single-use recovery codes for staff and client users; admins can require it for all staff.
  • Reports — revenue, leads, orders, fill rate and price per lead for any period (vs. the previous one), by client, pool and lead age; export as a multi-sheet Excel workbook or daily CSV.
  • Email notifications — order delivered / declined / unfilled, funds added, low balance (clients); approval needed, import finished (staff). Transactional outbox with retries; each user can mute them.
  • Delivery channels — send order files to clients by Email, WhatsApp, Microsoft Teams, OneDrive or Dropbox: staff connect each once, set per-client destinations, send from any order or automatically on delivery; clients can manage their own destinations in the portal.
  • Two looks — Modern (frosted clay, light/dark) or Classic (flat enterprise CRM), menu on the side or top, full or compact width — per browser, in both apps.

Architecture

apps/api        Fastify 5 REST API (OpenAPI docs at /api/docs in dev)
apps/worker     pg-boss jobs: imports, fulfillment, CSV exports, suppression, index builds, email,
                channel sends (packages/db/src/channels), maintenance
                (Excel copies of an order are built by the API on first request, then cached)
apps/admin      Staff console (Vite + React 19 + TanStack Router/Query + Tailwind 4)   → :5173
apps/portal     Client portal (same stack and design system)                          → :5174
packages/db     SQL migrations, Drizzle schema, fulfillment engine, wallet, exports, scripts
packages/shared Zod API contract, money/format helpers, identity normalisation, permissions
packages/ui     "Frosted clay" design system shared by both apps

PostgreSQL 18 is the only infrastructure dependency (the job queue also lives in Postgres). Leads are LIST-partitioned per pool; order picking walks an index freshest-first in bounded pages, so cost depends on the order size — not on how many leads a pool holds.

Quick start — everything runs in Docker (Windows, macOS, Linux)

The only thing the machine needs is Docker Desktop (or Docker Engine with Compose 2.22+). The database, migrations, API, worker, admin console and client portal all run in containers, so the same folder runs the same way on any computer.

cp .env.example .env
# Put 64 random hex characters in IDENTITY_PEPPER — for example from:
docker run --rm node:24-alpine node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"

docker compose up -d --build                 # database, migrations, API, worker, admin, portal
docker compose exec -w /app api pnpm db:seed # first time only: demo data (~40 s)
docker compose watch                         # leave running while you work: saved files sync in, apps reload
App URL Demo login (password ChangeMe!2026)
Admin console http://localhost:5173 admin@leadpool.local (super admin) · maya@leadpool.local (manager) · ops@leadpool.local (viewer)
Client portal http://localhost:5174 jordan@apexauto.example (auto-deliver) · dana@harborhealth.example (approval flow) · chris@blueriver.example

The API (http://localhost:4000) and the database (port 5432) are reachable from this computer only.

How live editing works: docker compose watch copies every file you save into the containers — the API and worker restart, the admin and portal hot-reload in the browser. Changing pnpm-lock.yaml (a new dependency) rebuilds the image automatically. The containers keep their own Linux node_modules; nothing from the host's is used. After changing docker-compose.yml or the Dockerfile, start with docker compose up -d --build — a plain up -d recreates the containers from the last built image, which leaves out everything synced in since then.

Everyday commands

Command What it does
docker compose up -d --build Start or update everything (rebuilds only what changed; migrations run first)
docker compose watch Live code sync while you edit
docker compose ps · docker compose logs -f api worker Status · follow logs
docker compose restart api Restart one service
docker compose down Stop everything — data stays (the pgdata volume and ./storage)
docker compose down -v Stop and delete the database

Tools, run inside the stack

exec api … runs in the API container, which holds every package; -w /app runs from the repo root.

Command What it does
docker compose exec -w /app api pnpm db:seed [--reset] Demo data: 3 staff, 4 pools, ~200k leads, 5 clients, 30 days of real orders (--reset wipes first)
docker compose exec -w /app api pnpm db:migrate Apply pending SQL migrations (packages/db/migrations; also automatic on up)
docker compose exec -w /app api pnpm typecheck TypeScript across the monorepo
docker compose exec -w /app api pnpm test Unit tests + fulfillment concurrency tests (creates a throwaway leadpool_test DB)
docker compose exec -w /app api pnpm lint Biome lint + format check
docker compose exec api pnpm smoke End-to-end HTTP check against the running stack (103 assertions: archiving + restoring pool columns and tiers, import, orders, mixed orders, repeat orders, lead returns and their on/off switches, table sorting + filtering, dedup, CSV + Excel exports, suppression, approvals, emails, reports, money — each run imports its own fresh leads and tops up the demo client; expects the demo client Apex Auto Protect on instant delivery and lead returns switched on in Settings)
docker compose exec api pnpm smoke:mfa Two-factor end-to-end check (76 assertions; uses throwaway users and cleans up)
docker compose exec api pnpm smoke:channels Delivery-channels check (40 assertions: channel save/test, secrecy, client destinations, manual + automatic sends through the Email channel in file mode, portal access and permissions; restores what it touched)
docker compose exec api node --import tsx scripts/create-admin.ts --email … --name … --password … Bootstrap / reset a super admin — also clears their 2FA (break-glass)
docker compose exec -w /app/packages/db api pnpm bench EXPLAIN + timings for the pick/preview queries
docker compose exec postgres psql -U leadpool SQL shell

Testing from other devices (phones, other laptops on the same Wi-Fi/LAN): open http://<this-computer's-IP>:5173 (admin) or :5174 (portal) — find the IP with ipconfig (Windows) / ipconfig getifaddr en0 (macOS); on Windows the computer's name works too. The apps reach the API through the dev-server proxy. If another device can't connect, allow Docker (or TCP 5173–5174) through the firewall for the network you're on. Anyone on that network can reach the apps, so change the demo passwords before testing on a shared network.

Move it to another computer

Everything is in three places: the project folder, .env (it holds IDENTITY_PEPPER — the data's phone/email keys only match with the same value, so it must travel with the data) and the data (the database plus ./storage).

# old computer — export the database into the project folder
docker compose exec -T postgres pg_dump -U leadpool -d leadpool -Fc -f /tmp/leadpool.dump
docker compose cp postgres:/tmp/leadpool.dump ./leadpool.dump

# copy the project folder (with .env, storage/ and leadpool.dump) to the new computer, then there:
docker compose up -d postgres
docker compose cp ./leadpool.dump postgres:/tmp/leadpool.dump
docker compose exec -T postgres pg_restore -U leadpool -d leadpool --no-owner /tmp/leadpool.dump
docker compose up -d --build

The same dump restores into the production stack (docker-compose.prod.yml, service postgres) when you go live.

Without Docker

Node 24 LTS + pnpm (corepack enable): pnpm install, then either docker compose up -d postgres or pnpm db:local (embedded Postgres 18 in ./.pgdata), then pnpm db:seed and pnpm dev. Every tool command above works as pnpm … from the repo root. Don't run these apps on the host while the Docker apps are up (same ports). Moving data from the embedded database into Docker: start the embedded one on a spare port (pg_ctl -D .pgdata -o "-p 5433" start, from @embedded-postgres/*/native/bin), then docker compose exec postgres pg_dump -h host.docker.internal -p 5433 -U leadpool -d leadpool -Fc -f /tmp/db.dump and docker compose exec postgres pg_restore -U leadpool -d leadpool --no-owner /tmp/db.dump.

How the guarantees are enforced

Guarantee Mechanism
Never the same lead twice deliveries primary key (org, pool, lead) — inserts have no ON CONFLICT, a violation aborts the transaction
Never the same person twice Every phone/email is normalised and HMAC-hashed into leads.keys; picks exclude any key in client_delivered_keys
No race between a client's orders Per-client advisory lock + FOR NO KEY UPDATE SKIP LOCKED on picked leads
Exclusive / shared caps buyers_count updated in the same transaction; lead flips to sold-out at the pool's max_buyers
Suppression Client keys checked at pick time; global DNC flags leads as suppressed on upload and import
Money Hold at submit (conditional wallet update), refund of any shortfall on completion, idempotency keys, append-only ledger

packages/db/src/fulfillment.test.ts proves these under concurrent load.

Mixed orders. Every order has one or more order_lines (tier, snapshot, quantity, price); a mix has several, orders.tier_id is NULL and its snapshot label describes the mix. Fulfillment fills the lines freshest first, each from its own age window, inside the same per-client lock — keys delivered for one line are already in client_delivered_keys when the next line is picked, so a mix never repeats a person. deliveries.line_no records which line each lead was for (the file gets an age_tier column), orders.charged_micros accumulates line quantity × line price, and the refund is held − charged. All-or-nothing mixes roll back entirely if any line is short. In the portal: product → Mix ages (or ?mode=mix); "Order again" on a mixed order rebuilds it (?mix=tierId:qty,…).

Deploy on your own server

One Linux VM with Docker is enough to start (4 vCPU / 8 GB RAM handles tens of millions of leads).

cp deploy/.env.production.example deploy/.env.production     # set hosts, POSTGRES_PASSWORD, IDENTITY_PEPPER
docker compose -f docker-compose.prod.yml --env-file deploy/.env.production up -d --build
docker compose -f docker-compose.prod.yml --env-file deploy/.env.production run --rm -w /app/apps/api migrate \
  node --import tsx scripts/create-admin.ts --email you@company.com --name "Your Name" --password "a-long-passphrase"

Try it on one machine first: set ADMIN_HOST=:8080, PORTAL_HOST=:8081, COOKIE_SECURE=false (and the matching ADMIN_URL / PORTAL_URL) in deploy/.env.production, add -p leadpool-trial to each command, then open http://localhost:8080 and :8081. Demo data: … run --rm migrate node --import tsx src/scripts/seed.ts; throw the trial away with … down -v. The dev database (docker compose up -d) is a separate project and isn't touched.

This runs Postgres 18, a one-shot migration, the API, the worker, and Caddy (automatic HTTPS for ADMIN_HOST / PORTAL_HOST, security headers, SPA hosting, /api reverse proxy). Uploads and exports live in the storage volume; back up pgdata + storage + your .env.production.

Operations notes

  • IDENTITY_PEPPER must never change once data exists — dedup and suppression depend on it, and it also derives the keys that encrypt two-factor secrets and delivery-channel credentials. Back it up with your database backups.

  • Two-factor: users turn it on under Settings → Security (password re-check → QR code → code → recovery codes). A user who loses their phone and recovery codes is reset by an admin (Staff / client Users → edit → Reset 2FA) or, for client users, by their org owner. Settings → General → "Require two-factor for all staff" can only be switched on by an admin who already uses 2FA.

  • Production: set NODE_ENV=production, COOKIE_SECURE=true, serve admin/portal/API behind HTTPS on one origin per app (the SPAs call /api on their own origin), and enable Postgres backups with point-in-time recovery before real money flows.

  • Exports live in STORAGE_DIR/exports and are purged after the configured lifetime (Settings → General). An Excel copy expires with its CSV; "Rebuild export" drops both.

  • Archiving a pool column or age tier never deletes anything. Archived ones are listed under the live ones (pool → Columns / Age tiers → Archived …) with Restore. A restored column goes back to the end of the list with the values leads already carry — filtering and the lead-date role stay off until switched on; leads imported while it was archived simply don't have it. A restored tier comes back switched off and is refused while it would overlap a live tier. An archived column's key can't be reused for a new column (the values are stored under it) — restore it instead.

  • Email: connect your mail server in Settings → Delivery channels → Email (host, port, security, sign-in, from address — presets for Microsoft 365, Google Workspace, SendGrid, Mailgun, Amazon SES) and click Test; it is used for every email the platform sends. Alternatively set SMTP_URL + MAIL_FROM in the environment (the channel wins when both exist). Also set PORTAL_URL and ADMIN_URL (see deploy/.env.production.example). With neither, every email is written to STORAGE_DIR/mail/<day>/*.eml — Settings → Notifications shows which mode is active, lets you send a test and lists the last 50 emails with any errors. Failed sends retry with backoff (2 → 32 minutes) before being marked failed.

  • Both apps have a light and a dark theme (sun/moon button in the sidebar header), remembered per browser. The inline pre-paint script in apps/admin/index.html and apps/portal/index.html is identical, so one sha256-… in deploy/Caddyfile's CSP covers both — if you edit it, change both files and update that hash.

  • Sidebar — the panel button at the top of the sidebar hides it (remembered per browser); the small tab on the left edge of the page brings it back (in Classic, a blue tab with an arrow). Below 1024px wide the sidebar is a slide-in menu behind the ☰ button.

  • Appearance — both the admin and the client portal (palette button in the sidebar header / top bar, or Settings → Appearance), applied instantly and remembered per browser:

    • Style — Modern (the frosted-clay look) or Classic, a flat, dense enterprise-CRM skin (Tahoma 12px, blue applet bars, tab strips, yellow row selection; light only, no blur, shadows or animations).
    • Menu — icon rail + sidebar on the side, or tabs across the top with a row of contextual shortcuts.
    • Width — full width or a centred compact frame (1320px).

    They are data-kit / data-nav / data-width attributes on <html>. The classic skin is packages/ui/src/styles/classic.css: unlayered rules scoped to [data-kit="classic"] that restyle the shared components through their lp-* hook classes — a new screen built from @leadpool/ui components gets both looks for free.

Repeat orders

A client saves any order as a repeat order from the product page (step 4, Repeat) — or from /repeat in the portal — and it's placed for them automatically:

  • When — every day, weekdays, chosen weekdays or monthly (day 1–28), at a time in their own time zone (daylight saving handled; a time skipped by the clock change runs just after it). Optional end date.
  • How — the worker's schedule-run job sweeps every minute and places each due run through the normal order path: balance hold, approval queue for approval-mode clients, fulfilment, file, emails. A slot runs at most once (row claim + an idempotency key per slot), and missed slots aren't made up.
  • Where — each run's file goes to the channels picked on the repeat order (the client's destinations under Settings → Delivery), on top of any auto-send destinations.
  • Money — a run the balance can't cover is skipped (nothing charged, the client is emailed); after 3 skips in a row it pauses itself. A product, tier or filter that no longer works pauses it too.
  • Controls — the client can edit quantities / schedule / destinations, pause, resume, "Run now" (an extra order) or end it; staff see every client's repeat orders on the client → Repeat orders tab and can pause, resume or end them. Every order a repeat order places links back to it.

Tables: order_schedules, order_schedule_runs, orders.schedule_id (packages/db/src/schedules.ts; the cadence maths is packages/shared/src/schedule.ts).

Lead returns

  • Policy — Settings → Lead returns: the on/off switch for every client, the return window (days after delivery), the return limit (% of an order, at least one lead) and which reasons clients can pick. The same tab shows returns waiting, the last 30 days (credited, returned, return rate) and the clients with their own rules. Turning it off hides returns from every portal at once and drops them from the staff menus once nothing is left to review; returns already sent stay in the queue.
  • Per client — client → Settings → Lead returns for this client: switch it off for one client, or give them their own window / limit (empty = the default).
  • Client — order page → Returns → Return leads: paste lead IDs from the file, phones (any format) or emails, or load the edited CSV; the portal shows what matched, what didn't and what's over the limit. Pick a reason (disconnected, wrong person, duplicate, bad email, fake, no consent, other).
  • Staff — Lead returns (rail badge) → each return shows the leads with their details, the share of the order and the client's 90-day return rate. Untick leads that aren't bad and Approve — approved leads are credited at the price paid for them (per line for mixed orders, return_credit in the ledger) and set to archived so nobody else buys them; the rest are rejected. The client gets an email.
  • A delivered lead can be returned once; the allowance is enforced under the order's row lock, and rejected leads free their share of it again.

Tables: lead_returns, lead_return_items, orders.returned_qty / returned_micros, client_orgs.returns_enabled (packages/db/src/returns.ts).

Tables: sort and filter

Every data table uses the shared DataTable (packages/ui/src/components/table.tsx). A column with a value (accessorFn / accessorKey) sorts when its heading is clicked; meta.filter adds the filter button — text, select (with meta.filterOptions), number, money (micro-dollars) or date.

  • Small lists (clients, staff, imports, suppression lists, reports, …) sort and filter in the browser; stateKey remembers the view for the tab.
  • Big lists (orders, returns, wallet history, audit log, a pool's leads) pass useTableState() to the table and its query: the API gets sort, dir and filter (JSON) and planTableQuery (packages/db/src/table-query.ts) turns them into SQL. Each list declares its sortable / filterable columns up front, so a column id can only pick a known expression and every value is a bound parameter; paging stays keyset-based (the cursor carries the sort it was made for). Sorting a pool's leads by a lead field has a 10-second statement limit.

Guided tours

Both apps explain themselves with driver.js tours, themed per kit (Modern light / dark: frosted card, soft entrance, mint ring around the highlighted part; Classic: flat applet window, no motion).

  • Welcome tour — on first sign-in: the menu, status counts, account, appearance and the compass button. When it ends with Done, the current page's tour follows.
  • Page tours — every page (and every tab of a pool, client or Settings) has one, shown once on the first visit. Replay any time with the compass button in the page header or the account menu (Take the tour of this page / Welcome tour).
  • Preferences — Settings → Appearance → Guided tours: turn first-visit tours off, restart the welcome tour, or Show every tour again.
  • Never in the way — a tour waits while a dialog, menu or drawer is open, closes when you navigate, skips steps whose part isn't on screen (a hidden column, a button your role doesn't have, the rail on a phone) and only starts automatically once the page's data has loaded. Esc, the × or Skip tour close it; a skipped tour counts as seen.
  • Where it's stored — per user on the server (staff_users / client_users .tours_seen, .tours_auto; GET / PATCH /api/{admin,portal}/auth/tours), so it follows people across browsers.

Writing a tour: add data-tour="name" to the element (or the tour prop on Tabs, PillTabs, Segmented, DataTable), describe the steps in apps/<app>/src/tours/<page>.ts (TourDefinition: id admin.<page> / portal.<page>, **bold** in bodies) and call usePageTour(tour, { ready }) in the page — or in each tab, so every tab gets its own. Engine and styles: packages/ui/src/tour/tour.tsx, packages/ui/src/styles/tour.css.

Delivery channels

Besides the portal download, delivered orders can be pushed to clients through five channels. Staff connect them once; each client then gets its own destination.

Channel What the client receives You need
Email The file attached (or a portal link above the size limit you set) SMTP host + sign-in, or SMTP_URL
WhatsApp The file as a document message from your business number Meta for Developers → WhatsApp: Phone Number ID + a permanent access token. Free-form messages only reach people who wrote to you in the last 24 h — outside that window pick Approved template and enter its name, language and variables. Without a number on file, the send dialog offers a wa.me link instead
Microsoft Teams An Adaptive Card in their channel (order, product, rows, portal button, cloud links) In Teams: channel → Connectors (or Workflows) → Incoming Webhook → copy the URL. A test posts a card titled "Lead Portal test"
OneDrive The file uploaded to your drive under a folder per client, plus a share link Token mode: a Graph Explorer token with Files.ReadWrite (expires after ~1 h). App mode (recommended): an Azure app with Files.ReadWrite.All application permission — tenant ID, client ID, secret and the drive owner — which never expires
Dropbox The file uploaded to your Dropbox under a folder per client, plus a shared link An app at Dropbox Developers with files.content.write + sharing.write; paste its key + secret and click Connect (refresh token — never expires), or paste a generated access token (expires after ~4 h)
  • Connect — Settings → Delivery channels (admins with settings.manage). Each tab has step-by-step setup, a Test button (sends a real test message or uploads a small test file) and 30-day stats. Credentials are write-only: they are encrypted at rest (AES-256-GCM) and the UI only ever shows "Saved …last 4". Changing anything resets the channel to untested.
  • Per client — client → Delivery tab: email addresses, WhatsApp number, Teams webhook, cloud folder name, file format (CSV / Excel) and Auto-send. Each card has its own Send test.
  • Send — order page → Send: tick channels, choose the format, add an optional note. The history under the delivery file shows every send with status, link and Retry. Needs the orders.send permission (managers and admins).
  • Auto-send — when an order's file is ready, every destination with Auto-send on is sent once.
  • Client portal — turn on Clients can use … in a channel's "Client portal" section and it appears under the client's Settings → Delivery: the owner edits their addresses / number / webhook and auto-send, and any user can Send to… from an order. Cloud folders stay under staff control.
  • How sends run — rows in delivery_sends, processed by the worker (channel-send queue, swept every minute). Cloud uploads go first so the email / WhatsApp / Teams messages carry their links. Failures retry up to 5 times (1 → 3 → 10 → 30 min); an authentication error marks the channel needs attention. Every send and failure is in the audit log.
  • Testing without real accounts — CHANNEL_DEV_WEBHOOK_HOSTS (non-production only) allows extra Teams webhook hosts, and WHATSAPP_API_BASE, GRAPH_API_BASE, MS_LOGIN_BASE, DROPBOX_API_BASE and DROPBOX_CONTENT_BASE point the adapters at a mock server. packages/db/src/channels/channels.test.ts covers each provider's request shapes and error handling.