leadclean

On-prem, LAN-only web app: ingest huge lead files (100M+ rows), clean, dedupe, suppress DNC/blacklist, filter, export dialer-ready lists. Nothing discarded silently — every record lands in a bucket (Clean / Review / Malformed / Removed / Flagged).

Language
Python
Stars
0
Last updated

From the README

LeadClean — Local Lead Data Cleaning & DNC Filtering Platform

Specs: see docs/ (BRD → RTM, shared requirement IDs).

Run locally (one command)

pip install -r backend\requirements.txt
python run.py

Open http://localhost:8080 → login admin / admin123 (change it in Admin tab).

  • LAN access: http://<server-ip>:8080 (open port 8080 in Windows Firewall).
  • Config: copy backend\.env.example → backend\.env (optional — defaults auto-detect machine RAM/cores; works on low-end machines, DuckDB spills to disk).
  • Tests: cd backend && python -m pytest tests/ -q

Workflow

  1. Datasets — upload CSV/TXT/TSV/XLSX (chunked, resumable), any layout, headers optional.
  2. Map — column → canonical mapping; auto-suggests (knows VICIdial repurposed columns: address2=make, address3=model, province=year, postal_code=zip); save as per-vendor template, auto-applies to future files.
  3. Process — vectorized cleaning (phone→10-digit, ZIP pad, Title Case, state codes, car-make aliases, junk→null) → Parquet buckets.
  4. DNC & Rules — upload persistent DNC/blacklists (auto-apply to every job); field exclusion rules (dispositions, area codes...).
  5. Jobs — pick datasets (cross-file dedup), dedup key, merge-fill, fuzzy thresholds, AND/OR filters, export format/chunking/split-by/naming → run. Progress live; results: export files + all 5 buckets downloadable + manifest.json.

Stack

Python 3.11+ · FastAPI · DuckDB (all heavy set-based ops) · Polars · SQLite (metadata only) · Parquet (zstd) · Vue 3 web UI (vendored locally, no build step, works offline) served by the backend.

Fuzzy dedup: DuckDB jaro_winkler_similarity self-join blocked on ZIP (same blocking strategy as Splink, no extra dependency).

Layout (TDD §2)

run.py              one-command start (creates data/ layout, boots API+UI)
backend/app/        FastAPI + domain pipeline
                    ingest → mapping → clean → validate → dedup → suppress → filter → export
backend/tests/      25 tests incl. full E2E pipeline
frontend/dist/      web UI (static, no build step needed)
data/               raw / processed / dnc / jobs / exports  (PRD §13)
deploy/             windows-service (NSSM) · docker (later, per user decision)

Notes

  • Default admin is created on first boot: admin / admin123 — change it.
  • Retention: nothing auto-deletes; datasets/jobs have manual Delete buttons.
  • Docker deferred by decision — native run now; deploy/docker/ reserved.
  • UI is a dependency-free static page (React/AG Grid upgrade path preserved under frontend/src/); all grids page server-side, browser never loads full data.