leadclean
On-prem, LAN-only web app: ingest huge lead files (100M+ rows), clean, dedupe, suppress DNC/blacklist, filter, export dialer-ready lists. Nothing discarded silently — every record lands in a bucket (Clean / Review / Malformed / Removed / Flagged).
- Language
- Python
- Stars
- 0
- Last updated
From the README
LeadClean — Local Lead Data Cleaning & DNC Filtering Platform
Specs: see docs/ (BRD → RTM, shared requirement IDs).
Run locally (one command)
pip install -r backend\requirements.txt python run.py
Open http://localhost:8080 → login admin / admin123 (change it in Admin tab).
- LAN access:
http://<server-ip>:8080(open port 8080 in Windows Firewall). - Config: copy
backend\.env.example→backend\.env(optional — defaults auto-detect machine RAM/cores; works on low-end machines, DuckDB spills to disk). - Tests:
cd backend && python -m pytest tests/ -q
Workflow
- Datasets — upload CSV/TXT/TSV/XLSX (chunked, resumable), any layout, headers optional.
- Map — column → canonical mapping; auto-suggests (knows VICIdial repurposed columns: address2=make, address3=model, province=year, postal_code=zip); save as per-vendor template, auto-applies to future files.
- Process — vectorized cleaning (phone→10-digit, ZIP pad, Title Case, state codes, car-make aliases, junk→null) → Parquet buckets.
- DNC & Rules — upload persistent DNC/blacklists (auto-apply to every job); field exclusion rules (dispositions, area codes...).
- Jobs — pick datasets (cross-file dedup), dedup key, merge-fill, fuzzy thresholds, AND/OR filters, export format/chunking/split-by/naming → run. Progress live; results: export files + all 5 buckets downloadable + manifest.json.
Stack
Python 3.11+ · FastAPI · DuckDB (all heavy set-based ops) · Polars · SQLite (metadata only) · Parquet (zstd) · Vue 3 web UI (vendored locally, no build step, works offline) served by the backend.
Fuzzy dedup: DuckDB jaro_winkler_similarity self-join blocked on ZIP
(same blocking strategy as Splink, no extra dependency).
Layout (TDD §2)
run.py one-command start (creates data/ layout, boots API+UI)
backend/app/ FastAPI + domain pipeline
ingest → mapping → clean → validate → dedup → suppress → filter → export
backend/tests/ 25 tests incl. full E2E pipeline
frontend/dist/ web UI (static, no build step needed)
data/ raw / processed / dnc / jobs / exports (PRD §13)
deploy/ windows-service (NSSM) · docker (later, per user decision)
Notes
- Default admin is created on first boot: admin / admin123 — change it.
- Retention: nothing auto-deletes; datasets/jobs have manual Delete buttons.
- Docker deferred by decision — native run now;
deploy/docker/reserved. - UI is a dependency-free static page (React/AG Grid upgrade path preserved
under
frontend/src/); all grids page server-side, browser never loads full data.