VayaPin Admin Dashboard Enhancement Plan¶

Overview¶

Enhance the VayaPin admin dashboard with additional testing and debugging features.

Base System (Already Implemented): - Admin dashboard at /admin/vayapin ✓ - Auto-export scheduler (every 5 minutes) ✓ - KPI cards, config modal, export run history ✓ - Toggle enable/disable, manual export trigger ✓

New Enhancement Requirements: 1. Show leads with extracted emails even if not yet verified by SpiderVerify 2. Button to manually export a single specific lead 3. Store request/response JSON for debugging (JSONL files vs database) 4. List of all created VayaPins with search functionality

Implementation Steps¶

Enhancement 1: Show Leads with Unverified Emails¶

Problem: Current pending leads query only shows leads where emails_verified IS NOT NULL. User wants to also see leads that have extracted emails from SpiderSite but haven't gone through SpiderVerify yet.

Data Flow:

SpiderSite extracts emails → stored in jobs.results.emails[]
SpiderVerify verifies → stored in campaign_workflow_jobs.emails_verified[]

Changes:

1.1 Add config option - app/database/migrations/061_vayapin_show_unverified.sql

ALTER TABLE vayapin_export_config
ADD COLUMN IF NOT EXISTS show_unverified_emails BOOLEAN DEFAULT false;

1.2 Update service - app/services/vayapin_export_service.py - Add show_unverified_emails to config response - Modify get_pending_leads() to optionally include unverified leads: - When show_unverified_emails=true: Also query leads where spidersite_job_id IS NOT NULL and join to get extracted emails from jobs.results - Add verification_status field: "verified", "unverified", "none"

1.3 Update hooks - apps/web/src/hooks/useVayapin.ts - Add show_unverified_emails to VayapinConfig type - Add verification_status to VayapinPendingLead type

1.4 Update UI - apps/web/src/routes/admin/vayapin.tsx - Add toggle in config modal: "Show leads with unverified emails" - Add badge on pending leads: "Verified" (green) / "Unverified" (yellow)

Enhancement 2: Single Lead Manual Export¶

Problem: Current manual export processes a batch. User wants to click on a specific lead and export just that one.

Changes:

2.1 Add API endpoint - app/api/v1/admin_vayapin.py

@router.post("/leads/{source}/{lead_id}/export")
async def export_single_lead(
    source: str,  # "campaign" or "playbook"
    lead_id: int,
    dry_run: bool = Query(False),
    db: AsyncSession = Depends(get_db),
    _: bool = Depends(require_admin)
):
    """Export a single specific lead to VayaPin."""

2.2 Add service method - app/services/vayapin_export_service.py

async def export_single_lead(self, source: str, lead_id: int, dry_run: bool = False):
    # 1. Fetch the specific lead
    # 2. Build payload
    # 3. Submit job (or simulate if dry_run)
    # 4. Return result

2.3 Add hook - apps/web/src/hooks/useVayapin.ts

export function useExportSingleLead() {
  return useMutation({
    mutationFn: async ({ source, leadId, dryRun }: { source: string; leadId: number; dryRun?: boolean }) => {
      return api.post(`/api/v1/admin/vayapin/leads/${source}/${leadId}/export`, { dry_run: dryRun });
    },
  });
}

2.4 Add Pending Leads tab in UI - apps/web/src/routes/admin/vayapin.tsx - Add tabbed interface: "Overview" | "Pending Leads" | "Exported Leads" - Pending Leads table with columns: Business Name, Domain, Source, Emails, Verification Status, Actions - "Export" button on each row (opens confirm dialog with dry-run option)

Enhancement 3: Store Request/Response JSON for Debugging¶

Decision: JSONL files (user preference to avoid database bloat)

Location: /var/log/spideriq/vayapin-export.jsonl

Benefits: - No database storage overhead - Easy to grep/analyze offline - Can be rotated/archived separately - Standard log location

JSONL Format:

{"timestamp": "2026-02-28T10:30:00Z", "job_id": "abc123", "source": "campaign", "source_id": 456, "request": {...}, "response": {...}, "status": "exported"}

Changes:

3.1 Create logging utility - app/services/vayapin_logger.py

import json
from datetime import datetime
from pathlib import Path

LOG_PATH = Path("/var/log/spideriq/vayapin-export.jsonl")

def log_vayapin_export(job_id: str, source: str, source_id: int, request: dict, response: dict, status: str):
    """Append export record to JSONL file."""
    LOG_PATH.parent.mkdir(parents=True, exist_ok=True)

    record = {
        "timestamp": datetime.utcnow().isoformat() + "Z",
        "job_id": job_id,
        "source": source,
        "source_id": source_id,
        "request": request,
        "response": response,
        "status": status,
    }

    with open(LOG_PATH, "a") as f:
        f.write(json.dumps(record) + "\n")

def read_recent_logs(limit: int = 100, job_id: str = None) -> list:
    """Read recent log entries (for UI display)."""
    if not LOG_PATH.exists():
        return []

    # Read last N lines efficiently
    lines = []
    with open(LOG_PATH, "rb") as f:
        # Seek to end and read backwards
        ...
    return lines

3.2 Add API endpoint - app/api/v1/admin_vayapin.py

@router.get("/logs")
async def get_export_logs(
    limit: int = Query(50, ge=1, le=200),
    job_id: Optional[str] = Query(None),
    _: bool = Depends(require_admin)
):
    """Read recent VayaPin export logs from JSONL file."""
    from app.services.vayapin_logger import read_recent_logs
    return {"logs": read_recent_logs(limit=limit, job_id=job_id)}

3.3 Update export logic - app/api/v1/admin_vayapin.py and app/workflows/functions/scheduled.py

from app.services.vayapin_logger import log_vayapin_export

# After building payload, before dispatch:
# (request is logged)

# After job completes (in callback):
log_vayapin_export(
    job_id=job_id,
    source=payload.get("source_type"),
    source_id=payload.get("source_id"),
    request=payload,
    response=result,
    status="exported" if result.get("exported") else "skipped"
)

3.4 Add Docker volume - docker-compose.yml

api-gateway:
  volumes:
    - vayapin-logs:/var/log/spideriq

volumes:
  vayapin-logs:

3.5 Add log rotation - /etc/logrotate.d/vayapin (via Ansible)

/var/log/spideriq/vayapin-export.jsonl {
    daily
    rotate 7
    compress
    missingok
    notifempty
}

3.6 Add to UI - apps/web/src/routes/admin/vayapin.tsx - Add "Export Logs" tab or section - Table showing recent logs with expandable JSON - Filter by job_id for debugging specific exports

Enhancement 4: List of All VayaPins with Search¶

Problem: No UI currently shows the exported leads list with search.

Changes:

4.1 Add search parameter - app/api/v1/admin_vayapin.py

@router.get("/exported")
async def get_exported_leads(
    search: Optional[str] = Query(None, description="Search by business name, domain, or PIN name"),
    ...existing params...
):

4.2 Update service - app/services/vayapin_export_service.py

async def get_exported_leads(self, search: str = None, ...):
    # Add WHERE clause:
    # business_name ILIKE '%search%'
    # OR domain ILIKE '%search%'
    # OR vayapin_pin_name ILIKE '%search%'

4.3 Add hook parameter - apps/web/src/hooks/useVayapin.ts

interface LeadsQuery {
  page?: number;
  page_size?: number;
  source?: string;
  status?: string;
  search?: string;  // ADD
}

4.4 Add Exported Leads tab in UI - apps/web/src/routes/admin/vayapin.tsx - Table columns: Business Name, Domain, PIN Name, Account ID, Status, Skip Reason, Exported At, Actions - Search input above table - Status filter dropdown (exported, skipped, failed, queued) - Source filter (campaign, playbook) - Expandable row showing SEO content preview and request/response JSON - Click on PIN name to copy

Files to Create/Modify¶

New Files¶

File	Purpose
`app/database/migrations/061_vayapin_enhancements.sql`	Add `show_unverified_emails` config column
`app/services/vayapin_logger.py`	JSONL logging utility for request/response

Files to Modify¶

File	Changes
`app/api/v1/admin_vayapin.py`	Add `/leads/{source}/{lead_id}/export`, `/logs`, add `search` param
`app/services/vayapin_export_service.py`	Add `export_single_lead()`, update queries for unverified, add search
`app/schemas/vayapin_export.py`	Add new fields to schemas
`app/workflows/functions/scheduled.py`	Call logger after building payload
`app/workflows/functions/standalone.py`	Call logger with response in callback
`apps/web/src/hooks/useVayapin.ts`	Add `useExportSingleLead`, `useVayapinLogs`, `search` param, new types
`apps/web/src/routes/admin/vayapin.tsx`	Add tabbed UI, pending leads table, exported leads table with search, logs viewer
`docker-compose.yml`	Add volume mount for `/var/log/spideriq`
`CHANGELOG.md`	Document enhancements

UI Design¶

Tab Structure¶

┌─────────────────────────────────────────────────────────────────┐
│  [Overview]  [Pending Leads]  [Exported Leads]  [Export Logs]   │
└─────────────────────────────────────────────────────────────────┘

Overview Tab (Current)¶

KPI Cards
Config Summary
Export Run History

Pending Leads Tab (New)¶

┌─────────────────────────────────────────────────────────────────┐
│  Filter: [All ▾] [Campaign ▾] [Playbook ▾]   [🔄 Refresh]      │
├─────────────────────────────────────────────────────────────────┤
│  BUSINESS NAME    DOMAIN           EMAILS  STATUS   ACTIONS    │
│  ─────────────────────────────────────────────────────────────  │
│  Acme Corp        acme.com         3       ✓ Verified  [Export]│
│  Beta Inc         beta.io          2       ⚠ Unverified[Export]│
│  ...                                                            │
└─────────────────────────────────────────────────────────────────┘

Exported Leads Tab (New)¶

┌─────────────────────────────────────────────────────────────────┐
│  🔍 [Search by name, domain, PIN...]                            │
│  Status: [All ▾]  Source: [All ▾]                              │
├─────────────────────────────────────────────────────────────────┤
│  BUSINESS    PIN NAME    STATUS     EXPORTED AT    ACTIONS     │
│  ─────────────────────────────────────────────────────────────  │
│  Acme Corp   DK:ACME     exported   2h ago         [▼ Details] │
│  Beta Inc    --          skipped    1h ago         [▼ Details] │
│  ...                                                            │
├─────────────────────────────────────────────────────────────────┤
│  ▼ Expanded Details:                                            │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │ PIN: DK:ACME  Account: acc_123  Subscription: sub_456      ││
│  │ SEO Preview: "Professional services company..."            ││
│  └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘

Export Logs Tab (New)¶

┌─────────────────────────────────────────────────────────────────┐
│  🔍 [Filter by Job ID...]                     [🔄 Refresh]     │
├─────────────────────────────────────────────────────────────────┤
│  TIMESTAMP          JOB ID      SOURCE    STATUS    ACTIONS    │
│  ─────────────────────────────────────────────────────────────  │
│  2026-02-28 10:30   abc123      campaign  exported  [▼ JSON]   │
│  2026-02-28 10:25   def456      playbook  skipped   [▼ JSON]   │
│  ...                                                            │
├─────────────────────────────────────────────────────────────────┤
│  ▼ Expanded JSON:                                               │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │ Request:                                                    ││
│  │ {                                                           ││
│  │   "business_name": "Acme Corp",                            ││
│  │   "country_code": "DK",                                    ││
│  │   ...                                                       ││
│  │ }                                                           ││
│  │                                                             ││
│  │ Response:                                                   ││
│  │ {                                                           ││
│  │   "exported": true,                                         ││
│  │   "data": { "pin_name": "DK:ACME", ... }                   ││
│  │ }                                                           ││
│  └─────────────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────────────┘

Verification¶

1. Run Migration¶

docker exec spideriq-postgres psql -U spideriq -d spideriq \
  -f /migrations/061_vayapin_enhancements.sql

2. Check New Columns¶

docker exec spideriq-postgres psql -U spideriq -d spideriq -c "
  SELECT column_name FROM information_schema.columns
  WHERE table_name = 'campaign_workflow_jobs'
  AND column_name LIKE 'vayapin%';"

3. Rebuild API Gateway¶

docker compose build api-gateway && docker compose up -d api-gateway

4. Test Single Lead Export (Dry Run)¶

curl -X POST "https://spideriq.ai/api/v1/admin/vayapin/leads/campaign/123/export?dry_run=true" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

5. Test Search¶

curl "https://spideriq.ai/api/v1/admin/vayapin/exported?search=acme" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

6. Rebuild Dashboard¶

cd apps/web && npm run build
docker compose up -d --build dashboard

7. Verify UI¶

Navigate to /admin/vayapin
Click "Pending Leads" tab - should show leads
Click "Export" on a lead - should trigger export
Click "Exported Leads" tab - should show exports with search
Expand a row - should show request/response JSON

8. Verify JSONL Log File¶

# Check log file exists
ls -la /var/log/spideriq/vayapin-export.jsonl

# View recent entries
tail -5 /var/log/spideriq/vayapin-export.jsonl | jq .

# Search for specific job
grep "abc123" /var/log/spideriq/vayapin-export.jsonl | jq .

9. Test Logs API Endpoint¶

curl "https://spideriq.ai/api/v1/admin/vayapin/logs?limit=10" \
  -H "Authorization: Bearer $ADMIN_TOKEN" | jq .

Configuration Defaults (Updated)¶

Setting	Default	Description
`enabled`	false	Auto-export disabled for testing
`rate_per_hour`	100	Max exports per hour
`batch_size`	10	Leads per scheduler run
`require_deliverable_email`	true	Only export with deliverable email
`require_markdown_content`	true	Only export with markdown content
`show_unverified_emails`	false	Show leads with unverified emails in pending

VayaPin_Admin_Dashboard_Enhancement_Plan

VayaPin Admin Dashboard Enhancement Plan¶

Overview¶

Implementation Steps¶

Enhancement 1: Show Leads with Unverified Emails¶

Enhancement 2: Single Lead Manual Export¶

Enhancement 3: Store Request/Response JSON for Debugging¶

Enhancement 4: List of All VayaPins with Search¶

Files to Create/Modify¶

New Files¶

Files to Modify¶

UI Design¶

Tab Structure¶

Overview Tab (Current)¶

Pending Leads Tab (New)¶

Exported Leads Tab (New)¶

Export Logs Tab (New)¶

Verification¶

1. Run Migration¶

2. Check New Columns¶

3. Rebuild API Gateway¶

4. Test Single Lead Export (Dry Run)¶

5. Test Search¶

6. Rebuild Dashboard¶

7. Verify UI¶

8. Verify JSONL Log File¶

9. Test Logs API Endpoint¶

Configuration Defaults (Updated)¶