SpiderVayaPin Worker - SpiderIQ Integration Guide¶

Overview¶

SpiderVayaPin is a worker that runs inside the SpiderIQ ecosystem. It receives enriched business data from SpiderIQ workflows and creates complete VayaPin profiles automatically.

Docker Image: martinshein/spideriq-vayapin-worker:latest

Architecture¶

SpiderIQ Workflow Pipeline
         ↓
   [SpiderMaps] → [SpiderSite] → [SpiderVerify]
         ↓
   Workflow complete - business enriched
         ↓
   Submit job to PostgreSQL queue (worker_type: "vayapin")
         ↓
┌─────────────────────────────────────────────────┐
│  SpiderVayaPin Worker                           │
│                                                 │
│  1. Receive job from queue                      │
│  2. Fetch markdown content from URL             │
│  3. VayaPin: Create account → get account_id    │
│  4. AI: Generate PIN name (OpenRouter)          │
│  5. VayaPin: Check PIN availability             │
│  6. VayaPin: Create subscription → pin_data_set │
│  7. AI: Generate SEO content (7 languages)      │
│  8. VayaPin: Update pin_data_set                │
│  9. Return success with created IDs             │
└─────────────────────────────────────────────────┘
         ↓
   Job completed - VayaPin profile created

Worker Type Registration¶

Add to SpiderIQ worker types:

// In your worker type enum/config
export const WORKER_TYPES = {
  // ... existing workers
  VAYAPIN: "vayapin",
} as const;

Job Payload Structure¶

When submitting a job to the SpiderVayaPin worker, use this payload structure:

interface VayaPinJobPayload {
  // Business basics (required)
  business_name: string;
  country_code: string;  // 2-letter ISO code (e.g., "DK", "US", "DE")

  // Location (required)
  gmaps_coordinates: {
    latitude: number;
    longitude: number;
  };

  // Website content (required)
  markdown_url: string;  // URL to crawled website markdown file

  // Contact info (optional but recommended)
  business_phone?: string;
  business_address?: string;
  original_website?: string;
  domain?: string;

  // Address components (optional)
  street?: string;
  city?: string;
  postal_code?: string;
  state?: string;
  country?: string;

  // Verified emails (optional)
  emails_verified?: Array<{
    email: string;
    status: string;
    is_deliverable: boolean;
  }>;

  // Social media (optional)
  facebook?: string;
  instagram?: string;
  linkedin?: string;
  twitter?: string;
  youtube?: string;
  tiktok?: string;

  // Booking/ordering links (optional)
  booking_appointment_link?: string;
  reservation_links?: string;
  menu_link?: string;
  order_links?: string;
  gmaps_link?: string;

  // Company info from SpiderSite AI extraction (optional but recommended)
  company_info?: {
    industry?: string;
    key_services?: string[];
    target_audience?: string;
    one_sentence_summary?: string;
  };

  // Images (optional)
  logo?: string;           // URL to logo image
  gmaps_image_url?: string; // URL to Google Maps photo

  // Lead scoring (optional, for tracking)
  lead_scoring?: {
    icp_fit_grade?: string;
    engagement_score?: number;
  };
}

interface VayaPinJobInternal {
  // OpenRouter API key from pool (required)
  openrouter_api_key: string;
  openrouter_account_id?: number;
}

Submitting Jobs¶

Via API Gateway¶

// POST /api/v1/internal/queue/submit
const response = await fetch(`${API_GATEWAY_URL}/internal/queue/submit`, {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${INTERNAL_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    worker_type: "vayapin",
    client_id: "vayapin",  // or your client identifier
    payload: {
      business_name: "Restaurant Tight",
      country_code: "DK",
      gmaps_coordinates: { latitude: 55.6761, longitude: 12.5683 },
      markdown_url: "https://media.spideriq.ai/crawls/abc123.md",
      business_phone: "+4533116996",
      original_website: "https://restauranttight.dk",
      // ... other fields
    },
    _internal: {
      openrouter_api_key: "sk-or-v1-xxx",  // From your OpenRouter pool
    },
  }),
});

Via Inngest Function¶

// In your Inngest workflow
import { inngest } from "./client";

export const submitToVayaPin = inngest.createFunction(
  { id: "submit-vayapin-job" },
  { event: "workflow/enrichment.complete" },
  async ({ event, step }) => {
    const { business } = event.data;

    // Get OpenRouter key from pool
    const openrouterKey = await step.run("get-openrouter-key", async () => {
      return getNextOpenRouterKey();
    });

    // Submit to VayaPin worker
    await step.run("submit-vayapin", async () => {
      return submitJob({
        worker_type: "vayapin",
        client_id: "vayapin",
        payload: {
          business_name: business.name,
          country_code: business.country_code,
          gmaps_coordinates: business.coordinates,
          markdown_url: business.markdown_url,
          business_phone: business.phone,
          original_website: business.website,
          emails_verified: business.verified_emails,
          company_info: business.ai_extracted_info,
          logo: business.logo_url,
          gmaps_image_url: business.gmaps_photo,
          // ... map other fields
        },
        _internal: {
          openrouter_api_key: openrouterKey,
        },
      });
    });
  }
);

Job Result Structure¶

When the job completes successfully:

interface VayaPinJobResult {
  success: true;
  data: {
    pin_name: string;           // e.g., "DK:TIGHT"
    account_id: string;         // VayaPin account ID
    pin_subscription_id: string; // VayaPin subscription ID
    pin_data_set_id: string;    // VayaPin data set ID
    business_name: string;
    country_code: string;
  };
  metadata: {
    worker_version: string;     // e.g., "1.0.0"
    processing_time_seconds: number;
    worker_id: string;
  };
}

On failure:

interface VayaPinJobError {
  success: false;
  error: string;
  metadata: {
    worker_version: string;
    worker_id: string;
  };
}

Docker Deployment¶

docker-compose.workers.yml¶

services:
  spider-vayapin-worker:
    image: martinshein/spideriq-vayapin-worker:latest
    container_name: spider-vayapin-worker
    restart: unless-stopped
    environment:
      # SpiderIQ connection
      USE_PG_QUEUE: "true"
      API_GATEWAY_URL: http://api-gateway:3000/api/v1
      INTERNAL_API_KEY: ${INTERNAL_API_KEY}

      # Worker identification
      WORKER_ID: spider-vayapin-1
      VPS_HOST: ${VPS_HOST:-vps1}

      # Polling configuration
      POLL_INTERVAL: "2.0"

      # VayaPin API
      VAYAPIN_API_TOKEN: ${VAYAPIN_API_TOKEN}

      # Note: OpenRouter key comes from SpiderIQ in _internal
    networks:
      - spideriq
    deploy:
      replicas: 3
      resources:
        limits:
          memory: 512M
          cpus: '0.5'

networks:
  spideriq:
    external: true

Environment Variables¶

Variable	Required	Description
`API_GATEWAY_URL`	Yes	SpiderIQ API Gateway URL
`INTERNAL_API_KEY`	Yes	SpiderIQ internal API key
`VAYAPIN_API_TOKEN`	Yes	VayaPin API authentication token
`WORKER_ID`	No	Unique worker identifier (default: auto-generated)
`VPS_HOST`	No	VPS hostname for tracking (default: "local")
`POLL_INTERVAL`	No	Seconds between queue polls (default: 2.0)

VayaPin API Flow¶

The worker executes this exact sequence:

Create Account → POST /accounts
Returns account_id and created_at
Generate PIN Name (AI)
Uses OpenRouter with free models
Max 15 chars, uppercase, ASCII only
Converts special characters (Ø→O, Æ→AE, etc.)
Check PIN Availability → POST /pin_availability_check
If taken, uses first alternative from response
Create PIN Subscription → POST /pin_subscriptions
Returns pin_subscription_id and pin_data_set_id
Generate SEO Content (AI)
Creates multilingual descriptions (EN, ES, NL, PT, FR, DE, DA)
600-800 characters per language
Generates meta tags, keywords, Open Graph, schema.org
Update PIN Data Set → PATCH /pin_data_sets/{id}
Sets all content in one request

OpenRouter Configuration¶

The worker uses OpenRouter for AI generation. Keys come from SpiderIQ's pool via _internal.openrouter_api_key.

Free models used:

mistralai/mistral-7b-instruct:free (default)
meta-llama/llama-3.2-3b-instruct:free
google/gemma-2-9b-it:free

Fallback paid models:

openai/gpt-4o-mini
anthropic/claude-3-haiku

Concurrency & Rate Limits¶

Recommended settings:

3 worker replicas for standard load
5-10 replicas for high volume
OpenRouter rate limits apply per API key

VayaPin API:

No documented rate limits
Worker includes retry with exponential backoff (3 attempts)

Monitoring¶

The worker logs structured JSON via structlog:

{
  "event": "Job claimed",
  "job_id": "abc123",
  "worker_type": "vayapin",
  "worker_id": "spider-vayapin-1",
  "timestamp": "2024-01-15T10:00:00Z"
}

Key log events:

Worker starting...
Worker initialized, polling for jobs...
Job claimed
Processing VayaPin job
Step X: [action] (for each step)
VayaPin profile created successfully
Job completed

Error Handling¶

The worker handles errors gracefully:

Network errors → Retry with exponential backoff (3 attempts)
VayaPin API errors → Logged and returned in job result
AI generation failures → Falls back to basic content generation
PIN name taken → Uses alternative name from VayaPin response

Testing¶

Run the test suite:

cd SpiderVayaPin
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python tests/test_worker.py

Updating the Worker¶

# Pull latest
docker pull martinshein/spideriq-vayapin-worker:latest

# Or build from source
cd /path/to/vayapin
docker build -t martinshein/spideriq-vayapin-worker:v1.0.1 -f SpiderVayaPin/Dockerfile .
docker push martinshein/spideriq-vayapin-worker:v1.0.1

Troubleshooting¶

Worker not claiming jobs¶

Check worker_type is exactly "vayapin"
Verify INTERNAL_API_KEY is correct
Check API Gateway connectivity

VayaPin API errors¶

Verify VAYAPIN_API_TOKEN is valid
Check VayaPin API is reachable from worker

AI generation failures¶

Verify OpenRouter key in _internal is valid
Check OpenRouter rate limits
Worker falls back to basic content on failure

PIN name conflicts¶

Worker automatically uses alternatives
If all alternatives fail, job will error

Contact¶

VayaPin API Base: https://cs.vayapin.com/api/automations_v1
Docker Hub: martinshein/spideriq-vayapin-worker

SPIDERIQ_INTEGRATION