JCS Data Pipeline Manager

← Back to Intranet

Client & Project

Engagement metadata

Engagement ID *

Format: JCS-DP-YYYYMM-SEQ · auto-generated, editable

Project Name *

Client Company *

Client Contact

Client Email

Engagement Type

Start Date

End Date (or open-ended)

Use Case

Lead Generation

Competitor Pricing

Public Records

Market Research

Content Aggregation

Custom

Project Budget (optional · USD)

Appears on the Client SOW if set

Engagement Status

Target Sites & Pages

One row per starting URL

Start URL

Page Type

JS Render

Auth

Pagination

Est. Pages

Anti-Bot Strategy & Tech Stack

Engine selection & protective measures

Primary Engine *

Headless Mode

Proxy Strategy

Captcha Solver

User Agent Strategy

Fingerprint Profile

Rate Limit (requests/min/host)

Concurrency

Max Retries per Page

Backoff Strategy

Compliance Review

Legal & ethical checkpoints

robots.txt URL

robots.txt Status

ToS Review Status

Personal Data Risk

Applicable Privacy Regime

Self-Identify as Research Bot

Client Legal Approval

Compliance Notes

Schedule & Monitoring

When it runs and how alerts fire

Frequency *

Cron Expression

Editable — auto-fills from frequency above

Timezone

Run Window (optional)

Change Detection Mode

Threshold / Field Trigger

Alert Recipients (comma-separated)

Alert Channels

Slack / Webhook URL (if applicable)

Data Delivery

Format, destination, validation

Output Format *

Delivery Method

Delivery Destination (URL, path, or address)

Schema Validation Level

Cleaning / Normalization

Hold Ctrl/Cmd to select multiple

Retention Period

Delivery Notes

SLA & Terms

Re-delivery guarantee & engagement terms

30-day re-delivery guarantee — if any extracted record fails validation or the target site changes structure within 30 days of delivery, JCS will re-extract and re-deliver at no additional cost.

Active site-structure monitoring during engagement (HTML schema diff between runs)

Failure escalation: alert within 1 hour of 2 consecutive failed runs

Client owns all extracted data; JCS retains no copy after engagement ends

Mutual NDA on file (engagement-specific)

NDA Reference (if checked)

Termination Notice

Produces Client SOW (PDF), Engineering Spec (DOCX via DocGen), and JSON job spec. JSON spec is also registered to the engagement registry.

Schema Metadata

Schema Name *

Schema Version

Description

Primary Key Field

Used for dedupe and change detection

Records per Run (estimate)

Field Definitions

One row per output field

Field Name

Type

Required

Validation

Example

Description

JSON Schema Preview

Live · updates as you edit

(add fields above to populate)

Loading engagements from registry…

Data Pipeline Manager