Data workflows · an example with n8n

Gather the web on a schedule. Pipe it into your data.

A lot of products depend on data that lives on other people’s web pages and changes without notice. Gourdo plus a workflow tool like n8n turns that into a scheduled pipeline: fetch the pages, extract what matters, store it, repeat. This page shows the pattern and one pipeline we run ourselves.

See the exampleRead the API docs

Who does this, and why

The same pipeline, four jobs.

Market and price watching

A set of pages checked every day and diffed against yesterday. Prices, inventory, rates, availability: trend lines instead of screenshots.

Unstructured data at volume

Hundreds of listings, filings or profiles that nobody publishes as an API, turned into rows an analyst can query.

Bringing related sources together

Five sites that each hold a piece of the picture, merged into one view people can actually read.

Signal feeds

Job postings, regulatory notices, competitor changelogs: scraped on a cadence, filtered by an LLM, delivered as a feed to the people who act on them.

The example below is the smallest version of all four.

The example

A startup events calendar, kept current by a pipeline.

It runs Philadelphia’s community startup calendar at philly.ue.community: a dozen event sites, read three times a day, every event extracted by an LLM, duplicates dropped, new ones posted. It is an n8n cloud workflow with Gourdo as the scraper.

Nothing here is special to events. Swap the sources and the extraction schema and the same shape gathers prices, listings, or filings. It is deliberately simple; every piece scales.

Part one

A reusable Gourdo tool in n8n.

Five nodes. Build it once; every workflow in your account calls it with a URL and an output format and gets a clean page back.

  1. When executed by another workflowinputs: url, output
  2. Workflow configurationdefaults
  3. Gourdo API requestPOST /api/scrape
  4. Close the sessionDELETE /api/scrape/sessions/{id}
  5. Clean and truncatemarkdown → content
The Gourdo Scraper Tool: trigger, configuration, scrape request, close session, clean and truncate
  1. 1
    Trigger: "When executed by another workflow"The tool takes two inputs, url and output. Any workflow in your n8n account calls it with those two values and gets a page back. That is the whole interface.
  2. 2
    Workflow configurationA Set node for defaults you may want to change later, such as the output format when the caller does not specify one. Keep configuration here, not scattered through the nodes.
  3. 3
    Gourdo API requestAn HTTP Request node: POST to https://gourdo.ai/api/scrape with a JSON body of { url, output }. The API key goes in the X-API-Key header, and the header comes from an n8n Header Auth credential, not a value typed into the node. Create the credential once; every node that talks to Gourdo references it. Turn on "never error" so a failed page comes back as data you can inspect instead of stopping the run.
  4. 4
    Close the sessionA second HTTP Request node: DELETE https://gourdo.ai/api/scrape/sessions/ followed by the sessionId from the previous response. Every scrape opens a browser session, and an open session keeps metering a little data until it expires. Closing it the moment you have the page is the single cheapest habit in the whole pipeline.
  5. 5
    Clean and truncateA Code node that takes data.markdown from the scrape response, caps it at a length your LLM step can handle, and passes the final url through alongside it. The tool returns { content, url }. Callers never see the raw API response.

The request

POST https://gourdo.ai/api/scrape
X-API-Key: {{ credential }}
Content-Type: application/json

{
  "url": "{{ $json.url }}",
  "output": "{{ $json.output }}"
}

What comes back

{
  "data": {
    "url": "https://…/final-url",
    "statusCode": 200,
    "sessionId": "7f3c…e21a",
    "usage": { "bytes": 412380 },
    "markdown": "# …"
  }
}

Clean and truncate

const res = $('Gourdo API request')
  .first().json;
const content = (res.data?.markdown || '')
  .substring(0, 100000);
return [{
  json: { content, url: res.data?.url }
}];
Two things people get wrong: the key belongs in an n8n credential, never typed into a node; and the session gets closed right after the fetch, because an open one keeps metering a little data until it expires.

Part two

The aggregator.

A simplified sketch of the real pipeline, with the error handling and retries left out so the shape is clear.

  1. Schedulethree times a day
  2. Sourcesthe list of pages to read
  3. For each sourcesplit out
  4. Gourdo Scraper Tooloutput: markdown-clean
  5. Clean and chunksmall pieces, with context
  6. LLM extractionfixed JSON schema
  7. Flattenone item per event
  8. Dedupe against the storeurl + title fingerprint
  9. Saveyour sheet, DB or API
The aggregator: schedule, sources, for each source call the Gourdo tool, clean and chunk, LLM extraction, flatten, dedupe against the store, save
  1. 1
    ScheduleA Schedule Trigger. The calendar runs three times a day; a price watcher might run once a night. The cadence is the only thing that changes.
  2. 2
    SourcesThe list of pages to read. In the real pipeline this comes from an API so sources can be added without editing the workflow; a Set node with a hard-coded list works to start.
  3. 3
    For each source, call the toolSplit the list and call the Gourdo Scraper Tool from part one with the source url and markdown-clean as the output. One call, one rendered page, session closed.
  4. 4
    Clean and chunkListing pages are long and repetitive. A Code node strips boilerplate, then splits the Markdown into chunks of a few thousand characters, carrying the last date heading into each chunk so an event is never separated from its date.
  5. 5
    LLM extractionAn LLM node with a system prompt that defines the job and a fixed JSON schema for the answer: title, registration URL, raw date text, location hint, source URL. The rules that matter: return only JSON, extract only what is stated, never invent, and build absolute URLs from the source domain.
  6. 6
    FlattenParse each chunk’s JSON and emit one item per event, dropping anything without a title and a URL, and anything whose URL is an image.
  7. 7
    Dedupe against the storeSources overlap and the same event reappears on every run, so a within-run dedupe is not enough. Fetch what the store already holds and drop matches by normalized URL and by a title-plus-source fingerprint.
  8. 8
    SaveWrite what is left to wherever your data lives: a sheet, a database, your own API. The real pipeline posts each new event through a second workflow that verifies it first; that is the piece to add once the basics run.

Why chunk

A rendered listing page can be 50,000 characters. Extraction gets measurably worse as the input grows, and small chunks with the date heading carried forward keep every event attached to its date.

Why dedupe against the store, not within the run

The run only knows what it just read. The store knows what it has already published. Comparing against the store is what makes a scheduled pipeline safe to run as often as you like.

Make it yours

Four things to swap. Three things to know.

  • Swap the sourcesAny page a browser can render. Add one by adding a URL.
  • Change the schemaThe LLM step extracts whatever shape you define: a price and SKU, a filing and date, a listing and address.
  • Change the cadenceHourly, nightly, weekly. The pipeline does not care.
  • Change the storeA sheet to start, a database when it grows, a vector store if you mean to search it.
  • Read-only: Gourdo renders and reads pages; it does not click, type or log in at this time. The feature is planned for a future release.
  • No captcha or block detection at this time: a block page comes back as content with the status the site returned, so check the status code before extracting. Detection is planned for a future release.
  • Every call is metered, failed ones too. Close sessions, and don’t retry in a loop against a site that is blocking you.

The limits in full are on the FAQ; the request schema is in the docs.

Start with one source.

Get startedPrefer an agent to a workflow? Connect over MCP