Turn Your Website Into a Knowledge Base

You paste your website address. FlowCastle finds the pages, reads them, strips the menus and cookie banners, and saves each page as a document in a knowledge base. Attach that base to an AI Chat Agent block and the bot answers from your real pages — your prices, your services, your delivery rules.

It is the fastest way to fill a knowledge base, and usually the most accurate one, because the text is already the text you show customers.

Open Settings → AI in the left menu (the page is titled AI Settings) and scroll to Knowledge bases. Press Import from website. You will see it in two places: at the top of the section (imports into a new base) and on a base you already selected (adds pages to that one).

The form asks for five things:

• Website address — https://example.com. A deeper address imports only that section: give it https://example.com/docs and nothing outside /docs is touched.
• Where to put the pages — add them to the base you selected, or create a new one.
• Knowledge base name — leave it empty and the base is named after the domain.
• Include blog and articles — on by default. Blog posts are what answer "how do I…" questions. Turn it off if you want a smaller base with only the commercial pages.
• Page limit — 200 by default, 500 at most. If the site has more eligible pages than the limit, the important ones are kept: home, pricing, about, contacts, services, FAQ, then features and docs, then everything else.

Press Find pages.

Discovery runs first and then stops. The card shows Found 137 pages on example.com. Import them? with a sample of the addresses it plans to read.

This step exists so you can catch a wrong domain, a staging site or a shop with ten thousand product pages before anything is fetched. Press Import 137 pages to continue, or Cancel import.

The card shows the stage — Finding pages, Reading pages, Saving pages, Indexing (preparing the pages for search), then Ready — and a counter like 42 of 137 pages read.

You can leave the page. The import keeps running on the server, and the card picks it back up when you return.

Not every page belongs in a knowledge base. The importer drops the ones that would only add noise:

Cart, checkout, login, search, tag and author pages are filtered out before fetching, along with images and files.

When the import finishes, Pages that were not imported lists every one of them with the reason. If that list is long and full of Rendered by JavaScript, your site builds its pages in the browser, and the importer cannot read them. It says so directly, and the answer is to paste the key text in as a document instead.

Email addresses and phone numbers usually sit in the footer of every page. Importing them 600 times would make the bot answer "here are our contacts" to almost anything.

So they are collected across the whole site and written into one document: Contact details for example.com. It is a normal document — open it, edit it, delete a number that should not be public.

The base is not connected to anything by itself. Open your flow, click the AI Chat Agent block, go to Settings → Knowledge, and pick the base under Knowledge bases.

Wait for indexing to finish before you test. A document is only searched once it reaches Ready — the bot cannot answer from a page that is not ready yet, and a test run too early looks like the import failed.

The rest of the setup — system prompt, model, intents, what happens when nothing matches — is in The AI Chat Agent Node: Answer Questions From Your Knowledge Base.

Websites change. Press Re-crawl on the base and the whole thing runs again with the same settings.

It is not a re-import from scratch:

• Pages that did not change are left alone and not re-indexed.
• Pages that changed are updated in place.
• Pages that no longer exist on the site are deleted from the base.

So a re-crawl of a mostly-static site is cheap and finishes quickly. Documents you wrote by hand are never touched — only pages that came from that website.

You can also just say it in the AI chat: "make an assistant for example.com". The builder starts the same import, tells you what it is reading, and attaches the finished base to the agent it builds. It runs in the background, so you can keep chatting while it works.

Import the site when the pages already say what you want the bot to say. Write documents by hand when they do not.

Your public pages are written to sell. A support bot often needs the boring version — the exact return window, the exact delivery times, the sentence you are comfortable sending to a customer at 2am. Import the site first, then add those few documents by hand. The two live side by side in the same base.

How the pages are found

1. robots.txt — the file where a site tells crawlers what they may read. It is read first, for the rules and for the sitemap links.
2. The sitemaps it points at (lists of all the site's pages), including sitemap indexes.
3. If there is no sitemap at all: the links on the home page, one level deep. The card warns you when this happens, because the result is usually thinner.

Pages that robots.txt disallows are never fetched, and the card tells you how many were skipped for that reason. The importer identifies itself to websites as FlowCastleKBBot/1.0.

Limits

• 500 pages per import, 200 by default.
• 2 MB per page, 20 MB per import in total.
• 20 seconds per request, three pages at a time.
• http and https only. Private and internal addresses are refused.
• If the site asks for a crawl delay longer than 2 seconds, 2 seconds is used and you are told.

MCP

MCP is a way to connect an outside AI agent to FlowCastle. Agents connected over MCP have the same tool: import_website_into_knowledge_base starts the crawl and returns a job id, and get_website_import reports progress.

One bot, several languages — including the pages you just imported: Multilingual Bots: One Bot, Many Languages.