Content Sources

Content sources are the data inputs that populate your knowledge bases with up-to-date information. Seclai automatically fetches, processes, and indexes content from various sources, keeping your knowledge bases current without manual intervention.

What are Sources?

A source is a connection to external content that you want to make searchable. When you add a source to a knowledge base:

1. Initial Polling: Seclai fetches content based on your seeding preferences
2. Content Processing: Text is extracted, chunked, and embedded for vector search
3. Automatic Updates: Content is refreshed based on your polling schedule
4. Multi-Phase Processing: Audio/video is transcribed, and all content is indexed

Sources can be shared across multiple knowledge bases or organizations, with each connection maintaining independent settings for polling, retention, and indexing.

Source Types

RSS Feeds

RSS feeds automatically pull content from blogs, podcasts, news sites, and other syndicated content.

Best For:

• Blog posts and articles
• Podcast episodes
• News feeds
• YouTube channels (via RSS)

Features:

• Automatic detection of RSS/Atom feeds
• Content metadata extraction (title, author, date)
• Support for full content or summary feeds
• Historical data seeding options

Example Use Cases:

• Monitor industry news and trends
• Track competitor blog posts
• Index podcast transcripts for searchability
• Aggregate content from multiple sources

Websites

A website source indexes a start page and any other pages chosen when it was created. Each scheduled poll fetches those same pages again; links on them are not followed.

Best For:

• Documentation sites
• Help centers
• Product pages
• Knowledge bases

Features:

• In the app, index the start page alone, the pages it links to on the same site up to a number you set, or pages you pick from those links.
• Your plan can limit how many linked pages you can add when you create a website source in the app.
• Main content extraction (removes headers, footers, navigation)

To index a page the start page does not link to, add it as a source of its own.

Example Use Cases:

• Index your product documentation
• Monitor competitor websites
• Create searchable help center
• Track regulatory or compliance documentation

Content Store

Content stores (API type: custom_index) let you upload documents, push content via API, or build your own index programmatically. Each content store uses an index mode that controls the trade-off between embedding quality, search speed, and cost.

Index Modes:

All preset modes use the default embedding model. The Custom mode lets you pick any available embedding model, dimension count, chunk size, overlap, and language-specific or custom separators.

Best For:

• Internal documents and custom data
• API-driven content from external systems
• Programmatic content management
• Optimizing retrieval quality with custom embeddings

Supported File Formats:

• Text: plain text, Markdown, HTML, CSV and TSV, JSON, XML, YAML and other configuration files, logs, subtitles, calendar invites, contact cards, and source code
• Documents: .pdf, Word, PowerPoint and Excel, .rtf, .epub, Jupyter notebooks, RSS/Atom feeds, and Outlook (.msg) and forwarded (.eml) emails
• Images: .png, .jpg, .gif, .bmp, .tiff, .webp, .svg
• Audio (with transcription): .mp3, .wav, .m4a, .flac, .ogg, .aac, .opus, .amr and WebM audio
• Video (with transcription): .mp4, .mov, .avi, .webm, .mkv, .mpeg, .3gp and MPEG transport streams
• Archives: .zip

Legacy Word .doc files: Microsoft Word 97–2003 binary documents (.doc) are not supported. Open the file in Microsoft Word, Google Docs, LibreOffice, or Apple Pages, save it as .docx, and upload the new file.

Features:

• Custom or preset embedding configuration
• API-driven content creation with full control over metadata
• Embedding migration to change models or dimensions after creation
• Direct file storage integration

Note: The legacy file_uploads source type is accepted as an alias for custom_index with index_mode=fast_and_cheap.

Cloud Drives

Cloud drive sources (type: cloud_drive) ingest a folder from a connected cloud drive into a knowledge base and keep it in sync — new and changed files are pulled in automatically, near-real-time via the provider's change webhook (a periodic poll is the backstop). Connect a drive once under Integrations → Cloud Drives (Dropbox or Google Drive), then point a source at a folder. See the Cloud Drives guide for the connection walkthrough.

Best For:

• Keeping a knowledge base in sync with a shared team folder
• Ingesting reports, meeting notes, and proposals as they land
• Research papers and technical documentation kept on a drive

Features:

• Automatic add/update ingestion — each file flows through the normal extract → index pipeline, versioned by the provider's file id
• Near-real-time sync via the provider webhook, with a scheduled poll as the backstop
• Optional filters: a folder path, a path glob (e.g. **/*.pdf), a MIME allowlist (e.g. application/pdf, image/*), and a per-file size cap
• The same file support as uploads, including audio/video transcription and document/image text extraction

Configuration:

• Connection (required) — the cloud-drive connection the files come from
• Folder path — the folder to ingest (blank = the connection's own folder). Use Browse… to pick one rather than typing it; on Google Drive a folder on a shared drive is written /Shared drives/<drive>/<folder>
• Path pattern — optional case-insensitive glob to keep only matching files
• Content types — optional comma-separated MIME allowlist
• Max file size (MB) — optional per-file cap (bounded by the platform ceiling)

Supported providers: Dropbox and Google Drive (more as they ship). Google Docs, Sheets and Slides are exported to text, CSV and PDF automatically.

Deletions are not propagated: new and changed files are ingested, but deleting a file on the drive does not remove it from the knowledge base — remove it there if you need it gone. File types match the Content Store formats above.

Note: v1 ingests adds and updates only — deleting a file on the drive does not remove it from the knowledge base yet. Governance source-content screening applies to cloud-drive files just like any other source.

Email

Email sources (type: email) store mail sent to the source's own address. Each accepted message becomes a content item and flows through the normal extract → index pipeline, so a knowledge base can be fed by newsletters, mailing lists, a shared inbox or forwarded mail.

Best For:

• Newsletters and mailing lists that have no feed or page to poll
• Reports and digests that other systems deliver by email
• Mail your team forwards to keep on record

An email source is never an open inbox. It needs at least one allowed sender, and it stores a message only when both of these hold:

• The address in the message's From line is on the list. An entry is a full address (news@example.com) or a bare domain (example.com), and a domain also matches its sub-domains.
• The message passed DMARC, which is what proves the From line was not forged. Mail without a DMARC pass is refused, including mail from a domain that publishes no DMARC policy.

A newsletter delivered through a bulk-mail provider is matched on the address it shows in its From line, not on the provider's delivery address. Newsletters and other bulk or list mail are always accepted from an allowed sender.

Configuration:

• Allowed senders (required) — the addresses and domains whose mail is stored. The list can be changed later but never emptied
• Retention — optional number of days to keep each message

The address. Open the source to copy its address. On an account with a verified email domain the address is on that domain; the address on the shared domain keeps working too. A branded domain is a convenience, never the access control: the allowed senders decide what is stored on every domain.

What is stored. The message body, which you can open from the source's content list with its links kept (a very large message is stored as plain text), plus this metadata, available to metadata filters:

Sender, recipient, subject, both times and thread are on every item: a message with no readable Date header takes the time it was received, and one with no readable Message-ID, References or In-Reply-To header is a thread of its own. The rest are present when the message has them.

Attached files. Up to 100 files sent with a message are kept as attachments of its item and indexed, so a search of the source finds what is in them; a match is returned under the message, with the file name added to its title. A text file is indexed as text. A PDF or Office document is indexed through its extracted text, and through the images inside it when the source indexes them; an image, audio or video file embedded whole when the source's embedding model indexes its kind, else through its text (OCR or a transcript). A file of a type a content source does not accept as an upload is kept without being indexed. Open the message from the content list to see its Attachments: each file shows whether search can find it and, when indexing failed, why. Images displayed in the message body, files past the first 100, and attachments with no usable file name or that could not be decoded are not kept; the message lists each one under Not kept as attachments, with the reason. A message with files and no text is stored under its subject.

Attached files are screened like the message: a file whose text a prompt scan flags, or that an enforcing governance policy blocks, is kept and not indexed. The files of a message held for review are attached when review releases it. An agent reaches an attached file through retrieval; the load content attachment step does not read files that arrived by email. Indexing a file is charged to the source's account: file processing and text extraction where the file needs them, embedding, and daily storage.

Repeated messages. Each source identifies a message by its Message-ID header, and one it has received before is refused as a repeat. The entry under Refused mail says what became of the earlier one: stored, held for review, deleted or expired, or not stored yet. The first three are final, and the message is refused however often it is sent. "Not stored yet" is not: once the earlier delivery shows as failed, or an hour has passed without it being stored, sending the message again stores it. A message with no Message-ID header cannot be recognised when it is sent a second time, and each copy is stored.

Refused mail. The source's Refused mail tab lists mail that was turned away and why: a sender that is not on the list, a message that failed DMARC, a message from which no single From address can be read unambiguously, a blocked sender, spam or a virus, a message with no readable body, a message the prompt-injection scan flagged, mail over the account's hourly limit, mail that arrives while the account cannot be charged, or a message the source has received before. Each entry records who sent the message, never its subject or body. A message from an allowed sender that the prompt-injection scan flags is also kept in that scan's own record, like any other scanned content. Account owners and administrators can allow a refused sender or its whole domain from there. A subscription confirmation often comes from a different address than the newsletter itself: allow that sender, then ask for the confirmation again.

Limits. Mail to email sources counts against the same per-plan hourly quota as inbound email to agents. Mail over the limit is refused, not queued. Governance source-content screening applies to each message like any other source.

Adding a Source

Basic Setup

// Using the TypeScript SDK
const source = await seclai.sources.create({
  name: "Company Blog",
  source_type: "rss_feed",
  url: "https://example.com/blog/rss.xml",
  polling: "daily",
  retention: 90, // Keep content for 90 days
});

// Create a Content Store with a preset index mode
const contentStore = await seclai.sources.create({
  name: "Internal Docs",
  source_type: "custom_index",
  index_mode: "balanced", // or "fast_and_cheap", "slow_and_thorough", "custom"
});

URL Detection

Seclai automatically detects whether a URL is an RSS feed or a website:

// Detect source type
const detection = await seclai.sources.detect({
  url: "https://example.com",
});

console.log(detection.source_type); // "website" or "rss_feed"
console.log(detection.final_url); // Resolved URL after redirects

The detection process:

1. Fetches the URL and checks content type
2. Analyzes response to identify RSS/Atom feeds
3. Returns metadata about the source
4. Checks for existing sources at the same URL

Seeding Options

When you first add a source, you can control which content is initially indexed. For an RSS feed, scheduled polls after that take items the source has not yet taken or passed over, in feed order, up to its polling limit.

Full History

Index all available content from the source.

{
  seed: "full_history";
}

Best For:

• Small content sources
• When you need complete historical context
• Documentation sites with limited pages

Note: Large sources may incur significant embedding costs.

Latest N Items

Index only the most recent items.

{
  seed: "latest_n",
  latest_n: 10  // Last 10 items (default: 5)
}

Best For:

• Large RSS feeds with frequent updates
• When historical content isn't relevant
• Testing new sources before full import

Selected Items

Choose specific items to index from RSS feeds.

{
  seed: "selected_items",
  selected_items: ["item-guid-1", "item-guid-2"]
}

Best For:

• Cherry-picking specific episodes or articles
• Manual content curation
• Importing only relevant historical content

Polling Schedules

Control how often Seclai checks for new content:

Manually

No automatic polling. Trigger updates via API or UI.

{
  polling: "manually";
}

Use When:

• Content rarely changes
• You want manual control
• Content stores

Once

Poll immediately when added, then never again.

{
  polling: "once";
}

Use When:

• One-time content import
• Archived content sources
• Testing source configuration

Hourly

Check for new content every hour.

{
  polling: "hourly";
}

Use When:

• Breaking news feeds
• Real-time monitoring
• Frequently updated sources

Note: Higher polling frequency increases costs.

Daily

Check for new content once per day.

{
  polling: "daily";
}

Use When:

• Blog posts and articles
• Most RSS feeds
• Documentation sites
• Recommended default for most sources

Weekly

Check for new content once per week.

{
  polling: "weekly";
}

Use When:

• Infrequently updated content
• Newsletters or weekly publications
• Cost optimization for low-priority sources

Polling Actions

Control what happens when Seclai finds existing content during polling:

New Only

Only index new content items.

{
  polling_action: "new";
}

Behavior:

• Skips items that were previously indexed
• Lower costs (no reprocessing)
• Doesn't catch content updates

Default for most source types.

New and Updated

Index new content and reprocess updated items.

{
  polling_action: "new_and_updated";
}

Behavior:

• Detects content changes via ETag and Last-Modified headers
• Reprocesses updated content
• Higher accuracy for dynamic content
• Increased costs for reprocessing

Use When:

• Content frequently changes (edited posts, updated documentation)
• Accuracy is more important than cost
• You need the latest version of content

Polling Limits

Control how many new items a scheduled poll of an RSS feed takes. Other source types do not use this setting.

{
  polling_max_items: 50; // At most 50 new feed items per scheduled poll
}

Benefits:

• Prevents cost spikes from large batches
• Gradual processing of backlog
• Predictable resource usage

Pull Status & Errors

Every time Seclai polls a source it creates a pull record describing what happened. The Pulls tab on the source detail page (and the list_source_pulls MCP tool / GET /authenticated/sources/{id}/pulls endpoint) shows the most recent pulls along with a Status column.

Pull Statuses

Each pull also carries:

• items_attempted — how many URLs the pull tried to fetch.
• source_connection_content_count — how many of those produced indexed content for this source connection.
• errors_count — how many URLs failed during this pull.
• error — a short aggregated summary of up to ten failing URLs (each line is url: message), with a … marker when more errors were truncated. The per-pull errors endpoint returns a per-URL breakdown parsed from this same stored summary, so it reflects the same truncation and is not guaranteed to include every failed URL when the pull had more than ten failures.

When the most recent pull failed, the source detail page also surfaces a red banner with a link to the Pulls tab so failures are not missed.

Reading the Status Column

• Green badge — completed with errors_count = 0: All URLs in the pull succeeded.
• Yellow badge — completed with errors_count > 0: Partial success. Hover the badge for an error summary, or click the row to see per-URL details.
• Red badge — failed: No URLs succeeded. Inspect the error summary and the per-URL details to decide whether to retry, change the source URL, or remove the source.

Common Pull Errors

If a pull keeps failing for the same reason, consider:

• Editing the source URL or filters.
• Lowering the polling frequency.
• Removing the source if the underlying site no longer exists.
• Contacting support so the site can be reviewed for compatibility.

File Upload Processing Errors

When you upload a file to a Content Store, the file is accepted immediately and then processed asynchronously.

A file that fails processing is never added to the source. It is not counted by the source's content count, so the count stops short of the number of files you uploaded.

The item is still listed on the Content tab, with a failed badge next to its title — hover the badge for the reason. Use the status filter above the table to narrow the list to just the failures, or to watch items that are still indexing, scanning, or fetching. An item that has not finished indexing has no content page yet, so its title is not a link until it does; a failed item never becomes one.

The same view is available over the API — each item carries the error below plus its title and content_url:

curl -G "https://api.seclai.com/sources/$SOURCE_ID/contents" \
  -H "X-API-Key: $SECLAI_API_KEY" \
  --data-urlencode "status=failed"

Drop the filter to see every item the source has attempted, including ones still being processed — see Track Indexing Progress Per Item. From an MCP client, use the list_source_content_status tool.

Tips for successful uploads:

• Ensure PDFs contain selectable text (not just scanned images).
• Use standard document formats — exotic or proprietary formats may not be supported.
• Keep individual files under 200 MB for reliable processing.
• For image files (.png, .jpg, etc.), OCR is attempted automatically but works best with clear, high-contrast text.

Content Retention

Automatically remove old content from your knowledge base:

{
  retention_days: 90; // Delete content older than 90 days
}

Settings:

• Null (default): Keep content indefinitely
• Number: Delete content after N days

How it works:

• A background sweep runs hourly, so content is removed shortly after it passes the window rather than at a fixed time of day.
• Deletion is permanent. The extracted text and its embeddings are removed, so the item stops appearing in search and cannot be recovered. Deleted items also stop accruing storage charges.
• Age is measured from when the item was added to this connection, not from the item's own published date. A feed that back-fills older articles keeps them for the full retention window from the day they were ingested, rather than deleting them on arrival. This is also the age storage is billed on, so retention and billing always agree.
• Retention is per connection. The same source shared into two knowledge bases can keep content for different periods in each, and an item is only fully removed once no connection still holds it.
• The first sweep after you shorten a retention window can remove a large backlog at once — everything already older than the new period.
• This window belongs to the connection. For the account-wide windows — agent traces and governance evaluations — see Data Retention, which also links back to each per-resource setting.

If you set a retention period before this release, check it now. Retention periods were configurable, and shown throughout the app and API, but were not previously applied — content was kept regardless of the setting. That has been fixed, so a connection carrying a period it has never actually enforced will delete everything older than that period on its first sweep, permanently.

Your settings have not changed, so no action is needed if the period you configured is still the one you want. Review any connection where the value was set long ago, or was never a deliberate choice — a period that has been silently ignored for months may not be the one you would pick today. Set it to Indefinite to keep everything.

One upside: storage is billed per item per day for as long as an item is kept, so content that outlived its retention period was also being billed for that whole time. Enforcing the setting stops that.

Use Cases:

• GDPR compliance (automatic data deletion)
• Keep only recent content relevant
• Control storage and search costs
• Maintain fresh results

Example: Set to 30 days for news feeds, 365 days for documentation.

Content Processing Pipeline

Phase 1: Fetching

Content is fetched from the source URL.

For RSS Feeds:

• Parses feed XML
• Extracts item metadata (title, author, date, GUID)
• Fetches full article content from item URLs
• Attaches the files an item lists beside its content, when the source chose their kind

For Websites:

• Fetches the start page and the source's selected linked pages
• Extracts main content
• Removes navigation, headers/footers
• Stores HTML for processing

Phase 2: Transcription (Audio/Video)

Audio and video content is transcribed to text. A recording attached to an item, such as a second enclosure or a Media RSS file, is transcribed only when the source's embedding model cannot embed it whole or a governance policy needs its text. The formats are those listed under Supported File Formats.

Features:

• Automatic language detection
• Speaker diarization
• Timestamp preservation
• Word-level confidence scores

Costs: Based on audio duration (minutes).

Phase 3: Indexing

Text content is chunked and embedded for vector search.

Process:

1. Text Extraction: Clean and prepare text
2. Chunking: Split into optimal segments (default: 1000 characters, 200 overlap)
3. Embedding: Generate vector embeddings using configured model
4. Storage: Store in PostgreSQL with pgvector

Costs: Text extraction is billed at 1 credit per second of processing time.

Configuration:

• Chunk Size: Length of text segments (default: 1000 characters)
• Chunk Overlap: Overlap between chunks (default: 200 characters)
• Embedding Model: Model used for vector generation
• Dimensions: Vector size (256-3584 based on model)

Phase 4: Linking

Content is linked to knowledge bases and triggers agents.

Actions:

1. Links content to all knowledge bases using the source
2. Triggers agents with "content_added" or "content_updated" triggers
3. Records usage costs for embeddings and processing
4. Updates content counts and metadata

Content Filtering

Page headers, footers and navigation menus are removed from web pages before they are indexed. The content_filter setting is stored and returned, but every value indexes the same text. To see what a page keeps, use Preview page when you create a website source or an RSS feed whose items link to pages, or when you edit a website source.

Embedding Configuration

Each source can use different embedding models:

{
  embedding_model: "text-embedding-3-small",
  dimensions: 1536,
  chunk_size: 1000,
  chunk_overlap: 200
}

Available Models:

• text-embedding-3-small: 512 or 1536 dimensions (recommended)
• text-embedding-3-large: 256 or 3072 dimensions (highest quality)
• amazon-nova-2-multimodal: 256 or 1024 dimensions
• voyage-3: 512 or 1024 dimensions
• And 200+ other models from OpenAI, Cohere, Voyage, Jina, Mistral

Considerations:

• Higher dimensions: Better accuracy, higher costs
• Lower dimensions: Faster search, lower costs
• Model consistency: Use the same model across sources in a knowledge base for best results

Managing Sources

Update Source Settings

await seclai.sources.update(sourceId, {
  name: "Updated Name",
  polling: "weekly",
  retention_days: 180,
  polling_action: "new_and_updated",
});

Manual Polling

Trigger an immediate content update:

await seclai.sources.pull(sourceId, {
  seed: "latest_n",
  latest_n: 5,
});

View Source Content

Each source produces indexed content items (articles, files, documents). You can list, inspect, replace, and delete individual items via the API. See Contents for the full content management API.

const content = await seclai.sources.listContent(sourceId, {
  page: 1,
  limit: 20,
  sort: "created_at",
  order: "desc",
});

Monitor Processing Progress

const progress = await seclai.sources.getProgress(sourceId);

console.log(progress.total); // Total items
console.log(progress.completed); // Completed items
console.log(progress.phases); // Phase-by-phase breakdown

Phases:

• fetching: Content retrieval from source
• transcribing: Audio/video transcription
• indexing: Embedding generation

Delete a Source

await seclai.sources.delete(sourceId);

Note: Deleting a source removes it from all knowledge bases and deletes all associated content.

Best Practices

Choose the Right Polling Frequency

• Hourly: Only for critical, frequently updated sources
• Daily: Default for most RSS feeds and blogs
• Weekly: Documentation, newsletters, low-priority sources
• Manually: Content stores, one-time imports

Why: Higher polling frequency increases costs and processing overhead.

Use Retention Policies

• Set retention periods for time-sensitive content
• Keep 30-90 days for news feeds
• Keep 365+ days for documentation
• Null (infinite) for permanent content

Why: Reduces storage costs and keeps results relevant.

Optimize Seeding

• Start with "latest_n" (5-10 items) for testing
• Expand to "full_history" once verified
• Use "selected_items" for specific content

Why: Prevents expensive initial indexing of unwanted content.

Monitor Processing Costs

• Check embedding costs before full import
• Use smaller embedding dimensions for large sources
• Limit polling_max_items for high-volume RSS feeds

Why: Large sources can quickly accumulate costs.

Limits and Quotas

• Maximum sources per account: Plan-dependent
• Polling frequency: Hourly minimum (no real-time)
• Content size: Maximum 10MB per item
• Retention: Minimum 1 day

Check your plan limits in the account under Settings → Usage.

Source Connection Sharing

Public sources (RSS feeds, websites) are shared across accounts to optimize resources:

• Source: Shared base entity with URL and type
• Source Connection: Your personal connection with unique settings
• Benefits: Reduced redundant polling, shared storage costs, faster initial setup

Private sources (content stores) are never shared between accounts.

Agent Integration

Sources can trigger agents automatically when content is added or updated:

// Create agent with content trigger
const agent = await seclai.agents.create({
  name: "Summarize New Posts",
  triggers: [
    {
      type: "content_added",
      knowledge_base_id: kbId,
    },
  ],
  steps: [
    /* ... */
  ],
});

Trigger Types:

• content_added: Runs when new content is indexed
• content_updated: Runs when content is reprocessed
• content_added_or_updated: Runs on either event

Use Cases:

• Generate summaries of new articles
• Send notifications for important updates
• Extract structured data from content
• Categorize and tag content

Troubleshooting

Source Not Polling

Check:

• Polling schedule is not set to "manually"
• Source hasn't failed recently (check error status)
• Account has active subscription
• No maintenance mode active

Content Not Appearing

Possible Causes:

• Still processing (check progress endpoint)
• Content filtered out by seeding rules
• Duplicate content (already indexed)
• Content failed processing (check errors)

High Costs

Solutions:

• Reduce polling frequency
• Lower embedding dimensions
• Set retention policies
• Use polling_max_items limits on RSS feeds
• Switch to smaller embedding models

Website Page Missing

A website source indexes only its start page and the other pages chosen when it was created.

Solutions:

• Add a page the start page does not link to as a source of its own
• Check the source's pulls for a page the site refused to serve

API Reference

For complete API documentation, see API Reference.