Query Engine¶

The EntryQueryEngine provides a fluent interface for building complex queries against regulatory feed data using method chaining. It is the most convenient way to filter entries across categories, topics, date ranges, and keywords.

Basic Usage¶

from dotenv import load_dotenv
from carver_feeds import create_query_engine
from datetime import datetime

load_dotenv()
qe = create_query_engine()

# Simple topic filter
results = qe.filter_by_topic(topic_name="SEC").to_dataframe()

# Multiple filters chained together
results = qe \
    .filter_by_category(category_name="Finance") \
    .filter_by_date(start_date=datetime(2024, 1, 1)) \
    .search_entries("capital requirements") \
    .to_dataframe()

print(f"Found {len(results)} entries")

Filter Methods¶

filter_by_category¶

Fetch all entries across every topic within a category:

results = qe.filter_by_category(category_name="Finance").to_dataframe()

This uses the optimized category endpoint to fetch only topics in the category, then entries per topic.

filter_by_topic¶

Fetch entries for a single regulatory body:

results = qe.filter_by_topic(topic_name="SEC").to_dataframe()

# Or by topic ID
results = qe.filter_by_topic(topic_id="topic-uuid").to_dataframe()

filter_by_date¶

Apply a date range filter (applied in-memory after fetching):

from datetime import datetime

# Since a start date
results = qe \
    .filter_by_topic(topic_name="RBI") \
    .filter_by_date(start_date=datetime(2024, 6, 1)) \
    .to_dataframe()

# Within a date range
results = qe \
    .filter_by_topic(topic_name="RBI") \
    .filter_by_date(
        start_date=datetime(2024, 1, 1),
        end_date=datetime(2024, 12, 31)
    ) \
    .to_dataframe()

Date filtering is in-memory

The API does not support server-side date filtering. All entries for the topic/category are fetched first, then filtered by date. Always combine with a topic or category filter to limit the data loaded.

search_entries¶

Keyword search across entry fields. Case-insensitive by default:

# Single keyword
results = qe \
    .filter_by_category(category_name="Finance") \
    .search_entries("sanctions") \
    .to_dataframe()

# Multiple keywords (OR logic by default)
results = qe \
    .filter_by_topic(topic_name="SEC") \
    .search_entries(["reporting", "disclosure", "quarterly"]) \
    .to_dataframe()

# AND logic (all keywords must match)
results = qe \
    .filter_by_topic(topic_name="SEC") \
    .search_entries(["reporting", "investment-advisers"], match_all=True) \
    .to_dataframe()

# Case-sensitive search
results = qe \
    .filter_by_topic(topic_name="SEC") \
    .search_entries("SEC", case_sensitive=True) \
    .to_dataframe()

Export Formats¶

The query engine supports multiple output formats:

qe = create_query_engine()
qe.filter_by_topic(topic_name="SEC")

# pandas DataFrame (most common)
df = qe.to_dataframe()

# Export to CSV
qe.to_csv("results.csv")

# Export to JSON
qe.to_json("results.json")

# Plain Python list of dicts
data = qe.to_dict()

Fetching Entry Content¶

Entry content (full article text) is stored in S3 and not fetched by default. Request it explicitly after filtering:

qe = create_query_engine()

# Good: filter first, then fetch content only for matches
results = qe \
    .filter_by_topic(topic_name="SEC") \
    .search_entries("capital requirements") \
    .fetch_content() \
    .to_dataframe()

# The results DataFrame will include a content_markdown column
print(results[['title', 'content_markdown']].head())

Performance

Content fetching makes additional S3 requests per entry. For large result sets this can take tens of seconds. Always filter to a smaller set before calling .fetch_content().

Approximate times: - <100 entries: 1–5 seconds - 100–500 entries: 10–30 seconds - 500+ entries: 60+ seconds

Running Multiple Queries¶

Use chain() to reset the query state and run a second query without re-initializing the engine (the engine caches topic/category data from the first query):

qe = create_query_engine()

# First query
finance_results = qe.filter_by_category(category_name="Finance").to_dataframe()

# Second query — resets entry results but reuses cached category/topic data
health_results = qe.chain().filter_by_category(category_name="Healthcare").to_dataframe()

Performance Tips¶

Always filter by category or topic first. The query engine is optimized for this path and will use the most efficient endpoint.
Filter before fetching content. Keyword searches and date filters run in-memory — apply them before calling .fetch_content() to minimize S3 requests.
Use chain() for multiple queries. Reusing the same engine instance avoids re-fetching category and topic metadata.
Avoid fetching all entries without a topic filter. Without a topic or category filter, the engine must paginate through the entire dataset, which can take several minutes.