Building an Agentic AI Content System to Streamline Research-Led Content Production

The problem

Research-backed content is only as good as its weakest cited fact, and checking facts by hand doesn't scale. Every piece needed a person to read the source paper, verify each stat and quote, and decide what was safe to publish. That process was slow, undocumented and never reused: the sources we'd already checked for one article gave us nothing for the next.

4 specific problems drove the build:

• 5-6 hours from idea to publishable draft, for a single piece

• No record keeping. Every fact-check started from zero, with nothing saved about what had already been verified

• No reuse of sources, extracted claims or prior verification work

• It doesn't scale. More content meant more manual hours, in a straight line

The system: 3 subgraphs, 1 parent graph

A LangGraph parent graph orchestrates 3 independent subgraphs with conditional branches, retry loops, discard paths and human checkpoints throughout. Research runs first. If a batch of claims passes the output gate, it moves to a 3-node GEO subgraph for machine-readability rewriting, then to content writing, then to either an automatic content dispatcher or a manual review step before publishing. If the quality gate fails or the system detects topic drift, the pipeline exits cleanly through a discard handler, with a full audit trail and no partial or silently-broken output.

01 Research: 24 nodes, 2 retry loops

The research subgraph carries the bulk of the pipeline's complexity across 4 stages.

Stage 1, the enrichment loop: a topic goes through light enrichment and an initial research pass, then a research gate. If it passes, it moves to extraction. If it fails, the pipeline escalates to heavier enrichment and retries once; 2 failures in a row send the topic to a discard handler rather than forcing a low-quality result through.

Stage 2, extraction: a source scorer rates each source, a claim extractor pulls out individual factual claims, a deduplicator removes repeats, and a prioritiser orders what gets verified first.

Stage 3, the verification loop: a batch verifier and aggregator check every claim. Claims that come back wrong enter a 2-pass correction sequence, covered below, before re-verification.

Stage 4, quality gates: a claim diffuser hedges any claims still marked weak, an argument checker and intent validator confirm the piece still argues what it set out to argue, and a topic gate checks for drift. On-topic content proceeds through a quality gate to GEO rewriting and storage; off-topic content is discarded with a logged reason rather than silently reshaped to fit.

02 Source scoring, weighted by tier and freshness

Not every source is trusted equally. Sources are scored into 5 tiers, academic sources at the top, then quality press, government sites, corporate sites and Wikipedia, then blogs and Medium posts at the bottom, and every score is adjusted downward the older a source gets. A stat from a 3-year-old blog post and the same stat from a current peer-reviewed paper are not treated as equivalent evidence.

03 4-layer verification, not one

Every batch of claims runs through 4 distinct verification layers, each catching a different failure mode:

• L1, regex, zero cost: extracts and normalises numbers ("1.8 billion" becomes 1,800,000,000), applies a 2% numeric tolerance, checks years, and runs fuzzy quote similarity and entity overlap. It outputs MATCH, MISMATCH, NO_ENTITY_OVERLAP or INCONCLUSIVE, and on a mismatch it hands the rewriter the exact source sentence to work from.

• L2, embeddings, low cost: chunks the source, embeds it, and builds a cosine similarity matrix between claims and chunks so the top-matching chunks feed into the next layer.

• L3, LLM-as-judge: receives the source URL, the L2 chunks, the L1 signal and the L2 similarity score, and returns a structured verdict, confidence and reasoning, batched 10 claims per call for cost efficiency.

• L4, citation accuracy: checks that every [cite: N] marker in the output actually maps to a real source and that the cited source genuinely supports the sentence attached to it. This is the layer that catches citation fabrication before publishing.

Confidence moves additively rather than compounding: an unsourced claim is capped at 0.6, conflicting sources cap it at 0.4, a causal claim takes a 0.10 penalty, a low-tier source takes up to a 0.15 penalty, low L2 similarity takes a further 0.15 penalty, and no entity overlap takes 0.10 off. A claim that passes L1 and is marked supported by L3 is boosted to 0.9.

04 Correction: rewrite before you discard

When a claim fails verification, the pipeline doesn't drop it. It rewrites, re-sources and, only as a last resort, hedges, re-verifying after every correction attempt.

• Pass 1, rewrite and fresh sources: a fact rewriter uses the exact source sentence the verifier flagged to rewrite the claim, then a fresh Exa web search finds replacement sources before re-verifying.

• Pass 2, aggressive re-source: a second, broader Exa search runs with wider queries, and any claim that still can't be resolved is explicitly marked "weak" rather than passed off as verified.

• Final safety net: a claim diffuser adds hedging language to every remaining weak claim, phrases like "some evidence suggests", "may indicate" or "early findings hint", so uncertainty is visible on the page rather than hidden behind confident prose.

Every claim that clears verification is written into a persistent verified-claims memory, so the next piece on a related topic starts from what we already know instead of from zero.

The system gets smarter with use

A separate feedback loop reads the signal every review generates, a thumbs up, a thumbs down with an optional note, or a regeneration, which counts as an implicit negative, and turns it into rules the pipeline actually applies.

• L1, structural analysis, zero API cost: measures features like sentence length, bullet ratio and citation density by regex, and directly compares liked versus disliked output, for example "liked pieces average 14 words per sentence, disliked pieces average 22".

• L2, embedding clustering: groups similar feedback together by cosine similarity, so 3 separate comments about length get recognised as 1 signal rather than 3.

• L3, an LLM learner: auto-triggers once 3 or more signals go unapplied, a reviewer writes "always" or "never" or "from now on", or 5 or more thumbs-up accumulate. It produces a measurable rule, for example "keep sentences under 18 words", and saves it as a per-user skill overlay appended to the base prompt. Overlays never replace the base skill, only add to it.

A safety valve prevents the loop from running away with itself: if the regeneration rate climbs above 50% after an overlay update, learning pauses rather than compounding a bad rule. Instructions stack across 4 layers, a global base skill, admin-set instructions, user preferences captured at onboarding, and these auto-learned rules on top. The metric we track is first_accept_rate, the share of content accepted without a single regeneration, which improves per user as their overlays accumulate.

Where AI helped, and where it did not

Where automation earned its keep

• Running deep research sweeps across sources at a volume no person checks by hand

• Numeric and quote verification against source text, at 2% tolerance, on every claim

• Rewriting and re-sourcing failed claims automatically instead of discarding them

• Turning scattered, one-off feedback into rules the system actually keeps

Where judgement still matters

• Deciding what counts as an acceptable hedge versus what should be cut entirely

• Reviewing anything the pipeline itself flags as weak before it reaches a client

• Setting the source-tier weightings and confidence thresholds in the first place

• Deciding when a rising regeneration rate means the safety valve should have tripped

The system removed the repetitive, high-volume part of fact-checking. It did not remove the need for a person to own the final call on what gets published.

Results

• 81% average accuracy end to end, across a deduplicated average of 73 claims extracted per batch

• Fewer than 1% wrong claims in the final output, down from an initial 72% strong / 27% weak / 1% wrong split before correction

• Across a sample batch, the correction loop moved strong claims from 31 to 61, cut weak claims from 29 to 19, and cut wrong claims from 21 to 1

• 40-90 minutes average runtime from research to an audit-ready draft, versus the 5-6 hours it replaced

• A flywheel where more usage produces more feedback, which produces better overlays, which raises first-accept rate and lowers regeneration and fact-checking effort per piece

What this means if you're not building it yourself

This is the engine behind ReCo's Content System offering. Clients don't operate the pipeline, log in to the admin panel or manage prompts; we do. What a client receives is a week-by-week content map, then a monthly drop of finished, on-brand content, social posts, blog articles, newsletters, ad copy, with sources fully traceable back to what the pipeline verified. The judgement calls, what to hedge, what to cut, what's ready to publish under a client's name, stay with a person on our side throughout.

Tech stack

Gemini Deep Research for research, GPT-5.4-mini with reasoning off for fast agents, GPT-5.4 with high reasoning for quality-critical agents, text-embedding-3-small for embeddings, Exa (AsyncExa) for web search, LangGraph as the orchestration framework, FastAPI running fully async as the backend, a server-rendered Jinja2 and HTMX frontend with no client-side framework, PostgreSQL on Railway with pgvector for embeddings, and invite-only Google OAuth for access.

FAQs

What makes this "agentic" rather than just using ChatGPT?

A single chat with a model produces one draft you still have to check yourself. This is a graph of specialised agents, research, extraction, verification, correction, GEO rewriting and content writing, each with a narrow job, retry logic and a defined pass or fail condition, coordinated by a parent graph rather than one long conversation. No single step is trusted to get everything right; the structure is built assuming individual steps will sometimes fail.

How do you stop the system from just hallucinating confidently?

By verifying every claim 4 separate ways before it's trusted: regex-level number and quote matching, embedding similarity against the actual source text, an LLM judge given the source and the other layers' signals, and a final check that every citation marker maps to a source that genuinely supports it. A claim that fails gets rewritten against fresh sources, not published anyway.

What happens to a claim that still can't be verified after correction?

It gets marked weak and hedged, with language like "some evidence suggests" or "early findings hint", rather than stated as settled fact. The system is built to surface uncertainty, not hide it behind confident phrasing.

Do you sell us the software, or the content it produces?

The content. This system runs inside ReCo as the engine behind our Content System offering. You get a content map, then finished, sourced, on-brand content on a monthly cadence. You don't need to run a pipeline, manage prompts or fact-check anything yourself; that's the part we've built this to handle.

How accurate is "81% average accuracy" in practice?

It means that, averaged across batches, 81% of extracted claims are confirmed accurate end to end after verification and correction. The remainder splits between claims that get hedged as weak rather than stated as fact, and under 1% that are wrong in the final published draft. We report the number including the failures it corrected, not just the successes.

Does the system replace the person doing the fact-checking?

No. It replaces the repetitive part: reading every source and checking every number by hand. A person still reviews anything the pipeline flags as weak, sets the thresholds the pipeline verifies against, and owns the decision on what actually gets published under a client's name.