# Track email reply rates for AI agents

Source: https://developer.nylas.com/docs/cookbook/agents/agent-track-reply-rates/

An autonomous email agent that never learns from outcomes keeps making the same mistakes. It sends a follow-up, gets ignored, and sends an almost identical one to the next prospect. The fix is a feedback signal the agent can act on: a reply rate measured per thread and per campaign, computed by matching the replies that land in the mailbox against the messages the agent sent.

This recipe wires that signal up with the Nylas Email API. You read sent messages, watch the threads they belong to, and count which ones earned an inbound reply. The result is a number the agent can optimize against, not a dashboard a human reads once a quarter. It's the difference between an agent that acts and an agent that improves.

## How can an email API track reply rates for an AI agent?

An email API tracks reply rates by giving the agent read access to both sides of every conversation: the messages it sent and the replies that come back. With Nylas, each conversation is a thread, and a thread exposes `latest_message_sent_date` and `latest_message_received_date`. When the received date is newer than the sent date, that thread got a reply. Reply rate is replied threads divided by sent threads.

That single comparison is the whole engine. A thread the agent started but nobody answered has a `latest_message_sent_date` and no later `latest_message_received_date`. One that earned a reply flips the inequality. Threads can't be filtered by metadata directly, so the agent finds a campaign's conversations by filtering its tagged messages (up to 200 per page), then reads each parent thread to compare the two dates. The same thread shape returns across Gmail, Outlook, Yahoo, and IMAP, so one counter covers every connected mailbox.

## Tag every message the agent sends

Reply rate per campaign needs a way to group sends, and the cleanest way is to tag each outbound message with metadata at send time. The send endpoint, `POST /v3/grants/{grant_id}/messages/send`, accepts a `metadata` object of up to 50 key-value pairs and a `tracking_options` object. Set `tracking_options.thread_replies` to `true` so the `thread.replied` webhook fires for the conversation. One detail trips people up: only five reserved keys (`key1` through `key5`) are indexed and filterable, so store the campaign in `key1` and the agent run in `key2` rather than inventing free-form names.

The request below sends a message and attaches metadata that ties it to a campaign. Use this on every agent send, not just tracked ones, because a message with no campaign tag is a hole in your later rate calculation. The `metadata_pair` filter on the messages endpoint reads these reserved keys back, which is how you scope a reply-rate query to one campaign.

```bash
curl --request POST \
  --url 'https://api.us.nylas.com/v3/grants/<NYLAS_GRANT_ID>/messages/send' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "to": [{ "email": "prospect@example.com" }],
    "subject": "Following up on your trial",
    "body": "Hi there, checking in to see how the trial is going.",
    "metadata": {
      "key1": "q2-trial-nudge",
      "key2": "run-3187"
    },
    "tracking_options": {
      "thread_replies": true
    }
  }'
```

Each value can hold up to 500 characters, which is plenty for a campaign slug. The tag travels with the message and surfaces on the thread, so a later `metadata_pair=key1:q2-trial-nudge` query returns exactly the conversations this campaign produced. Keep the slug stable: renaming a campaign value mid-run splits your cohort and corrupts the rate.

## How does email threading help an agent understand conversation context?

Email threading groups every message in a back-and-forth into one object, so the agent sees a conversation instead of disconnected messages. A Nylas thread carries `message_ids`, `participants`, `subject`, `earliest_message_date`, and both the latest sent and received timestamps. The agent reads one thread and knows who's involved, how long the exchange has run, and whether the last move was the agent's or the human's.

That context is what makes reply detection reliable. Without threading, the agent would match replies by parsing `Subject: Re:` prefixes and `in_reply_to` headers by hand, which breaks the moment someone edits the subject or a provider rewrites the header. Nylas does that stitching server-side and assigns a stable `thread_id`, so an agent that sent message A and later sees the thread's `latest_message_received_date` advance knows a reply arrived without reconstructing the chain. Threading also tells the agent when to stop: if a thread already has an inbound reply, sending another nudge into it would annoy a person who already answered.

## Read sent threads and count the replies

With sends tagged, the reply count comes in two steps. First call `GET /v3/grants/{grant_id}/messages?metadata_pair=key1:q2-trial-nudge` to list the campaign's sent messages, each carrying the `thread_id` of its conversation. Then read each parent thread and compare its two date fields: a thread where `latest_message_received_date` is greater than `latest_message_sent_date` earned a reply. The messages list returns up to 200 per page; the example below reads the first page, so page through `page_token` for a campaign with more than 200 sends.

The code below pulls the campaign's messages, dedupes their `thread_id` values, and reads each thread once. The agent increments a replied counter when the received date wins, then divides by the thread count to get the campaign's reply rate. Run it on a schedule, hourly or nightly, so the signal stays fresh as replies trickle in over days.

```python [countReplies-Python SDK]
from nylas import Client

nylas = Client(api_key="<NYLAS_API_KEY>")

# 1. List the campaign's sent messages by their metadata tag
messages = nylas.messages.list(
    "<NYLAS_GRANT_ID>",
    query_params={
        "metadata_pair": "key1:q2-trial-nudge",
        "limit": 200,
    },
)

# 2. Each tagged message belongs to a thread; dedupe the thread IDs
thread_ids = {message.thread_id for message in messages.data}

# 3. Read each thread once and compare its two date fields
replied = 0
for thread_id in thread_ids:
    thread = nylas.threads.find("<NYLAS_GRANT_ID>", thread_id)
    received = thread.data.latest_message_received_date or 0
    last_sent = thread.data.latest_message_sent_date or 0
    if received > last_sent:
        replied += 1

sent = len(thread_ids)
rate = (replied / sent) if sent else 0
print(f"Reply rate: {rate:.1%} ({replied}/{sent})")
```

A very low reply rate on cold outreach usually means the agent's copy or targeting is off, not that the math is wrong. Feed the number back into the agent's prompt or its send decision so a low rate throttles the campaign instead of burning more goodwill.

## React to replies in real time with webhooks

Polling threads on a schedule works, but a webhook turns reply rate into a live signal. Subscribe to the `thread.replied` trigger through `POST /v3/webhooks`, and Nylas posts to your endpoint the moment a participant answers a tracked thread. A thread counts as tracked only when the original send set `tracking_options.thread_replies` to `true`, so enable it on every agent send. The agent updates its counter on the event instead of waiting for the next poll, which matters when a reply should pause an in-flight campaign within minutes.

Set up the subscription once per application. Nylas truncates webhook payloads larger than 1 MB and appends a `.truncated` suffix to the trigger name, so treat the payload as a notification and re-fetch the thread by ID for the full picture rather than trusting every field in the event body.

```bash
curl --request POST \
  --url 'https://api.us.nylas.com/v3/webhooks' \
  --header 'Authorization: Bearer <NYLAS_API_KEY>' \
  --header 'Content-Type: application/json' \
  --data '{
    "trigger_types": ["thread.replied"],
    "webhook_url": "https://your-app.example.com/nylas/replies",
    "description": "Agent reply-rate feedback signal"
  }'
```

One caveat the [notifications reference](/docs/reference/notifications/) is explicit about: if a grant is out of service for more than 72 hours, you can't backfill missed `thread.replied` events. Keep the scheduled thread poll as a reconciliation pass so a webhook outage doesn't silently undercount replies.

## Reply-rate signal: polling vs webhooks vs a dashboard

The right approach depends on how fast the agent needs to react. Polling threads is simple and self-healing but lags reality by your poll interval. Webhooks are near real-time but need an always-on endpoint and a reconciliation fallback. A human analytics dashboard answers a different question entirely: it reports to people, not to the agent's control loop.

| Capability | Scheduled thread poll | `thread.replied` webhook | **Nylas agent feedback loop** |
| --- | --- | --- | --- |
| Latency | Poll interval (minutes to hours) | Seconds | **Mix: webhook live, poll reconciles** |
| Always-on endpoint | Not required | Required | **Required for the live half** |
| Survives 72h+ grant outage | Yes, on next poll | No backfill | **Yes, poll covers the gap** |
| Drives agent decisions | Batch | Real-time | **Real-time with a safety net** |
| Cross-provider | Gmail, Outlook, Yahoo, IMAP | Same | **Same one code path** |

Most agents want the combined column: webhooks for speed, a nightly poll so a dropped event never corrupts the rate. If you only need a number a person reads weekly, a plain poll is enough, and the [email analytics dashboard](/docs/cookbook/email/email-analytics-dashboard/) recipe covers that human-facing view.

## What's next

- [Build an autonomous email agent](/docs/cookbook/agents/autonomous-email-agent/) to wrap this signal in send caps and a kill switch
- [Email analytics dashboard](/docs/cookbook/email/email-analytics-dashboard/) for the human-facing reporting view of the same data
- [Track email opens](/docs/cookbook/use-cases/build/track-email-opens/) to add open tracking alongside reply rates
- [Getting started with Nylas](/docs/v3/getting-started/) to create a project, connector, and your first grant