Skip to content
Skip to main content

Extract action items with your own LLM

Last updated:

Notetaker already gives you an action_items list out of the box, but it’s a flat array of strings like ["Test the Notetaker implementation."]. That reads fine in an email recap, yet it falls apart the moment you try to push tasks into Jira, Linear, or a CRM, because there’s no assignee, no due date, and nothing typed. When you need structured records instead of a bulleted list, the move is to fetch the raw transcript and run your own LLM over it.

This recipe shows how to pull the speaker-labelled transcript, prompt an LLM to return typed action items as JSON, and validate the result before it hits a task tracker. It builds on the media fetch in Get a meeting transcript and recording and follows the same extraction pattern as Extract structured data from email.

When should you skip the built-in action items?

Section titled “When should you skip the built-in action items?”

Use the built-in action_items when a plain checklist is enough, like a recap email. Reach for your own LLM when downstream systems need typed fields. A task tracker row usually wants at least 3 fields the built-in list never provides: who owns it, what the task is, and when it’s due.

The built-in summary and action_items are media objects Nylas generates automatically, each delivered as a small JSON file under 1 KB. They’re great for human reading and cost you nothing extra. The tradeoff is shape: a string array can’t carry an assignee or a date, so a task tracker import either drops those fields or forces a human to fill them in by hand. Running your own model over the transcript lets you define the exact schema your tracker expects, including an owner pulled from the speaker labels and a due date parsed from phrases like “by next Friday.” For the format of the built-in outputs, see Handling Notetaker media files.

How do you fetch the raw transcript?

Section titled “How do you fetch the raw transcript?”

Call the media endpoint, then download the transcript object’s url. The transcript is a JSON file, usually type: "speaker_labelled", with a transcript array where each entry carries a speaker, text, and start/end times in milliseconds. That speaker attribution is what lets your LLM assign owners.

Make a GET request to /v3/grants/{grant_id}/notetakers/{notetaker_id}/media, grab data.transcript.url, and fetch it. The signed link is valid for 3,600 seconds, so download the content right away rather than storing the URL. The media fetch itself is covered in depth in Get a meeting transcript and recording; here you only need the transcript object. The snippet below pulls the transcript and flattens it into a single labelled string the model can read in one pass.

import os, requests


NYLAS = "https://api.us.nylas.com"
HEADERS = {"Authorization": f"Bearer {os.environ['NYLAS_API_KEY']}"}


def get_transcript_text(grant_id, notetaker_id):
    media = requests.get(
        f"{NYLAS}/v3/grants/{grant_id}/notetakers/{notetaker_id}/media",
        headers=HEADERS).json()["data"]
    doc = requests.get(media["transcript"]["url"]).json()  # link valid 3,600 seconds
    if doc["type"] == "raw":
        return doc["transcript"]                            # plain string, no speakers
    return "\n".join(f'{seg["speaker"]}: {seg["text"]}' for seg in doc["transcript"])

How do you prompt an LLM for typed action items?

Section titled “How do you prompt an LLM for typed action items?”

Define a strict JSON schema, send the transcript with temperature set to 0 for repeatable output, and ask the model to return only items it can ground in the text. The schema below asks for 4 fields per item: an assignee, a task, a due_date, and a decision flag.

The prompt does the real work here. Give the model the exact fields you want, tell it to use null when a value isn’t stated rather than inventing one, and constrain it to JSON so you can parse the response directly. Setting temperature to 0 keeps the same transcript producing the same items across runs, which matters when the output feeds an automated pipeline. The example uses an OpenAI-style client, but any chat model with JSON output works the same way. This is the identical pattern used to extract structured data from email, just pointed at a transcript instead of a message body.

import json
from openai import OpenAI


client = OpenAI()


SCHEMA_HINT = """Return JSON: {"action_items": [
  {"assignee": string|null, "task": string, "due_date": string|null, "decision": bool}
]}. Use null when a value is not stated. Do not invent owners or dates."""


def extract_items(transcript_text):
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        response_format={"type": "json_object"},
        messages=[
            {"role": "system", "content": "You extract action items from meeting transcripts."},
            {"role": "user", "content": SCHEMA_HINT + "\n\nTranscript:\n" + transcript_text},
        ],
    )
    return json.loads(resp.choices[0].message.content)["action_items"]

How do you validate before sending to a task tracker?

Section titled “How do you validate before sending to a task tracker?”

Never trust raw model output. Check each item has a non-empty task, coerce due_date to a real date or null, and drop anything malformed before it reaches your tracker. One bad row can fail a whole batch import, so filter first.

Validation is the difference between a pipeline that runs unattended and one that pages you at 2 AM. The model returns JSON, but it can still hand back an empty task, a due_date like “soon” that isn’t a date, or an assignee who wasn’t in the meeting. Run every item through a small guard that enforces your tracker’s 2 required fields, a task and a due date, normalizes dates to ISO 8601, and discards the rest. The check below keeps only items with a usable task and a parseable date.

from datetime import date


def clean(items):
    valid = []
    for it in items:
        task = (it.get("task") or "").strip()
        if not task:
            continue                                    # skip empty tasks
        due = it.get("due_date")
        try:
            due = date.fromisoformat(due).isoformat() if due else None
        except (ValueError, TypeError):
            due = None                                  # unparseable date becomes null
        valid.append({"assignee": it.get("assignee"), "task": task, "due_date": due})
    return valid


# clean(items) returns the validated list; push it to Jira, Linear, or your CRM, one row per action item

What’s next

Section titled “What’s next”