Data & Retrieval
What the model knows, and whether it is the right thing.
The question this role arrives with: “Why does it know the wrong thing, or nothing?”
23 tools · 5 sections · 10 also owned by another role
Retrieval & Vector Stores
all 15 →- pgvectordatabasePostgreSQL
Use when Under a few million vectors, or when retrieval joins rows you already have.
Skip when Vector search has become the workload rather than a side feature.
- QdrantdatabaseApache-2.0
Use when Vector-first workloads with heavy metadata filtering at scale.
Skip when You want no new datastore to operate.
- WeaviatedatabaseBSD-3-Clause
Use when Hybrid or multi-vector search with a graph-shaped data model.
Skip when Your data is genuinely relational and joins matter more than vectors.
- ChromadatabaseApache-2.0
Use when Getting a prototype working in an afternoon.
Skip when You need durability, scaling or concurrent writes.
- Pineconedatabaseproprietary
Use when Zero operational work, and vector search is not your core competence.
Skip when Residency, cost predictability or a query language you control.
- MilvusdatabaseApache-2.0
Use when Billion-scale indexes with a cloud-native deployment.
Skip when You need a small, comprehensible system.
- Turbopufferdatabaseproprietary
Use when High-recall retrieval at volume, without operating the index yourself.
Skip when You need to tune the index directly.
- UnstructuredlibraryApache-2.0
Use when Partitioning messy documents into retrievable chunks before indexing.
Skip when Your input is already structured.
- ElasticsearchdatabaseElastic-License-2.0
Use when One index carrying BM25 and dense vectors, with ranking you can tune.
Skip when You need a permissive licence — the Elastic licence is not OSI-approved.
- VespaplatformApache-2.0
Use when Ranking-heavy retrieval where the ranking function is the product.
Skip when You want a small, low-operations component.
- Rerankersserviceproprietary
Use when A cross-encoder in the second retrieval stage, to lift precision cheaply.
Skip when You have no way to measure whether recall actually improved.
- Jina AIserviceproprietary
Use when Multimodal embeddings and a reranker behind one endpoint.
Skip when Embeddings must stay in-house.
- LangChain Text SplitterslibraryMIT
Use when Document loaders and splitters, if you are already in that ecosystem.
Skip when Structure matters — fixed-size splitting is rarely the right boundary.
- DoclinglibraryMIT
Use when Layout-aware parsing that preserves tables and heading structure.
Skip when Your documents are plain text.
- LlamaParseserviceproprietary
Use when Managed parsing for PDFs, especially the ugly ones.
Skip when Document contents cannot leave your infrastructure.
Agent Frameworks
all 13 →- LlamaIndexframeworkMIT
Use when Data-centric work where retrieval is the centre of the problem.
Skip when You need agent orchestration more than data access.
- LettaplatformApache-2.0
Use when Agents whose memory is a first-class, inspectable and editable state.
Skip when You want memory to be a detail the framework handles for you.
Workflow Orchestration
all 10 →- DagsterplatformApache-2.0
Use when Data and ML pipelines framed as assets with explicit dependencies.
Skip when You want a task queue rather than a lineage-aware orchestrator.
- PrefectplatformApache-2.0
Use when Python-native flows with a managed cloud option behind them.
Skip when You need deep lineage modelling.
- Apache AirflowplatformApache-2.0
Use when Scheduled batch data engineering, where DAGs are the shared language.
Skip when You are building a latency-sensitive product — it was not designed for one.
Evaluation & Observability
all 13 →Learning & Reference
all 10 →Shared with another role
These are part of what you own and part of what someone else owns. That overlap is where the interesting arguments happen — who decides the eval threshold, who carries the pager.
Roles overlap on purpose, so these counts do not sum to the index. All 5 specialisations are listed in /roles.