AI Transparency and Explainer
Understanding how artificial intelligence is used within DevExplorer Dashboard
Overview
DevExplorer Dashboard is an AI-powered search and analysis tool designed to help international development professionals find and analyse programmes and projects from the IATI registry. The application leverages multiple Large Language Models (LLMs) through the Vercel AI SDK to provide intelligent search, document analysis, and content generation capabilities.
Key Principle: AI is used to augment human decision-making, not replace it. All AI-generated content should be reviewed and verified by users before being used for critical decisions.
AI Models Used
Primary Models
| Provider | Model | Primary Use |
|---|---|---|
| Anthropic | Claude 4 Sonnet | Query variation generation, complex analysis |
| Gemini 2.5 Flash | Summary generation, context analysis, document ingestion (see Ingestion) | |
| Groq (inference provider) | Meta Llama 3.3 70B (open source) | Toolkit selection, country filter extraction |
Model Selection Logic
The application uses an intelligent routing system to select the appropriate toolkit based on user intent:
- Document toolkit: Selected when users want to create or edit documents
- Database toolkit: Selected when users want to search programmes/activities
- Analysis toolkit: Selected when users want summaries, insights, or comparisons
This routing decision is made by the fast Meta Llama model (via Groq inference) before the main interaction begins.
Document Ingestion
DevExplorer processes all programme documents from the IATI public registry through a structured ingestion pipeline:
1. Summary Chunk Generation
All IATI programme documents are processed through Gemini 2.5 Flash to generate high-quality summary chunks. These AI-generated summaries capture the key information from documents and are optimized for semantic search and retrieval.
2. Page Chunking
A simple, deterministic chunker (no AI) splits documents into page-level chunks. This non-AI approach ensures consistent, predictable segmentation for granular document retrieval.
3. Retrieval for Analysis
Both summary chunks and page chunks are used during the retrieval process when performing Analysis. This hybrid approach combines AI-generated semantic summaries with precise page-level content for comprehensive document understanding.
Frugal AI
DevExplorer is committed to responsible and efficient AI usage. We continually focus on:
Building with Smaller Models
We prioritize using smaller, more efficient models wherever possible. For example, we use the lightweight Gemini 2.5 Flash for summary generation and Meta Llama 3.3 70B (via fast Groq inference) for routing decisions rather than always defaulting to larger, more expensive models.
Replacing LLMs Where Appropriate
We actively work to identify tasks where traditional algorithms can replace LLM usage. Examples include our deterministic page chunker for document ingestion and hybrid search algorithms that combine traditional full-text search with semantic similarity. This approach reduces costs, improves reliability, and decreases environmental impact.
Cost and Environmental Awareness
By optimizing model selection and reducing unnecessary AI inference, we minimize both operational costs and the carbon footprint of our AI operations.
AI-Powered Features
Users enter search queries in natural language. AI generates query variations to improve search coverage, extracts country filters automatically, and retrieves results using hybrid search (full-text + semantic).
Example:
Searching for "climate adaptation projects in East Africa" will:
- Generate variations like "climate change adaptation Kenya", "resilience programmes Tanzania"
- Automatically detect and filter by relevant country codes
- Return semantically relevant results even if exact keywords don't match
Search results are enhanced using text embeddings (256 dimensions). Cosine similarity measures how close results are to the query semantically. Hybrid scoring combines full-text search relevance (60% weight) with semantic similarity (40% weight) using Reciprocal Rank Fusion (RRF) with k=60.
Why: Traditional keyword search may miss relevant documents that use different terminology. Semantic reranking helps surface conceptually related content.
The system (see Ingestion):
- Retrieves relevant document chunks for each activity/programme
- Uses Google Gemini Flash to generate extractive summaries
- Focuses on user-specified topics (e.g., "lessons learned", "outcomes")
- Includes source citations linking back to original documents
- Reranks summaries by relevance to the user's query
Output format includes:
- Main summary text (under 300 words by default)
- One-line summary (15 words max)
- Source references with document IDs and page numbers
Aggregates insights across multiple activities or summaries (see Ingestion). Creates high-level synthesis for briefs and reports. Useful for understanding patterns across a collection of programmes.
Uses AI for real-time document streaming. Creates content based on search results and user specifications. Supports markdown formatting with automatic heading suggestions.
Use cases: Draft reports, briefings, analysis documents based on search findings.
Tracks user actions and current state. Provides personalised capability explanations. Suggests relevant next actions based on conversation context.
Data Flow and Privacy
- Search queries: Every search (for query expansion and filtering)
- Document excerpts: Analysis requests (for summary generation, see Ingestion)
- Activity metadata: Analysis requests (for context, see Ingestion)
- • User credentials or authentication tokens
- • Personal account information
- • Raw database contents (only relevant excerpts)
Third-Party API Providers
Data is processed by the following external AI providers:
- Anthropic - Claude models
- Google - Gemini models
- Groq - Inference provider for open source Meta Llama models
Each provider has their own privacy policy and data handling practices. We recommend reviewing:
Data Retention
- AI-generated analysis results are persisted for user access
- No training data is provided to AI model providers
Observability and Monitoring
Langfuse Integration
The application uses Langfuse for AI observability:
- • Search and analysis operations
- • Tool executions
- • Model performance metrics
- • User interaction patterns (anonymised)
- • Error rates and latency
To improve AI response quality, identify issues, and optimise costs.
Telemetry metadata includes: Feature type (search, analysis), Model provider and name, Operation identifiers, User ID (for session correlation)
User Controls
Rate Limiting
Analysis operations are rate-limited per user to:
- Ensure fair usage across all users
- Manage API costs
- Prevent system overload
Limitations and Considerations
AI Limitations
1. Hallucination risk
AI may generate plausible-sounding but incorrect information. Always verify critical facts against source documents.
2. Context window limits
Very long interactions may lose earlier context. Start fresh for unrelated queries.
3. Training data cutoff
Models have knowledge cutoffs and may not reflect the most recent developments.
4. Language bias
Models perform best in English; other languages may have reduced accuracy.
Best Practices
- Review AI outputs: Use AI-generated summaries as a starting point, not final analysis
- Check sources: Follow citation links to verify information
- Iterative refinement: Use natural language to refine searches and analyses
- Specific queries: More specific queries yield more relevant results
When AI May Not Help
- Highly specialised technical queries outside training data
- Real-time information requirements
- Tasks requiring human judgement on ethical/policy matters
- Verification of data accuracy (AI cannot access live databases beyond what's indexed)
Technical Reference
Key Files
| File | Purpose |
|---|---|
| ai/models.ts | Model configuration and providers |
| ai/prompts.ts | System prompts for different modes |
| ai/modes.tsx | User interface mode definitions |
| lib/agents/tools/ | AI tool implementations |
| lib/ailibs/ | AI utility functions (embeddings, reranking) |
| instrumentation.ts | Observability setup |
Environment Variables
The following AI-related environment variables are required:
ANTHROPIC_API_KEY # Required for Claude models GOOGLE_API_KEY # Required for Gemini models GROQ_API_KEY # Required for fast Llama inference via Groq LANGFUSE_SECRET_KEY # Required for observability LANGFUSE_PUBLIC_KEY # Required for observability LANGFUSE_BASEURL # Langfuse instance URL
Questions and Feedback
For questions about AI usage in DevExplorer: