Introduction: The Fatal Flaw of Traditional RAG
If you’re building an enterprise knowledge base, AI search system, or document Q&A application, you’ve definitely encountered RAG (Retrieval-Augmented Generation). Everyone’s familiar with the traditional RAG workflow: crawl web pages → parse HTML → chunk text → vectorize → retrieve → feed to LLM.
The problem lies in step two.
Imagine a Wikipedia page with a stock price table:
| Year | Price |
|---|---|
| 1990 | 12.4 |
| 1991 | 18.7 |
| 1995 | 42.3 |
Humans can instantly see “the highest price before 1995 was 18.7.” But after an HTML parser converts the table to plain text, the column alignment disappears, and the table becomes:
Year Price 1990 12.4 1991 18.7 1995 42.3Faced with this string of text, it’s hard for an LLM to accurately answer “what was the highest price before 1995?” Worse still, charts, infographics, multi-column layouts, mixed text-image PDFs—these pieces of information that are directly lost during HTML parsing are precisely the content types users most frequently ask about.
This is the motivation behind UC Berkeley’s creation of PixelRAG: Since web pages are inherently visual, why not use screenshots for retrieval?
PixelRAG Core Architecture: From Pixels to Answers
PixelRAG’s paper title is straightforward—“Web Screenshots Beat Text for Retrieval-Augmented Generation.” It comes from three top Berkeley labs: Sky Computing Lab, BAIR, and Berkeley NLP, led by Yichuan Wang, Zhifei Li, and others, with Matei Zaharia (Spark creator), Joseph Gonzalez, and Sewon Min as co-advisors.
Two Core Components
1. The Renderer (pixelshot)
pixelshot is PixelRAG’s “eyes.” It uses Playwright + Chrome DevTools Protocol (CDP) to render any web page or PDF into screenshot tiles. Each tile is a viewport-sized image of the page at screen resolution.
# Render Wikipedia Python page to screenshot tiles
pixelshot https://en.wikipedia.org/wiki/Python --output ./tilesKey advantages:
- JavaScript-rendered content—SPAs, dynamically loaded data, all visible
- Tables, charts, infographics—complete visual structure preserved
- Multi-column layouts—spatial relationships intact
- PDF documents—page-by-page rendering via poppler
2. Visual Embedding Model
Screenshot tiles are vectorized using the Qwen3-VL-Embedding-2B model. This model was LoRA fine-tuned on screenshot data (both training dataset and adapters are open-sourced), enabling screenshots to be retrieved by visual content in vector space.
When querying “What is the capital of France?”, the system doesn’t match text keywords—it finds the screenshot tile containing the answer, with the infobox, tables, and context that humans can directly read.
Workflow
Document → pixelshot renders screenshots → Qwen3-VL embedding → FAISS index
↓
Query → Qwen3-VL embed query → FAISS retrieval → return screenshot tiles → VLM reads image for answerNote: The entire pipeline has no text intermediate representation. Screenshots go in, screenshots come out, and VLMs (like Claude, GPT-4o, Qwen-VL) read answers directly from images.
Comparison with Mainstream RAG Frameworks
PixelRAG isn’t trying to replace Firecrawl or Jina Reader—it solves a different layer of the problem. Let’s clarify their relationships:
| Dimension | Firecrawl | Jina Reader | RAGFlow | PixelRAG |
|---|---|---|---|---|
| Core Function | Web crawling + structured extraction | Single URL content extraction | End-to-end RAG engine | Visual retrieval + screenshot reading |
| Data Processing | HTML → Markdown/JSON | HTML → Text | Multi-format parsing | HTML → Screenshot tiles |
| Table Handling | Partially preserved | Column alignment lost | Parser-dependent | Fully preserved (as image) |
| Chart Handling | Lost | Lost | Partial OCR | Fully preserved |
| Visual Layout | Lost | Lost | Partial | Fully preserved |
| Use Cases | Large-scale crawling | Quick content extraction | Enterprise document Q&A | Structured content retrieval |
| Open Source | Yes | Yes | Yes | Yes (Apache 2.0) |
Key Insight: Firecrawl and Jina Reader are data acquisition layer tools; PixelRAG is an innovation at the retrieval method layer. They can be complementary—use Firecrawl for large-scale web crawling, and PixelRAG for visual retrieval of structured content.
Performance Benchmarks
The experimental data in the paper is impressive:
| Benchmark | Text RAG (Best Baseline) | PixelRAG | Improvement |
|---|---|---|---|
| SimpleQA | 71.6% | 78.8% | +7.2% |
| NQ-Tables | 42.5% | 48.8% | +6.3% |
| EVQA | 29.6% | 45.1% | +15.5% |
| MMSearch | — | Significant improvement | — |
| LiveVQA | — | Significant improvement | — |
| MoNaCo (Agent benchmark) | Baseline | 3x fewer tokens | Efficiency gain |
The biggest gains are on tables and structured content—EVQA (visual QA) improved by 15.5 percentage points, precisely because traditional text RAG cannot handle chart information at all.
Installation and Quick Start Guide
Minimal Installation
pip install pixelragThis gives you pixelshot (the renderer) and the core library. Add functional modules as needed:
pip install 'pixelrag[embed]' # chunk, embed, build-index commands
pip install 'pixelrag[index]' # Full pipeline orchestration
pip install 'pixelrag[serve]' # FastAPI search server
pip install 'pixelrag[pdf]' # PDF rendering support (requires poppler)Zero-Config Experience: Hosted Wikipedia API
The fastest way to try it—the Berkeley team hosts a pre-built index of 8.28 million Wikipedia pages:
# No installation needed, query directly
curl -X POST https://api.pixelrag.ai/search \
-H "Content-Type: application/json" \
-d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'Results include matching screenshot tiles (base64-encoded images) and document metadata. You can try it directly in your browser: pixelrag.ai.
Building a Local Index
Build an index for your own documents:
1. Create configuration file pixelrag.yaml:
source:
type: local
path: ./my_docs
embed:
model: Qwen/Qwen3-VL-Embedding-2B
device: auto # Automatically selects CUDA on Linux, MPS on macOS
output: ./my_index2. Build and start the service:
# Build index (~3 minutes on Apple M-series, ~1 minute on GPU)
pixelrag index build
# Start search service
pixelrag serve --index-dir ./my_index --port 300013. Query:
curl -X POST http://localhost:30001/search \
-H "Content-Type: application/json" \
-d '{"queries": [{"text": "What is the core principle of PixelRAG?"}], "n_docs": 5}'Hands-On: Indexing a PDF
pip install 'pixelrag[index,pdf]'
# Download sample PDF (the PixelRAG paper itself)
curl -L -o paper.pdf https://raw.githubusercontent.com/StarTrail-org/PixelRAG/main/assets/pixelrag-paper.pdf
# Create configuration
cat > pixelrag.yaml << 'EOF'
source:
type: local
path: ./paper.pdf
embed:
model: Qwen/Qwen3-VL-Embedding-2B
device: auto
output: ./paper_index
EOF
# Build → Serve → Query
pixelrag index build
pixelrag serve --index-dir ./paper_index --port 30001
curl -X POST http://localhost:30001/search \
-H "Content-Type: application/json" \
-d '{"queries": [{"text": "Overview of PixelRAG pipeline diagram"}], "n_docs": 1}'Give Claude Code “Eyes”
PixelRAG also provides a Claude Code plugin pixelbrowse that lets Claude directly “see” web pages:
# Install pixelshot CLI
uv tool install pixelrag # or pipx install pixelrag
# Install Claude plugin
claude plugin marketplace add StarTrail-org/PixelRAG
claude plugin install pixelbrowse@pixelrag-pluginsThen:
claude -p "screenshot https://news.ycombinator.com and summarize the top stories"
claude -p "screenshot https://arxiv.org/abs/2404.12387 and explain the key findings"Claude screenshots the page and reads the content in the image like a human—tables, charts, and layouts all visible at a glance.
Real-World Cases: Complex Table and Chart Retrieval
Case 1: Enterprise Financial Data Tables
Suppose you have an HTML table with multi-year financial data. After traditional RAG parsing, column alignment is lost, making it hard for LLMs to answer questions like “What was the YoY revenue growth rate for Q3 2023?” that require cross-column calculations.
PixelRAG’s screenshots preserve the complete visual structure of the table. The VLM can locate the “2023 Q3” cell like a human, read the corresponding value horizontally, then compare and calculate against “2022 Q3.”
Case 2: Experimental Comparison Charts in Academic Papers
Bar charts and line graphs in papers are completely lost in text parsing. PixelRAG renders charts as screenshots, allowing VLMs to directly read trends and compare values from the images.
Case 3: E-commerce Product Comparison Pages
Multi-column product specification comparison tables often get parameters from different products mixed together by traditional parsers. PixelRAG’s screenshots preserve spatial layout, enabling VLMs to accurately distinguish specifications of different products.
Use Case Analysis
Strongly Recommended Scenarios
| Scenario | Reason |
|---|---|
| Enterprise Knowledge Base | Internal documents filled with tables, flowcharts, approval forms |
| Academic Literature Retrieval | Experimental data, charts in papers are core information |
| E-commerce Data | Product specification tables, price comparisons, user review screenshots |
| Government/Legal Documents | Tables, attachments, seals in regulations |
| Financial Report Analysis | Financial statements, data visualization charts |
Less Suitable Scenarios
| Scenario | Reason |
|---|---|
| Plain text blog posts | Text RAG is already good enough; visual RAG adds unnecessary overhead |
| Code documentation | Code blocks are more precise in text representation |
| Large-scale real-time crawling | Screenshot rendering is much slower than text extraction |
Limitations and Future Directions
Current Limitations
1. Storage Costs
The pre-built index of 8.28 million Wikipedia pages is approximately 217 GB. For enterprise applications with large document volumes, storing screenshot indexes costs far more than text indexes. However, the project reports achieving 97% storage savings through image compression while maintaining retrieval accuracy.
2. Rendering Latency
pixelshot needs to launch a headless browser to render each page, which is slower than directly parsing HTML. For scenarios requiring real-time processing of thousands of URLs, this is a bottleneck.
3. GPU Dependency
The embedding model Qwen3-VL-Embedding-2B performs best on GPUs. While it supports CPU and Apple Silicon (MPS), large-scale index building still requires GPU resources.
4. Query Costs
Image tokens are more expensive than text tokens. Although PixelRAG achieves 3x token savings through precise retrieval (because returned screenshots are more focused than multiple text chunks), the VLM inference cost per query is still higher than pure text solutions.
Future Directions
- More efficient visual compression: Further reduce storage and transmission costs
- Streaming rendering: Support incremental screenshot indexing for large-scale websites
- Multimodal fusion: Combine text and visual signals for hybrid retrieval
- Edge deployment: Optimize models to support edge device operation
Conclusion
PixelRAG proposes a bold but intuitively correct viewpoint: Web pages are meant to be seen by humans—why first “translate” them into machine text and then have machines read them? By preserving the original visual form of web pages, PixelRAG achieves retrieval accuracy that traditional text RAG cannot match in scenarios with tables, charts, and complex layouts.
It’s not trying to replace Firecrawl or Jina Reader, but rather fills the gap of “visual retrieval” in the RAG ecosystem. If your knowledge base contains大量 structured content (tables, charts, forms), PixelRAG deserves serious evaluation.
Project Links:
- GitHub: StarTrail-org/PixelRAG
- Live Demo: pixelrag.ai
- Paper: arXiv:2606.28344
- License: Apache 2.0
Frequently Asked Questions (FAQ)
Q1: What’s the relationship between PixelRAG and Firecrawl/Jina Reader? Can they be used together?
PixelRAG is an innovation in retrieval method (using screenshots instead of text); Firecrawl/Jina Reader are data acquisition tools. They can be complementary—use Firecrawl for large-scale web crawling, and PixelRAG for visual retrieval of structured content.
Q2: Does PixelRAG require a GPU?
GPU is recommended for index building (3x+ faster), but it supports Apple Silicon (MPS) and CPU modes. The query service can also run on CPU.
Q3: The 8.28 million page Wikipedia index is 217GB—what if storage costs are too high?
PixelRAG reports achieving 97% storage savings through image compression. It also supports Qdrant backend quantization configuration (like int8 quantization) to further reduce memory usage.
Q4: What document formats does PixelRAG support?
Currently supports web pages (HTML/JS rendering), PDFs (requires poppler installation), and local images. Can mix processing URLs and local files.
Q5: What benchmark is the 18% accuracy improvement based on?
On the SimpleQA benchmark, PixelRAG achieves 78.8% accuracy, 7.2 percentage points higher than the strongest text RAG baseline (71.6%). On table-intensive NQ-Tables, it improves by 6.3 percentage points, and on visual QA EVQA, by 15.5 percentage points.
Q6: How to add visual web access capability to Claude Code?
Install pixelshot (uv tool install pixelrag), then install the Claude plugin (claude plugin install pixelbrowse@pixelrag-plugins). After that, Claude can directly screenshot and read any web page.
Hope this deep dive is helpful! If you’re building a RAG system involving structured content like tables and charts, PixelRAG offers a completely new approach—letting machines “see” web pages like humans, rather than forcing web pages to adapt to machines’ text preferences.
