Benchmarking

As we prepare Spykio for the V1 release, we're conducting extensive benchmarking to assess and optimize its performance in long-context scenarios. This page presents our current benchmarking efforts and will be continuously updated with new results.

We're exploring how Spykio performs on several long-context LLM benchmarks, including:

• LongBench v2 - Designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning across real-world multitasks.
• OpenAI MRCR - A benchmark for evaluating the multi-round contextual retrieval capabilities of models.

We will publish comprehensive results here, comparing Spykio against standard vector database solutions such as Weaviate and Pinecone, as well as general long-context models like Gemini, OpenAI GPT-4o, Llama 4 Scout, and others.

We've conducted initial tests using a real-world dataset to evaluate Spykio's retrieval capabilities:

Ongoing Improvements

We are continuously improving the Spykio engine to enhance its performance across various retrieval scenarios. Our team is working on:

• Optimizing the knowledge graph generation for better contextual understanding
• Improving the query processing pipeline
• Enhancing document chunking strategies for more accurate retrieval
• Expanding our test suite to cover more diverse document types and query patterns