Ekonoomix - Investment Dashboard

Finance dashboard with stock, ETF, and crypto data, featuring automated ETL pipelines via Prefect, MLOps agentic LLM workflows, and a multi-database architecture with PostgreSQL/TimescaleDB, MongoDB, and Qdrant.

Knowledge EngineeringPythonPrefectPostgreSQLTimescaleDBMongoDBQdrantMLOpsLLMs
View on GitHub

Introduction & Problem Statement

Individual retail investors frequently face significant information asymmetry: financial knowledge remains fragmented across volatile markets, technical metrics are complex to interpret, and online financial content often lacks verified rigor.

Ekonoomix was designed as an intelligent decision-support system and personal investment dashboard that consolidates heterogeneous financial streams into actionable knowledge. The platform couples real-time and historical market data, company metadata, and news sentiment analysis with an autonomous conversational agent capable of natural-language reasoning and live dashboard manipulation.

Ekonoomix Container Architecture

Architecture & Storage Tier

Ekonoomix adopts a distributed microservices architecture orchestrated via Docker Compose, adhering to a hybrid neuro-symbolic design: factual financial calculations are strictly delegated to deterministic database engines, while natural language understanding is handled by Large Language Models.

Multi-Database Storage Tier

To satisfy disparate operational workloads without performance bottlenecks, data is partitioned across specialized database engines:

  • Relational Operations (PostgreSQL): Handles core platform domain entities—user profiles, authentication (BCrypt hashing), portfolio definitions, and buy/sell transaction ledgers. All CRUD interactions are strictly encapsulated behind PL/pgSQL stored procedures and functions, preventing the LLM agent from issuing arbitrary or destructive SQL queries.
  • Time-Series Analytics (TimescaleDB): Extends PostgreSQL with indexed hypertables for high-frequency daily OHLCV market metrics (open, high, low, close, volume, splits, dividends) and system audit telemetry (timescale_logs), enabling sub-second analytical queries across millions of rows.
  • Raw Data Lakehouse (MongoDB): Serves as an immutable document store capturing raw JSON/XML payloads from external APIs and web-scraping jobs before downstream normalization, ensuring complete data auditability.
  • Vector Search (Qdrant): High-performance vector database indexing financial news articles and ticker profile descriptions. Uses INT8 Scalar Quantization for a 4x reduction in RAM usage while retaining high search precision.

Data Engineering & ETL Pipelines (Prefect)

Data ingestion is orchestrated through Prefect utilizing a resilient manager-worker pattern with concurrent task execution:

  1. Ticker Metadata Pipeline: Ingests S&P 500 and Nasdaq asset directories, maps industry, sector, and currency attributes, and updates PostgreSQL (tbl_instrument), MongoDB, and Qdrant.
  2. Historical Price Pipeline: Daily automated flows pull market metrics via yfinance, deduplicate against existing timestamps, and stream data into TimescaleDB hypertables.
  3. Financial News & Sentiment Pipeline: Fetches RSS feeds from Investing.com and yfinance. To bypass anti-scraping protections, tasks use curl_cffi for browser TLS fingerprint impersonation and BeautifulSoup for semantic article extraction.
  4. Hierarchical Text Chunking: Articles are split using RecursiveCharacterTextSplitter with dynamic overlap, prepending the headline to each fragment to preserve crucial contextual boundaries.

Vector Embeddings & Hybrid Search

Text representations are generated through a dedicated FastAPI embedding microservice:

  • Model: BAAI/bge-m3, an open-source multilingual model supporting 1024-dimensional embeddings with native support for European Portuguese.
  • CUDA Acceleration: Containerized with nvidia/cuda runtime, achieving 0.048s average latency per embedding on GPU (versus 0.386s on CPU).
  • Hybrid Search (Dense + Sparse): Combines dense semantic vectors with sparse lexical term weights, merged in Qdrant through Reciprocal Rank Fusion (RRF). This guarantees that exact keyword queries (e.g., specific ticker symbols or company names) receive top ranking alongside semantic topical matches.

Intelligent Agent & Function Calling

The conversational assistant operates through an iterative Agentic Loop structured around two distinct operational phases:

  • Phase 1 (Intent Routing): The user prompt is classified by an initial router LLM to isolate the query scope: structured market queries (SQL), semantic news retrieval (RAG), or portfolio management.
  • Phase 2 (Tool Execution): The agent accesses a restricted, validated toolset (AgentTools) via OpenAI-compatible function calling schemas:
    • Data Retrieval: Deterministic procedures such as get_latest_stock_price_sql, simulate_historical_invest_sql, and search_financial_news_qdrant.
    • Live UI Mutation: The agent can directly alter the Streamlit dashboard state—plotting technical indicators (add_technical_indicator_ui), displaying volatility overlays, or generating multi-ticker comparison charts.
    • Self-Healing Parser: When malformed JSON is returned, the agent invokes _try_repair_json or feeds the error trace back into the model context, prompting immediate autonomous self-correction within a 5-step loop limit.

LLM Fine-Tuning & Evaluation

To explore offline inference viability, the team conducted local parameter-efficient fine-tuning on Mistral 7B using QLoRA (4-bit quantization) and Unsloth on a native WSL environment (NVIDIA GPU with CUDA and Triton kernels), subsequently converting the fused weights to GGUF format for deployment under Ollama.

The model was rigorously benchmarked across 20 real-world financial prompts in both European Portuguese and English against established reference answers and rubric criteria (Financial Accuracy, Financial Literacy, Risk Prudence, and Practical Applicability):

  • Base Mistral 7B: Achieved an impressive global average of 4.11/5.00 in English and 3.63/5.00 in Portuguese.
  • Fine-Tuned Checkpoint: Experienced catastrophic forgetting on instruction-following when trained on raw financial domain text without structured instruction pairs, dropping to 1.56/5.00 in Portuguese.
  • Key Takeaway: Validated that the neuro-symbolic hybrid approach—combining high-capacity base foundation models with deterministic SQL tools and hybrid RAG—is drastically superior and more reliable than unstructured domain fine-tuning for mission-critical financial applications.