Streaming Smarter: Real-Time Market Data with Kola, Python, and Databento
A few weeks back, I shared a blog post on how to stream real-time data from Yahoo Finance into KDB using Python and the yfinance API. Following that, my former colleague Nathan Swann, instead of using qpython
, why not try the kola
API?
This is a perfect example of the power of the KDB/Q community, sharing knowledge, tools, and better ways to do things. So, you asked, and we delivered. I gave kola
a try, refactored the code to use it instead of qpython
, and was impressed with the results.
But I didn’t stop there. One of the limitations of using Yahoo Finance is the restriction on request volume, a real challenge for real-time data feeds. So I explored other options and found a great alternative: Databento, a modern market data provider offering $125 in free credits for new users.
In this updated blog post, I walk through how to:
- Replace
qpython
withkola
- Swap out Yahoo Finance for Databento
- Build a flexible, scalable pipeline from real-time data sources into KDB
It’s a great demonstration of just how powerful and adaptable KDB/Q is, easily integrating with other programming languages and data providers.
Meet Kola: The Blazing-Fast Python Interface for KDB/Q You Didn’t Know You Needed
Kola is a free and open-source Python interface for KDB/Q designed with performance and modern development in mind. Built on top of Rust or speed and using Polars for ultra-fast DataFrame operations, Kola delivers a powerful and efficient toolkit for working with KDB data in Python.
The project is the brainchild of Jo Shinonome, a well-known contributor to the KDB/Q ecosystem and the developer behind the popular KDB/Q plugin for Visual Studio Code. Kola combines his deep understanding of both KDB/Q and modern Python tooling.
If you’re looking for a high-performance alternative to qpython
or simply want a more efficient and modern way to work with KDB/Q in Python, Kola is worth checking out. You can explore the source code on Jo’s GitHub and read more about Kola’s performance and internals in this excellent Timestored blog post.
Whether you’re building research tools, ingesting real-time data, or just tinkering with KDB from Python, Kola is a fantastic addition to your toolbox.
How do you install Kola?
Just like any standard Python library, getting started with Kola is incredibly simple. All it takes is a quick pip install
and you're ready to roll. Since Kola is built on top of Polars, the Polars library will be automatically installed alongside it, no extra steps needed.
pip install kola
Lean, Mean, and Lightning Fast: Upgrading Your KDB/Q Pipeline with Kola and Polars
We’ve built on the foundation laid in the previous tutorial, this time, with some powerful upgrades. With just a few strategic changes, we replaced legacy components and supercharged our pipeline using modern tools. Specifically, we phased out qpython
in favour of kola
, and swapped out pandas
for polars
.
These updates require minimal code changes but deliver significant improvements in performance and memory efficiency. If you've been following my blog, you’ll find the transition straightforward. Here's a breakdown of the core modifications: