The Hidden Cost of a Good Idea - Where Quant Research Loses Time
From Research to Live Markets · Part 1
Why this series
DefconQ has always been about making difficult technical ideas easier to understand and, more importantly, useful. Much of that conversation has centred on KDB/Q. With From Research to Live Markets, I want to widen the lens to the investment decisions those systems support.
This recurring series will explore what happens between an interesting research question and a system that can act on the answer. We will look at market data, backtesting, real-time risk and the engineering choices that connect them. The aim is to give researchers and engineers a better understanding of each other’s problems, with examples they can question, reproduce and use.
Financial engineering spans pricing, risk, numerical methods and much more. Here, I am particularly interested in the point where financial reasoning meets implementation. A model’s assumptions have to survive the data pipeline, the portfolio constraints and the trading environment.
The right tools make that work easier. The right architecture makes it repeatable. Getting either wrong can consume expensive research time, produce misleading evidence or expose capital to decisions nobody intended to make.
For the first article, let’s start with a deceptively simple question: where does the time actually go?
The model finished. The research did not.
Consider a familiar-looking research task.
A researcher has an idea about short-term price behaviour. The initial calculation is straightforward. Getting the right history takes longer. Instrument identifiers need reconciling. Someone asks whether the timestamps represent exchange events or internal receipt. A suspicious interval sends the work back to the data team.
Eventually, the backtest runs. Then another team has to reproduce it, implement the feature for live data and explain why its output differs from the research version.
This is an illustrative workflow, not a claim about how every fund operates. But notice how little of it involves estimating the model.
A profiler can tell us which function consumed CPU time. It cannot tell us how many afternoons disappeared while three people established what a column meant.
Both costs belong in the research budget.
Good ideas have an economic clock
There is a reason this matters beyond developer convenience.
In The Journal of Finance, McLean and Pontiff (2016) studied 97 published stock-return predictors. They reported portfolio returns that were 26% lower out of sample and 58% lower after publication, relative to the original in-sample results. Their analysis distinguishes statistical concerns from the effects of investors learning about published predictors.
Those numbers are not a depreciation schedule for your research backlog. The study does not tell us what one extra week of internal delay costs. It does establish that historical predictability cannot be treated as an indefinitely available asset.
There is also a separate clock inside the trading strategy. Gârleanu and Pedersen (2013), also in The Journal of Finance, show how transaction costs and the persistence of predictive signals jointly affect optimal trading. In their framework, slower-decaying predictors receive more weight in the portfolio the investor aims towards.
Project lead time and signal holding period are different problems. Together, they explain why an investment idea needs an engineering plan early: how long will it take to evaluate, and what will implementation require if it survives?
Data access is only the first hurdle
Having permission to query a dataset does not mean it is ready for research.
Before trusting a result, we need to know what the observations represent, how identifiers behave through time and which information was available at the decision date. Corrections, missing periods and changes in methodology all need a policy.
This is an old problem. Croushore and Stark (2001), writing in the Journal of Econometrics, describe a dataset of historical macroeconomic vintages and demonstrate how revisions can affect forecasts. Their work makes a distinction that remains essential: today’s historical series can differ from the series available to someone making a decision in the past.
For a research team, the practical consequence is repeated work. If vintage handling exists only inside individual notebooks, every new project has to rediscover the rules. Two researchers can produce different results while believing they used the same data.
A useful data interface should therefore carry more than values. It should make the relevant time conventions and version history visible, with documented treatment of corrections and missing observations.
That does not remove the need to investigate a new dataset. It prevents the fifth investigation from starting where the first one did.
Faster experimentation needs a better memory
Once the data are usable, another temptation appears: run more experiments.
That can be valuable. It also makes it easier to lose track of how much searching preceded the attractive result.
In The Review of Financial Studies, Harvey, Liu and Zhu (2016) examine the multiple-testing problem in the cross-section of expected returns. They argue that, given the extensive search for factors, conventional significance thresholds are inadequate. Their framework calls for a new factor to clear a t-statistic above 3.0.
That is a result in a specific research setting, rather than a universal acceptance rule for every strategy. The engineering implication is nevertheless useful: an experiment platform should preserve the history of the search, including unsuccessful variants.
Record which data snapshot, feature definition and sample produced each result. Keep the cost assumptions and the validation procedure alongside them. A promising chart detached from that history is difficult to evaluate and surprisingly easy to rediscover as a “new” idea.
The objective is to make a well-supported decision sooner. Sometimes that decision should be to stop working on the idea. Cheap, credible rejection is a perfectly respectable research outcome.
The handoff can become a second research project
A particularly expensive delay appears when production has to reinterpret the research.
The notebook calculates a feature one way. The streaming system calculates it another. Historical data arrive in complete batches, while live observations can be late, duplicated or corrected. Defaults that seemed harmless during exploration now influence trading decisions.
The resulting reconciliation is necessary. Reconstructing the intended behaviour from scratch is avoidable.
Sculley and colleagues (2015), in their NeurIPS paper Hidden Technical Debt in Machine Learning Systems, describe how data dependencies, feedback loops and interactions between components create maintenance costs beyond the model code itself. This is a systems paper, not an estimate of financial firms’ deployment costs. Its relevance is the location of the debt: much of it lives between components.
My preference is to involve production engineering while the research assumptions are still taking shape. Agree on the feature’s meaning, its timing and its behaviour when inputs are incomplete. Give both implementations the same recorded events and compare their outputs.
Research still needs room to explore. A team should be able to discard an unsuccessful experiment without first building a production service. But a promising experiment should carry enough of its assumptions with it that the next team can understand what it is being asked to build.
A faster query can leave the real bottleneck untouched
Suppose a research cycle takes five working days: one day of computation and four days of preparation, waiting and reconciliation. Make the computation ten times faster and the cycle still takes 4.1 days.
That is an 18% reduction in elapsed time, despite a tenfold computational improvement. It is an illustrative calculation, but it captures why local benchmarks can disappoint at the workflow level.
Queues deserve particular attention. Little’s 1961 paper in Operations Research establishes the relationship between the average number of items in a queuing system, its flow rate and average time in the system, under the stated conditions.
For a stable research workflow, an illustrative average of 12 open questions and two completed decisions per week implies six weeks in the system. Use consistent boundaries and include rejected ideas among completed decisions. The relationship alone does not identify the cause of the delay.
It does give us a useful question for planning: if we start more projects without increasing the rate at which we resolve them, what should we expect to happen to lead time?
The cost can reach beyond the research budget
At the extreme, weak implementation controls become a capital problem.
The SEC’s findings on Knight Capital describe an automated routing failure on 1 August 2012. While processing 212 customer orders, the defective system generated millions of orders and more than four million executions in approximately 45 minutes. Knight accumulated unintended positions and ultimately lost more than $460 million.
The findings included inadequate deployment and testing controls. The SEC also described 97 automated messages before the market opened that identified an error but were not designed or treated as actionable system alerts.
Knight is an extreme operational failure, not a typical consequence of a slow research process. Its relevance here is that code, deployment, monitoring and risk controls have to work together.
A platform that accelerates deployment while leaving those responsibilities unclear can make an expensive mistake happen sooner. Necessary validation should be easier to perform and harder to bypass. Removing it is a different proposition entirely.
Where the tools fit
If you have followed DefconQ for a while, you will know why I keep returning to q.
Its array-oriented approach and time-series operations make it a natural place to express many financial-data problems. An as-of join, for example, gives a researcher a direct way to ask which eligible observation preceded a decision. The data model still has to establish what “eligible” means.
The architectural attraction, for me, is reducing the distance between expressing an analytical idea and running it against substantial historical or live data. Every unnecessary export, conversion and reimplementation introduces another place to wait or disagree.
That is also why I am following PeachQ, L and Korze. Their different approaches around q and the k family create useful questions about runtimes, storage and integration. Compatibility, maturity and operational suitability need checking against the actual workload.
The interesting test is whether a tool helps a team carry a clear, inspectable calculation through the research process with fewer opportunities to change its meaning. A faster implementation has much more value when the surrounding architecture lets the team use it.
What I would measure first
Before buying another platform or rewriting the stack, I would follow a few research questions from proposal to decision. Include the ones rejected along the way.
For each, record time spent obtaining usable data, running analysis, validating results and preparing implementation. Separate hands-on work from time waiting for a dependency. Record when work returns to an earlier stage and why.
Then ask a few more specific questions:
- How long does it take to answer a new question with an existing dataset?
- How often does another person reproduce the result without help from its author?
- How much reconciliation follows a handoff to production?
- Which recurring data or infrastructure problems force teams to repeat work?
- Can an unsuccessful idea be closed with enough evidence that someone else does not unknowingly repeat it?
These are proposed operating measures, not industry benchmarks. Their purpose is to reveal where your own process loses time. A team struggling with query latency needs a different intervention from one waiting weeks for an agreed identifier mapping.
The same applies to architecture. A shared data service is useful when it removes repeated integration work. It becomes another queue if every new question requires a bespoke ticket. Common libraries help when their behaviour is explicit and versioned. An undocumented dependency shared by everyone simply gives the confusion a larger audience.
I would start with one recurring source of rework, establish its owner and fix it properly. The value should appear in the next researcher’s workflow, not just in a platform diagram.
The question behind the benchmark
Quantitative finance rightly rewards good ideas. We should pay closer attention to the systems that determine how much effort it takes to evaluate them.
The same dataset should not need rediscovering for every project. A promising result should reach the next team with its assumptions intact. An expensive simulation should be worth running before we spend time making it faster.
That is the conversation I want From Research to Live Markets to develop.
In the next article, we will examine a specific source of misplaced confidence: a backtest that knows more about the past than anyone could have known at the time.
For now, I would be interested in your experience: where does your research process lose the most time, and what architectural change actually helped? An anonymised example is more useful than another perfect benchmark.
Research cited
- McLean, R. D., & Pontiff, J. (2016). Does Academic Research Destroy Stock Return Predictability? The Journal of Finance, 71(1), 5–32.
- Gârleanu, N., & Pedersen, L. H. (2013). Dynamic Trading with Predictable Returns and Transaction Costs. The Journal of Finance, 68(6), 2309–2340.
- Croushore, D., & Stark, T. (2001). A Real-Time Data Set for Macroeconomists. Journal of Econometrics, 105(1), 111–130.
- Harvey, C. R., Liu, Y., & Zhu, H. (2016). … and the Cross-Section of Expected Returns. The Review of Financial Studies, 29(1), 5–68.
- Sculley, D., et al. (2015). Hidden Technical Debt in Machine Learning Systems. Advances in Neural Information Processing Systems, 28. Conference paper.
- Little, J. D. C. (1961). A Proof for the Queuing Formula: L = λW. Operations Research, 9(3), 383–387.
Operational case: U.S. Securities and Exchange Commission (2013). In the Matter of Knight Capital Americas LLC, Release No. 70694.