Backtesting in python continue build or buy available software

External question — Quantitative Finance Stack Exchange Author: MMsmithH Original post: https://quant.stackexchange.com/questions/80756 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I started building a backtesting application in Python to backtest and optimize trading strategies, but I've paused to assess whether to continue development or purchase software to speed up further progress. My considerations include processing speed & data, cost & time to implement, and the time to use my software for trading. I would like advice and opinions on whether it makes sense to migrate my testing to a platform or to continue building, buying data, and scaling speed by setting up infrastructure in the cloud. Processing speed: My initial speed tests suggest that software (e.g., TradeStation) processes much faster, as expected. I'm using threading/parallel processing and algorithms to increase speed, but I think I may need to configure cloud computing infrastructure to improve speed further (if needed). Data sources: I want to backtest options strategies, but getting this data isn't as straightforward . For example, TradeStation doesn’t seem to have historical options data for expired contracts , and some data sources aren’t "real" but calculated using models (e.g., Black-Scholes). Some brokers offer options data, some offer backtesting, but few offer both. Exporting results of backtests: I don’t want this to limit scaling backtesting. Local Python code is only limited by my GPUs, but external software may not allow data downloads or may impose restrictions. Other considerations Loss of customizability — Will I lose the ability to customize aspects that might affect results, like changing optimization algorithms (genetic, vector, grid, etc.)? Other advantages — There may be additional benefits like security, updates, and more. Cost/Time — To obtain reliable options data, I might need to subscribe to a service. Exporting options data could be more expensive than running tests on their platform. Possibilities QuantConnect offers historical options data for expired contracts, an API, and Python compatibility. It advertises "unlimited backtests," but users can run into log limits related to saving results. TradingView is customizable but uses Pine Script, not Python. It has an API but no options data—though importing options data from other platforms might be possible. Continuing development will require setting up the right dev ops environment and cloud computing to increase speed. I will also need to buy and integrate reliable options data. I summarized what I believe the pros and cons are of the different approaches, as well a what I think about each approach in the table below. Options Pro Cons Between lines QuantConnect Speed, equity, historical options data, API, uses Python "Unlimited backtests" may be limited by log limits Fastest but potentially the most expensive to scale. TradingView Equity data, possible to import options data, unlimited backtesting Uses Pine Script (not Python), no historical options data Workarounds needed to import/export options data and backtest results Continue development Unlimited backtesting, more economic Must set up cloud infrastructure and software Slowest, with potential unknown roadblocks beyond other solutions Questions Should I prioritize building backtesting infrastructure in the cloud or migrate to an existing platform like QuantConnect or TradingView or another provider? How significant are the log limits or restrictions on downloading data from platforms like QuantConnect or TradingView for large-scale backtesting? Thank you in advance for your feedback.

External answer — Quantitative Finance Stack Exchange Author: Mikko Ohtamaa Original post: https://quant.stackexchange.com/a/80775 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I, with my team, have written Python-based algorithmic trading framework and infrastructure at TradingStrategy.ai for decentralised finance protocols , so let me share my thoughts. Primarily: For common use cases, like options trading on established markets, you should find existing Python libraries available, even if not full backtesting frameworks You can customise these for your needs Unless you are an expert Python developer, developing your framework is a good learning experience, but it is a large time sink Time to market If you can get started with TradingView, TradeStation, MetaTrader or other similar software you should go for it Better to have zero to one moment and then later figure out how to do it properly On the performance of Python code: For data crunching, Python uses native libraries like NumPy and Polars and speed is not an issue, unless you do high-frequency trading For example, we rent a heavy Google Cloud server for backtesting, running Datalore for Python notebooks, with 64 CPUs, where we pay by the hour and we have a script to automatically turn the server off Most backtesting is in optimization/grid search and these tasks are trivial to parallerise Our approach is hybrid: we build our indicator data using vectorised calculations with Pandas, but the trade decision loop is still event driven as this way we can take the unmodified backtest code and put it to the live trade execution On data sources for options trading: Options trading is more niche, so datasets are naturally more scarce Someone collects this data, but you might need to pay for it Usually more sophisticated datasets and larger downloads are behind "professional" or "enterprise plan" That's why I love cryptocurrency markets, as all data is public and free I am not an expert on options trading on traditional finance, so I cannot give you any pointers on data sources for your specific needs On customizability: We are using our own datasets, trade order types, which are not compatible with any existing services like TradingView Because DeFi data is somewhat special, we had to write our own backtester We can still utilise existing Python libraries like pandas_ta and talib Even if you get backtester with a ready made tool, you need to think the live trade execution (unless you plan to do it by hand) On devops For live trading, you are going to need some kind of devops solutions any case Most issues are related failing third party APIs and how to handle and recover these situations This may mean servers, alerts, etc. and people - people are the most important one here We are "logging" from the live execution to Discord channels - one per strategy, but Telegram group chats and bots are also popular for alerts and status updates Servers are cheap, it is mostly that you might need a sysadmin knowledge with any Python solution However this can come afer "1.0" when you have something in place, even if it is put together with a duct tape

External answer — Quantitative Finance Stack Exchange Author: Con Fluentsy Original post: https://quant.stackexchange.com/a/80999 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. R language has extensive finance libraries with backtesting ,performance measurement and attribution and things that take months to build in python that are ready to go and run fast in seconds, I know this is not what people want to hear but R is the language of professional quants. Python is a swiss army knife a jack of all trades for creating things from scratch, but R is for specific professions because it is for vector mathematics and functional programming it is the mathematical and statistical language of science and finance. There is a lot in python, but not for specific things, ready built, like extreme value theory, and all the methods of active trading from Grinold and Kahn, and Carl Bacon's performance measurement book,Quantmod in R downloads current and future options strikes and expiries data free, as well as FRED economic data, and share prices.You can buy historical option data, from a number of sources, but ORATS has an extensive options backtester and historical prices going back to 2007, it is a subscripton service POA depending on requirements,it is not related to R in anyway. I have been trading options since 2007 and tried dozens of software as a service options tools and they are not good or realistic, they find marvelous anomalies which are not repeatable trades and they are not in real time. The main workhorses in R are Rmetrics libraries PerformanceAnalytics libraries, Quantstrat, and blotter, and Rquantlib, and they are very comprehensive and there is at least a dozen more I have not tried. They have from traditional markowitz portfolios to Kelly criterion, sharp ratios, American and European options, unfortunately the Heston Nandi volatility pricing model for options has been discontinued, but I rewrote the code from git hub, I find tree models better, so I do not use it. I have a parallel library of python open source code and it is not very robust or functional or as useful for real trading. In R there is built in libraries for event studies and metrics, Risk management , plus a vast array of finance related things ready to go, no programming required just code commands. There is not a lot of secrets to trading options, as arbitrage is limited to few types the main type is volatility arbitrage also known as statistical arbitrage, it is persistent and consistent and provides steady returns. Also if you use tastytrade as a broker they released a good free options trading backtester with their platform to anyone who signs up, you can use their data without even trading or depositing any money, they are simply a very good but not the cheapest option broker, and you can trades shares, and futures with them also, and I believe crypto for what thats worth. This is not what people want to hear in python universe but it saves reinventing the wheel, if you can program python, then R is a bit more like C syntax, but still clean and simple, but it has some unique tricks for vectors and arrays and graphics.

External answer — Quantitative Finance Stack Exchange Author: bakunet Original post: https://quant.stackexchange.com/a/81198 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I wrote my first backtester in Python. After this I moved to C# that I knew already. About your question, remember that using someone's frameworks you are limiting yourself to someone's solution. If you want to scale your backtester or experiment with different solutions, I suggest to have your own.

External answer — Quantitative Finance Stack Exchange Author: RndmSymbl Original post: https://quant.stackexchange.com/a/81376 License: CC BY-SA 4.0 — https://creativecommons.org/licenses/by-sa/4.0/ Adaptation: HTML converted to plain text; contact email addresses removed. I have had a similar challenge a few times and even been starting to write a backtester in R a few years ago. Abandoned the project because, as mentioned earlier, its only really worthwhile if building a trading platform or if you need to be really specific for what you are doing. If you need to run a few ideas there is enough work in thinking hard about bias, order execution etc. Regarding your first question: I recently looked at the market to test a quick options on futures hedging strategy and QuantConnect looked like a decent package to me. Depending on the markets you need, you might find that many markets offer data (for a fee). CME has an offering, including full options and futures data, called DataMine . So an important consideration might be if the platform you identify allows good data import features. On the second question: The log limits are important in two cases. First if you are not yet experienced enough with the platform and need to log logs of debug during design/build - rather then using the Jupyter notebook for that. Second, when you run live strategies you want to know what is happening. QuantConnect has plans that should give you want you needs if restricting the critical logs (orders are tracked separately) and you can always send notifications to a an other service.