Tools of the trade
|Tools of the trade|
We need tools for:
- trade execution and management
Data: getting it, preparing it and saving it
|Data: get it, prepare it, save it.|
Forms of dataData comes in many forms, shapes, and sizes. Mostly we will be looking for:
- historical price data (time series)
- fundamental data
So where to get data?
ChartsPrice charts are most probably offered by your broker. If you trade FX and CFDs then your broker will most likely offer the MetaTrader platform or maybe the cTrader platform. Both of these offer good charting in my opinion.
If you want to get your charting from another source then your broker, then you have a few options. My favorite option is TradingView. This is a free online charting platform. I have written an article and recorded youtube videos about how to backtest strategies using this platform here: http://davetromp.blogspot.com/2015/08/how-to-back-test-manually-traded.html
If you trade or invest in stocks and ETFs based on the daily closing prices, then most online finance sites like Google finance, Yahoo finance, Blomberg, etc. offer charts that will do for getting an overview of the market. These charts are not good enough to do advanced chart analysis.
Another site that you can use for advanced chart analysis is Investing.com, but all things considered, I think that TradingView is the best charting option for any market.
Historical Price DataAll charts are nothing more than visual representations of prices over time, also called price time series. This is interesting data to have as it will allow us to analyze prices in a different way, then just visually.
It will also enable the automation of backtesting. This will speed up our trading analysis process. Always mind data errors in the freely available data you find online. Especially daily highs and lows can be way off. Therefore it is most safe to only use closing prices from free online sources.
Manual data retrievalYou can download data from financial websites and load them into a spreadsheet for analysis. I use the following sources to get historical price data manually:
|Getting data on Investing.com|
Just look up a security and go to the "Historical Prices" page: http://finance.yahoo.com/q/hp?s=AAPL+Historical+Prices
You can select daily, weekly, monthly and yearly data. Yahoo Finance will give you volume, open, high, low, close prices plus adjusted closing prices. Adjusted closing prices are historical prices that have been adjusted for events like stock splits. For instance, if a stock price was 200 USD and was split to become 2 times 100 USD alll historical prices before this event would be recalculated by dividing it by 2. This way it is possible to make a correct comparison of the prices over time.
Investing offers more or less the same kind of data as Yahoo Finance, but in addition, it offers historical data on currencies, metals, and cryptocurrencies in daily weekly and monthly timeframes. See https://www.investing.com/currencies/eur-usd-historical-data.
- StooqAt www.stooq.com you can download price data for almost any security there is. The site is based in Poland. It is set up in Polish, but the main navigation will translate to English if your browser/os is set to English. I have once contacted their customer support to ask some questions about the data they offer and I got a prompt reply in English. They offer download options for daily, weekly, monthly, quarterly and yearly data on the historical data page much like Yahoo and Investing. For example http://stooq.com/q/d/?s=eurusd
Dukascopy is a Swiss broker. They offer free downloads of complete historical forex price data down to ticks. You do need to register to download. You can find the download options at www.dukascopy.com --> market info --> historical data feed.
Automated data retrievalGetting the data manually can become a problem if you want to test a strategy on many different securities. Luckily, there are several services that allow us to easily automate the retrieval of data.
InvestPy is a Python library that will let you retrieve historical price data from Investing.com. See https://github.com/alvarob96/investpy/
AlphaVantage.co is a service provider that offers access to realtime and historical stock, fx and cryptocurrency price data. You need to authenticate via API-key. You can signup for free and get an API-key at https://www.alphavantage.co/
- Pandas DataReader
I use python with the pandas library to do many of my analysis and data handling. Pandas has a (now) separate library that allows easy access to free data from Yahoo and others. Check out https://github.com/pydata/pandas-datareader
- Oanda API
The broker Oanda has a rest-API. Among other things, it allows for accessing historical price data up to 5000 data points. If you have a practice account you can access the API and retrieve all FX CFD data this way (please note that US customers do not have access to CFDs). See http://developer.oanda.com/
How to prepare historical price data?Sometimes the data you get will contain errors. So the first thing you want to do after you got the data is to plot it, inspect it visually and adjust/remove any obvious out-layers. You can do this by simply reading in the data into a spreadsheet and make a graph. I do this in LibreOffice Calc or in Python, but you can do this in any software you like, such as Excell, for instance.
|Reading in Google Finance data in LibreOffice Calc|
|Sorting the data in LibreOffice Calc|
|Plotting the data for inspection|
I usually just save the data as a CSV file (comma separated file), but you can also build a securities database in a SQL database. This way it is easy to reuse the data. You can retrieve and clean the data once and then use it over and over again. Also over time, you will be able to build up a database of data that goes back in time. I like CSV files as they are easy to use and understand. They can be read into a spreadsheet and a spreadsheet can be saved as a CSV file.
How to save data, so you can handle it?
|Python with the pandas library. My tool of choice.|
df = pandas.read_csv("file.csv")The data frame is more or less like a spreadsheet but can be manipulated using a few simple lines of Python code. The result can then be saved to a CSV file like this:
df.to_csv("newfile.csv")And then this file can be opened again in LibreOffice Calc or in a text/code editor. Pandas also has read and write functions for excel, sql, json and others. For me, the CSV format has been most efficient to use. For large sets of data databases or the HDF5 datastore format might be more convenient.
More info on the read and write options in Pandas can be found here: http://pandas.pydata.org/pandas-docs/stable/io.html
Fundamental DataTradingView and Investing offer screeners that allow you to screen stocks by fundamental data like P/E ratios, etc.
Two other interesting sources, that allow for programmatic access to data, including fundamental data are also mentioned above:
- Pandas DataReader
NewsI do not trade the news, but I do mind big news events as they may cause higher than normal volatility. Most trading platforms incorporate some sort of news feed. I use TradingView's platform and this platform has a nice feature that plots future and past news events on the price charts. This way it is easy to see which events tend to move the markets.
A similar feature is available on the charts of Investing. Investing also offers a news feed and alerts of breaking news events and scheduled events like announcements by central banks, etc.
The next step is that I try to translate the pattern I think, I have found, into a few simple rules. For example, a rule can be: "if today's closing price is more than 2% higher than yesterday's close price, then the next day will be a down day, meaning I will sell short at the next days open and buyback just before the close.".
I write down the rules and then I manually backtest the rules using software like TradingView if possible. I then model the rule set in a spreadsheet. The following step is to write a backtest in Python. This will allow me to do the testing on more different data samples more quickly, then when I would use only spreadsheets. If you want to read more about why I use Python for this, then go check out this article I wrote:
|Execution of your trade|
- Through trading platforms
This is straight forward enough. You just enter your trades in your broker's platforms.
- Through online trading APIs
Many brokers offer access via trading APIs. There are different kinds of APIs, but the most commonly used are rest APIs and the FIX protocol API. Restful APIs are APIs that are typically used on the web to have web applications interact with each other. The FIX Protocol is a protocol used mainly by professionals to interact with financial platforms like exchanges, brokers and liquidity providers. I have not yet familiarized my self with the protocol, but there is a great Youtube channel FixProtocol with a great series on how to read Fix Protocol messages if you are interested to know more.
- Through local API to the trading platform
Some platforms allow you to hook up excel to the platform, so you can get data into excel from the trading platform and generate trading signals that can then be executed via the trading platform. This means you will have to have everything on your computer or on a server. Some platforms also allow for programmatic access. For instance, cTrader has cAlgo that allows you to program your strategies in cSharp and InterActive Brokers has a Java-based API.
I also have used Darwinex ZeroMQ Connector to access Metatrader4 from Python.
Github is your friend here:
I know I only briefly touched on many different things and I referenced a lot of sources. My intention is now to get you familiarized with the tools I use and to get you researching all the tools I reference.
Next in this getting started trading series:
If you like this "getting started with trading" series and wanna stay in the loop on new articles, then be sure to subscribe using the social media buttons and by subscribing to this blog.