Step 1 - Collecting Data

Open the Recipe for a Walk-through on How to Code

🦉

Step1 Collecting Data

Open Recipe

In this tutorial, we will go through the data collection phase and provide a guide to obtaining Token Metrics and market data from exchanges.

1.1 Preparation Before the Action

Here are the packages needed:

import requests
from pandas import json_normalize
from urllib.parse import urlencode

from typing import Dict, List, Union, Optional, Any
import warnings

warnings.filterwarnings("ignore")

Here is a valuable function built to help connect to our platform. Just enter your API key in <your API key> and explore the desired endpoints.

API_key = '<your API key>'

def tm_API(endpoint: str, payload: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
	"""Obtain data from the Token Metrics Data API

	Args:
		endpoint (str): The endpoint of the API
		payload (Optional[Dict[str, Any]], optional): The parameters to send to the API. Defaults to None.

	Returns:
		Dict[str, Any]: The response from the API
	"""

	if payload:
		url = 'https://alpha.data-api.tokenmetrics.com/v1/' + endpoint + '?' + urlencode(payload)
	else:
		url = 'https://alpha.data-api.tokenmetrics.com/v1/' + endpoint 
	headers = {'Accept': 'application/json', 'Content-Type': 'application/json', 'api_key': API_key}
	response = requests.get(url, headers=headers)
	return response.json()

1.2 Obtain the Tokens Index from `Tokens` Endpoint

Tokens Index is a table with the Name, Symbol, and ID.

Once we have all our tools ready, what do we need first? It would be the Tokens !

As shown in our Guide, all the tokens or symbol information are stored in the Tokens endpoint. We can extract the token information with the following code:

endpoint = 'tokens'
params = {}
response = tm_API(endpoint,params)
coins = json_normalize(response['data'])
coins = coins.sort_values(by = 'TOKEN_ID').reset_index(drop = True)
coins[coins.NAME.isin(['Bitcoin','Ethereum','Litecoin'])].reset_index(drop = True)

Index	TOKEN_ID	SYMBOL	NAME
0	3306	ETH	Ethereum
1	3375	BTC	Bitcoin
2	3377	LTC	Litecoin

We filtered the table with three coins: Bitcoin, Ethereum, and Litecoin for demo purposes.

To ensure that this tutorial presents the entire process in an easily understandable format from a professional trader's perspective, we will use a simple strategy rather than complex ones. So, before moving on to the following steps, we aim to provide a clear overview of the process. Long-Short Single asset trading strategy.

Here is an example of a simple Long-Short Single asset trading strategy:

We focus on Bitcoin only
There are only two positions : Long, Short

1.3 Obtain the Market Price from Exchange

import pandas as pd
import datetime as dt
import requests

Next, we need to collect the market quotes for Bitcoin.

# Obtain data from the Binance US API
params = {'symbol': 'BTCUSDT', 
       'interval': '1d', 
       'startTime': int(dt.datetime(2020,5,2).timestamp())*1000,
       'endTime': int(dt.datetime(2023,1,28).timestamp())*1000,
       'limit': 1000}

r = requests.get('https://api.binance.us/api/v3/klines', params = params).json()

# Convert to pandas dataframe
col = ['Open time','Open','High','Low','Close','Volume','Close time','Quote asset volume','Number of trades',
       'Taker buy volume','Taker buy quote asset volume','Ignore']
btcusdt = pd.DataFrame(r, columns = col)

btcusdt['Open time'] = btcusdt['Open time'].apply(lambda x: dt.datetime.fromtimestamp(x/1000)).dt.strftime('%Y-%m-%d')
btcusdt['Close time'] = btcusdt['Close time'].apply(lambda x: dt.datetime.fromtimestamp(x/1000)).dt.strftime('%Y-%m-%d')

# Clean up the data
btcusdt.rename(columns = {'Close time': 'Date'}, inplace = True)
btcusdt[['Open', 'High', 'Low', 'Close', 'Volume']] = btcusdt[['Open', 'High', 'Low', 'Close', 'Volume']].astype(float)
btcusdt = btcusdt[['Date','Open', 'High', 'Low', 'Close', 'Volume']]
btcusdt

We obtain the historical price of BTCUSDT from Binance US. After some data-cleaning steps, we get the following table:

Date	Open	High	Low	Close	Volume
0	2020-05-10	9531.93	9569.31	8127.14	8726.30	600.659830
1	2020-05-11	8730.15	9162.08	8200.00	8561.03	399.574609
2	2020-05-12	8560.18	8973.14	8533.72	8810.00	153.606379
3	2020-05-13	8808.62	9398.12	8800.91	9295.00	167.297195

And here is the plot of the data:

fig = plt.figure(figsize=(10,5))
sns.lineplot(x=pd.to_datetime(btcusdt['Date']), y=btcusdt['Close'])
plt.title('BTC/USDT')
plt.savefig('../Plots/btcust.png')

1.4 Obtain the Data from `Trader Grades` endpoint

Our Trader Grades endpoint provides a comprehensive overview of price movements. This table can inspire many different trading strategies based on this endpoint.

endpoint = 'trader-grades'
params = {
	'tokens': '3375',
	'startDate': '2020-05-03T00:00:00.000Z',
	'endDate': '2023-01-27T00:00:00.000Z',
	'limit': 1000

}
r = tm_API(endpoint,params)
trader_grades = json_normalize(r['data'])

Once we extract the data from our API, we will only select the following features[['DATE','TA_GRADE','QUANT_GRADE','TM_TRADER_GRADE',]]. Next, we will do a simple transformation on the data.

# select the columns we want
btc_tg = trader_grades[['DATE','TA_GRADE','QUANT_GRADE','TM_TRADER_GRADE',]]

# merge the data
data = pd.merge(btc_tg, btcusdt, left_on='DATE', right_on='Date', how='left').drop('Date', axis=1)

# do some transformations
data['DayReturnPCT'] = (data.Close/data.Close.shift(-1) - 1)*100
data['ta_gradePCT'] = (data['TA_GRADE']/data['TA_GRADE'].shift(-1) - 1)*100
data['quant_gradePCT'] = (data['QUANT_GRADE']/data['QUANT_GRADE'].shift(-1) - 1)*100
data['tm_trader_gradePCT'] = (data['TM_TRADER_GRADE']/data['TM_TRADER_GRADE'].shift(-1) - 1)*100
data['DailyReturnPCT'] = (data.Close/data.Open - 1)*100

Now we have collected all the data we need. Please refer to the btc_tg.csv.

<class 'pandas.core.frame.DataFrame'>
Int64Index: 748 entries, 0 to 747
Data columns (total 14 columns):
 #   Column              Non-Null Count  Dtype         
---  ------              --------------  -----         
 0   DATE                748 non-null    datetime64[ns]
 1   TA_GRADE            747 non-null    float64       
 2   QUANT_GRADE         747 non-null    float64       
 3   TM_TRADER_GRADE     746 non-null    float64       
 4   Open                748 non-null    float64       
 5   High                748 non-null    float64       
 6   Low                 748 non-null    float64       
 7   Close               748 non-null    float64       
 8   Volume              748 non-null    float64       
 9   DayReturnPCT        747 non-null    float64       
 10  ta_gradePCT         745 non-null    float64       
 11  quant_gradePCT      745 non-null    float64       
 12  tm_trader_gradePCT  743 non-null    float64       
 13  DailyReturnPCT      748 non-null    float64       
dtypes: datetime64[ns](1), float64(13)
memory usage: 87.7 KB

Open the Recipe for a Walk-through on How to Code

1.1 Preparation Before the Action

1.2 Obtain the Tokens Index from Tokens Endpoint

1.3 Obtain the Market Price from Exchange

1.4 Obtain the Data from Trader Grades endpoint

1.2 Obtain the Tokens Index from `Tokens` Endpoint

1.4 Obtain the Data from `Trader Grades` endpoint