Using ElasticNetCV for Stock Price Prediction

Purpose of participation

Stock price prediction is pretty distant from the field of my interest, which is NLP.

But there are several purpose that I spent time participating in no-reward competition.

• Data wrangling practice: whatever datatype it be, I think it is important to wrangle as many data as possible. It is for the numpy & pandas practice
• Refreshing: I love studying the field of machine learning and deep learning. Although it is important to study deep enough, I think some refreshment helps the continuity of studying.

• Somehow connected!

  • The usage of cross validation
  • Markov-chain vs matrix approach
  • usage of sine, cosine function for timeseries annotation
  • Regularization at ElasticNet might be also regarded as weight decay.

Using ElasticNetCV for stock price prediction

Adding more data

• external data: WTI, NASDAQ, was tested, but it was no good
• derivative data: after @SangHoeKim constructed derivative data, I added VWAP and SMI but didn't increase the model's performance

Choosing Adaquate Model

• When choosing a model, it was important that the model 1) don't fluctuate its performance over periods of time, 2) has higher performance than Linear Regression.

• The original plan was to ensemble 1) time-series forecast model with 2) regression model. Regression model ElasticNetCV turned out to be the best model, but ARIMA model's performance was too overfit for specific periods only to use. For example, ARIMA was good for predicting overall bear market / overall bull market but was bad at predicting fluctuation between the spand of 5 days.

  • ElasticNetCV was the right choice. It was advanced version of Linear Regression, and it has provided K-fold validation structure so that it comparably had more robustness compared to ElasticNet.
  • ARIMA(0,1,1) showed pretty good performance. This was out of expectation, since (p, q, d) = (0, 1, 1) is just moving average model with constant uprise/downfall of differentiation.
  • RandomForestRegressor and pmdARIMA was selected since its criteria can be customized as MAE instead of MSE.
  • pmdARIMA is implementation of AutoARIMA from RStudio, but its performance was too low to use.
• XGBoostRegressor turned out to be not good. Not only this was my team's conclusion, but the general consensus over the Dacon community was similar. However @tjy3090 pointed out that XGB captured the turning point and the pattern of the graph, even though its NMAE score was low.
• Compared between ffill vs dropna(), but concluded that ffill outperformed on more recend dates.

Model	NMAE (2021-09-06 ~ 2021-09-10)	NMAE (2021-09-15 ~ 2021-09-24)
ElasticNetCV	3.02	2.93
ARIMA(0,1,1)	3.03	-
ElasticNet	3.12	-
XGBoost	3.87	-
Linear Regression	4.03	-
RFRegressor	4.11	-
pmdARIMA	8.81	-

• Experimented with ARIMA, pmdARIMA, ElasticNetCV, RandomForestRegressor and XGBoost.
• Competition's criteria is based on NMAE. Grading criteria function was constructed as following:

def NMAE(true_df, pred_df_input):
    """ grading criteria for public leader board """
    return (abs(true_df_copy - pred_df_input) / true_df_copy * 100).iloc[:5].values.mean()

• Thanks for setting up grading criteria, the team was able to experiment different kinds of models without being limited by submission quota.

Code and Outputs

# import python modules
import os
import copy

# import data wrangling modules
import pandas as pd
import numpy as np

# import visualization modules
from tqdm import tqdm
import matplotlib.pyplot as plt
import seaborn as sns

# import dataloader & stock data downloader
import FinanceDataReader as fdr

# import machine learning modules
from sklearn.preprocessing import MinMaxScaler, RobustScaler, StandardScaler
from sklearn.model_selection import train_test_split

# import machine learning models
from sklearn.linear_model import LinearRegression, ElasticNet, ElasticNetCV
from xgboost import XGBRegressor

# import cell output cleaner
from IPython.display import clear_output

# import stock data derivatives module
from talib import RSI, BBANDS, MACD, ATR

import warnings
warnings.filterwarnings('ignore') # remove warnings from pandas

# set options pandas: display more columns and reduced digits of float numbers
pd.set_option('display.max_columns', 100)
pd.options.display.float_format = '{:.4f}'.format

path = './open'
list_name = 'Stock_List.csv'
stock_list = pd.read_csv(os.path.join(path,list_name))
stock_list['종목코드'] = stock_list['종목코드'].apply(lambda x : str(x).zfill(6))
display(stock_list.head())
display(stock_list.tail())

# define dates
""" Make dataset for training & evaluation """
from datetime import datetime
import os

# determines whether it is public leaderboard period or new_public period
BOOL_PUBLIC_PERIOD = False

# Determines whether to use public way or private way to utilize data
BOOL_PUBLIC = False

# set data window range
start_date = None # if this was set to 2021-01-04, then the mae shoots up to from 3.0 to 4.7
today = datetime.now()
data_end_date = today.strftime('%Y-%m-%d')

# public_start_date = '2021-09-10'
public_start_4th = '2021-09-06'
public_end_date = '2021-09-10'

check_point = '2021-09-14'
recent_known_date = '2021-09-24'

private_start_4th = '2021-09-27'
private_end_4th = '2021-10-01'

if BOOL_PUBLIC_PERIOD:
    sample_name = 'sample_submission.csv' # week 4 submission
    pred_df = pd.read_csv(os.path.join(path,sample_name)).set_index('Day')
else:
    new_public_name = "new_public.csv"
    pred_df = pd.read_csv(os.path.join(path,new_public_name)).set_index('Day')
pred_df

""" fill out ground truth stock price of the companies for the public submission period """

if BOOL_PUBLIC_PERIOD:
  blank_submission = pd.read_csv(os.path.join(path,sample_name)).set_index('Day')
  true_df = blank_submission.copy()
  true_df_first_day = public_start_4th
  true_df_last_day = public_end_date
else:
  df_new_public = pd.read_csv(os.path.join(path,new_public_name)).set_index('Day')
  true_df = df_new_public.copy()
  true_df_first_day = true_df.iloc[0].name
  true_df_last_day = true_df.iloc[-1].name
print(true_df_first_day, true_df_last_day)

2021-09-15 2021-10-01

for company_code in tqdm(true_df.columns):
  data_raw = fdr.DataReader(company_code, start = start_date, end = private_start_4th).reset_index()
  data_raw = data_raw.drop(columns = ['Change'])
  data_raw = data_raw.replace(0, np.nan).ffill()
  data_raw.index = data_raw.Date
  # print(data_raw.head())
  public_true_closes = data_raw.loc[true_df_first_day:true_df_last_day].Close.iloc[:]
  # print(public_true_closes)
  true_df.loc[:,company_code] = public_true_closes.to_list() * 2

true_df.head()

100%|██████████| 376/376 [00:50<00:00,  7.50it/s]

def makeData(data):
    """
    - make derivative data from the fdr dataset
    - not using external dataset which increases noise that harms the performance
    """

    data = data.copy()

    # Korean won volume
    data['KRW_Vol'] = data[['Close', 'Volume']].prod(axis=1)
    data['KRW_Vol_1m'] = data.KRW_Vol.rolling(21).mean()

    # RSI
    data['RSI'] = RSI(data.Close)

    # Bollinger Bands
    def compute_bb(close):
        high, mid, low = BBANDS(close, timeperiod=20)
        return pd.DataFrame({'bb_high': high, 'bb_low': low}, index=close.index)

    data[['bb_high', 'bb_low']] = compute_bb(data.Close)
    data['bb_high'] = data.bb_high.sub(data.Close).div(data.bb_high).apply(np.log1p)
    data['bb_low'] = data.Close.sub(data.bb_low).div(data.Close).apply(np.log1p)

    # ATR
    def compute_atr(stock_data):
        df = ATR(stock_data.High, stock_data.Low,
                 stock_data.Close, timeperiod=14)
        return df.sub(df.mean()).div(df.std())

    data['ATR'] = compute_atr(data)

    #MACD
    def compute_macd(close):
        macd = MACD(close)[0]
        return (macd - np.mean(macd))/np.std(macd)

    data['MACD'] = compute_macd(data.Close)

    # Lagged Returns
    lags = [1, 2, 3, 4, 5, 10, 21, 42, 63]
    for lag in lags:
        data[f'return_{lag}d'] = data.Close.pct_change(lag).add(1).pow(1 / lag).sub(1)

    for t in [1, 2, 3, 4, 5]:
        for lag in [1, 5, 10, 21]:
            data[f'return_{lag}d_lag{t}'] = data[f'return_{lag}d'].shift(t * lag)

    # target return
    for t in [1, 2, 3, 4, 5]:
        data[f'target_{t}d'] = data[f'return_{t}d'].shift(-t)

    # volume change
    q = 0.01
    data[data.filter(like='Vol').columns] = data.filter(like='Vol').pct_change().apply(lambda x: x.clip(lower=x.quantile(q),upper=x.quantile(1 - q)))

    data = data.drop(['Date', 'Open', 'High', 'Low', 'Close'], axis= 1)
    data = data.fillna(method="ffill")
    data = data.fillna(method="bfill")
    # display(data.tail())

    return data

def mape(true_df, pred_df_input):
    """ grading criteria for public leader board """
    # extract columns from true_df same as pred_df_input
    true_df_copy = true_df.copy()
    true_df_copy = true_df_copy.loc[:, pred_df_input.columns]
    true_df_copy = true_df_copy.iloc[:5]
    pred_df_input = pred_df_input.iloc[:5]
    return (abs(true_df_copy - pred_df_input) / true_df_copy * 100).iloc[:5].values.mean()

def sklearn_predict(
    pred_df_input,
    model,
    bool_public,
    start_date,
    recent_known_date,
    clip=True,
    scaler=RobustScaler()
    ):
    """ make prediction dataframe """

    if bool_public:
        # get first five rows of pred_df_input
        pred_df_input = pred_df_input.iloc[:5]
    else:
        # dropping first 5 days of public dates from the submission dataframe
        pred_df_input = pred_df_input.iloc[5:]

    for company_code in tqdm(pred_df_input.columns):
        # clear_output()
        data_raw = fdr.DataReader(
            company_code,
            start = start_date,
            end = recent_known_date
            ).reset_index()
        data_raw = data_raw.drop(columns = 'Change')
        data_raw = data_raw.replace(0, np.nan).ffill()
        data_raw.index = data_raw.Date
        # clear_output() # clear output for the jupyter notebook

        if bool_public: # if public submission
            # make necessary data for the prediction
            data = makeData(data_raw)

            # get the last date of the public submission period
            public_last_close = data_raw.loc[recent_known_date].Close
            public = data.loc[[recent_known_date]]

            # loc[] is inclusive for the end slicing date, unlike list slicing
            train_indv_comp = data.loc[:recent_known_date]
            Ys = train_indv_comp.filter(like='target')
            X = train_indv_comp.drop(Ys.columns, axis=1)
            Ys_public = public.filter(like='target')
            X_public = public.drop(Ys_public.columns, axis=1)
            X = scaler.fit_transform(X)
            X_public = scaler.transform(X_public)
            pred_public = []
            public_close = public_last_close
            for y_col in Ys.columns:
                model.fit(X, Ys[y_col])
                r_pred_public = model.predict(X_public)

                public_close = public_close * (1+r_pred_public[0])

                pred_public.append(public_close)
            # print(pred_df_input.shape)
            # display(pred_df_input)
            pred_df_input.loc[:,company_code] = pred_public

        else: # if private submission
            data = makeData(data_raw)
            # display(pred_df_input)
            private_last_close = data_raw.loc[recent_known_date].Close

            # display(data)
            private = data.loc[[recent_known_date]]
            # display(private)
            train_indv_comp = data.loc[:recent_known_date]
            # display(train_indv_comp)

            # make train_indv_comp data
            Ys = train_indv_comp.filter(like='target') #Consisted of mon, tue, wed, thur, fri data columns
            X = train_indv_comp.drop(Ys.columns, axis=1)

            # make private data
            Ys_private = private.filter(like='target')
            X_private = private.drop(Ys_private.columns, axis=1)

            # fit scaler
            X = scaler.fit_transform(X)
            X_private = scaler.transform(X_private)

            pred_private = []
            private_close = private_last_close

            for y_col in Ys.columns:
                model.fit(X, Ys[y_col])
                r_pred_private = model.predict(X_private)
                private_close = private_close * (1+r_pred_private[0])
                pred_private.append(private_close)
            # print(pred_private)
            pred_df_input.loc[:,company_code] = pred_private
            # display(pred_df_input)
    return pred_df_input

Check the performance for sampled 20 companies

""" sample prediction """

# Utilizing ElasticnetCV to automatically tune the alpha hyperparameter with cross validation.
# (rho is left at default value 0.5)

# ElasticnetCV Documentation: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNetCV.html#sklearn.linear_model.ElasticNetCV
# Cross Validation documentation: https://scikit-learn.org/stable/modules/cross_validation.html#cross-validation
# Builtin Cross Validation provided models: https://scikit-learn.org/stable/glossary.html#term-cross-validation-estimator

"""
https://stackoverflow.com/questions/12283184/how-is-elastic-net-used
To automatically tune the value of alpha it is indeed possible to use ElasticNetCV which will spare redundant computation as apposed to using GridSearchCV in the ElasticNet class for tuning alpha.
In complement, you can use a regular GridSearchCV for finding the optimal value of rho.
"""

# name = "ElasticNet_ALL_External_Dataset"
name = "ElaticNetCV_private"

# random sample 20 companies
PRED_NUM = 20

if PRED_NUM != None:
    pred_df_sampled = pred_df.copy()
    pred_df_sampled = pred_df_sampled.sample(PRED_NUM, axis=1, random_state=616)

scaler = StandardScaler()
model = ElasticNetCV(max_iter=1000000) # default K-Fold is 5 folds

df_result_sampled = sklearn_predict(
    pred_df_input = pred_df_sampled,
    model = model,
    bool_public= True,
    start_date=start_date,
    recent_known_date = check_point,
    clip=True,
    scaler=scaler
    )

df_result_sampled_copy = df_result_sampled.copy()
df_result_sampled_copy

mape_value = mape(true_df, df_result_sampled_copy)
print(mape_value)

2.9318977420110794

전체

""" training time: 3444 second(or 57 minutes) """

# name = "ElasticNet_ALL_External_Dataset"
name = "ElasticNetCV_private"

# predict for all 376 companies
PRED_NUM = None

scaler = StandardScaler()
model = ElasticNetCV(max_iter=1000000) # default K-Fold is 5 folds

df_result_all = sklearn_predict(
    pred_df,
    model,
    bool_public= BOOL_PUBLIC,
    start_date=start_date,
    recent_known_date = recent_known_date,
    clip=True,
    scaler=scaler
    )

100%|██████████| 376/376 [1:02:57<00:00, 10.05s/it]

display(df_result_all)

name = "ElasticNetCV_private_last"
df_result_all.to_csv(f'predict/{name}.csv')

# load result from csv
df_result_all = pd.read_csv(f'predict/{name}.csv')

# apply retina display for clearer visualization
%config InlineBackend.figure_format = 'retina'

# Korean font path designation
import matplotlib.font_manager as fm
fontpath = './font/NanumBarunGothic.ttf'
font = fm.FontProperties(fname=fontpath, size=14)

# sample 9 companies from stock_list
SAMPLE_NUM = 16
sampled_data = stock_list.sample(SAMPLE_NUM)

# visualize 16 companies subplots from df_result_all
fig, ax = plt.subplots(nrows=4, ncols=4, figsize=(16,16))
# make margin between figures
plt.subplots_adjust(hspace=0.5, wspace=0.5)

# visualize df_result_all
for i, code in enumerate(sampled_data["종목코드"]):
    ax[i//4, i%4].plot(df_result_all.loc[:,code])

    # find the matching row of sampled_data from code
    company_name = stock_list[stock_list["종목코드"] == code]["종목명"].values[0]
    ax[i//4, i%4].set_title(company_name, fontproperties=font)
    ax[i//4, i%4].set_xlabel('Date')
    ax[i//4, i%4].set_ylabel('Close Price')
    ax[i//4, i%4].grid()

# check submission's validity
submission_last = pd.read_csv("./predict/ElasticNetCV_2021-09-26.csv")
# show information of the submission
submission_last.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Columns: 377 entries, Day to 950130
dtypes: float64(376), object(1)
memory usage: 29.6+ KB