FINA 5840: Financial Modeling Assignment 4
Halis Sak
Due Date: May 7, 2021
Question. We want to use Sklearn package to implement feed-forward neural networks on firm
characteristics data. This time we want to use all the features except “permno”, “year”, “month”,
“next_ret”, and “pe_op_dil”. Please train two different models; a single hidden layer of 32 neurons
(NN1) and two hidden layers with 32 and 16 neurons (NN2). We also want to see the factor importance
for NN1 and NN2. For this we need to use permutation importance (it simply looks at how much
the mean squared error increases when a feature is not available - its values are randomized). Please
report top ten features for NN1 and NN2. Top ten features that we get for NN1 and NN2 are given
below. (The list that you get might be different depending on the hyperparameters of the
model and also the random seed that you choose.)
NN1 NN2
rank 1 divyield curr_debt
rank 2 mmt6 gpm
rank 3 gprof gprof
rank 4 lt_debt cash_debt
rank 5 accrual peg_trailing
rank 6 max accrual
rank 7 cash_conversion at_turn
rank 8 debt_at curr_ratio
rank 9 rd_sale cash_lt
rank 10 roa intcov
Please do the following in the given order.
• Step 1. Remove the features “permno”, “year”, “month”, “next_ret”, and “pe_op_dil” from
columns of “df”. Name this new list as “feats”.
• Step 2. Assign mean of the columns for missing values and normalize the features in “feats” for
“df_train” and “df_test”.
• Step 3. Install eli5 package to your computer.
• Step 4. Please see the class definition (“MLPRegressor_O”) given in Appendix. We override
the _init_coef function ito nitialize the weights of MLPRegressor in this class. Please copy and
paste this class definition to your Python session.
• Step 5. Set the seed of random number generator of numpy using np.random.seed(12345).
• Step 6. Construct an “MLPRegressor_O” model with hidden layer size equal to 32 for NN1
(“MLPRegressor” is from sklearn.neural_network). (I set activation=“tanh”, solver=“sgd”,
batch_size=100, learning_rate_init=0.004, momentum=0, n_iter_no_change=10. You can
play with these parameters to increase out-of-sample R^2 that you will get at Step 8.)
• Step 7. Train the “MLPRegressor_O” model on the train data.
• Step 8. Make predictions on test data using the trained models and compute out-of-sample R^2.
1
• Step 9. If you are satisfied with out-of-sample R^2 (0.6% seems to be a good value based on
Gu. et. al. 2020), use “PermutationImportance” function with “MLPRegressor_O” model and
train data for fitting (“PermutationImportance” is from eli5.sklearn). See https://eli5.
readthedocs.io/en/latest/blackbox/permutation_importance.html
• Step 10. List the factor importance using the command below
>>> print(eli5.format_as_text(eli5.explain_weights(perm,feature_names = df_train[
feats].columns.tolist())))
where “perm” is the object returned from “PermutationImportance” function call.
Appendix.
>>> class MLPRegressor_O(MLPRegressor):
def _init_coef(self, fan_in, fan_out):
if self.activation == 'logistic':
init_bound = np.sqrt(2. / (fan_in + fan_out))
elif self.activation in ('identity', 'tanh', 'relu'):
init_bound = np.sqrt(6. / (fan_in + fan_out))
else:
raise ValueError("Unknown activation function %s" %self.activation)
coef_init = np.random.uniform(-0.01,0.01,(fan_in,fan_out))
intercept_init = np.random.uniform(-0.01,0.01,fan_out)
return coef_init, intercept_init
2