Is this stock advise worth taking seriously?

 Introduction

Business related TV channels and newspapers are replete with stock advisory from investment firms and certified individual advisors. How seriously should the average viewer and retail investor take these advices? There is the risk of a loss if an advise is acted upon, and the risk of lost opportunity if it is ignored. To heed or not to heed is then the question. What if we had a database of advices from different advisors, for different stocks, to either buy or sell them at a certain price and then wait and close that position when the stock hits a certain target price or a stop loss price, these advices being valid for the day ( meant for Day trade, that is), the short term (3 months) or the long term(1 year)? Surely, a lot could be learnt from this data; in particular, whether or not to take the advise seriously; and act - or not. Interested?

Tools of trade

Apart of course, from your laptop or desktop (preferably Ubuntu - just kidding!), we are going to need Python 3.7.0 or higher (if nothing because I have tried all my code on 3.7.0 and I am cautiously optimistic that they'll run on higher versions) and  Jupyter Notebooks 4.4.0. You could use anaconda as the big-brother package and environment manager for your Python installation as I tend to do, or you might not - up to you. Whether you use anaconda (conda install) or pip install to install the Python modules I am soon going to mention, just ensure that Jupyter notebooks, when you start it, uses the Python kernel from the same Python installation where you installed the modules. So, for example, if you have several Python virtual environments set up using anaconda, and you have installed all your modules into a particular Python 3 environment - let's call it python3env, then ensure that you run jupyter notebook only after you have run conda activate python3env in the shell. The conda activate... command ensures that the correct Python image is run when Jupyter runs it inside of itself. The Python modules I was going to mention are

• pandas
• numpy
• scipy
• matplotlib

You will also need a dataset of advices collected over a period of time. For the purpose of this blog, let's use a scaled down, representative dataset from my GitHub site. 

Now we have all the data in a pandas DataFrame called data

Staring at the data

The more you stare the better, but let it not be an empty gaze. Here's what I have come up with. First let's look at the data as a table. Just typing and executing data

in a Jupyter notebook cell displays it as follows:

What do we have here? Let's see. The 'buysell' attribute tells you if it's a buy advise (value '1') or a sell advise (value '2'). The 'durationtype' attribute tells us if it's a long term advise (valid for a year and value of the attribute is '3'), a short term advise ( valid for 3 months, value '2') or a day trade advise (valid for the trading day it was created on and value '1'). The advisor attribute identifies the advisor . The symbolname attribute identifies the stock symbol on the exchange (NSE, the Indian National Stock Exchange in this case). The 'otheradvices' attribute tells us if there has been a similar advice (by way of the buysell attribute and symbolname being the same) within 3 days of an advice. The niftysentiment attribute takes values '1' for 'up' and '2' for 'down', meaning that the Nifty 50 stock index opened higher or lower than the previous day's close respectively. Finally, the success attribute which is also the predicted variable tells us whether the advice actually went to make a profit (value '1') or loss (value '2'). 'Profit' in this case means the advice first hit the target price mentioned in the advice, while 'Loss' signifies that the stock price hit the stop loss first, within the advice duration period. If you are wondering why we don't see the target price, entry price and stop loss price for the advices in the table, then yes, you would be right to wonder. But we'll leave incorporating that important information for another day and blog. For now, we are going to focus only on the categorical attributes. Yes, the ones and twos and even the Trues and Falses shown as values in that table are all string values.

 One of the first things I like to see about the data at hand is the correlations between attributes. If only we had numeric attributes, then we could have used 

data.corr() to get correlations between all of them , but that is not to be! Categorical attributes need other means to get an idea of correlations. The following function which implements the Cramer's V test will give us this information. Explaining it is beyond the scope of this blog, since it involves the Chi-Square test. More explanation may be found here.

from scipy import stats as ss

def cramers_v(x, y):
    confusion_matrix = pd.crosstab(x,y)
    chi2 = ss.chi2_contingency(confusion_matrix)[0]
    n = confusion_matrix.sum().sum()
    phi2 = chi2/n
    r,k = confusion_matrix.shape
    phi2corr = max(0, phi2-((k-1)*(r-1))/(n-1))
    rcorr = r-((r-1)**2)/(n-1)
    kcorr = k-((k-1)**2)/(n-1)
    return np.sqrt(phi2corr/min((kcorr-1),(rcorr-1)))

 This function takes two array-like arguments for two attributes whose mutual correlation is being checked and returns a real number between 0 and 1, 0 signifying no correlation and 1 signifying a high correlation, with a continuous gradation in between. The results of using this to find correlations of other attributes with the success attribute are in the first several lines of the following image:

The values above reveal that the advisor attribute has a medium correlation with success. It leads the pack anyway. This is to say that of all the attributes we have considered, the advisor that gave the advice is the strongest indicator of whether the advice will be a success or not. The advisor attribute is followed by durationtype, and then symbolname. There is something strange you will notice about the niftysentiment attribute. By itself it shows a significantly lower correlation (0.11) than if concatenated with the buysell attribute (0.19). This is not as strange as it seems. After all, a niftysentiment value of 1 will tend to make a buy advice succeed ( since Nifty index going up will probably help this  stock to achieve its target price too), but a sell advice fail, since in the latter case the movement may well lead to hitting the stop loss price for the sell advice ! This is why the niftysentiment attribute in combination with the buysell attribute is a better indicator, if at all, of success, than the niftysentiment by itself.

But is there an easier way to figure out that two attributes are correlated - something that gives you more insight? Let's fall back on school level probability for this. We learnt there that two events are correlated if they are not independent. Independence means is that the probability of one event is the same irrespective of the other having occurred or not. Let us check if in our dataset the frequency of a success depends on whether it is, let's say, a buy advice or a sell advice. What we are really talking about here is to check whether the total probability of success and the probability of success for buy advices is the same. One graphical way to check this is as in the image below:

The total probability of success can be estimated to be the total green area divided by the total area of the bars in the left-hand side plot above. This is the same as the area of the first bar (blue as well as green) divided by the total area of both the bars in the right-hand side plot. The value, as you can estimate, is around 0.5. Now the probability of success conditional to an advice being a buy advice, is estimated by dividing the green area in the first bar in the left plot with total area of that bar. It is clear that this value is also around 0.5. Similarly, the probability of success given a sell advice is also around 0.5. So, knowledge that it was a buy or sell advice made little difference to the probability of success. This means attributes buysell and success are almost independent not well-correlated, which vindicates the low value of 0.02 that we obtained earlier for the Cramer's V value.

One more way to check the same thing is revealed in the snippet below

Note how the total probability of success and the probability of success conditional to whether it is a buy or sell advice, are all almost the same value - around 0.5. Again this means that the buysell attribute and success of an advice are almost independent and hence not strongly correlated, corroborated by the low Cramer's V value of 0.02 for this case. The reader is urged to try out correlations of other attributes like symbolname etc. with the success attribute.

In the next blog we look at further preprocessing steps like binarizing the categorical attributes etc.