Contents

6. Gap Analysis with Categorical Variables

6.1. Preliminaries¶

I include the data import and library import commands at the start of each lesson so that the lessons are self-contained.

import pandas as pd
from scipy import stats
bank = pd.read_csv('Data/Bank.csv')

# Recode JobGrade to Manager
grades = [1,2,3,4,5,6]
status = ["non-mgmt", "non-mgmt", "non-mgmt", "non-mgmt", "mgmt", "mgmt"]
bank['Manager'] = bank['JobGrade'].replace(grades, status)

6.2. Creating a contingency table

Pandas has a very simple contingency table feature. Below, I specify the two variables of interest (Gender and Manager) and set margins=True so I get marginal totals (“All”).

contab_freq = pd.crosstab(
    bank['Gender'],
    bank['Manager'],
    margins = True
   )
contab_freq
Manager mgmt non-mgmt All
Gender
Female 10 130 140
Male 25 43 68
All 35 173 208

6.3. Showing row percentages

Typically, showing frequencies is less useful than relative frequencies. Here, I am interested in the row percentages: what is the probability that a female is a manager versus the probability a male is a manager.

We can get relative frequencies using the normalize argument. If normalize  = True, then we get the relative frequency in each cell relative to the total number of employees. This is not very useful. What we want instead is to normalize by row. The parameter for this is: normalize = 'index'. Why “index” instead of “row”? Because each row has a row number (or index).

conttab_relfreq = pd.crosstab(
    bank['Gender'],
    bank['Manager'],
    margins = True,
    normalize='index'
   )
conttab_relfreq
Manager mgmt non-mgmt
Gender
Female 0.071429 0.928571
Male 0.367647 0.632353
All 0.168269 0.831731

Here, each row sums to 100%. Thus, for the total set of female employees, 7% are managers and 94% are non-managers. For males, 37% are managers and 63% are non-managers. The advantage of this presentation is that these percentages are directly comparable even though the majority (140/208) employees of the bank are female.

6.4. Chi-squared test of independence

The row percentages leave us with the impression that managerial status depends on gender. We can test this more formally using the \(\chi^2\) (/ˈkaɪ skweə(r)) test of independence.

Scipy has a method called chi2_contingency() that takes a contingency table of observed frequencies as input. Note that this table cannot include marginal totals or marginal frequencies. Instead, it must consist of m x n observations:

contab_obs = pd.crosstab(
    bank['Gender'],
    bank['Manager'],
    margins = False)
chi = stats.chi2_contingency(contab_obs)
chi

(26.617776266575998,
 2.479518719230249e-07,
 1,
 array([[ 23.55769231, 116.44230769],
        [ 11.44230769,  56.55769231]]))

The output of the chi2_contingency() method is not particularly attractive but it contains what we need:

  1. The first line is the \(\chi^2\) statistic, which we can safely ignore

  2. The second line is the probability of getting a \(\chi^2\) statistic that large if the two variables are independent. This p-value is very small (\(10^{-7}\)) so we conclude there is almost zero chance that gender and managerial status are independent at this bank.

  3. The third line is the degrees of freedom, which we can safely ignore.

  4. The remainder of the output is a matrix showing the expected frequencies under the assumption in independence. These expected values are quite different from the observed values above.

You may notice that the \(\chi^2\) statistic and p-value are different from those provided by R. This is because scipy defaults to the “Pearson’s Chi-squared test with Yates’ continuity correction” version of the test.

5. Gap analysis with Continuous Variables 7. Correlation and Scatterplots