This is the concluding part of my article devoted to a statistical analysis of police shootings and criminality among the white and the black population of the United States. In the first part, we talked about the research background, goals, assumptions, and source data; in the second part, we investigated the national use-of-force and crime data and tracked their connection with race.

Let's recall the intermediate inferences that we were able to make from the available data for 2000 — 2018:

• White police victims outnumber black victims in absolute figures.
• Use of lethal force results in an average of 5.9 per one million Black deaths and 2.3 per one million White deaths (Black victim count is 2.6 greater in unit values).
• Year-to-year scatter in Black lethal force fatalities is nearly twice the scatter in White fatalities.
• White fatalities grow continuously from year to year (by 0.1 — 0.2 per million on average), while Black fatalities rolled back to their 2009 level after climaxing in 2011 — 2013.
• Whites commit twice as many offenses as Blacks in absolute numbers, but three times as fewer in per capita numbers (per 1 million population within that race).
• Criminality among Whites grows more or less steadily over the entire period of investigation (doubled over 19 years). Criminality among Blacks also grows, but by leaps and starts; over the entire period, however, the growth factor is also 2, like with Whites.
• Fatal encounters with law enforcement are connected with criminality (number of offenses committed). The correlation though differs between the two races: for Whites, it is almost perfect, for Blacks — far from perfect.
• Lethal force victims grow 'in reply to' criminality growth, generally with a few years' lag (this is more conspicuous in the Black data).
• White offenders tend to meet death from the police a little more frequently than Black offenders.

Today, as I promised, we'll be looking at the geographical distribution of these data across the states, which ought to either confirm or confute the previous conclusions.

However, before we take up geography, let's make a step back and see what happens if we analyze only the most violent offenses instead of 'All Offenses' as the source data for criminality. Many of my readers have pointed out in their comments that this would have been more proper, since 'All Offenses' incorporate those which should not (in practice) be associated with aggressive behavior provoking police shooting, such as petty larceny or selling drugs. I cannot whole-heartedly agree with this reasoning because, as I see it, any offense can arouse or heighten attention from the law enforcement, which in turn may wind up sadly… Still, let's just be curious enough to check!

Assault and Murder Instead of All Offenses

We just need to change one line of code where we form the crime dataset. Replace this line

df_crimes1 = df_crimes1.loc[df_crimes1['Offense'] == 'All Offenses']

with this:

df_crimes1 = df_crimes1.loc[df_crimes1['Offense'].str.contains('Assault|Murder')]

Our new filter now lets through only offenses connected with assault (simple and aggravated) and murder / non-negligent homicide (negligent / justifiable homicide / manslaughter cases are not included).

We leave the rest of the code as it was.

The number of crimes per 1 million population within each race now looks as follows:



We can see that, though the scale (Y-axis) is much lower, the shape of the curves is almost identical to the All Offenses ones we saw previously.

The criminality vs. lethal force victims curves for both races:





And the correlation matrix:


The correlation between criminality and lethal force fatalities is worse this time (0.68 against 0.88 and 0.72 for All Offenses). But the silver lining here is the fact that the correlation coefficients for Whites and Blacks are almost equal, which gives reason to say there is some constant correlation between crime and police shootings / victims (regardless of race).

Now for our 'DIY' index — the ratio of lethal force deaths to the number of crimes (both per capita):



The difference here is even more apparent. The inference is the same: White criminals are more likely to get killed by the police than Black criminals.

The summary is that all our prior conclusions hold true.

Well, down to geography lessons now! :)

Source Data

To investigate criminality in individual states, I used different source endpoints in the FBI database:

• State level UCR Estimated Crime Data Endpoint — without race classification (the resulting CSV can be downloaded from here)
• State level Arrest Demographic Count By Offense Endpoint — with race classification (the resulting CSV can be downloaded from here)

Unfortunately, I didn't manage to get complete data on committed offenses with the offense state, year and offender race, much as I tried. The returned results had large gaps, for example, some states were totally omitted. But the alternative data on arrests is quite sufficient for our humble research.

The first dataset contains crime counts for all the 51 states from 1991 to 2018, for the following offense categories:

1. violent crime (murder, rape, robbery and aggravated assault)
2. homicide (all types, including negligent / justifiable)
3. rape legacy (using outdated metrics — before 2013)
4. rape revised (using updated metrics — from 2013 on)
5. robbery
6. aggravated assault
7. property crime
8. burglary
9. larceny
10. motor vehicle theft
11. arson

For our purposes, we'll be using the 'violent crime' category, in keeping with the rest of the research.

The second dataset features the number of arrests for the 51 states from 2000 to 2018, with details on the arrested persons' race (refer to the previous part for the race categories). Since the arrest dataset uses a different offense classification and doesn't provide the combined 'violent crime' category, the requests and retrieved results are for the four constituent offenses — murder / non-negligent manslaughter, robbery, rape, and aggravated assault.

Crime Distribution (No Racial Factor)

First, we'll look at the distribution of violent crimes across the states regardless of the offenders' race:

import pandas as pd, numpy as np

CRIME_STATES_FILE = ROOT_FOLDER + '\\crimes_by_state.csv'
df_crime_states = pd.read_csv(CRIME_STATES_FILE, sep=';', header=0, 
                         usecols=['year', 'state_abbr', 'population', 'violent_crime'])

The resulting dataset:

1428 rows ? 4 columns

Adding the full state names (the list of states we already used in our research — CSV) and optimizing / sorting the data:

df_crime_states = df_crime_states.merge(df_state_names, on='state_abbr')
df_crime_states.dropna(inplace=True)
df_crime_states.sort_values(by=['year', 'state_abbr'], inplace=True)

Since the dataset already has population values, let's calculate the number of crimes per million people:

df_crime_states['crime_promln'] = df_crime_states['violent_crime'] * 1e6 / 
                                             df_crime_states['population']

Finally, we'll turn the data into a table spanning the 2000 — 2018 period transposing the state names and dropping the redundant columns:

df_crime_states_agg = df_crime_states.groupby(['state_name', 
                                 'year'])['violent_crime'].sum().unstack(level=1).T
df_crime_states_agg.fillna(0, inplace=True)
df_crime_states_agg = df_crime_states_agg.astype('uint32').loc[2000:2018, :]

The resulting table contains 19 rows (year observations from 2000 through 2018) and 51 columns (by the number of states).

Let's display the top 10 states for the average number of crimes:

df_crime_states_agg_top10 = df_crime_states_agg.describe().T.nlargest(10, 'mean').                                                                                                         astype('uint32')

We'll also make it more graphic with a box plot:

df_crime_states_top10 = df_crime_states_agg.loc[:, df_crime_states_agg_top10.index]
plt = df_crime_states_top10.plot.box(figsize=(12, 10))
plt.set_ylabel('Violent crime count (2000 - 2018)')

The 'Hollywood' state easily and notoriously beats the rest 9. The 'prizewinners' are California, Texas and Florida, all three in the South, the regular settings for most Hollywood criminal blockbusters.

You can also see that criminality has changed considerably over the observed period in some states (California, Florida and Illinois), whereas in others (like Georgia) it has remained almost constant.

I tend to think the crime rates are in some way connected with population :) Let's see the top 10 states by population in 2018:

df_crime_states_2018 = df_crime_states.loc[df_crime_states['year'] == 2018]
plt = df_crime_states_2018.nlargest(10, 'population').                                       sort_values(by='population').plot.barh(x='state_name', 
                                       y='population', legend=False, figsize=(10,5))
plt.set_xlabel('2018 Population')
plt.set_ylabel('')

Same old mugs here :) Let's check the correlation between crimes and population:

df_corr = df_crime_states[df_crime_states['year']>=2000].groupby(['state_name']).mean()
df_corr = df_corr.loc[:, ['population', 'violent_crime']]
df_corr.corr(method='pearson').at['population', 'violent_crime']

The calculated Pearson correlation coefficient is 0.98. Q.E.D.

But the per capita crime counts give a staringly different picture:

plt = df_crime_states_2018.nlargest(10, 'crime_promln').                                sort_values(by='crime_promln').plot.barh(x='state_name', 
                                y='crime_promln', legend=False, figsize=(10,5))
plt.set_xlabel('Number of violent crimes per 1 mln. population (2018)')
plt.set_ylabel('')

There's a pretty kettle of fish! The leaders by per capita crimes are the least populated states: District Columbia (with the US capital) and Alaska (both home to some 700+ thousand people as of 2018), as well as one medium-populated state — New Mexico, with 2 mln. people. Only one state from our previous toplist is featured here — Tennessee, which gives this state a less-than-desirable reputation.

We will then display these results on the US map. To do this, we need the folium library:

import folium

First, the 2018 absolute crime counts:

FOLIUM_URL = 'https://raw.githubusercontent.com/python-visualization/folium/master/examples/data'
FOLIUM_US_MAP = f'{FOLIUM_URL}/us-states.json'

m = folium.Map(location=[48, -102], zoom_start=3)

folium.Choropleth(
    geo_data=FOLIUM_US_MAP,
    name='choropleth',
    data=df_crime_states_2018,
    columns=['state_abbr', 'violent_crime'],
    key_on='feature.id',
    fill_color='YlOrRd',
    fill_opacity=0.7,
    line_opacity=0.2,
    legend_name='Violent crimes in 2018',
    bins=df_crime_states_2018['violent_crime'].quantile(
                                            list(np.linspace(0.0, 1.0, 5))).to_list(),
    reset=True
).add_to(m)

folium.LayerControl().add_to(m)

m

The same in per capita values (per 1 million):

m = folium.Map(location=[48, -102], zoom_start=3)

folium.Choropleth(
    geo_data=FOLIUM_US_MAP,
    name='choropleth',
    data=df_crime_states_2018,
    columns=['state_abbr', 'crime_promln'],
    key_on='feature.id',
    fill_color='YlOrRd',
    fill_opacity=0.7,
    line_opacity=0.2,
    legend_name='Violent crimes in 2018 (per 1 mln. population)',
    bins=df_crime_states_2018['crime_promln'].quantile(
                                             list(np.linspace(0.0, 1.0, 5))).to_list(),
    reset=True
).add_to(m)

folium.LayerControl().add_to(m)

m

In the first case, as we can see, crimes are more or less evenly distributed in the North to South direction. In the second case, it's mostly the Southern states plus DC and Alaska that make the trend.

Lethal Force Fatalities Across States (No Racial Factor)

We are now going to look at lethal force used in individual states across the country.

To prepare the dataset, we'll complement the UOF (Use Of Force) data we used previously by the full state names, group the cases by states, and constrain the observations to years 2000 through 2018:

df_fenc_agg_states = df_fenc.merge(df_state_names, how='inner', 
                                         left_on='State', right_on='state_abbr')
df_fenc_agg_states.fillna(0, inplace=True)
df_fenc_agg_states = df_fenc_agg_states.rename(columns={'state_name_x': 'State Name'})
df_fenc_agg_states = df_fenc_agg_states.loc[:, ['Year', 'Race', 'State', 
                                                'State Name', 'Cause', 'UOF']]

df_fenc_agg_states = df_fenc_agg_states.                                groupby(['Year', 'State Name', 'State'])['UOF'].                                count().unstack(level=0)
df_fenc_agg_states.fillna(0, inplace=True)
df_fenc_agg_states = df_fenc_agg_states.astype('uint16').loc[:, :2018]
df_fenc_agg_states = df_fenc_agg_states.reset_index()

Top 10 states for police victims in 2018:

df_fenc_agg_states_2018 = df_fenc_agg_states.loc[:, ['State Name', 2018]]
plt = df_fenc_agg_states_2018.nlargest(10, 2018).sort_values(2018).plot.barh(
                                   x='State Name', y=2018, legend=False, figsize=(10,5))
plt.set_xlabel('Number of UOF victims in 2018')
plt.set_ylabel('')

Let's also review the data for the entire period as a box plot:

fenc_top10 = df_fenc_agg_states.loc[df_fenc_agg_states['State Name'].            isin(df_fenc_agg_states_2018.nlargest(10, 2018)['State Name'])]
fenc_top10 = fenc_top10.T
fenc_top10.columns = fenc_top10.loc['State Name', :]
fenc_top10 = fenc_top10.reset_index().loc[2:, :].set_index('Year')
df_sorted = fenc_top10.mean().sort_values(ascending=False)
fenc_top10 = fenc_top10.loc[:, df_sorted.index]

plt = fenc_top10.plot.box(figsize=(12, 6))
plt.set_ylabel('Number of UOF victims (2000 - 2018)')

Yep! The same 'unholy trio' of California, Texas and Florida, with their other two Southern sidekicks — Arizona and Georgia. The leaders again show large scatter indicative of year-to-year changes.

Connection Between Lethal Force Fatalities and Crimes

As in the previous part of this research, we are investigating the possible connection between criminality and deaths at the hands of law enforcement. We'll start without the racial factor, to see if such a connection exists in principle and how it varies from state to state.

At first, we must merge the UOF and (violent) crime datasets, setting the observation period to 2000 — 2018:

# add full state names
df_fenc_crime_states = df_fenc.merge(df_state_names, how='inner', 
                                   left_on='State', right_on='state_abbr')
# rename some columns
df_fenc_crime_states = df_fenc_crime_states.rename(columns={'Year': 'year', 
                                   'state_name_x': 'state_name'})
# truncate period to 2000-2018
df_fenc_crime_states = df_fenc_crime_states[df_fenc_crime_states['year'].between(2000, 
                                                                            2018)]
# group by year and state
df_fenc_crime_states = df_fenc_crime_states.groupby(['year', 'state_name'])['UOF'].                                  count().reset_index()
# join with crime data
df_fenc_crime_states = df_fenc_crime_states.merge(df_crime_states[df_crime_states['year'].                             between(2000, 2018)], how='outer', on=['year', 'state_name'])
# set missing data to zero
df_fenc_crime_states.fillna({'UOF': 0}, inplace=True)
# unify data types
df_fenc_crime_states = df_fenc_crime_states.astype({'year': 'uint16', 'UOF': 'uint16', 
                                  'population': 'uint32', 'violent_crime': 'uint32'})
# sort data
df_fenc_crime_states = df_fenc_crime_states.sort_values(by=['year', 'state_name'])

As you will remember, the UOF column contains the number of deaths from encounters with law enforcement officers (who I sometimes call here just 'the police', but who include, of course, other agencies such as the FBI) where lethal force was used intentionally.

We will also make a separate dataset with year-average values:

df_fenc_crime_states_agg = df_fenc_crime_states.groupby(['state_name']).                            mean().loc[:, ['UOF', 'violent_crime']]

Now let's look at the year averages for crimes and lethal force fatalities for all the 51 states on one plot:

plt = df_fenc_crime_states_agg['violent_crime'].plot.bar(legend=True, figsize=(15,5))
plt.set_ylabel('Number of violent crimes (year average)')
plt2 = df_fenc_crime_states_agg['UOF'].plot(secondary_y=True, style='g', legend=True)
plt2.set_ylabel('Number of UOF victims (year average)', rotation=90)
plt2.set_xlabel('')
plt.set_xlabel('')
plt.set_xticklabels(df_fenc_crime_states_agg.index, rotation='vertical')

Looking closely at this combined chart, one can see the following:

• the connection between crime and use of force is plainly trackable: the green UOF curve tends to repeat the shape of the crime bars
• the more criminal states (such as Florida, Illinois, Michigan, New York and Texas) evince proportionately less use of force compared to the less criminal states

Let's also make a scatterplot:

plt = df_fenc_crime_states_agg.plot.scatter(x='violent_crime', y='UOF')
plt.set_xlabel('Number of violent crimes (year average)')
plt.set_ylabel('Number of UOF victims (year average)')

Here it becomes conspicuous that the ratio between crime and use of lethal force is affected by the crime rate. Speaking crudely, in states with the number of violent crimes below 75k the number of police victims grows more slowly; whereas in the states with the crime count above 75k this growth is quite steep. This latter group includes, as we can see, only four states. Let's look them 'in the face':

df_fenc_crime_states_agg[df_fenc_crime_states_agg['violent_crime'] > 75000]

Will you be surprised? We've got the same 'four horsemen of the Apocalypse': California, Florida, Texas and New York.

Correspondingly, let's calculate the correlation coefficients between our data for three cases:

1. states with the year average crime count up to 75,000
2. states with the year average crime count above 75,000
3. all the states

For the first case:

df_fenc_crime_states_agg[df_fenc_crime_states_agg['violent_crime'] <=                   75000].corr(method='pearson').at['UOF', 'violent_crime']

— we obtain 0.839 as the correlation coefficient. This is a statistically valid value, although it doesn't reach 0.9 due to scatter across the 47 states.

For the first case:

df_fenc_crime_states_agg[df_fenc_crime_states_agg['violent_crime'] >                   75000].corr(method='pearson').at['UOF', 'violent_crime']

— we get 0.999 — an ideal correlation!

For the last case (all states):

df_fenc_crime_states_agg.corr(method='pearson').at['UOF', 'violent_crime']

— the correlation is estimated at 0.935. This overall correlation may be considered very good.

Let's now look at the geographical distribution of our 'offender shootdown' index (the term is coined here for brevity). As before, we divide the number of lethal force fatalities by the number of crimes:

df_fenc_crime_states_agg['uof_by_crime'] = df_fenc_crime_states_agg['UOF'] / 
                                  df_fenc_crime_states_agg['violent_crime']
plt = df_fenc_crime_states_agg.loc[:, 'uof_by_crime'].sort_values(ascending=False).                                  plot.bar(figsize=(15,5))
plt.set_xlabel('')
plt.set_ylabel('Ratio of UOF victims to number of violent crimes')

It is interesting to observe that our erstwhile leaders have shifted toward the center or even the rightmost end of the chart, which must mean that the most criminal states don't have the most 'bloodthirsty' police (towards real or potential offenders).

Intermediate conclusions:

1. The number of violent crimes is directly proportionate to population (good call, Captain Obvious!)
2. The most populated states (California, Florida, Texas and New York) are also the most criminal, in absolute values.
3. In per capita values, Southern states are more criminal than Northern states, with the exception of Alaska and District Columbia.
4. Lethal force deaths are correlated to criminality with an average coefficient of 0.93 across all the states. This correlation reaches almost unity (strictly linear) for the most criminal states and only 0.84 for the rest.

Racial Factor in Criminality and Lethal Force Fatalities Across States

Proving that crime rates do affect police victim rates, let's add the racial factor and see what it affects. As I explained above, we'll be using the arrest data for this purpose as being the most complete and covering the main offenses for all the states. There is, of course, no such state or country where one could equate the number of committed crimes to the number of arrests; yet these parameters are closely related. As such, we can do very well with arrest data for our statistical analysis. And, as we already agreed, only violent offenses (murder, rape, robbery, aggravated assault) will be taken into account.

Let's load the source data from the CSV file and routinely add the full state names:

ARRESTS_FILE = ROOT_FOLDER + '\\arrests_by_state_race.csv'
# arrests of Blacks and Whites only
df_arrests = pd.read_csv(ARRESTS_FILE, sep=';', header=0, 
                      usecols=['data_year', 'state', 'white', 'black'])
# sum the four offenses and group by states
df_arrests = df_arrests.groupby(['data_year', 'state']).sum().reset_index()
# add state names
df_arrests = df_arrests.merge(df_state_names, left_on='state', right_on='state_abbr')
# rename / remove columns
df_arrests = df_arrests.rename(columns={'data_year': 'year'}).drop(columns='state_abbr')
# peek at the result
df_arrests.head()

We'll also create a dataframe with year average values:

df_arrests_agg = df_arrests.groupby(['state_name']).mean().drop(columns='year')

Looking at this table, one can't overlook some oddities. In some states the arrest counts reach hundreds and thousands, while in others — only dozens or fewer. That's the case with Florida, one of the most populated states: it counts only 19 arrests per year (12 Blacks and 7 Whites). Surely, some data is missing here; let's check:

df_arrests[df_arrests['state'] == 'FL']

And indeed we see that data for Florida is available only for 2017. Well, we'll have to put up with this, I suppose. All the other states have complete data. But the ten / hundred-fold difference should be accounted for by population. Let's add population-by-race data and have a look.

The population data was taken from the US Census Bureau website (which is for some reason not accessible in Russia). You can download the prepared CSV file with 2010 — 2019 data from here.

Unfortunately, no state population data exist for prior periods (2000 — 2009). We have therefore to narrow down our observation period to 9 years (from 2010 through 2018) for this part of the research.

POP_STATES_FILES = ROOT_FOLDER + '\\us_pop_states_race_2010-2019.csv'
df_pop_states = pd.read_csv(POP_STATES_FILES, sep=';', header=0)
# the source CSV has a specific format, so some trickery is required :)
df_pop_states = df_pop_states.melt('state_name', var_name='r_year', value_name='pop')
df_pop_states['race'] = df_pop_states['r_year'].str[0]
df_pop_states['year'] = df_pop_states['r_year'].str[2:].astype('uint16')
df_pop_states.drop(columns='r_year', inplace=True)
df_pop_states = df_pop_states[df_pop_states['year'].between(2000, 2018)]
df_pop_states = df_pop_states.groupby(['state_name', 'year', 'race']).sum().                                         unstack().reset_index()
df_pop_states.columns = ['state_name', 'year', 'black_pop', 'white_pop']

Merging this data with the arrests dataset, we can calculate the per-million arrest counts:

df_arrests_2010_2018 = df_arrests.merge(df_pop_states, how='inner', 
                                             on=['year', 'state_name'])
df_arrests_2010_2018['white_arrests_promln'] = df_arrests_2010_2018['white'] * 1e6 / 
                                             df_arrests_2010_2018['white_pop']
df_arrests_2010_2018['black_arrests_promln'] = df_arrests_2010_2018['black'] * 1e6 / 
                                             df_arrests_2010_2018['black_pop']

And again let's calculate the year averages:

df_arrests_2010_2018_agg = df_arrests_2010_2018.groupby(
         ['state_name', 'state']).mean().drop(columns='year').reset_index()
df_arrests_2010_2018_agg = df_arrests_2010_2018_agg.set_index('state_name')

Let's visualize this stuff.

1. Absolute arrest counts

plt = df_arrests_2010_2018_agg[['white', 'black']].sort_index(ascending=False).                   plot.barh(color=['g', 'olive'], figsize=(10, 20))
plt.set_ylabel('')
plt.set_xlabel('Year-average arrest count (2010-2018)')

2. Arrest counts per million population (for each race)

plt = df_arrests_2010_2018_agg[['white_arrests_promln', 'black_arrests_promln']].          sort_index(ascending=False).plot.barh(color=['g', 'olive'], figsize=(10, 20))
plt.set_ylabel('')
plt.set_xlabel('Year-average arrest count per 1 mln. within race (2010-2018)')

What can we infer from this data?

First of all, we see that the number of arrests is affected by population — this is observed for both races.

Secondly, Whites get busted somewhat more often than Blacks in absolute figures. The 'somewhat' — because this rule isn't universal for all the states (exclusions are North Carolina, Georgia, Louisiana, etc.); at the same time, the difference is but slight in most states, except a few (like California, Texas, Colorado, Massachusetts and a few others).

Last but not least, Blacks get arrested much more often in all the states in per capita values.

Let's back these observations by numbers.

Difference between the average White and Black arrest counts:

df_arrests_2010_2018['white'].mean() / df_arrests_2010_2018['black'].mean()

— we get 1.56. That is, the observed 9 years saw on average one and a half times more Whites being arrested than Blacks.

Then in per capita values:

df_arrests_2010_2018['white_arrests_promln'].mean() / 
                                     df_arrests_2010_2018['black_arrests_promln'].mean()

— the ratio is 0.183. That is, a Black person is on average 5.5 times more likely to get arrested than a White person.

Thus, the previous conclusion of higher criminality among Blacks (compared to Whites) is confirmed by the arrest data for all the states of the USA.

To understand how race and criminality are connected with lethal force victims, let's merge the two datasets.

First, we prepare the use-of-force data with the victims' race details:

df_fenc_agg_states1 = df_fenc.merge(df_state_names, how='inner', 
                                          left_on='State', right_on='state_abbr')
df_fenc_agg_states1.fillna(0, inplace=True)
df_fenc_agg_states1 = df_fenc_agg_states1.rename(columns={
                                          'state_name_x': 'state_name', 'Year': 'year'})
df_fenc_agg_states1 = df_fenc_agg_states1.loc[df_fenc_agg_states1['year'].                                  between(2000, 2018), ['year', 'Race', 'state_name', 'UOF']]
df_fenc_agg_states1 = df_fenc_agg_states1.groupby(['year', 'state_name', 'Race'])['UOF'].                                    count().unstack().reset_index()
df_fenc_agg_states1 = df_fenc_agg_states1.rename(columns={
                                   'Black': 'black_uof', 'White': 'white_uof'})
df_fenc_agg_states1 = df_fenc_agg_states1.fillna(0).astype({
                                    'black_uof': 'uint32', 'white_uof': 'uint32'})

Then we're merging it with the arrest data:

df_arrests_fenc = df_arrests.merge(df_fenc_agg_states1, 
                    on=['state_name', 'year'])
df_arrests_fenc = df_arrests_fenc.rename(columns={
                   'white': 'white_arrests', 'black': 'black_arrests'})

OK, time to calculate the correlation coefficients between arrests and lethal force fatalities, as we did before:

df_corr = df_arrests_fenc.loc[:, ['white_arrests', 'black_arrests', 
                      'white_uof', 'black_uof']].corr(method='pearson').iloc[:2, 2:]
df_corr.style.background_gradient(cmap='PuBu')

Again we've produced quite good correlations: 0.87 for Whites and 0.77 for Blacks. It's curious that these values are very close to those we obtained for All Offenses in the previous part of the article (0.88 for Whites and 0.72 for Blacks).

What about our 'offender shootdown' index? Let's check:

df_arrests_fenc['white_uof_by_arr'] = df_arrests_fenc['white_uof'] / 
                            df_arrests_fenc['white_arrests']
df_arrests_fenc['black_uof_by_arr'] = df_arrests_fenc['black_uof'] / 
                            df_arrests_fenc['black_arrests']
df_arrests_fenc.replace([np.inf, -np.inf], np.nan, inplace=True)
df_arrests_fenc.fillna({'white_uof_by_arr': 0, 'black_uof_by_arr': 0}, inplace=True)

To see how this index is distributed geographically, let's take the 2018 data point:

plt = df_arrests_fenc.loc[df_arrests_fenc['year'] == 2018, 
                             ['state_name', 'white_uof_by_arr', 'black_uof_by_arr']].                             sort_values(by='state_name', ascending=False).                             plot.barh(x='state_name', color=['g', 'olive'], figsize=(10, 20))
plt.set_ylabel('')
plt.set_xlabel('Ratio of UOF victims to violent crimes (2018)')

The index for Whites is greater in most states, with some exclusions (Utah, West Virginia, Kansas, Idaho, and District Columbia).

Let's compare the values for Whites and Blacks averaged for all the states:

plt = df_arrests_fenc.loc[:, ['white_uof_by_arr', 'black_uof_by_arr']].                                   mean().plot.bar(color=['g', 'olive'])
plt.set_ylabel('Ratio of UOF victims to violent crimes (2018)')
plt.set_xticklabels(['White', 'Black'], rotation=0)

The index is 2.5 times greater for Whites than for Blacks. If this index really says something, it means that a White criminal is on average 2.5 times more likely to meet death from the police than a Black criminal. Of course, this index varies much from state to state: for example, in Idaho a Black criminal is twice as likely to become a law enforcement victim, whereas in Mississippi — four times less likely.

Well, that's it really. Time to summarize our research.

Conclusions

1. In the US, criminality is a function of population. The most 'criminal' states that we are used to watching movies or read about are simply the most populated. When analyzing per capita crime rates, the top positions are taken by some quite unexpected states like Alaska, District Columbia (with Washington City) and New Mexico.
2. Southern states are on average more criminal than Northern states (in per capita crime values).
3. Per capita crimes and arrests are unevenly distributed among the US white and black populations: black persons commit 3 times more crimes and are 5 times more often arrested than white persons.
4. A black person is on average 2.5 times more likely to get killed in an encounter with law enforcement than a white person.
5. Lethal force fatalities correlate well with criminality: the higher the crime rate, the more people get killed by the police. This correlation holds true for most states and for both races, although it is somewhat more pronounced among the white population. This is also confirmed by the difference in the victim-to-crime ratio between the races: white criminals are more likely to get killed by the police.

As a final word, I'd like to say thanks to my readers for their valuable comments and advice.

P.S. In a future (separate) article I am planning to continue analyzing crime and its connection with race in the US. We can first look into hate crimes and then discuss the law enforcement / offender interfaces from a reversed point of view, investigating line-of-duty fatalities among US police officers. I'd appreciate if you let me know in the comments if this subject is of interest.