How do you eat an elephant? One bite at a time.
In this allegory we will look at 20 million deer hunters as the elephant.
Let us assume that there are 20 Million American deer hunters. Let us further assume that they are relatively evenly spread across all 50 states (not true, but it will make this logic easier to follow). This means that we are looking at around 400,000 per state. In another calculation, 20 million hunters divided by 3.7 million square miles is around 5 hunters per square mile.
How does the military fight? Do we deploy individuals and spread them evenly over terrain? No, we deploy units and put them into advantageous terrain to dominate the battlespace.
That is how a small number of Soldiers can control a larger, relatively well armed, population (every household in Iraq is authorized a fully automatic AK-47, which did them no good relative to the coalition forces).
Doesn't matter how well armed you are if you are starving, or can't get to where the fight is. Doesn't matter how well you can shoot if the enemy controls your supply lines and mobility corridors.
In the big picture of a campaign, a battle, or a war, individual marksmanship doesn't even register on the scale of importance.
If I were to tap into 20 million deer hunters to become a field Army, the old Minutemen model would have to be severely revamped. Training a modern Minuteman would end up looking a lot like a National Guardsman than a big game hunter.