Neil Oliver: Governments amount to hundreds, we amount to millions - they are few and we are many