Ex-Foreign Office ministers and diplomats said the decision was petty and could set a bad precedent.
But the Foreign, Commonwealth and Development Office insisted EU delegation staff would still receive the privileges needed to do their job.
The government has been criticised for downgrading the diplomatic status of the EU's ambassador in London.