"We don’t allow the EU to sell oil and gas licences in our territorial waters so why would we treat fishing access any differently?"