We need more EU workers, admits leading Tory Brexiter