Though they be under the external administration of the covenant, they were never truly part of the Covenant (not IN Christ).
Does John 15:2 teach that there are those in covenant (in Christ, in a non-saving sense!) who will be removed?: "Every branch in me that beareth not fruit he taketh away: and every branch that beareth fruit, he purgeth it, that it may bring forth more fruit." This seems to accord with passages like Hebrews 6:4, for instance.