If you own a phone number with an address requirement, you will not be able to delete that address while it is linked to the phone number.
To resolve this, you can add a new address that satisfies the requirements of the phone number, or you can release the phone number which requires the address.
You can see which phone numbers require the address by navigating to that address in the Twilio Console or by using the DependentPhoneNumbers resource in the API.