I have opened a case for this recently and it was closed regarding the functionality of the IP Prefix Rule in the DNS Load Balancer > Load Balancing Rules > Location > IP Prefix Sets.

The notation under edit mode in the Load Balancing Rule > Client Selection for the "IP Prefix Sets" dropdown states the following: "Selects the clients for which the DNS Load Balancing Rule will be applied. The conditions are limited to the geo location, ASN, and IP prefix from where the client DNS request originates. This geo location, ASN, and IP prefix is derived from the EDNS0 client subnet"

When populated with IP's any IP is still able to resolve the record if resolving against the nameserver itself.

It would be a great asset to have if this worked as a source lookup control to allow only a specified source to resolve a record over the default set in the zone.