Does my wallet card put customers' phone numbers in Google's hands?

No — the pass carries no name, number or address, and its update mechanism is built so wallet infrastructure never learns your customer list. Structural, not a setting.