If you have an iPhone, tap the three dots on the top-right of the Apple Wallet screen, click on the "Pass Details" button and then hit "Remove Card".
On Google Wallet, click on the three dots on the top-right of the screen, then tap the bin icon at the bottom left of the screen.