The rfid can work trough fabric. So a shielded wallet is a smart choice. Technically someone can make you pay by tapping your pants with a terminal device. Does this happen allot? No not really. But technically possible and more likely when more people use tap.
Here there is a maximum amount that can be paid with only contactless payment without a PIN code. If the price is higher a pin prompt will show up automatically (max $25,-) If you pay more things in a short time with the rfid than the pin prompt will eventually show up. So an unlimited amount of low payments isn’t possible. (Do check your bank for these details.) So when you lose your card your account can’t be drained. (Unless you have less than the minimum amount on your account)
Phones can also be setup as a contactless payment method, and would physically work the same. With the benefits of having a stronger signal, and being locked behind your phone lock (facial recognition or whatever.)