Hai is not “yes”. It means “I am with you”, and Japanese speakers use it while listening, several times a sentence, without agreeing to anything. Answering a question with hai when you meant no is the most common misunderstanding a visitor causes without noticing.

Refusing is the other half. A flat iie is rare; the normal shapes are chotto… trailing off, or daijoubu desu, which means “I am fine, thank you” and is how you decline a bag at a till. Saying you did not understand — wakarimasen — is not rude and saves everyone time.