If you live in Japan, you have to bring your passport of recidence card with you at all time. The police can sometimes randomly ask you to show your card when you're out.
Otherwise you might be brought to the police office until someone can help you get you your card.