When someone is feeling threatened by you (and you are not threatening them) stop, be still, ask questions and help them explore how they are feeling. 

Often times defences incite defences. One person is defending themselves and the other person automatically starts building their own defences. I am attacked - Well no, now I am attacked because you think I am attacking you! 

Two walls make for a BIG misunderstanding and a conflict which might just last longer than you intend.  

When someone starts building a defensive wall with no need. Stop. Be still. Allow that person to fully explore what is happening for them - without judgement - full of compassion - seeking only clarity. 

You might gain clarity that an apology from you after all is warranted. They might find clarity that it was their own stuff colouring their perception of you. You might find a bit of both :)