Game of Thrones season 6: the White Walkers, explained

"Yet it's noteworthy that the Others have never been seen south of the Wall. And outside of their possible inability to swim explaining their failure to attack the ships at Hardhome, maybe they simply chose not to attack. Perhaps, from their point of view, they are not evil conquerors at all but a put-upon minority (maybe they are actually the Children of the Forest) simply defending their territory against human aggression. It's at least conceivable that the whole conflict is a result of an enormous misunderstanding of some kind."

read analysis by Matthew Yglesias

Game of Thrones season 6: the White Walkers, explained - Vox