Viking mariners largely navigated by a mix of their senses and practical knowledge. This gave them an intuitive sense of where they were on their mental map.
Using their senses, they would note navigation marks such as the highest hills or an unusually-shaped rock.
They could see whales feeding in certain currents, and experienced mariners could hear birds calling, waves breaking on shore or rocks, and could taste if fresh water was flowing into the sea.
They could smell land in a sea breeze and feel the prevailing wind on their skin, as even modern, experienced sailors can. By looking at birds and the colour of the sea, they could tell if land was close.
When the night sky was clear, Viking mariners would use the stars and constellations.
The Vikings probably also used a sun compass, which always shows the correct direction. A sun compass comprises a vertical pointer on a horizontal surface, on which the shadow of the pointer is drawn through the day. This shadow curve is different at different latitudes and at different times of the year, so in order to use it for navigation, a series of curves is necessary.
When the sun is obscured by clouds even the sun compass has its limits. But we know, thanks to the sagas, that so-called “sun stones”, which could determine the direction of the sun even under cloud cover, were also used. There is evidence to suggest these consisted of a particular mineral which polarises light.