Most of the early settlers of Woody Island came from England. Samuel Williams, who was born on Woody Island in 1901 said his great-grandfather Samuel came to Woody Island around 1820 with his brother. It was around this time also that other Englishmen were settling on the island. One of these was Robert Barrett, who became a very prominent figure in early Woody Island.

An old man named George Stainer also moved from England and settled on the island, he claimed to be 115 years old and he still had a strong British accent when he died.

The other two men who moved to Woody Island in the 1800s were Sam Crab and Thomas Russell, neither of whom was married. Both men were fishermen, then one of them drowned and the other moved away. It seems that they did not move from the island before 1880, as old Mrs. Collins, a resident of Woody Island born in 1832, met Sam Crab on a footpath in 1880 and she describes his outfit that day as "white duck overhauls." Courtneys, Smiths, Husseys, Allens, Marshalls, Pearces, and Lockyers also inhabited the island sometime in the 1800s.