The man was taken to shore, where paramedics performed first aid, but his life could not be saved.
Authorities say he was bitten by a four-meter-long shark.
Rottnest Island is a nature reserve and popular tourist destination outside Perth. The island is best known for its beaches and the opportunity to see the marsupial quokka.
Australia has been hit by several shark attacks this year. In New South Wales, four people were attacked over a 48-hour period in January during torrential rains, killing a boy.
