From rugby (“Rugby football”) + -er (Oxford “-er”). Compare contemporary soccer, from Association football. Both words seem to have originated at the University of Oxford during the 1880s. See Oxford -er.
Improve your pronunciation
Practice speaking and memorizing "rugger" and many other words and sentences in English.