Pryse's mother died when he was four and his father when he was 11. He and his four brothers and sisters were raised in dire poverty and he had a total of four days of formal education in his life. He worked first as a farm laborer and then as a weaver.
The fine quality of his weaving made him famous and he married and had seven children. At night he studied alone, teaching himself English, Latin, Greek and Welsh history and literature. In 1857 he joined the staff of the publishers Gee and worked on their Welsh encyclopedia and dictionaries, also contributing prolifically to Welsh journals. He won many prizes for his writing and poetry and published a number of books of history and five dictionaries.