If a builder ruins my property while working on it is he responsible for the cost of replacing it?

I had some building work done, on day one the builder pulled down the ceiling (as agreed) and black soot went everywhere. This soot sunk into two carpets and could not be hoovered up. He tried to clean one of the carpets with a vax and scrubbed it so hard he ruined the pile. I told him to leave the other carpet and get it professionally cleaned. The work is now finished and I am waiting for the carpets to be cleaned by a carpet cleaner to see if they can be restored but I fear the worse. I have not yet paid the builder for all the work, am I within my rights to withhold the money so I can use it to replace the carpets?