-
Don’t pay. He sounds like a nasty bully. Tell the Conservation Officer at your local council what has happened. They have enforcement powers. He will probably have to get consent to rebuild as it’s fallen over.
Be careful in erecting a wall yourself as you are on the curtilage of a listed building so you could get into trouble. Ask the Conservation Officer for advice here.