The Taliban on Tuesday broke off talks with the Afghan government on a prisoner exchange, a main step in peace talks being brokered by the United States after it agreed on a troop withdrawal pact with the militants.