The Kurdish SDF confirmed the withdrawal and blamed it on "the outside world's indifference to IS" and the international community's "failure" to take responsibility.
Thousands of relatives of IS members, mainly women and children, are in the al-Hol camp in northeastern Syria.
Earlier on Tuesday, negotiations between Syria's interim president Ahmad al-Sharaa and Mazloum Abdi, commander of the Kurdish-led SDF, were reported to have broken down.
This follows al-Sharaa's declaration of a ceasefire on Sunday after a major offensive against Kurdish strongholds in Syria. He said Kurdish forces would be included in the Syrian army and Kurdish-controlled areas would be handed over.
