In the 28th minute, Ellborg, signed from Malmö FF last winter, conceded the only goal in the round of 16 away match. Ben Waine then headed in the winning goal after the Swedish 22-year-old goalkeeper was left exposed when the ball bounced around in the penalty area.
Newcomer Port Vale are top of League One - the third division - with 27 points from 32 games played.
Ellborg made his debut for Sunderland in a 1–0 win over Leeds earlier this week after first-choice goalkeeper Robin Roefs was injured.
Fellow Premier League club Fulham have also been eliminated from the FA Cup after a 1-0 home loss to Southampton, who play in the second division.
