The Fire (2-3-6) had a two-game winning streak stopped. Columbus (4-4-4) ended a 0-4-4 stretch with its first win since March 29.
Forward Federico Higuain assisted on the goal by Finlay in the 10th minute and again on Arrieta's 15 minutes later with leading balls that left the Fire's backline exposed.
Finlay ran onto the pass down the right side as Fire defender Greg Cochrane pursued. Goalkeeper Sean Johnson charged toward the edge of the box but avoided making contact with Finlay and a possible red card.
That's when Finlay continued his dribble toward the end line before putting the ball out of the reach of a sliding Cochrane.
Finlay has scored in two straight games after going scoreless in his first 40 MLS games, to support the four saves by Steve Clark, who had his second shutout of the season.
The second goal was scored in similar fashion on the left side as Higuain got the ball in space to Arrieta. He slipped past Fire defender Patrick Ianni and once again Johnson came off his line only to see Arrieta slot the ball to the far netting for his third goal of the season.
The Fire had several chances in the first half, most notably a header off the post by Juan Luis Anangano in the 27th minute that would have cut the two-goal deficit in half.
Anangano started at forward for injured forward Mike Magee, who is second on the team with four goals.
The Fire was also without defender Lovel Palmer and midfielder Patrick Nyarko because of injuries.
Columbus defender and captain Michael Parkhurst, who was among the final seven cuts Wednesday for the U.S. World Cup roster, came off the bench in the 87th minute.
NOTES: Fire technical director Brian Bliss missed a return to Columbus because he is overseeing the U.S. Men's Under 20 team training camp. Bliss is in his first season with the Fire after spending the previous six with Columbus. ... Former Crew and Fire forward Brian McBride is on a short-term coaching assignment helping the Chicago Fire trikers. He is second all-time in Crew goals with 62.