Directed from a script by Aaron Guzikowski, the movie slyly explores questions of revenge, parental responsibility, individualism and the suspicions that rattle around the human brain. Gyllenhaal's Loki swaggeringly tries to crack the difficult case of two missing girls on behalf of a contractor named Keller Dover (Hugh Jackman), whose daughter disappears along with a friend outside their small-town Pennsylvania home. Convinced Loki is not doing enough, Dover secretly kidnaps the man he believes abducted the girls (Paul Dano) and tortures him to find the truth. As Loki is thwarted by the case, he and Dover become mistrustful of each other.

Gyllenhaal plays the part with a mixture of sympathy and bravado; prominent tattoos and references to a tough childhood suggest a criminal past that further colors the character. The performance is rife with subtle contradictions. Loki is full of bravado about his ability to find the kidnapper but also is manifestly unsure of how to do it; he directs righteous rage toward those who get in his way even as he intimates he's hardly been righteous himself.

PHOTOS: Billion-dollar movie club

Those who worked with Gyllenhaal on the movie say the actor seems to be growing out of a boyishness that marked many of his roles.

"When I knew him years before he seemed like such a boy," said Maria Bello, who costars in "Prisoners" as Dover's wife. "And as soon as I saw him at the reading [of the 'Prisoners' script], I sensed a depth, an anger, a sense of experience, a sense of having lived more than I ever had sensed, a fight within himself that made his work so interesting."

In many scenes, he can also be seen doing something less expected: blinking obsessively, a tic that adds a layer of humanity and that only came about after he talked Villeneuve into letting him try it.

"I remember meeting Denis before we shot the movie, and I said, 'I think the character has some physical tic or attribute,'" Gyllenhaal said. "And I could see his reaction," the actor recalled, laughing. "Directors can have a sense of terror when you suggest something like that."

PHOTOS: Celebrities by The Times

That filmmaking instinct has been getting sharper lately. Many days on set of “Prisoners,” Gyllenhaal, Villeneuve and the film’s decorated cinematographer Roger Deakins would huddle and plan shots that weren’t in the script. “It was just the three of us, calling a play. You go deep, you cut over the middle, “ Gyllenhaal said.

Dano, who worked so intensively with Gyllenhaal that they once filmed a single-take 20-minute interrogation scene (about a minute of it can be seen in the film) which Gyllenhaal crafted from weeks of studying interrogation techniques, said he sees a kind of directorial impulse in the actor.

“On ‘Prisoners’ he was always conscious of what works not just for his character but for everyone in the scene, which is rare,” said Dano. (Indeed, “Night Crawler,” which will mark the directorial debut of “Bourne Legacy” writer Dan Gilroy, has Gyllenhaal halfway there, producing for the first time.)

On the set of "Prisoners," Gyllenhaal, Villeneuve and the film's decorated cinematographer Roger Deakins would huddle and plan shots that weren't in the script. "It was just the three of us, calling a play. You go deep, you cut over the middle," Gyllenhaal said.

Still, the actor-filmmaker segue can be a rocky one. What’s more, dark roles don’t always mean profitable ones—despite its acclaim, "End of Watch” was a modest box-office performer.

Besides his turn to moodier fare, he has been branching out in other ways. After living in Los Angeles for most of his life, he moved to New York a few years ago. It was a welcome change of scenery for someone who was born into Hollywood. His father, Stephen Gyllenhaal, is a prolific director, and his mother, Naomi Foner, has written scripts for numerous movies. His sister, the actress Maggie Gyllenhaal, is married to Peter Sarsgaard.

Jake Gyllenhaal began acting when he was 10, appearing in “City Slickers” as the son of Billy Crystal’s character. Roles in his father’s movies followed. New York, where his mother and sister live, offers a bit of a respite from the industry.

"I heard about the movie business before I even knew what it was," said the actor. "So I surround myself now with people who are like, 'Can we not talk about movies for an hour?'" That is true — to an extent. Gyllenhaal, who is not married, has been romantically linked to a number of high-profile entertainers over the years, including Reese Witherspoon, Kirsten Dunst and Taylor Swift.

He'll soon be returning to L.A. for a few months to shoot "Nightcrawler." At Toronto, the actor was also coming off a hectic Labor Day weekend packing up his New York apartment in preparation for the move. Sipping coffee to quell a stomach virus possibly brought on by his  filming schedule, he showed a sentimental side.

PHOTOS: Greatest box office flops

"If it was a choice given to me, I would have been at Rosh Hashanah," he said. "I would have been eating apple and honey with my nieces." Instead he watched a group of videos of his nieces--Maggie Gyllenhaal's daughters, ages 6 and 1--that his mom sent him.

Gyllenhaal has begun to lose weight for "Nightcrawler," trying to pick up on a theme of hunger, spiritual and physical, in the film. And he is ready to go a little bigger in "Everest," a survival-themed action movie in which he'll play the leader of a team of doomed explorers, because it contains some existentialist themes and will be directed by the Icelandic auteur Baltasar Kormakur.

"I had a boxing coach tell me something the other day: He said that as boxers, we create time. When you practice your combinations, time slows down and you have an infinite amount of time to throw a punch," the actor said. "And I want to connect to the projects emotionally, to have more time to throw a punch."

steve.zeitchik@latimes.com