It was just something he could sense, not just from fans and reporters but from the people he was around every day.
"Even your teammates, they may not say anything to you but I kind of felt like they were thinking, 'You're a first-round draft pick. You should be out here doing this. You should be out here killing it,' " Smith said. "But it just wasn't the path I was on at the time."
Smith's path since the Ravens drafted him 27th overall in 2011 has been bumpy. Over his first two seasons, injuries forced him to the sideline. Inconsistent play kept him out of the starting lineup. But as the Ravens prepare to face the Buffalo Bills on Sunday afternoon at Ralph Wilson Stadium, Smith has finally found smoother terrain.
The 25-year-old is healthy for one of the first times in his NFL career. He's entrenched as a starter, the team's No. 2 cornerback opposite Lardarius Webb. He's also recommitted to the physical, press-coverage style that originally made him a top NFL prospect.
"Coming into my third year, I was like, 'Well, if I'm going to go out of the league, I'm going to go out the way I want to go out and play the game that I want to play,' " said Smith who has broken up four passes and made nine tackles over the past two weeks for a defense that has held consecutive opponents without a touchdown. "I'm a long, tall, fast guy. If I'm playing off, I might as well be like every 5-foot-9 corner. To play physical and to be up on the line and be able to press guys with my length and my speed, I just think that's a bigger advantage."
Smith hasn't been perfect, but over the past two weeks, he's stood up to frequent tests, stringing together two of the better games of his career. On Sunday, he'll have to do it all over again against Stevie Johnson, the Bills' top receiving threat.
By now, Smith understands that's the life of an NFL cornerback. Every week is another challenge. Every play is another test. Smith is comfortable with that pressure but there was a time not long ago where it left him questioning himself.
"A lot of things your rookie season get thrown at you as a corner — a lot of things that you've never seen," said Smith who has started eight games in his career and has two interceptions. "You get beat on a big play and you lose a little confidence. The team loses a little confidence in you. You can tank or you figure out what you are going to do. For me, a big play got caught on me here, a big play there. I'd forget about it until after the game and then I'd hear things. It was like, 'OK, maybe, I'm not as good as I thought.' But as I kept progressing, I just started honing in on what I needed to do."
At his best, Smith uses his size (6-foot-2, 200 pounds) to jam wide receivers at the line of scrimmage and knock them off their routes. On the defining play of his career so far, Smith muscled San Francisco 49ers wide receiver Michael Crabtree, preventing him from making a touchdown catch and finalizing the Ravens' goal-line stand in Super Bowl XLVII.
At his worst, he's prone to penalties and slow to react to the ball, and he gives receivers far too much space.
"The thing he really had to keep working on was just being consistent," Ravens defensive coordinator Dean Pees said. "When you're a big corner, that's quite a tool you have when you can play physical and get your hands on guys. Some you don't want to try to jam, because they're going to slip you at the line and get on top of you and build speed. You've just got to know who you're playing against. That's one of his talents, being physical, because he is big [and] he's got long arms."
Even at the height of his struggles, Pees and coach John Harbaugh have defended Smith, knowing how difficult it is for young cornerbacks to succeed in the NFL. From 2009 to 2012, thirteen cornerbacks have been selected in the first round of the draft, and just two of them — the New England Patriots' Devin McCourty and the Arizona Cardinals' Patrick Peterson — have made the Pro Bowl.
"I can't think of one corner that I've ever seen come into the league their first year and play at a Pro Bowl-status," Smith said. "You don't see the same schemes and you don't know the game that well when you first come up. It takes a little time to adjust and understand the game. It's a huge jump, actually. In college, it was very simple. When I got up here, they come at you consistently and nonstop. They kind of found the weaknesses in my game that I had to try and make better."
Smith acknowledged that he faced a particularly tough adjustment given his background at the University of Colorado. Smith was one of the best players on a struggling team and the opposition rarely threw his way or even dug too deep in its playbook.
When he got to the NFL, he quickly realized he could no longer just rely on his size and speed. He had to learn the nuances of the position and work on being disciplined with his eyes and his techniques.
"I think a guy has to look himself in the mirror and be humbled a little bit and understand that, 'Man, I have to up my game. I have to go to another level,' " said Dan Hawkins, Smith's head coach at Colorado. "Jimmy was always kind of on that path. He was always very open, always very honest and he was a guy that was willing to learn and communicate with his coaches. It doesn't surprise me that he's improving."
Smith doesn't want to be singled out, suggesting his improved play is a result of the defensive backs jelling rather than anything he's done. However, his teammates and coaches have recognized some of the changes he's made.
He dropped about 10 pounds in the offseason and took boxing classes to strengthen his core and avoid some of the nagging injuries that have affected him in the past. Off the practice field, Webb said his teammate is "taking the game more serious," while Corey Graham, whose starting job Smith now holds, said that Smith is "understanding more about concepts and playing smart football.
Smith, meanwhile, knows better than to get too comfortable. He understands he's just one bad game away from the narrative shifting, from again being labeled a first-round bust. It's not something he thinks about or hears. But if it happens, he'll undoubtedly sense it.
"I'd be lying if I said I didn't kind of plan on it being this way," Smith said. "This is the same path that I had in college. My first couple of years, I didn't play. My third year, I came on and my fourth year is when I played really well and played my way into the first round. It's not a huge surprise to me. I knew I was going to have some ups and downs but when I'm out there playing, I'm doing pretty well."