Usher and Tameka Foster. PHOTO:

JOHNNY NUNEZ/WIREIMAGE

Love in the limelight hasn’t always been easy for Usher.

The star, 45, reflects on that and more in this week’s PEOPLE cover story, all about his ups and down and happy life now as a newlywed and Super Bowl-approved superstar.

Back in the late ’00s Usher had fallen for stylist Tameka Foster, a few years after his fan-beloved relationship with TLC’s Rozonda “Chilli” Thomas has ended. But upon marrying Foster in 2007 he started to face backlash from fans who didn’t approve of the relationship.

“I think I became very closed and then felt attacked in many ways,” he says of that period. “It was a bit unconventional in a time when I felt like my female fans were judgmental of the woman that I chose.”

Usher and Tameka Foster at the 15th annual Trumpet Awards at the Bellagio in January 2007.

ETHAN MILLER/GETTY

He still isn’t sure what all of the public scrutiny over Foster, who’s seven years his senior, stemmed from, though he has his speculations.

“She was a dark-skinned woman, and maybe they didn’t want to love her in a way because of maybe some reason that had to do with themselves,” he hypothesizes, “I don’t know. Or maybe she was not the best reference because of how she handled herself. That’s not for me to be concerned about. I did my part and what I thought was right.”

Usher Reflects on the ‘Hurt’ of Stepson’s Death: ‘It’s a Real Hard Thing to Lose a Child’ (Exclusive)Usher for PEOPLE.

Around that time the couple sat for an in-depth interview with Essence where Usher reflected on how the backlash impacted their relationship. “Ours is not a typical love story, but it is a true one,” he told the publication at the time. “Tameka and I have been fortunate enough to go through the thick of it in the beginning. We’ve had that opportunity to huddle up as a team, to make sure that we’re clear and speak as one voice.”

The star followed his heart but in the end things didn’t work out between the pair, who divorced in 2009.

How Usher’s Soul Searching Solo ‘Journey’ Around the World Led to Love and His Super Bowl Triumph (Exclusive) Cinco, Usher and Naviyd in Hollywood in September 2016. AXELLE/BAUER-GRIFFIN/FILMMAGIC

“We ended up having two children together,” he says now of sons Usher V, 16, and Naviyd, 15. He and Foster struggled through a difficult custody battle years ago, but now, “We are cordial with each other,” he says. “We ain’t good, good, but we good. You know what I’m saying?”

Usher recently tied the knot with music executive Jenn Goicoechea, with whom he shares daughter Sovereign, 3, and son Sire, 2, following his hit Super Bowl performance. Foster is now starring on new WE tv reality series Bold & Bougie.

As for their relationship now, “It’s an 18 year process of which we all understand that, and we’re doing our best to raise two responsible, respectful and happy children.”

For more on Usher’s past and happy life now, pick up this week’s issue of PEOPLE, on newsstands everywhere Friday.