A major police operation was in place for Mr Erdogan’s visit to Cologne, which has a large Turkish community.
His three-day trip to Germany has been controversial, with the president making remarks criticising his hosts.
Germany is home to a 3 million-strong Turkish diaspora.
Both supporters and opponents of Mr Erdogan held gatherings in Cologne on Saturday.
But plans for up to 25,000 people to be allowed to gather outside the mosque were cancelled by city authorities cancelled over safety fears.
Cologne’s Central Mosque was built by an Islamic religious group with close ties to the Turkish state.
“We need to put aside our differences and focus on our common interests,” he said.
But the trip also highlighted differences on issues including Turkey’s crackdown after the failed coup attempt of 2016.
At a state banquet on Friday night the Turkish leader, departing from his prepared remarks, accused Germany of harbouring terrorists, those present told the Reuters news agency.
Germany’s best-selling newspaper Bild picked up on the remarks and was critical of Mr Erdogan.
It emblazoned its front page on Saturday with the words “Hate speech against Germany”.
On Friday, Mr Erdogan met Chancellor Angela Merkel for talks about the Syrian conflict.
But he used the occasion to urge Mrs Merkel to extradite critics of his government – known as Gulenists – that he sees as “terrorists”.
The chairman of the German parliament’s foreign affairs committee, Norbert Roettgen, told the Funke newspaper group that “the timing of this visit was wrong – it was far too early”.