Merkel managed to revive her fortunes over the last two months with a victory in Saarland on March 26 and an upset defeat of the ruling SPD in Schleswig-Holstein a week ago.
"We are going very, very confidently with a tailwind into the Bundestag election", said Michael Grosse-Broemer, a national parliamentary leader of Merkel's Christian Democrats.
"The CDU has won the heartland of the Social Democrats", said the Christian Democrats' general secretary, Peter Tauber, calling it a great day.
It was seen as a test for Mrs Merkel ahead of September's general election. The two parties now rule together nationally in a "grand coalition".
In 2005-2009, Merkel also ruled in a grand right-left coalition with the SPD, while in 2009-2013, she relied on the liberal Free Democratic Party (FDP) for running another coalition Cabinet.
Given the abysmal loss, SPD officials will spend the coming days analyzing the mistakes made in North Rhine-Westphalia - especially given the previous two crushing regional defeats.
According to initial projections made by German local media ZDF and ARD, CDU secures 34.3 to 34.4 percent of the votes while the Social Democratic Party (SPD) lag behind, getting 30.6 percent of the votes.
The shock result comes four months before the German federal election, with polls showing the Social Democrats trailing CDU by up to 10 points after drawing level earlier this year.
The pro-business Free Democrats won a strong 12.6 percent of the vote Sunday after a campaign headed by their national leader, Christian Lindner.
The populist, anti-immigrant AfD party will now enter the NRW parliament, meaning the party is now in 13 out of Germany's 16 state parliaments.
Social Democrat governor Hannelore Kraft's coalition lost its majority as her junior governing partners, the Greens, took only 6.4%.
Voters were choosing candidates for the state legislature, which may now change from its current SPD-Greens coalition. Other parties gain in the amount of 3.3%.
The SPD has run the state for most of the post-war period.
The opinion polls were the latest indication that initial enthusiasm for the new SPD leader, Martin Schulz, could be fizzling out.
Kraft said she took "personal responsibility for this defeat" before announcing her decision to step down.
"I gave it my best".
"This was a committed election, but it wasn't enough".
She had urged voters to look at her government's economic record, noting that with 7.5% unemployment, the state fares worse than the national rate of 5.8%.
Martin Schulz, leader of the Social Democratic Party (SPD), resorted to his comfort zone as former president of the European parliament when asked today about his plans to revive his faltering election campaign.
"This is a hard day for the Social Democrats, a hard day for me personally as well", Schulz, who wasn't on the ballot Sunday, told reporters in Berlin.
The conservatives' improbable come-from-behind triumph in heavily industrial North Rhine-Westphalia, long a bastion for the Social Democratic party and home to almost a quarter of Germany's voters, gave the chancellor a powerful gust of tailwind just four months before the federal election on September 24.
During that period, the conservative CDU was in power only in 2005-2010, leading commentators to declare its Sunday's win historic.