Muchova, the seventh seed, overcame 50 unforced errors across the two-hour-and-52-minute match to beat Osaka 7-5, 1-6, 7-6 (9/7).
Osaka, the current world number 17, surrendered a three-game lead in the first set after a series of misfired shots in the seventh game.
The 28-year-old who represents Japan dropped just one game to win the second set and saved eight match points in the third but could not get past Muchova's match-winning volley.
"I think she played always great and steady," the 30-year-old Czech player told the crowd.
"I was shaking. I'm happy I made the last volley... Naomi could be standing here. It's me, and I'm really grateful for it."
In the quarter-finals, she will play fellow Czech Nikola Bartunkova, who later defeated Austria's Sinja Kraus 6-0, 4-6, 6-3.
Earlier, Russia's Mirra Andreeva beat Ukraine's Daria Snigur 6-2, 6-2 to book her quarter-final spot against fellow Russian Ekaterina Alexandrova.
Alexandrova, ranked 28th, upset Wimbledon champion Linda Noskova 6-2, 6-2, ending the Czech player's bid to return to the Beijing final that she reached last year.