URL with posts/pages slug in double-byte character results in 404

[iorippi]

I'm using Japanese language which tends to use the letters like "日本語" as in example.com/posts/日本語/, and that is causing 404 error both for posts and pages. I haven't tried in other CJK language or some others that can't be covered in ASCII characters like Arabic yet, so this report is pretty limited.

I didn't have this issue in simple vanilla Next.js implementation without any such special care, so I'm guessing string conversion is happening somewhere in the process?

I want to come back to this issue and see if I can suggest fix, but I'm still in learning process of Next.js, so I'm leaving this here as memo. (I'm pretty determined about using this in production for Japanese clients, so I'll come up with solution somehow anyways.)

[colbyfayock]

hey @iorippi interesting one! do you happen to have your wordpress instance public that i can spin it up? if you're still interested in actually fixing and putting together a PR, i'd be happy to at least try to point you in the right direction :)

if your WP isn't public, do i just simply copy those characters into the post title to reproduce?

[iorippi]

@colbyfayock Thanks for heads up! Here's URL for GraphQL API: https://cms.appy.hiup.dev/wp/graphql
There are several Posts and one of them has its slug in Japanese.

{
            "id": "cG9zdDo4",
            "slug": "%e3%81%aa%e3%81%9e%e3%81%aa%e3%81%9e%e3%81%a0%e3%82%88%ef%bc%81"
}

decodeURI() for the slug does actually return the string identical with the actual slug "なぞなぞだよ！", however, the result is "Not Found" on dev mode, and for deployed & exported static site, it returns "Not Found - The requested URL /posts/ã�ªã�žã�ªã�žã� よï¼�/ was not found on this server."

I get the seemingly identical string of gibberish when I convert UTF-8 string for the slug into ASCII, so I'm guessing that conversion is happening somewhere internally upon resolving the route??

edit: Just checked exported out/posts/ directory and directory name of correctly converted string was present.