Finding Video Games In Your Language

In a previous blog post I had complained not being able to find (or even search!) video games to play in Arabic. When recently I was searching for Arabic games again I had no luck, instead I found my own rant. So decided to take matters into my own hands.

The problem with stores like gog.com and Steampowered.com is simply that they don’t even allow you to search for games with Arabic language support. You can search for games in English, German, Romanian, Czech, … and others, but for some reason the choice is always arbitrarily limited, and excludes Arabic, among others.

So to take things into my own hands I decided to create a web crawler.

Finding Arabic Video Games

So as I said the search of the biggest game storefronts doesn’t allow you to search for it, but at least they the game’s detail pages list the supported languages. But going through the tens of thousands of games to find out by hand seemed quite arduous and little fun to me.

Enter the web crawler (also known as “spider”). That’s a program that is basically doing what Google’s search engine does: Automatically visiting countless websites and parsing their content to make it discoverable. Except this crawler has a vastly smaller scope and much easier search criterium.

When I was still in Uni I had already worked a bit on web crawlers to earn some money on the side. Back then it was a crawler for scraping real estate listings, but the idea is the same. Except back then I was using PHP, and nowadays I prefer using TypeScript and Node for these kinds of problems.

A quick search yielded various libs for web crawling and https://sdk.apify.com/ looked the most promising to me. Its API is simple yet it supports more advanced crawling techniques including rendering of the pages with headless browsers in cases where the raw HTML source is unhelpful (e.g. for Angular or other SPAs).

The basic idea is quite simple: Visit a storefront like https://www.gog.com/games and find all links to games on all pages. Then for each game find the name and whether it supports Arabic by looking at the source code of the page.

In case of GOG I was lucky since there is an API that I could use. The advantage of the API is that is returns structured data. So I don’t need to manually parse relevant data from the source using regex or selectors. Also this simpler than rendering in a headless browser if the site happens to be using Angular – where the HTML source is just a template that is filled in via JavaScript on the client side, like in the case of GOG.

The Web Crawler Code

The full source code can be found on GitHub and effectively amounts to less than a 100 lines of code (thanks to the Apify SDK).

The crawler begins by parsing https://www.gog.com/games/ajax/filtered?mediaType=game&page=1&sort=release_desc&language=en – a JSON endpoint that yields a paginated list of games.

Apify.main(async () => {
    const requestQueue = await Apify.openRequestQueue();

    await requestQueue.addRequest({
        url: 'https://www.gog.com/games/ajax/filtered?mediaType=game&page=1&sort=release_desc&language=en',
    });

    const crawler = new Apify.CheerioCrawler({
        requestQueue,
        handlePageFunction: handlePage,
    });

    await crawler.run();
});

The handlePage function is the actual meat of the crawler. It gets the links to the games and adds them to the queue for crawling. For the details another API endpoint is used that contains infos like the supported languages and operating systems. There’s also data like description and screenshots that one could query but that’s not needed for my purposes.

export const handlePage= async ({request, $, json, crawler}) => {
    if (json && 'products' in json) {
        if (json.page === 1) {
            log.info(`FOUND ${json.totalGamesFound} GAMES`)
        } else {
            log.info(`Crawling page ${json.page}/${json.totalPages}`)
        }

        for (let game of result.products) {
            // this url will be fetched and also passed to handlePage
            const url = `https://api.gog.com/products/${game.id}`
            crawler.requestQueue.addRequest({
                url,
                headers: {
                    "Accept-Language": "en"
                },
                // some infos are included in the listing already (including title) - this is passed on to handlePage
                userData: game, 
            })
        }

        if (json.page < json.totalPages) {
            const next = json.page + 1;
            requestQueue.addRequest({
                url: `https://www.gog.com/games/ajax/filtered?mediaType=game&page=${next}&sort=release_desc`,
            })
        }
    } else if (request.url.match(/api\.gog\.com\/products/)) {
        log.info('Crawling game at ' + request.url)
        // saves the crawled data locally as a JSON file
        await Apify.pushData({
            ...request.userData,
            ...json
        })
    }
};

Generating a HTML page

With the crawler having collected all the data as JSON I still wanted to create a more user-friendly representation of the data in form of an interactive HTML page.

To that end I used handlebars to create a template and wrote another simple script to generate the HTML page using the template and the crawled data.

Finally I also wrote a simple GitHub action that first runs the web crawler, then generates the HTML page, and commits the updated HTML page to my Git repository. The workflow can be triggered manually by me on GitHub to update the GitHub page.

The Result

The result can be found at https://bunkerbewohner.github.io/video-games-by-language/gog/ – a simple HTML page that lists all games that can be found on gog.com and allows you to filter them by supported languages.

Some Game Gems I Found

To my pleasant surprise the list of games that support Arabic includes blockbusters like The Witcher 3 and Cyberpunk 2077 (thanks CD Project Red!). But there also some small gems that I stumbled over which are worth playing regardless of which language you want to play it in.

Yaga

The first little gem I found is pictured above and is called Yaga. Arabic is supported well by the game’s UI. The dialogues are spoken in English but the text you can read along in Arabic. Unfortunately you can’t pause dialogues so you sometimes have to be a fast reader.

Smash, clobber and bash the murderous legends of Slavic mythology in this darkly funny action role-playing game that changes every time you play. Play as Ivan, a one-handed blacksmith with incredibly bad luck, who must take on the impossible tasks given to him by the tzar. All the while the mysterious witch, Baba Yaga, watches over Ivan’s fate.

https://www.gog.com/game/yaga

Yaga offers English audio with text translations in German, Spanish, French, Italian, Dutch, Polish, Portuguese, Romanian, Turkish, Russian, Arabic, Chinese, Japanese and Korean.

Alba: A Wildlife Adventure

“Alba: A Wildlife Adventure” somehow completely passed by me before and I just found it in this endeavour to find games that support Arabic. It was developed by Ustwo games of Monument Valley fame (a game series that I love anyway!).

Join Alba as she visits her grandparents on a Mediterranean island. She is ready for a peaceful summer of wildlife exploration with her friend Ines, but when she sees an animal in danger, she realises she needs to do something about it.

https://www.gog.com/game/alba_a_wildlife_adventure

The game is very relaxed and casual. Dialogs don’t continue automatically unless you press a key / tap the screen which makes this perfect for taking your time to read and understand the Arabic texts.

The game’s text is translated into German, Spanish, French, Italian, Dutch, Portuguese, Turkish, Russian, Arabic, Chinese, Japanese and Korean.

Steam is next

GOG was an easy start. But that only covers about 5000 video games (107 of which are available in Arabic). My next goal is to do the same for Steam, where the majority of games can be found.