Collecting the links for the Pagoda database is an art form. No two websites are the same. Some websites try to prevent link harvesting (not unreasonably so). There are zillions of potential links for the Google Play Store and Apple App Store. Names for the same game vary widely. When websites restructure themselves the links become out of date.

Nicest is when a website has a Twitter feed. These can be scanned programmatically. Caching the redirect lookups is a must as the links in the Twitter feeds can be a long chain of redirects. Currently I scan these every night in an incremental process.

Steam currently provide a Web API which lets you list all the apps on Steam in JSON format. Kind of them.

Some websites one can see recent updates on their homepage. Some websites one can laboriously crawl over their indexes of game links.

Finding potential links on the Google Play Store and Apple App Store is where one gets into nightmare land. Truly horrible.