STEAMEXPERT was supposed to be a project about Steam Community item market data, it would make anyone excited who does item trading, selling and collecting. Providing in-depth statistics for not just for one game, but for all applications that utilized Steam item market. We were scraping so many pages (100,000+ pages, one for each market item) so that we could just calculate how much money Gabe Newell is earning per day from the virtual item transaction fee.
The project deemed to be too ambitious, we wanted to do too much. API for developers, pricing expensive items, inventory tracking and scraping. If you have experience with CS:GO trading, then think of CSGOstash.com + CSgro.com + SteamAnalyst.com and other sites, but just combined into one better one.
We have open-sourced the code for anyone to see because otherwise it would have been a waste. I am still proud of some of the things we did for that project to this day e.g. efficiently scraping many pages while swapping out proxies to avoid rate-limits, storing massive amount of data in the database, proxy testing, calculating values by parsing many history entries, scraping data with a head-less browser to combat anti-scraping methods, setting up simple caching to serve many requests. And credit where it’s due the server management (e.g. load balancing) was done by my friend who also co-developed this project with me.
