I am going to blog a series of survey posts on synthesis ways to produce datasets for computer vision or virtual environment for DL/RL, the idea first came into my mind when one of my upperclassman (FP, T) told me we can use data produced by simulation software for DL, and later on I realized that these synthesis ways are actually feasible when I encounter the works of openai, especially the gym and universe projects. And the survey starts when I was doing the project for Mrs. Liang Wan's Advanced Computer Graphics class.
There are (at least) three main fields of the intended survey:
- Physics Simulations, like Bullet, Havok, MuJoCo, ODE, and PhysX, etc
- Game Engines, like Unity3D, Unreal, cocos2d-x, godot and so on
- Closed source games with (community) API for developers, like GTA V
The first one can be part of the second one, and for synthesis images, the second and third can be more appealing since they provide abundant images of various scenarios.
The potential difficulty is that the lacking of game development experience, especially in the second and third one. But the promising outcome is really appealing and maybe not so much experience in game development is needed at all.
What's more, all of these three have successful prior work, the openai gym uses MuJoCo in their project, and there is unrealcv which aim to help computer vision researchers build virtual worlds using Unreal Engine 4, then there is one article Using Virtual Worlds, Specifically GTA5, to Learn Distance to Stop Signs. So I am going to survey these three in the first round.
To make things easy for referring and organized, all blogs are listed below with logical order: