Under what conditions do modern c++ compilers not perform rvo?

For performance reasons I'm interested in making sure that RVO is being performed on the return values of functions. What would prevent this optimization from being performed by a modern C++ compiler (such as gcc, clang, and msvc 2015) which I should worry about?