This is useful for the compiler tests, but otherwise it's not useful. Additionally if an application has external templates (as is common) then we should actually fetch these templates in tests. Fixes #4539 Closes #4682