Change yaml test suite testcase to automatically delete all data streams after each yaml test (#62214)

Backporting #62205 to 7.x branch.

This is similar to what happens for indices. Initially we decided to let each test cleanup the
data streams it created.

The reason behind this was that client yaml test runners would need to be modified to do this too and
because data steams were new, we waited with that and let each test cleanup the data stream it created.
However we sometimes have very hard to debug test failures, because many tests fail because another test
failed mid way and didn't clean up the data streams it created. Given that and data streams exist in
the code base for a while now, we should automatically delete all data streams after each yaml test.

Relates to #62190

* preserve data streams for rolling upgrade yaml tests

test/framework/src/main/java/org/elasticsearch/test/rest/yaml/ESClientYamlSuiteTestCase.java

@@ -439,13 +439,4 @@ public abstract class ESClientYamlSuiteTestCase extends ESRestTestCase {
         configureClient(builder, restClientSettings());
         return builder;
     }
-
-    protected final boolean preserveDataStreamsUponCompletion() {
-        // TODO: enable automatic deleting of data streams
-        // For now don't automatically try to remove all data streams after each yaml test.
-        // The client runners need to be adjust to remove data streams after each test too,
-        // otherwise rest yaml tests using data streams succeed in Elasticsearch, but may fail when clients run
-        // the yaml test suite. In the mean time we should delete data streams manually after each test.
-        return true;
-    }
 }

x-pack/qa/rolling-upgrade/src/test/java/org/elasticsearch/upgrades/UpgradeClusterClientYamlTestSuiteIT.java

@@ -83,6 +83,11 @@ public class UpgradeClusterClientYamlTestSuiteIT extends ESClientYamlSuiteTestCa
         return true;
     }
 
+    @Override
+    protected boolean preserveDataStreamsUponCompletion() {
+        return true;
+    }
+
     public UpgradeClusterClientYamlTestSuiteIT(ClientYamlTestCandidate testCandidate) {
         super(testCandidate);
     }