How to run Spark 2.0 from HDInsight using Azure Automation

Question

I cannot figure out how to run HDInsight Spark 2.0 from the Azure Automation graphical workbook. I have an existing workbook that works with HDInsight using Spark 1.6. Normally, I would update the version line from 3.4 to 3.5, but there seem to be two releases of HDInsight 3.5 - one with Spark 2.0 and one with 1.6.

Can someone tell me how can I get my workbook to use the version of Spark2.0?

Link to the version: https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-component-versioning#hadoop-components-available-with-different-hdinsight-versions

Answer 1

I determined how to do this using the ARM pattern. Hope this helps you regarding the workbook. Here is JSON to make cluster 2.0. Pay attention to the sections clusterVersion and componentVersion.

...
"resources": [
    {
        "apiVersion": "2015-03-01-preview",
        "name": "somename",
        "type": "Microsoft.HDInsight/clusters",
        "location": "northcentralus",
        "dependsOn": [],
        "properties": {
            "clusterVersion": "3.5",
            "osType": "Linux",
            "tier": "standard",
            "clusterDefinition": {
                "kind": "spark",
                "componentVersion": {
                    "Spark": "2.0"
                },
...