terraform-google-dataproc
The Terraform module handles the creation of Dataproc Cluster on Google Cloud.
Assumptions and prerequisites
This module assumes that below mentioned prerequisites are in place before consuming the module.
To deploy this blueprint you must have an active billing account and billing permissions.
APIs are enabled.
Permissions are available.
Dataproc Cluster
Basic usage of this module is as follows:
module "dataproc" {
source = " terraform-google-modules/dataproc/google"
version = " ~> 0.1"
project_id = " <PROJECT ID>"
name = " simple-example-cluster"
region = " us-central1"
}
Functional examples are included in the
examples directory.
Name
Description
Type
Default
Required
cluster_config
Allows you to configure various aspects of the cluster.
object({ staging_bucket = optional(string) temp_bucket = optional(string) gce_cluster_config = optional(object({ zone = optional(string) network = optional(string) subnetwork = optional(string) service_account = optional(string) service_account_scopes = optional(list(string)) tags = optional(list(string), []) internal_ip_only = optional(bool) metadata = optional(map(string), {}) reservation_affinity = optional(object({ consume_reservation_type = string key = string values = list(string) })) node_group_affinity = optional(object({ node_group_uri = string })) shielded_instance_config = optional(object({ enable_secure_boot = bool enable_vtpm = bool enable_integrity_monitoring = bool })) confidential_instance_config = optional(object({ enable_confidential_compute = bool })) })) master_config = optional(object({ num_instances = number machine_type = optional(string) min_cpu_platform = optional(string) image_uri = optional(string) disk_config = optional(object({ boot_disk_type = string boot_disk_size_gb = number num_local_ssds = number })) accelerators = optional(list(object({ accelerator_type = string accelerator_count = number }))) })) worker_config = optional(object({ num_instances = number machine_type = optional(string) min_cpu_platform = optional(string) disk_config = optional(object({ boot_disk_type = string boot_disk_size_gb = number num_local_ssds = number })) image_uri = optional(string) accelerators = optional(list(object({ accelerator_type = string accelerator_count = number }))) })) preemptible_worker_config = optional(object({ num_instances = number preemptibility = string disk_config = optional(object({ boot_disk_type = string boot_disk_size_gb = number num_local_ssds = number })) })) software_config = optional(object({ image_version = optional(string) override_properties = optional(map(string)) optional_components = optional(list(string)) })) security_config = optional(object({ kerberos_config = optional(object({ cross_realm_trust_admin_server = optional(string) cross_realm_trust_kdc = optional(string) cross_realm_trust_realm = optional(string) cross_realm_trust_shared_password_uri = optional(string) enable_kerberos = optional(bool) kdc_db_key_uri = optional(string) key_password_uri = optional(string) keystore_uri = optional(string) keystore_password_uri = optional(string) kms_key_uri = string realm = optional(string) root_principal_password_uri = string tgt_lifetime_hours = optional(number) truststore_password_uri = optional(string) truststore_uri = optional(string) })) })) autoscaling_config = optional(object({ policy_uri = string })) initialization_action = optional(object({ script = string timeout_sec = optional(number) })) encryption_config = optional(object({ kms_key_name = string })) lifecycle_config = optional(object({ idle_delete_ttl = optional(string) auto_delete_time = optional(string) })) endpoint_config = optional(object({ enable_http_port_access = bool })) dataproc_metric_config = optional(object({ metrics = list(object({ metric_source = string metric_overrides = optional(list(string)) })) })) metastore_config = optional(object({ dataproc_metastore_service = string })) })
null
no
graceful_decommission_timeout
Allows graceful decommissioning when you change the number of worker nodes directly through a terraform apply. Does not affect auto scaling decommissioning from an autoscaling policy. Graceful decommissioning allows removing nodes from the cluster without interrupting jobs in progress. Timeout specifies how long to wait for jobs in progress to finish before forcefully removing nodes (and potentially interrupting jobs). Default timeout is 0 (for forceful decommission), and the maximum allowed timeout is 1 day.
string
"0"
no
labels
The list of labels (key/value pairs) configured on the resource through Terraform and to be applied to instances in the cluster.
map(string)
{}
no
name
The name of the Dataproc Cluster.
string
n/a
yes
project_id
The ID of the project in which the Dataproc Cluster belongs.
string
n/a
yes
region
The region in which the cluster and associated nodes will be created.
string
n/a
yes
virtual_cluster_config
Allows you to configure a virtual Dataproc on GKE cluster
object({ staging_bucket = optional(string) auxiliary_services_config = optional(object({ metastore_config = optional(object({ dataproc_metastore_service = string })) spark_history_server_config = optional(object({ dataproc_cluster = string })) })) kubernetes_cluster_config = optional(object({ kubernetes_namespace = optional(string) kubernetes_software_config = object({ component_version = map(string) properties = optional(map(string)) }) gke_cluster_config = object({ gke_cluster_target = optional(string) node_pool_target = optional(object({ node_pool = string roles = list(string) node_pool_config = optional(object({ autoscaling = optional(object({ min_node_count = optional(number) max_node_count = optional(number) })) config = optional(object({ machine_type = optional(string) preemptible = optional(bool) local_ssd_count = optional(number) min_cpu_platform = optional(string) spot = optional(bool) })) locations = optional(list(string)) })) })) }) })) })
null
no
Name
Description
cluster_id
The ID of the Dataproc cluster
cluster_name
The name of the Dataproc cluster
cluster_project
The project ID of the Dataproc cluster
cluster_region
The region of the Dataproc cluster
These sections describe requirements for using this module.
The following dependencies must be available:
A service account with the following roles must be used to provision
the resources of this module:
Dataproc Admin: roles/dataproc.admin
The Project Factory module and the
IAM module may be used in combination to provision a
service account with the necessary roles applied.
A project with the following APIs enabled must be used to host the
resources of this module:
Google Cloud Storage JSON API: dataproc.googleapis.com
The Project Factory module can be used to
provision a project with the necessary APIs enabled.
Refer to the contribution guidelines for
information on contributing to this module.
Please see our security disclosure process .