Your Guide to Google Dataproc for Kubernetes

Cloud Dataproc is probably one of the most exciting products in the Google Cloud portfolio for people with interest in data management, but it's also one of the most underrated. With Google Cloud Dataproc, businesses can access a fully-managed service on the cloud that allows then to run Hadoop and Apache Spark clusters without having to worry about managing the underlying infrastructure in their environments at the same time. 

Today, Google announced that it would be launching the alpha of Cloud Dataproc to Kubernetes, and although that might not sound very interesting for some GCP customers - it's a big step forward for the Google Cloud team. Essentially, it means that Google is adapting more of its products to suit the growing demand for a hybrid cloud model. 

This new and improved version of Cloud Dataproc in Alpha for Kubernetes is aimed at enterprises that are currently struggling with hybrid systems of public and private clouds, as well as legacy infrastructure.

Cloud Dataproc for Kubernetes

Google's open-source Kubernetes solution was originally developed by the Google team to help businesses deploy and manage microservices within a cloud-native computing environment. This basically makes it easier to write applications in a range of situations. In a blog post produced by Google Cloud project managers James Malone and Christopher Crosbie, the company said that the demand for hybrid cloud computing is causing a few inefficiencies for certain brands, as machines can often sit around unused for long periods of time.

With this new announcement, Google is bringing enterprise-level support, security, and management to Apache Spark solutions in GKE clusters. The idea is to give enterprise customers the ability to run their Spark jobs on Kubernetes. With products like Anthos making GKE more available to anyone, anywhere, this means that customers can take the Cloud Dataproc experience right to their own data centres. For the time being, the service will only support Apache Spark, but Google is planning on delivering support for other projects in the long-term too. 

Matt Aslett, the VP for research at 451 research commented on the release by saying that today's enterprises are currently looking for services and products that allow them to manage data processing across a variety of platforms and locations. The arrival of the DataProc solution Kubernetes is a significant step forward for Google as it attempts to make the cloud environment offered by the GCP more appealing to enterprises.

Unlocking Cloud Dataproc for Kubernetes

Most of the time, Spark applications would run on Hadoop clusters, but Google notes that the Cloud Dataproc solution for Kubernetes will stop users from having to use two cluster management systems at once, and also provide them with a more cohesive single view across all of their clusters. Supporting different YARN and Kubernetes clusters at once will give enterprises on the GCP the flexibility and freedom they need to modernize modern workloads while businesses continue to manage their existing workloads too. 

The release of Cloud Dataproc for Kubernetes also appears to be a part of the ongoing race between all the major cloud providers, including Microsoft and Azure, to provide the most complete suite of hybrid cloud services for companies of all shapes and sizes in the age of digital transformation. Although it's still in Alpha mode, the Cloud Dataproc update on Kubernetes should make it easier for businesses to engage in their own digital transformation strategies with as little hassle as possible. Google hopes that by delivering this new option to enterprises, it will make itself more appealing to companies that want to simplify the transition to containers in the cloud and microservices. 

In the long-term, many experts believe that the shift to microservices and containers will lead to greater security and stability for all cloud users. 

The Cloud Dataproc solution for Kubernetes is now available in alpha mode. To give it a try for yourself, you'll need to contact the Google team.