Apps Admins & Google Cloud Composer: Your Partner for Multi-Cloud Mastery

On the 1st of May 2018, Google announced a new solution for IT administrators who want to make the most out of the hybrid and multi-cloud architecture. The beta launch of the new 'Cloud Composer' solution - a managed service for Apache Airflow, makes workflow management and creation simple and consistent.

After exploring the challenges that admins, data engineers, and analysts face when it comes to workflow, Google decided that they needed to create a solution that minimized stress and confusion for the cloud community. The 'Composer' system allows you to monitor, schedule, and manage your workflows with:

• Hybrid and multi-cloud environment management. Handle multiple cloud systems at once with a single orchestration service that works across your environment.

• GCP integration throughout: Enjoy complete control over your Google Cloud Platform pipeline thanks to Cloud Composer's state-of-the-art integration.

• Easy workflow orchestration: Code your 'DAGs' (Directed acyclic graphs) using Python to improve the readability of your workflows, and pinpoint areas in need of assistance.

The unique features that Google Cloud Composer brings to Cloud apps admins mean that any mid-sized enterprise can make the most of a single, fully-managed solution for productivity and performance. Cloud Composer allows you to pull your workflows together from wherever they live, giving you the support, you need for a fully-functioning and connected cloud environment.

What Can You Do with Google Cloud Composer?

Cloud Composer gives Google users the opportunity to make the most out of the rich Apache Airflow open source project. Google built the new application to combine the benefits of the GCP with the flexibility of Airflow. Because the system is built on an open-source project and operated with the Python programming language, it's free from lock-in and ideal for a multi-cloud, flexible environment.

The pipelines you create with Cloud Composer will be configured as DAGs with Python, which means that users of almost any level will have the ability to create and schedule their own workflows. There's even one-click deployment built-in so that you'll have instant and easy access to a range of connectors and graphical representations that show your workflow in action.

What's more, because Cloud Composer is naturally integrated with the rest of the Google Cloud Platform, it can also be a fantastic way to upgrade the rest of your GCP experience. Even the beta release of the cloud Composer application includes various features ideal for helping you to save time on your projects, such as:

• Identity Access Management

• Stackdriver monitoring and logging

• Streamlined Airflow configuration

• Cloud Identity Aware Proxy

• Google Developer Console and Cloud SDK client tooling

Google is currently working on various efforts to support the extension of Airflow too, including work on the new KubernetesExecutor system.

Creating Flexible Hybrid & Multi-Cloud Environments

Google began their Cloud Composer project with a focus on developing their 'open cloud' philosophy. One of the must-have features they implemented from day one was a no-lock-in approach intended to help the mid-market enterprises who want to make the most out of a versatile multi-cloud environment.

When you create a workflow on the Google Cloud Platform, it will be portable to other environments too. What's more, Composer works seamlessly across a range of technologies outside of the GCP. That means that you can connect your pipeline through a single regardless of whether part of your workflow lives in a different cloud, on-premise, or anywhere else. This flexibility ensures that you can eliminate the silos in your business and focus more on growing your organization.

A number of incredible open-source projects have already been developed with a focus on workflow, including Luigi, Apache Azkaban, and Apache Oozie. Cloud Composer is based on Apache Airflow because Airflow comes with:

• A Python baseline and custom plugin support

• An active developer community

• An elegant web user interface with command-line tooling

• Connectors for various common technologies and clouds

• Support for multi-cloud and hybrid cloud networks

The Features of Google Cloud Composer

One of the great things about Google Cloud Composer is how easy it is to use. If you're already familiar with the ins and outs of Apache Airflow, then you should feel comfortable using Composer too. If you haven't worked with Apache Airflow before, then there are plenty of tutorials out there to help you, and an active community to speak to, as well.

The top features of Google's Cloud Composer solution include:

• Multi-cloud and hybrid support: Google's Cloud Composer service makes it easier for admins and leaders in the mid-market enterprise to build the cloud experience they need from scratch. With composer, you can create productive workflows that connect processing, data, and services across channels, for a more unified data environment. Whether you're interested in multi-cloud or you simply want to ease your transition to the cloud with a hybrid framework, the Cloud Composer system can help.

• Apache Open Source and Python Programming language: Making the most of Cloud Composer couldn't be simpler thanks to the fact that it's built on the user-friendly Apache Airflow project. Not only does Apache Airflow eliminate problems like lock-in, but it also gives you the freedom to adjust your cloud portfolio as you see fit. You can also leverage your Python language skills to schedule and author workflows within as little time as possible.

• Integrated with the GCP: Like many things on the Google Cloud Platform, the Cloud Composer application is designed to work seamlessly as part of the Google experience. That means it comes with built-in integration for Dataflow, Dataproc, Datastore, Cloud Storage, BigQuery, the Machine Learning Engine, and Pub/Sub.

• Fully-managed and reliable: Because Cloud Composer is fully managed, you can focus on making the most out of your workflows with scheduling, authoring, and monitoring. You'll also be able to increase the reliability of your workflows through simple charts for troubleshooting and monitoring the root cause of any issues.

The price for Google Cloud Composer varies depending on your consumption needs. This means that you pay for what you use according to vCPU per hour, as well as gigabytes per month, and gigabytes transferred per month. Google offers multiple pricing units because the Cloud Composer uses a host of different GCP products as essential building blocks.

This beta version of the Cloud Composer experience is a major milestone for Google, and the company suggests that there are many updates still to come. To get ahead of the trend and start exploring the benefits of Cloud Composer today, or to learn more about the multi-cloud environment, contact Coolhead Tech.