Google Cloud today announced the beta launch of Cloud AI Platform Pipelines, a new enterprise-grade service that is meant to give developers a single tool to deploy their machine learning pipelines, together with tools for monitoring and auditing them.When you&re just prototyping a machine learning (ML) model in a notebook, it can seem fairly straightforward,& Google notes in today announcement.
&But when you need to start paying attention to the other pieces required to make an ML workflow sustainable and scalable, things become more complex.& And as complexity grows, building a repeatable and auditable process becomes harder.That, of course, is where Pipelines comes in.
It gives developers the ability to build these repeatable processes.
As Google notes, there are two parts to the service: the infrastructure for deploying and running those workflows, and the tools for building and debugging the pipelines.
The service automates processes like setting up Kubernetes Engine clusters and storage, as well as manually configuring Kubeflow Pipelines.
It also uses TensorFlow Extended for building TensorFlow-based workflows and the Argo workflow engine for running the pipelines.In addition to the infrastructure services, you also get visual tools for building the pipelines, versioning, artifact tracking and more.With all of this, getting started only takes a few clicks, Google promises, though actually configuring the pipelines isn''t exactly trivial, of course.
Google Cloud is adding a bit of complexity (or flexibility, depending on your perspective) here, given that you can use both the Kubeflow Pipelines SDK and the TensorFlow Extended SDK for authoring pipelines.