Kubeflow for MLOps — A Field Guide

Kubeflow's pitch is that Kubernetes, already the thing running your services, can also run your ML workflows — training jobs, pipelines, and served models — using the same cluster, the same RBAC, and the same operational tooling your platform team already knows. It delivers on that, but it does it by bundling together several genuinely separate projects under one dashboard, and knowing where one project ends and the next begins is most of what you need to operate it well.

Kubeflow Pipelines and Argo Workflows

Kubeflow Pipelines is the orchestration layer, and it's built directly on top of Argo Workflows — each pipeline compiles down to an Argo Workflow custom resource, with pipeline steps becoming Argo DAG nodes. This matters operationally: when a pipeline run misbehaves, you're debugging an Argo Workflow, and `kubectl get workflows` and Argo's own UI are as useful as the Kubeflow dashboard for figuring out what happened. Pipelines are authored in Python using the Kubeflow Pipelines SDK, compiled into a YAML workflow spec, and each step runs as its own container — which means step isolation and dependency management look like any other containerized batch job, not like a notebook cell.

from kfp import dsl

@dsl.component(base_image="python:3.11")
def train_step(data_path: str, model_path: dsl.Output[dsl.Model]):
    # training logic here; runs as its own container
    pass

@dsl.pipeline(name="fraud-model-training")
def training_pipeline(data_path: str = "s3://bucket/train/"):
    train = train_step(data_path=data_path)
    train.set_cpu_limit("4").set_memory_limit("16Gi")

# compiles to an Argo Workflow CRD
from kfp import compiler
compiler.Compiler().compile(training_pipeline, "pipeline.yaml")

Training Operators for distributed jobs

Distributed training — a TensorFlow job split across workers and parameter servers, or a PyTorch job split across ranks — is handled by the Kubeflow Training Operator, which defines custom resources like TFJob and PyTorchJob. The operator's job is narrow but important: it creates the right number of pods with the right roles, injects the environment variables each framework needs to find its peers (MASTER_ADDR, WORLD_SIZE, rank), and manages the job's lifecycle as a single Kubernetes object instead of you hand-managing a pod-per-worker deployment. It doesn't do anything TensorFlow or PyTorch couldn't do if you wired up the networking yourself — it just removes the part where you wire up the networking yourself.

Model serving with KServe

Model serving used to be KFServing; it split out of the Kubeflow project and now lives as KServe, though most Kubeflow installs still integrate it as the serving layer. KServe wraps a trained model in an InferenceService custom resource, handles scale-to-zero for infrequently-used models, and supports canary rollouts between model versions natively — you can shift a percentage of traffic to a new model version and watch metrics before committing fully. It sits on top of Knative Serving for the autoscaling and revision-management pieces, which is another dependency worth knowing about when something in the serving path breaks.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: fraud-model
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: s3://bucket/models/fraud-v3/
    canaryTrafficPercent: 10
  predictorSpec:
    minReplicas: 0
    maxReplicas: 5

Notebooks and the Central Dashboard

The Kubeflow Central Dashboard ties all of this together with a UI for launching Jupyter notebook servers (backed by their own CRD and controller), browsing pipeline runs, and inspecting InferenceServices, all namespaced per user or team through Kubeflow's multi-tenancy model. The notebook servers are ordinary pods running a Jupyter image with a PVC attached for persistence — convenient for exploratory work, but the actual production path is still: write the pipeline in the SDK, compile it, submit it, and let the Training Operator and KServe handle the parts that need to survive a restart.

Wrapping up

Kubeflow's real value is giving ML workflows a home on infrastructure you already operate, using Argo for orchestration, the Training Operator for distributed jobs, and KServe for serving — but it's a composition of separately-maintained projects, not a monolith, and the operational cost is understanding each layer well enough to debug it directly when the dashboard doesn't tell you enough.