Routing to Gemini 3.5 Flash via Regional Endpoints and PSC

gemini-3.5-flash is not available on global Vertex AI endpoints. It is GA only in the EU multi-region — meaning all calls must reach aiplatform.eu.rep.googleapis.com, not the familiar aiplatform.googleapis.com that older models use.

Getting there from inside Agent Engine, behind a private VPC, turned out to require solving three independent problems in sequence.

---

The Challenge

Our Agent Engine runs in europe-west1 inside a VPC-Service-Controls perimeter. Outbound calls to Google APIs go through the restricted VIP (restricted.googleapis.com) — a private DNS path that serves a certificate covering *.googleapis.com.

The EU multi-region endpoint has a different hostname pattern: aiplatform.eu.rep.googleapis.com. The restricted VIP's certificate does not cover *.rep.googleapis.com, so the first call to Gemini 3.5 dies immediately:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED]
hostname 'aiplatform.eu.rep.googleapis.com' doesn't match

The model is there. The network can reach Google. But the TLS handshake is wrong — a Subject Alternative Name (SAN) mismatch between the certificate served and the hostname being dialled.

---

Part 1 — The Pickle Trap (Discovered First)

Before even hitting the TLS wall, we ran into a deployment crash.

The naive fix was to set GOOGLE_CLOUD_LOCATION=eu before constructing the Gemini(model="gemini-3.5-flash") instance. The ADK Gemini class builds its underlying google.genai.Client(...) lazily via a cached_property called api_client. If you resolve it at construct time (by touching .api_client or passing a location override eagerly), the resulting client holds httpx and gRPC thread locks internally.

agent_engines.create() then calls cloudpickle.dumps(agent) to ship the agent to Agent Engine, and it crashes:

TypeError: cannot pickle '_thread.lock' object

This only surfaced when deploying a real agent — the A2A deploy path uses a different SDK call and hides the bug.

The Fix: _EUPinnedGemini

We introduced a Gemini subclass that overrides api_client as a cached_property and applies the location override at resolve time, not construct time:

class _EUPinnedGemini(Gemini):
    @cached_property
    def api_client(self):
        target = os.environ.get("MODEL_LOCATION", DEFAULT_MODEL_LOCATION)
        original = os.environ.get("GOOGLE_CLOUD_LOCATION")
        os.environ["GOOGLE_CLOUD_LOCATION"] = target
        try:
            return Gemini.api_client.func(self)   # delegate to parent builder
        finally:
            if original is None:
                os.environ.pop("GOOGLE_CLOUD_LOCATION", None)
            else:
                os.environ["GOOGLE_CLOUD_LOCATION"] = original

Until the first model call, __dict__ has no api_client entry — the instance pickles cleanly. In the deployed container, the first request hits the override, reads MODEL_LOCATION from the container environment (forwarded from the deploy shell), briefly patches GOOGLE_CLOUD_LOCATION, builds the client pointing at eu, and restores the original.

The env-var mutation is safe because cached_property runs exactly once and does so from the request-handling thread before any concurrent model calls.

sequenceDiagram
    participant D as Deploy shell
    participant AE as Agent Engine SDK
    participant CP as cloudpickle
    participant C as Container (runtime)
    participant G as google.genai.Client

    D->>AE: agent_engines.create(AdkApp(agent))
    AE->>CP: cloudpickle.dumps(agent)
    Note over CP: _EUPinnedGemini.__dict__ has no api_client → pickles cleanly
    CP-->>AE: bytes

    AE-->>C: deploys container

    C->>C: first model call triggers cached_property
    C->>C: set GOOGLE_CLOUD_LOCATION = "eu"
    C->>G: Gemini.api_client.func(self)  → Client(api_endpoint=eu)
    G-->>C: client (cached)
    C->>C: restore GOOGLE_CLOUD_LOCATION = "europe-west1"
    C->>G: model call → aiplatform.eu.rep.googleapis.com

build_gemini() is the public factory — it always returns an _EUPinnedGemini and is safe to call anywhere in agent code:

def build_gemini(model: str = DEFAULT_MODEL, ...) -> Gemini:
    return _EUPinnedGemini(model=model, retry_options=retry_options)

---

Part 2 — The TLS Fix (Private Service Connect)

With the pickle crash fixed, deploys succeeded — but model calls still failed at runtime with the SAN mismatch. We needed to give Agent Engine a private network path to aiplatform.eu.rep.googleapis.com that presents the correct certificate.

The solution is Private Service Connect (PSC): three pieces of infrastructure, all guarded by var.enable_agent_engine_psc_interface in Terraform.

flowchart TB
    AE["Agent Engine\n(Google-managed project)\ncalls gemini-3.5-flash"]

    subgraph ours["our project / VPC (europe-west1)"]
        NA["① Network attachment\nagent-engine-psc-interface\nvpc-private-subnet-3"]
        DNS{{"③ DNS record\naiplatform.eu.rep.googleapis.com\n→ regional endpoint IP"}}
        EP["② Regional endpoint\nvertex-aiplatform-eu-mrep\naccess_type = GLOBAL\n→ aiplatform.eu.rep.googleapis.com"]
    end

    mREP["Google EU mREP\naiplatform.eu.rep.googleapis.com\n(correct *.rep cert)"]

    AE -->|egress via PSC interface| NA --> DNS --> EP -->|correct TLS| mREP

① Network Attachment

Gives Agent Engine's Google-managed compute a NIC in our VPC. Traffic from Agent Engine enters on vpc-private-subnet-3 (added alongside this work) in europe-west1.

resource "google_compute_network_attachment" "agent_engine" {
  count = var.enable_agent_engine_psc_interface ? 1 : 0

  name                  = "agent-engine-psc-interface"
  region                = var.region_eu1          # europe-west1
  connection_preference = "ACCEPT_AUTOMATIC"
  subnetworks = [
    module.vpc_private_subnets[0].subnets["${var.region_eu1}/vpc-private-subnet-3"].self_link,
  ]
}

② Regional Endpoint

A PSC endpoint that privately connects our VPC to aiplatform.eu.rep.googleapis.com. Google's PSC fabric terminates TLS with the correct *.rep.googleapis.com certificate.

access_type = GLOBAL is required: the endpoint lives in europe-west1 but the target is the eu multi-region. Without global access, the cross-region reachability is blocked by default (Google support flagged this during our setup).

resource "google_network_connectivity_regional_endpoint" "vertex_eu_mrep" {
  count = var.enable_agent_engine_psc_interface ? 1 : 0

  name              = "vertex-aiplatform-eu-mrep"
  location          = var.region_eu1
  target_google_api = "aiplatform.eu.rep.googleapis.com"
  access_type       = "GLOBAL"
  network           = data.google_compute_network.vpc_network.id
  subnetwork        = module.vpc_private_subnets[0].subnets["${var.region_eu1}/vpc-private-subnet-3"].id
}

Note on the subnetwork field: The Network Connectivity API rejects the full compute self_link (https://www.googleapis.com/compute/v1/...). You must use the subnet's relative resource id (.id, not .self_link).

③ DNS Record

One A record in our existing private zone points aiplatform.eu.rep.googleapis.com at the regional endpoint's IP. No DNS peering needed — we verified empirically that a fresh deploy with only the network attachment + this DNS record reaches the EU mREP without the cert error.

resource "google_dns_record_set" "vertex_eu_mrep_a" {
  count = var.create_vpc_private_subnets && var.enable_agent_engine_psc_interface ? 1 : 0

  name         = "aiplatform.eu.rep.${google_dns_managed_zone.private_googleapis[0].dns_name}"
  managed_zone = google_dns_managed_zone.private_googleapis[0].name
  type         = "A"
  ttl          = 300
  rrdatas      = [google_network_connectivity_regional_endpoint.vertex_eu_mrep[0].address]
}

IAM — Two Grants, Different Lifetimes

The org custom role holds exactly four permissions: compute.networkAttachments.{get,update,use} and compute.regionOperations.get. It is defined at org level in iam-gevgo (owned by Digital Security) and bound to the service agent via psc.tf — our Terraform never needs to be iam.roleAdmin.

---

Part 3 — Wiring It Into the Deploy

App Side: agent_engine.py

_psc_interface_config() reads AGENT_ENGINE_NETWORK_ATTACHMENT from the environment and passes it into every agent_engines.create() / update() call:

from google.cloud.aiplatform_v1.types import service_networking as service_networking_types

def _psc_interface_config(project_id: str):
    network_attachment = os.getenv("AGENT_ENGINE_NETWORK_ATTACHMENT")
    if not network_attachment:
        return None
    return service_networking_types.PscInterfaceConfig(
        network_attachment=network_attachment,
    )

# used in both add_agent_to_agent_engine() and update_agent_in_agent_engine():
psc_interface_config=_psc_interface_config(project_id),

When the env var is unset the function returns None and Agent Engine deploys without PSC — preserving backwards compatibility for local or PSC-less environments.

CI Side: .gitlab-ci.yml

.common_setup derives the URI from the per-job project ID (identical attachment name across all environments) and exports it for every deploy stage:

- export AGENT_ENGINE_NETWORK_ATTACHMENT="projects/${GOOGLE_CLOUD_PROJECT_ID}/regions/europe-west1/networkAttachments/agent-engine-psc-interface"

Env-Var Forwarding

MODEL, MODEL_LOCATION, ENABLE_THINKING_CONFIG, and THINKING_BUDGET are in _FORWARDED_ENV_VARS in agent_engine.py. Values set in the deploy shell are written into the Agent Engine container's environment — so _EUPinnedGemini reads MODEL_LOCATION=eu at first call from the container runtime, not from the deploy machine.

_FORWARDED_ENV_VARS: tuple[str, ...] = (
    "MODEL",
    "MODEL_LOCATION",
    "ENABLE_THINKING_CONFIG",
    "THINKING_BUDGET",
)

---

The Results

After MR !611 merged to DEV:

• Default model: gemini-3.5-flash / eu (was gemini-2.5-flash / europe-west1)
• Deploy path: every CI job attaches the PSC interface automatically — no per-deploy flag
• TLS: all calls to the EU mREP resolve through the regional endpoint and receive the correct certificate
• Serialisation: agents deploy without pickle errors; the lazy-pin pattern is tested with a cloudpickle.dumps() round-trip in the unit suite

The model upgrade also landed ThinkingConfig in the same MR: a BuiltInPlanner with ThinkingConfig(include_thoughts=True, thinking_budget=4096) is attached to every ADKAgent by default, surfacing intermediate reasoning in Gemini Enterprise's "Show thinking" disclosure.

---

Key Gotchas

1. api_client must not be resolved before pickling. If anything touches .api_client on a Gemini instance before cloudpickle.dumps(), the deploy will crash. The lazy subclass pattern is the only safe approach with the current ADK / Agent Engine SDK.

2. access_type = GLOBAL on the regional endpoint is non-obvious. The endpoint is created in europe-west1 but targets the eu multi-region. Without global access, the cross-region call is silently blocked.

3. DNS peering is not needed. Early docs and some Google samples suggest DNS peering. We tested it and a network attachment + private A record is sufficient. DNS peering adds complexity (requires roles/dns.peer on the service agent) with no benefit in this topology.

4. Subnetwork field uses .id, not .self_link. The Network Connectivity API rejects the full https://... self_link URI for subnetwork on a regional endpoint. Use the relative id instead.

5. tf-deploy-sa networkAdmin is time-limited by design. The role is granted via a conditional binding in iam-gevgo only while Terraform is building the PSC infra. Renew if the rollout window slips.