Feature: Central Platform Kubernetes Module + Optional Application Namespace Service for Landing Zones

[lweberru]

Summary

We want to extend the landing zone accelerator with a central Kubernetes capability in the platform context and an optional namespace service for application landing zones, following the existing module standards and schema conventions of this repository.

Background and Motivation

Current landing zone capabilities cover governance, management, connectivity, devops, and project landing zones.
A reusable central Kubernetes platform capability is required to host shared cluster infrastructure (for example external-dns and central cluster observability integration) and to enable optional namespace onboarding for application landing zones.
We already have implementation experience from a separate Kubernetes replatform repository and want to transfer only the suitable parts into this repository architecture and standards.

Goals

• Add a new optional central Kubernetes platform capability, similar in optionality and composition to existing platform modules.
• Deploy the SKE cluster centrally in a platform landing zone context.
• Support two cluster network modes:
  • with SNA-aware network configuration
  • without SNA, using the corresponding network creation path for SKE
• Add optional encrypted node volumes for SKE as a feature toggle.
• Integrate central SKE observability extension to a central observability instance in the same project as the cluster, explicitly for cluster/platform monitoring only.
• Reuse existing landing zone DNS setup by attaching DNS integration instead of redeploying base DNS services.
• Extend application landing zone capabilities with an optional namespace service in the central SKE cluster.
• Evaluate and implement integration path for DNS + workload namespace and Service Manager usage instead of classic secrets where applicable.
• Keep full compatibility with repository conventions:
  • module layout and naming conventions
  • variable object schema style with optional sections
  • output and tests style
  • provider/version and docs generation workflow

Non-Goals

• Not introducing a full workload deployment framework in this repo.
• Not replacing existing management/landing-zone responsibilities beyond necessary extension points.
• Not introducing unrelated network/firewall topology changes.

Scope Phase 1: Central Platform Kubernetes

• New optional module block at root level for central platform Kubernetes.
• Project creation and labels following existing naming and project patterns.
• SKE cluster creation with:
  • configurable node pools
  • network mode branching for SNA/non-SNA
  • optional encrypted volumes
  • observability extension integration
  • DNS integration by attaching to existing DNS zone strategy in landing zone context.
• Required outputs for cluster/project integration consumed by downstream optional services.

Scope Phase 2: Application Landing Zone Namespace Service

• Optional service toggle per application landing zone entry.
• Namespace creation in the central cluster.
• Optional DNS/workload integration model per namespace.
• Service Manager integration concept to avoid static Kubernetes secrets wherever possible.
• Role/permission model for tenant onboarding and namespace lifecycle.

High-Level Technical Approach

• Follow current module conventions:
  • numbered resource files per module
  • variables/outputs/terraform requirement files
  • tfdocs-generated README
  • tests in current tftest style
• Introduce a platform-kubernetes module as optional object configuration at root input level.
• Keep central cluster as single control point for app landing zone namespace onboarding.
• Add explicit integration contracts via outputs consumed by namespace service logic.
• Add/extend tests for standalone, hub-spoke, and hub-spoke without firewall relevant to the new optional features.

Acceptance Criteria

• Root module supports optional platform Kubernetes configuration without impacting existing deployments when unset.
• SKE cluster deploys in hub-spoke without firewall baseline and supports network mode branching for SNA/non-SNA.
• Encrypted volumes can be enabled/disabled via explicit toggle and validate in plan/apply.
• Central observability extension is wired and points to central instance in same project.
• DNS integration does not duplicate existing DNS foundational deployment and works with selected namespace/workload model.
• Application landing zone can optionally request namespace service creation in central cluster.
• Service Manager based secret flow is defined and implemented for supported use cases.
• Documentation and generated tfdocs are updated.
• E2E validation for a full hub-spoke without firewall landing zone setup is executed using test company service account access.

Implementation Task Breakdown

• Design input schema extensions at root and module boundaries.
• Implement platform-kubernetes module with SKE, observability extension, DNS attach path, optional encryption.
• Implement namespace service extension for application landing zones.
• Add IAM/RBAC and cross-project access model for DNS and namespace operations.
• Add tests and example config variants.
• Update docs and migration guidance.
• Run E2E hub-spoke without firewall validation and collect verification outputs.

Risks and Design Constraints

• Provider/resource capability constraints for namespace lifecycle and service-manager-backed secret flow.
• Cross-project IAM for DNS record management from central cluster.
• SNA network parameter correctness per provider schema and region constraints.
• Drift between central cluster intent and app landing zone optional services.
• Backward compatibility for existing users with null/optional defaults.

Open Questions (to be resolved before implementation)

• Central cluster tenancy model:
  • one central cluster globally
  • one per region
  • one per environment
• Module placement:
  • separate new module at root (recommended)
  • extension of existing devops module
• Namespace service implementation location:
  • in landing-zone module directly
  • separate namespace-service module consuming landing-zone map
• Kubernetes provider introduction:
  • acceptable to add kubernetes provider to this repo
  • or strict stackit-provider-only requirement
• SNA decision model:
  • explicit boolean mode flag
  • implicit behavior based on provided network fields
• DNS delegation strategy for workloads:
  • per-namespace subzone
  • per-landing-zone subzone
  • shared zone with naming convention and ownership boundaries
• External-dns ownership model:
  • one central external-dns instance
  • namespace-scoped tenancy with filters
• Service Manager integration target:
  • runtime secret retrieval in workloads
  • provisioning-time secret handover
  • both
• Encrypted volume default:
  • opt-in default false
  • opt-out default true for regulated environments
• Node pool baseline:
  • fixed two-AZ baseline
  • configurable list-based pool schema aligned with current repo style
• Observability ownership boundary:
  • cluster-only platform monitoring (as requested) confirmed
  • no app workload telemetry coupling in this module
• E2E scope and guardrails:
  • allowed budget/quota limits for test company
  • cleanup expectations after validation
  • acceptable apply runtime and retry policy

Definition of Ready
Implementation starts only after open questions above are answered and accepted in this issue.

---

Finalized Design Decisions (2026-06-11)

The following open points are now resolved and are binding for implementation:

1. Cluster topology: One central platform Kubernetes setup per region.
2. Module strategy: Introduce a new dedicated module (no DevOps module extension).
3. Providers: Kubernetes provider is allowed and required for Kubernetes objects (e.g. namespaces).
4. DNS model: Use the Landing Zone project DNS zone as the namespace/workload DNS base for that landing zone.
5. Secrets integration: Requirement is STACKIT Secrets Manager integration (not Service Manager).
6. Encrypted volumes: Optional feature, default = disabled, easy to enable.
7. Validation cadence: Run E2E checks after each milestone.
8. Destroy caveat: Not all objects are destroyable (e.g. folders with retention constraints), so cleanup must be conservative and explicit.

Milestone Tracking

• M1: Dedicated platform-kubernetes module scaffold + root wiring
• M1: Central SKE cluster resource with per-region design
• M1: SNA/non-SNA aware network handling in cluster config
• M1: Observability extension wiring (central cluster monitoring scope)
• M1: DNS extension wiring to Landing Zone DNS zone model
• M1: Optional encrypted volume support (default off)
• M1: Baseline tests/config updates + validation run
• M2: Application Landing Zone optional namespace service
• M2: Namespace onboarding + DNS combination model per landing zone
• M2: Secrets Manager integration path for namespace/workload setup
• M2: Extended tests + milestone validation run

Progress Log

• 2026-06-11: Requirements clarified and design decisions finalized with stakeholder.
• 2026-06-11: Dev container trust fixed for Zscaler MITM CA; HTTPS downloads working.
• 2026-06-11: Terraform and OpenTofu installed in container and verified.
• 2026-06-11: GitHub CLI installed and authenticated for issue lifecycle updates.

[lweberru]

Status update: Design decisions are now locked and reflected in the issue body. Next implementation step is Milestone 1 (dedicated platform-kubernetes module scaffold + root wiring), followed by milestone-level validation runs.

[lweberru]

Milestone 1 implementation update:

Completed in code:

• Added dedicated platform-kubernetes module (project, SKE cluster, region-scoped design).
• Added root wiring via new platform_kubernetes input map and module integration.
• Implemented SNA/public network mode handling in SKE network block.
• Implemented central observability extension wiring in the same cluster project.
• Implemented DNS extension wiring with fallback to Landing Zone DNS zones.
• Implemented optional encrypted volume foundation (KMS keyring/key, kms.admin role assignment, SKE internal SA act-as role assignment), default disabled.
• Added root outputs and hub-spoke test/config updates for the new capability.

Validation status:

• Module-level terraform validate for modules/platform-kubernetes: PASS
• Root-level / tftest full execution was initially blocked in this container by missing STACKIT credentials file at /root/.stackit/credentials.json (and repo-global firewall image prerequisite for connectivity module when validating full stack).

Next:

• Run milestone validation with available test credentials and standard preconditions, then mark M1 validation complete.

[lweberru]

Milestone 1 validation update:

Executed with STACKIT service account credentials and hub-spoke (no firewall) scope:

• terraform test -filter=tests/hub_spoke.tftest.hcl => PASS (1 passed, 0 failed)
• platform-kubernetes module validate => PASS

Notes:

• Existing provider warnings about beta/experimental resources are expected in current repo setup.
• For local test execution, a minimal placeholder file for firewall-image.qcow2 was temporarily needed because connectivity module currently validates local_file_path even when firewall is not enabled in the selected test variant.

Milestone 1 is now implementation-complete and validation-complete in the scoped test path.

[lweberru]

Default/HA baseline update applied for platform Kubernetes:\n\n- Updated defaults to a minimal HA baseline with two node pools across two AZs.\n- Each default node pool now uses minimum=2 and maximum=2 nodes.\n- Defaults are now module-enforced when cluster.node_pools is omitted (region-aware AZ defaults: -1 and -2).\n- Updated hub-spoke test fixture and hub-and-spoke example values to reflect the same HA baseline.\n\nValidation:\n- terraform fmt -recursive\n- module-level validate for modules/platform-kubernetes: PASS

[lweberru]

Test hardening update (DNS integration):

• Added explicit DNS configuration for platform_kubernetes in hub_spoke.tftest.hcl.
• Added assertion to verify the SKE DNS extension zones include apps.test-corp.stackit.run.
• Re-ran scoped test:
  • terraform test -filter=tests/hub_spoke.tftest.hcl => PASS (1 passed, 0 failed)

This now validates DNS integration behavior in tests, not only in module implementation.

[lweberru]

Milestone 2 progress update (namespace service):

Implemented:

• Added optional namespace_service configuration to landing_zones input.
• Added central namespace provisioning resource (kubernetes_namespace_v1) for enabled landing zones.
• Added metadata wiring for DNS and Secrets Manager context:
  • stackit.cloud/dns-fqdn
  • stackit.cloud/secretsmanager-instance-id
• Added regional platform-kubernetes kubeconfig output and root kubernetes provider wiring.
• Added landing-zone module output for secretsmanager_instance_id.
• Added root output landing_zone_namespace_services.

Validation:

• terraform validate: PASS
• terraform test -filter=tests/hub_spoke.tftest.hcl: PASS (1 passed, 0 failed)

Notes:

• Existing provider warnings for beta/experiments are unchanged from prior runs.

[lweberru]

Next TODOs (prioritized) and why they matter:

1. M2 hardening: namespace_service guardrails (High)

• Add strict input validation for namespace and dns_subdomain format.
• Add clear failures when regional platform Kubernetes is missing.
• Add deterministic checks for namespace uniqueness.

2. DNS policy hardening (High)

• Enforce explicit DNS behavior for namespace_service with predictable FQDN rules.
• Add tests validating DNS annotations and expected zone usage.

3. Secrets Manager integration hardening (High)

• Keep namespace-level metadata contract stable and testable.
• Ensure outputs and annotations remain consistent for workload onboarding.

4. Milestone validation and issue traceability (High)

• Re-run scoped milestone tests after guardrail changes.
• Keep this issue updated with implementation and validation status.

Starting now with items 1 and 2.

[lweberru]

M2 hardening update (guardrails + DNS policy checks):

Implemented:

• Added input validation for landing_zones[*].namespace_service:
  • namespace format validation (DNS-1123 label)
  • dns_subdomain format validation (DNS label)
  • dns_subdomain requires namespace_service.enabled=true
• Added namespace-service safety checks:
  • unique resolved namespace names across enabled services
  • non-empty resolved namespace names
  • explicit precondition: dns_subdomain requires landing zone DNS zone
  • explicit precondition: namespace service requires a platform_kubernetes deployment in var.region
• Added deterministic output for resolved namespace-service requests:
  • landing_zone_namespace_service_requests
• Extended test assertion to validate resolved DNS FQDN in plan-safe way.

Validation:

• terraform validate: PASS
• terraform test -filter=tests/hub_spoke.tftest.hcl: PASS (1 passed, 0 failed)

[lweberru]

Namespace-scoped Kubernetes access update:

Implemented for each enabled landing zone namespace_service:

• Namespace-scoped ServiceAccount creation
• Namespace-scoped Role (no ClusterRoleBinding, no cluster-wide grant)
• Namespace-scoped RoleBinding to that ServiceAccount
• ServiceAccount token secret resource
• Output map for namespace users and generated namespace-scoped kubeconfigs

Validation:

• terraform validate: PASS
• terraform test -filter=tests/hub_spoke.tftest.hcl: PASS (1 passed, 0 failed)

Secrets Manager status:

• Enforced metadata contract in namespace annotations remains in place via
  stackit.cloud/secretsmanager-instance-id
• This establishes deterministic binding to the app landing zone Secrets Manager instance.
• Workload-side mandatory consumption enforcement (admission policy/operator-level) is not part of this module yet and remains a follow-up hardening item.

[lweberru]

Tracking update:

• Created dedicated follow-up issue for Secrets Manager workload-side enforcement:
  • Follow-up: Enforce Secrets Manager consumption for namespace workloads #36
• Verified OpenTofu execution path in addition to Terraform:
  • tofu init -backend=false -upgrade=false: PASS
  • tofu validate: PASS
  • tofu test -filter=tests/hub_spoke.tftest.hcl: PASS (1 passed, 0 failed)

Note:

• OpenTofu updates provider source addresses in .terraform.lock.hcl to registry.opentofu.org for hashicorp providers. This is expected behavior when running OpenTofu.