Skip to content

Commit e07c029

Browse files
SinaChavoshirlakhtakia
SinaChavoshi
authored and
rlakhtakia
committed
feat: Initial setup for conformance test suite (#720)
* feat: Initial setup for conformance test suite * fix missing go.sum entry * Fix the API version and basic Inferencepool-basic-accepted Yaml definition. * exclude conformance tests from github acceptance test run. * Add support for multiple profiles, remove release channel and update version to use semver. * Adding another layer to the report hierarchy for category of conformance (gateway, epp, model server). * Add trailing new line to yaml files. * switch to use InferencePoolMustHaveCondition from /conformance/utils/kubernetes * remove extra godoc comments * Remove references to ExtensionChannel from reports readme * format readme * remove the service for the conformance backend. * update the namespace and EEP names to match the manifest. * Update PR based on review feedback including, change dir name to lower case, remove unused manifest, remove NamespaceLabels and NamespaceAnnotations * add a comment to clarify use of echo server
1 parent dd32f1c commit e07c029

File tree

14 files changed

+671
-31
lines changed

14 files changed

+671
-31
lines changed

‎Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ vet: ## Run go vet against code.
121121

122122
.PHONY: test
123123
test: manifests generate fmt vet envtest image-build ## Run tests.
124-
KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test $$(go list ./... | grep -v /e2e) -race -coverprofile cover.out
124+
KUBEBUILDER_ASSETS="$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(LOCALBIN) -p path)" go test $$(go list ./... | grep -v /e2e | grep -v /conformance) -race -coverprofile cover.out
125125

126126
.PHONY: test-unit
127127
test-unit: ## Run unit tests.

‎conformance/conformance.go

Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
// Package conformance contains the core setup and execution logic
18+
// for the Gateway API Inference Extension conformance test suite.
19+
package conformance
20+
21+
import (
22+
"fmt"
23+
"io/fs"
24+
"os"
25+
"testing"
26+
27+
"github.com/stretchr/testify/require"
28+
apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
29+
clientset "k8s.io/client-go/kubernetes"
30+
31+
// Import runtime package for scheme creation
32+
"k8s.io/apimachinery/pkg/runtime"
33+
"k8s.io/apimachinery/pkg/util/sets"
34+
"sigs.k8s.io/controller-runtime/pkg/client"
35+
"sigs.k8s.io/controller-runtime/pkg/client/config"
36+
"sigs.k8s.io/yaml"
37+
38+
// Import necessary types and utilities from the core Gateway API conformance suite.
39+
// Assumes sigs.k8s.io/gateway-api is a dependency in the go.mod.
40+
gatewayv1 "sigs.k8s.io/gateway-api/apis/v1" // Import core Gateway API types
41+
confapis "sigs.k8s.io/gateway-api/conformance/apis/v1" // Report struct definition
42+
confconfig "sigs.k8s.io/gateway-api/conformance/utils/config"
43+
confflags "sigs.k8s.io/gateway-api/conformance/utils/flags"
44+
confsuite "sigs.k8s.io/gateway-api/conformance/utils/suite"
45+
"sigs.k8s.io/gateway-api/pkg/features" // Using core features definitions if applicable
46+
47+
// Import the test definitions package to access the ConformanceTests slice
48+
"sigs.k8s.io/gateway-api-inference-extension/conformance/tests"
49+
50+
// Import test packages using blank identifier
51+
// This triggers the init() functions in these packages, which register the tests
52+
// by appending them to the tests.ConformanceTests slice.
53+
_ "sigs.k8s.io/gateway-api-inference-extension/conformance/tests/basic"
54+
// TODO: Add blank imports for other test categories as they are created.
55+
// _ "sigs.k8s.io/gateway-api-inference-extension/conformance/tests/model_routing"
56+
57+
// Import the Inference Extension API types
58+
inferencev1alpha2 "sigs.k8s.io/gateway-api-inference-extension/api/v1alpha2"
59+
)
60+
61+
// GatewayLayerProfileName defines the name for the conformance profile that tests
62+
// the Gateway API layer aspects of the Inference Extension (e.g., InferencePool, InferenceModel CRDs).
63+
// Future profiles will cover EPP and ModelServer layers.
64+
const GatewayLayerProfileName confsuite.ConformanceProfileName = "Gateway"
65+
66+
var InferenceCoreFeatures = sets.New[features.FeatureName]() // Placeholder - Populate with actual features specific to this profile or manage features per profile
67+
68+
// GatewayLayerProfile defines the conformance profile for the Gateway API layer
69+
// of the Inference Extension.
70+
// In future iterations, we will add constants and ConformanceProfile structs for
71+
// EPPProfileName ("EPP") and ModelServerProfileName ("ModelServer")
72+
// to cover their respective conformance layers.
73+
var GatewayLayerProfile = confsuite.ConformanceProfile{
74+
Name: GatewayLayerProfileName,
75+
CoreFeatures: InferenceCoreFeatures,
76+
}
77+
78+
// DefaultOptions parses command line flags and sets up the suite options.
79+
// Adapted from the core Gateway API conformance suite.
80+
func DefaultOptions(t *testing.T) confsuite.ConformanceOptions {
81+
t.Helper()
82+
83+
cfg, err := config.GetConfig()
84+
require.NoError(t, err, "error loading Kubernetes config")
85+
86+
// Initialize client options. The scheme must include Gateway API types
87+
// and the Inference Extension types.
88+
clientOptions := client.Options{}
89+
scheme := clientOptions.Scheme
90+
if scheme == nil {
91+
// If default options don't provide a scheme, create one using runtime.NewScheme().
92+
scheme = runtime.NewScheme()
93+
clientOptions.Scheme = scheme
94+
}
95+
96+
// Register necessary API Types
97+
require.NoError(t, gatewayv1.Install(scheme)) // Add core Gateway API types
98+
// Add the Inference Extension API types to the scheme using the correct import alias
99+
require.NoError(t, inferencev1alpha2.Install(scheme))
100+
require.NoError(t, apiextensionsv1.AddToScheme(scheme)) // Needed for CRD checks
101+
102+
// Create the Kubernetes clients
103+
c, err := client.New(cfg, clientOptions)
104+
require.NoError(t, err, "error initializing Kubernetes client")
105+
cs, err := clientset.NewForConfig(cfg)
106+
require.NoError(t, err, "error initializing Kubernetes clientset")
107+
108+
exemptFeatures := confsuite.ParseSupportedFeatures(*confflags.ExemptFeatures)
109+
skipTests := confsuite.ParseSkipTests(*confflags.SkipTests)
110+
// Initially, run the GatewayLayerProfile. This will expand as other profiles
111+
// (EPP, ModelServer) are added and can be selected via flags in future iterations.
112+
conformanceProfiles := sets.New(GatewayLayerProfileName)
113+
114+
// Implementation details from flags
115+
implementation := confsuite.ParseImplementation(
116+
*confflags.ImplementationOrganization,
117+
*confflags.ImplementationProject,
118+
*confflags.ImplementationURL,
119+
*confflags.ImplementationVersion,
120+
*confflags.ImplementationContact,
121+
)
122+
123+
// Inference Extension Specific Report Fields
124+
inferenceExtensionVersion := "v0.3.0"
125+
_ = inferenceExtensionVersion // Avoid unused variable error until implemented
126+
127+
// Create ConformanceOptions
128+
opts := confsuite.ConformanceOptions{
129+
Client: c,
130+
Clientset: cs,
131+
RestConfig: cfg,
132+
GatewayClassName: *confflags.GatewayClassName,
133+
Debug: *confflags.ShowDebug,
134+
CleanupBaseResources: *confflags.CleanupBaseResources,
135+
SupportedFeatures: sets.New[features.FeatureName](), // Initialize empty, will be populated below
136+
TimeoutConfig: confconfig.DefaultTimeoutConfig(),
137+
SkipTests: skipTests,
138+
ExemptFeatures: exemptFeatures,
139+
RunTest: *confflags.RunTest,
140+
Mode: *confflags.Mode,
141+
Implementation: implementation,
142+
ConformanceProfiles: conformanceProfiles,
143+
ManifestFS: []fs.FS{&Manifests}, // Assumes embed.go defines `Manifests`
144+
ReportOutputPath: *confflags.ReportOutput,
145+
SkipProvisionalTests: *confflags.SkipProvisionalTests,
146+
// TODO: Add the inference extension specific fields to ConformanceOptions struct if needed,
147+
// or handle them during report generation.
148+
// GatewayAPIInferenceExtensionChannel: inferenceExtensionChannel,
149+
// GatewayAPIInferenceExtensionVersion: inferenceExtensionVersion,
150+
}
151+
152+
// Populate SupportedFeatures based on the GatewayLayerProfile.
153+
// Since all features are mandatory for this profile, add all defined core features.
154+
if opts.ConformanceProfiles.Has(GatewayLayerProfileName) {
155+
for feature := range GatewayLayerProfile.CoreFeatures {
156+
opts.SupportedFeatures.Insert(feature)
157+
}
158+
}
159+
160+
// Remove any features explicitly exempted via flags.
161+
for feature := range opts.ExemptFeatures {
162+
opts.SupportedFeatures.Delete(feature)
163+
}
164+
165+
return opts
166+
}
167+
168+
// RunConformance runs the Inference Extension conformance tests using default options.
169+
func RunConformance(t *testing.T) {
170+
RunConformanceWithOptions(t, DefaultOptions(t))
171+
}
172+
173+
// RunConformanceWithOptions runs the Inference Extension conformance tests with specific options.
174+
func RunConformanceWithOptions(t *testing.T, opts confsuite.ConformanceOptions) {
175+
t.Logf("Running Inference Extension conformance tests with GatewayClass %s", opts.GatewayClassName)
176+
177+
// Register the GatewayLayerProfile with the suite runner.
178+
// In the future, other profiles (EPP, ModelServer) will also be registered here,
179+
// and the suite runner will execute tests based on the selected profiles.
180+
confsuite.RegisterConformanceProfile(GatewayLayerProfile)
181+
182+
// Initialize the test suite.
183+
cSuite, err := confsuite.NewConformanceTestSuite(opts)
184+
require.NoError(t, err, "error initializing conformance suite")
185+
186+
t.Log("Setting up Inference Extension conformance tests")
187+
// Setup requires the list of tests, which is populated by the init() functions
188+
// triggered by the blank imports at the top of this file.
189+
cSuite.Setup(t, tests.ConformanceTests)
190+
191+
t.Log("Running Inference Extension conformance tests")
192+
// Run the tests.
193+
err = cSuite.Run(t, tests.ConformanceTests)
194+
require.NoError(t, err, "error running conformance tests")
195+
196+
// Generate and write the report if requested.
197+
if opts.ReportOutputPath != "" {
198+
t.Log("Generating Inference Extension conformance report")
199+
report, err := cSuite.Report() // Use the existing report generation logic.
200+
require.NoError(t, err, "error generating conformance report")
201+
202+
// TODO: Modify the report struct here if channel, version need to be modified.
203+
// Example (requires adding fields to confapis.ConformanceReport):
204+
// report.GatewayAPIInferenceExtensionChannel = opts.GatewayAPIInferenceExtensionChannel
205+
// report.GatewayAPIInferenceExtensionVersion = opts.GatewayAPIInferenceExtensionVersion
206+
207+
err = writeReport(t.Logf, *report, opts.ReportOutputPath)
208+
require.NoError(t, err, "error writing conformance report")
209+
}
210+
}
211+
212+
// writeReport writes the generated conformance report to the specified output file or logs it.
213+
// Adapted from the core Gateway API suite.
214+
func writeReport(logf func(string, ...any), report confapis.ConformanceReport, output string) error {
215+
rawReport, err := yaml.Marshal(report)
216+
if err != nil {
217+
return fmt.Errorf("error marshaling report: %w", err)
218+
}
219+
220+
if output != "" {
221+
if err = os.WriteFile(output, rawReport, 0o600); err != nil {
222+
return fmt.Errorf("error writing report file %s: %w", output, err)
223+
}
224+
logf("Conformance report written to %s", output)
225+
} else {
226+
// Log the report YAML to stdout if no output file is specified.
227+
logf("Conformance report:\n%s", string(rawReport))
228+
}
229+
return nil
230+
}

‎conformance/conformance_test.go

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package conformance
18+
19+
import (
20+
"testing"
21+
)
22+
23+
// TestConformance is the top-level function that runs the conformance tests.
24+
// It calls the RunConformance function which sets up the suite and executes
25+
// the registered tests.
26+
func TestConformance(t *testing.T) {
27+
// RunConformance is defined in conformance.go
28+
RunConformance(t)
29+
}

‎conformance/embed.go

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
/*
2+
Copyright 2025 The Kubernetes Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package conformance
18+
19+
import "embed"
20+
21+
// Manifests embeds the contents of the conformance/resources directory making
22+
// the YAML files within them available to the test suite at runtime.
23+
//
24+
//go:embed resources/* tests/*
25+
var Manifests embed.FS

‎conformance/reports/README.md

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# Conformance Reports for Gateway API Inference Extension
2+
3+
This directory stores conformance reports submitted by various implementations of the Gateway API Inference Extension. This structure closely follows the [kubernetes-sigs/gateway-api/conformance/reports](https://github.com/kubernetes-sigs/gateway-api/blob/main/conformance/reports/README.md).
4+
5+
## How this folder is structured
6+
7+
This folder stores conformance reports organized first by the version of the Gateway API Inference Extension specification they were tested against, and then by the specific conformance profile (e.g., Gateway, EPP, Model Server):
8+
9+
|-- conformance/reports
10+
| |-- v0.3.0 # Example extension version
11+
| | |-- gateway # Conformance profile/category
12+
| | | |-- my-inference-gateway
13+
| | | | |-- README.md
14+
| | | | |-- experimental-v1.2.3-default-gateway-report.yaml # Example report file
15+
| | | |-- another-implementation
16+
| | | | |-- README.md
17+
| | | | |-- ...
18+
| | |-- epp # Future conformance profile/category
19+
| | | |-- my-epp-implementation
20+
| | | | |-- ...
21+
| | |-- model-server # Future conformance profile/category
22+
| | | |-- ...
23+
| |-- v0.4.0 # Future extension version
24+
| | |-- ...
25+
26+
## Implementation Submissions
27+
28+
Each implementation conformant with a specific profile of a specific version of the Gateway API Inference Extension should have its own folder within the corresponding version and profile directory (e.g., `/conformance/reports/v0.3.0/Gateway/my-implementation/`).
29+
30+
The implementation is the owner of its folder and is responsible for:
31+
32+
1. Uploading one or more conformance reports (YAML files).
33+
2. Maintaining a mandatory `README.md` file within their folder, structured as follows:
34+
35+
# My Inference Gateway Implementation (Gateway Profile Conformance)
36+
37+
General information about the My/Implementation project.
38+
39+
## Table of Contents
40+
41+
| Extension Version Tested | Profile Tested | Implementation Version | Mode | Report |
42+
|--------------------------|----------------|------------------------|---------|----------------------------------------------------------------------------|
43+
| v0.3.0 | Gateway | v1.2.3 | default | [v1.2.3 Gateway report](./experimental-v1.2.3-default-gateway-report.yaml) |
44+
| ... | ... | ... | ... | ... |
45+
46+
## Reproduce
47+
48+
Instructions on how to reproduce the claimed report(s).
49+
50+
### Table of Contents (within Implementation README)
51+
52+
The table of contents within an implementation's `README.md` should contain one row for each submitted report and include the following columns:
53+
54+
* **Extension Version Tested**: The version of the Gateway API Inference Extension specification tested against (e.g., `v0.3.0`). Must correspond to the `gatewayAPIInferenceExtensionVersion` field in the report.
55+
* **Profile Tested**: The specific conformance profile tested (e.g., `Gateway`, `EPP`, `ModelServer`). Must correspond to the `name` of the profile in the `profiles` list within the report.
56+
* **Implementation Version**: A link to the GitHub/website page for the specific release/commit of the implementation tested. The version value MUST correspond to the `implementation.version` field in the report.
57+
* **Mode**: The operating mode of the implementation used for the test run (default is `default`). Must correspond to the `mode` field in the report. If a mode other than `default` is used, the "Reproduce" section must explain how to configure it.
58+
* **Report**: A link to the corresponding report YAML file. Reports MUST be named according to the pattern: `<Implementation Version>-<Mode>-<Profile>-report.yaml` (e.g., `experimental-v1.2.3-default-gateway-report.yaml`).
59+
60+
### Reproduce Section (within Implementation README)
61+
62+
This section MUST exist and contain the manual or automatic steps required to reproduce the results claimed by the uploaded conformance reports for that specific implementation. If reproduction steps differ significantly between implementation versions, use sub-sections.
63+
64+
## Report Files
65+
66+
Conformance reports MUST be uploaded exactly as generated by the official Gateway API Inference Extension conformance test suite, without any modifications. The "Reproduce" section allows for verification of the submitted report against a fresh run.
67+
68+
### Report Rules
69+
70+
To be accepted, submitted conformance reports must comply with the following rules:
71+
72+
1. **Implementation Details:** All fields within the `implementation` block must have meaningful values:
73+
* `organization`: The entity maintaining the implementation (company, open source org, individual).
74+
* `project`: The name of the implementation project, unique within the organization.
75+
* `url`: A valid URL for the project (e.g., GitHub repository, product page).
76+
* `version`: A specific, reproducible snapshot of the implementation (e.g., tag, commit hash, release version). Branch names are not acceptable.
77+
* `contact`: A list of contact points (GitHub handles like `@maintainer`, team handles like `@org/team`, email addresses, or support URLs like an issue tracker).
78+
2. **Inference Extension Versioning:** The report MUST include:
79+
* `gatewayAPIInferenceExtensionVersion`: The specific version of the Gateway API Inference Extension specification tested against (e.g., `v0.3.0`).
80+
3. **Mode:** The `mode` field indicates the implementation's operating mode during the test run.
81+
4. **Test Profile & Result:**
82+
* The report MUST contain exactly one profile result under the `profiles` list for the specific conformance category being submitted (e.g., a report for "Gateway" conformance should only contain the "Gateway" profile result).
83+
* The profile's `name` MUST match the conformance category (e.g., `Gateway`, `EPP`, `ModelServer`).
84+
* The profile's `result` field MUST be `success`. A `success` result indicates that **all** tests defined within the Gateway API Inference Extension conformance suite for that specific profile and version passed.
85+
86+
## Submission Process
87+
88+
Conformance reports demonstrating a `success` result for a specific profile (e.g., `Gateway`) should be submitted via Pull Request directly to this repository (`kubernetes-sigs/gateway-api-inference-extension`).
89+
90+
1. Create a new folder structure under `/conformance/reports/<extension-version>/<profile-name>/` named after your implementation (e.g., `/conformance/reports/v0.3.0/Gateway/my-implementation/`).
91+
2. Add your implementation's `README.md` to this folder, following the structure described above.
92+
3. Add your generated conformance report YAML file(s) to this folder, ensuring they follow the naming convention `<Implementation Version>-<Mode>-<Profile>-report.yaml`.
93+
4. Submit the Pull Request.

0 commit comments

Comments
 (0)