This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Run Prow integration tests

Run all integration tests

./test/integration/integration-test.sh

Run a specific integration test

./test/integration/integration-test.sh -run=TestIWantToRun

Cleanup

./test/integration/teardown.sh -all

Adding new integration tests

New component

Assume we want to add most-awesome-component (source code in cmd/most-awesome-component).

  1. Add most-awesome-component to the PROW_COMPONENTS, PROW_IMAGES, and PROW_IMAGES_TO_COMPONENTS variables in lib.sh.

    • Add the line most-awesome-component to PROW_COMPONENTS.
    • Add the line [most-awesome-component]=cmd/most-awesome-component to PROW_IMAGES.
    • Add the line [most-awesome-component]=most-awesome-component to PROW_IMAGES_TO_COMPONENTS.
    • Explanation: PROW_COMPONENTS lists which components are deployed into the cluster, PROW_IMAGES describes where the source code is located for each component (in order to build them), and finally PROW_IMAGES_TO_COMPONENTS defines the relationship between the first two variables (so that the test framework knows what to redeploy depending on what image has changed). As an example, the deck and deck-tenanted components (in PROW_COMPONENTS) both use the deck image (defined in PROW_IMAGES_TO_COMPONENTS), so they are both redeployed every time you change something in cmd/deck (defined in PROW_IMAGES).

  2. Set up Kubernetes Deployment and Service configurations inside the [configuration folder][config/prow/cluster] for your new component. This way the test cluster will pick it up when it deploys Prow components.

    • If you want to deploy an existing Prow component used in production (i.e., https://prow.k8s.io), you can reuse (symlink) the configurations used in production. See the examples in [configuration folder][config/prow/cluster].

    • Remember to use localhost:5001/most-awesome-component for the image: ... field in the Kubernetes configurations to make the test cluster use the freshly-built image.

New tests

Tests are written under the test directory. They are named with the pattern <COMPONENT>_test.go*. Continuing the example above, you would add new tests in most-awesome-component_test.go

Check that your new test is working

  1. Add or edit new tests (e.g., func TestMostAwesomeComponent(t *testing.T) {...}) in most-awesome-component_test.go.
  2. Run ./test/integration/integration-test.sh -run=TestMostAwesomeComponent to bring up the test cluster and to only test your new test named TestMostAwesomeComponent.
  3. If you need to make changes to most-awesome-component_test.go (and not the component itself), run ./test/integration/integration-test.sh -run=TestMostAwesomeComponent -no-setup. The -no-setup will ensure that the test framework avoid redeploying the test cluster.
    • If you do need to make changes to the Prow component, run ./test/integration/integration-test.sh -run=TestMostAwesomeComponent -build=most-awesome-component so that cmd/most-awesome-component is recompiled and redeployed into the cluster before running TestMostAwesomeComponent.

If Step 2 succeeds and there is nothing more to do, you’re done! If not (and your tests still need some tweaking), repeat steps 1 and 3 as needed.

How it works

In short, the integration-test.sh script creates a KIND Kubernetes cluster, runs all available integration tests, and finally deletes the cluster.

Recall that Prow is a collection of services (Prow components) that can be deployed into a Kubernetes cluster. KIND provides an environment where we can deploy certain Prow components, and then from the integration tests we can create a Kubernetes Client to talk to this deployment of Prow.

Note that the integration tests do not test all components (we need to fix this). The PROW_COMPONENTS variable is a list of components currently tested. These components are compiled and deployed into the test cluster on every invocation of integration-test.sh.

Each tested component needs a Kubernetes configuration so that KIND understands how to deploy it into the cluster, but that’s about it (more on this below). The main thing to keep in mind is that the integration tests must be hermetic and reproducible. For this reason, all components that are tested must be configured so that they do not attempt to reach endpoints that are outside of the cluster. For example, this is why some Prow components have a -github-endpoint=... flag that you can use — this way these components can be instructed to talk to the fakeghserver deployed inside the cluster instead of trying to talk to GitHub.

Code layout

.
├── cmd # Binaries for fake services deployed into the test cluster along with actual Prow components.
│   ├── fakegerritserver # Fake Gerrit.
│   ├── fakeghserver # Fake GitHub.
│   └── fakegitserver # Fake low-level Git server. Can theoretically act as the backend for fakeghserver or fakegerritserver.
├── config # Kubernetes configuration files.
│   └── prow # Prow configuration for the test cluster.
│       ├── cluster # KIND test cluster configurations.
│       └── jobs # Static Prow jobs. Some tests use these definitions to run Prow jobs inside the test cluster.
├── internal
│   └── fakegitserver
└── test # The actual integration tests to run.
    └── testdata # Test data.

1 - Fake Git Server (FGS)

FGS is actually not a fake at all. It is a real web server that serves real Git repositories them over HTTP. FGS wraps around the vanilla git http-backend subcommand that comes with Git, calling it as a CGI executable. It supports both read (e.g., git clone, git fetch) and write (e.g., git push) operations against it.

FGS is used for integration tests. See TestClonerefs for an example.

Usage in Integration Testing

The fakegitserver.go file is built automatically by hack/prowimagebuilder, and we deploy it to the KIND cluster. Inside the cluster, it accepts web traffic at the endpoint http://fakegitserver.default (http://localhost/fakegitserver from outside of the KIND cluster).

There are 2 routes:

  • /repo/<REPO_NAME>: endpoint for Git clients to interact (git clone, git fetch, git push). E.g., git clone http://fakegitserver.default/repo/foo. Internally, FGS serves all Git repo folders under -git-repos-parent-dir on disk and serves them for the /repo route with the git-http-backend CGI script.
  • POST /setup-repo: endpoint for creating new Git repos on the server; you just need to send a JSON payload like this:
{
  "name": "foo",
  "overwrite": true,
  "script": "echo hello world > README; git add README; git commit -m update"
}

Here is a cURL example:

# mkFoo is a plaintext file containing the JSON from above.
$ curl http://localhost/fakegitserver/setup-repo -d @mkFoo
commit c1e4e5bb8ba0e5b16147450a75347a27e5980222
Author: abc <d@e.f>
Date:   Thu May 19 12:34:56 2022 +0000

    update

Notice how the server responds with a git log output of the just-created repo to ease debugging in case repos are not created the way you expect them to be created.

During integration tests, each test creates repo(s) using the /setup-repo endpoint as above. Care must be taken to not reuse the same repository name, as the test cases (e.g., the test cases in TestClonerefs) all run in parallel and can clobber each other’s repo creation setp.

Allowing Push Access

Although this is not (yet) used in tests, push access is enabled for all served repos. This is achieved by setting the http.receivepack Git configuration option to true for each repo found under -git-repos-parent-dir. This is because the git http-backend script does not by default allow anonymous push access unless the aforementioned option is set to true on a per-repo basis.

Allowing Fetching of Commit SHAs

By default the CGI script will only serve references that are “advertised” (such as those references under refs/heads/* or refs/pull/*/head). However, FGS also sets the uploadpack.allowAnySHA1InWant option to true to allow Git clients (such as clonerefs) to fetch commits by their SHA.

Local Usage (for debugging)

FGS has 2 requirements:

  1. the path to the local git binary installation, and
  2. the path to a folder containing Git repositories to be served (can be an empty directory, or pre-populated).

By default port 8888 is used, although this can also be configured with -port.

Example:

$ go run fakegitserver.go -h
Usage of /tmp/go-build2317700172/b001/exe/fakegitserver:
  -git-binary git
        Path to the git binary. (default "/usr/bin/git")
  -git-repos-parent-dir string
        Path to the parent folder containing all Git repos to serve over HTTP. (default "/git-repo")
  -port int
        Port to listen on. (default 8888)

$ go run fakegitserver.go -git-repos-parent-dir <PATH_TO_REPOS> -git-binary <PATH_TO_GIT>
{"component":"unset","file":"/home/someuser/go/src/sigs.k8s.io/prow/pkg/test/integration/fakegitserver/fakegitserver.go:111","func":"main.main","level":"info","msg":"Start server","severity":"info","time":"2022-05-22T20:31:38-07:00"}

In this example, http://localhost:8888 is the HTTP address of FGS:

# Clone "foo" repo, assuming it exists locally under `-git-repos-parent-dir`.
$ git clone http://localhost:8888/repo/foo
$ cd foo
$ git log # or any other arbitrary Git command
# ... do some Git operations
$ git push

That’s it!

Local Usage with Docker and Ko (for debugging)

It may be helpful to run FGS in a containerized environment for debugging. First install ko itself. Then cd to the fakegitserver folder (same folder as this README.md file), and run:

# First CD to the root of the repo, because the .ko.yaml configuration (unfortunately)
# depends on relative paths that can only work from the root of the repo.
$ cd ${PATH_TO_REPO_ROOT}
$ docker run -it --entrypoint=sh -p 8123:8888 $(ko build --local sigs.k8s.io/prow/pkg/test/integration/fakegitserver)

The -p 8123:8888 allows you to talk to the containerized instance of fakegitserver over port 8123 on the host.

Custom Base Image

To use a custom base image for FGS, change the baseImageOverrides entry for fakegitserver in .ko.yaml like this:

baseImageOverrides:
  # ... other entries ...
  sigs.k8s.io/prow/pkg/test/integration/fakegitserver: gcr.io/my/base/image:tag

If you want ko to pick up a local Docker image on your machine, rename the image to have a ko.local prefix. For example, like this:

baseImageOverrides:
  sigs.k8s.io/prow/pkg/test/integration/fakegitserver: ko.local/my/base/image:tag