test: proof of concept to replace BDD Gherkin tests

[susnux]

Motivation

Every time we discussed integration tests it was concluded that no backend engineer liked the BDD tests (behat integration tests) as the gherkin language just adds a artificial language layer on top of the PHP fixtures.
But no non-technical person ever worked on the tests, meaning the one argument for such gherkin tests is not fulfilled.

Writing new integration tests is a burden that prevents many needed regression tests and thus negatively affects the Nextcloud QA.

There are various solutions available as alternatives, I have considered following approaches:

• Use PEST + guzzle
• Use PHPUnit + guzzle
• Use Playwright requests
• Keep behat

PEST

Pest is a slim framework on-top of PHPUnit it allows for slimmer test case code as you no longer need to write full test classes but just use functional approach test('what i am testing', function () { expect(1)->toBe(1); }).

➕	➖
Reduces the boiler plate for every test code	Introduces a new dependency
Parallel test runner	New test styles can lead to inconsistencies in unit tests as well
Not that popular in PHP world

PHPUnit

➕	➖
Known framework in Nextcloud	Verbose test syntax
Same tech-stack for backend engineers	No parallel test runner
Write tests just like code	Tests require understanding code

Use PlayWright

➕	➖
Reuse infrastructure from e2e tests	Tests written in Typescript = different tech stack for backend engineers
Reuse fixtures and helpers from e2e tests	No direct access to server internal state anymore
Requests and JSON handling is language native
Fully parallel test runner

Comparison

As PEST only is a layer on top of PHPUnit I skipped this for the proof of concept, I personally also do not think it makes sense to introduce this new test pattern in Nextcloud backend.

For performance:

• Behat: 43s
• PHPUnit: 37s
• Playwright: 38s

The test was a bit unfair as it mostly waits for the rate limit to cooldown, so a test like the sharing features is most likely much faster on Playwright than on the others as it can make use of fully parallel web requests.

PHPUnit	Playwright
Keep PHP tech stack	Easier working with requests
Migration is very simple as we just call existing behat helpers	We can reuse and share helpers created for e2e tests
Still allows access to internals of the running server	Faster when many requests are done in one test

Summary

I think that both alternatives Playwright and PHPUnit have their benefits so this is up for a discussion of all contributors.
From my point of view both are better readable and maintainable as an software engineer when compared with behat - at least when working in the Nextcloud ecosystem.

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)
• Labels added where applicable (ex: bug/enhancement, 3. to review, feature component)
• Milestone added for target branch/version (ex: 32.x for stable32)

AI (if applicable)

• The content of this PR was partly or fully generated using AI

[provokateurin]

I'm strongly in favor of going with PHPUnit.

Would we be able to directly use code from server inside the tests, e.g. to set config options instead of having to go through the OCS APIs? That would be a major benefit IMO.

[artonge]

Would we be able to directly use code from server inside the tests, e.g. to set config options instead of having to go through the OCS APIs? That would be a major benefit IMO.

I think if those are integration tests, then I guess we should not access internals to set the state. Fredinand's example is using occ to set the config for example.

---

I would also be in favor of PHPUnit, it is faster than cypress, and it is already known. Only caveat, as it needs a live instance, we need to provide a good way of running the tests locally, just like we are doing with the current ones.

Using PHPUnit is the way we went for Circles (example). It is more barebone than your example, and uses internal code to set the state as Kate suggested. But I think your way of provinding primitives to set the state is much cleaner.

Also, unsure how to manage the test accross the entire test suite of server, which is massive. Not sure how the current ones are doing it. I think to remember that nothing specific is done, so changing one test could afect another one.

Small rant: I think all our PHPUnit tests that access the DB should eventually be integration tests like in your example, so that actual unit tests can run without a live instance.

[provokateurin]

I think if those are integration tests, then I guess we should not access internals to set the state

Yes, I just think it would make the tests execute faster and easier to write.