Test and benchmark KPHP code

Overview

Overview

ktest is a tool that makes kphp programs easier to test.

  • ktest phpunit can run PHPUnit tests using KPHP
  • ktest bench run benchmarks using KPHP
  • ktest bench-php run benchmarks using PHP
  • ktest bench-vs-php run benchmarks using both KPHP and PHP, compare the results
  • ktest benchstat compute and compare statistics about benchmark results (see benchstat)
  • ktest env print ktest-related env variables

Example - phpunit

Imagine that we have an ordinary PHPUnit test:

<?php

use PHPUnit\Framework\TestCase;
use ExampleLib\Strings;

class StringsTest extends TestCase {
    public function testContains() {
        $this->assertSame(Strings::contains('foo', 'bar'), false);
        $this->assertTrue(Strings::contains('foo', 'foo'));
    }

    public function testHasPrefix() {
        $this->assertSame(Strings::hasPrefix('Hello World', 'Hello'), true);
        $this->assertFalse(Strings::hasPrefix('Hello World', 'ello'));
    }
}

It comes without a surprise that you can run it with phpunit tool:

$ ./vendor/bin/phpunit tests

......                                                              6 / 6 (100%)

Time: 70 ms, Memory: 4.00 MB

OK (6 tests, 14 assertions)

When you're using phpunit, tests are executed as PHP, not KPHP.

ktest makes it possible to run your phpunit-compatible tests with KPHP:

$ ktest phpunit tests

.... 4 / 6 (66%) OK
.. 6 / 6 (100%) OK

Time: 10.74657386s

OK (6 tests, 14 assertions)

Note that running KPHP tests is slower: a separate binary is compiled per every Test class.

All you need is ktest utility and installed kphpunit package:

$ composer require --dev vkcom/kphpunit

Now let's do something more exciting.

Take a look at this Integers::getFirst method:

<?php

namespace Foo\Bar;

class Integers {
    /** @param int[] $xs */
    public static function getFirst(array $xs) {
        return $xs[0];
    }
}

It's intended to return the first int array item, or null, if index 0 is unset.

We can write a test for this method:

<?php

use PHPUnit\Framework\TestCase;
use Foo\Bar\Integers;

class IntegersTest extends TestCase {
    public function testGetFirst() {
        $this->assertSame(Integers::getFirst([]), null);
        $this->assertSame(Integers::getFirst([1]), 1);
    }
}

All tests are passing:

.                                                                   1 / 1 (100%)

Time: 36 ms, Memory: 4.00 MB

OK (1 test, 2 assertions)

But if you try to run it with ktest, you'll see how that code would behave in KPHP:

F 1 / 1 (100%) FAIL

Time: 4.59874429s

There was 1 failure:

1) IntegersTest::testGetFirst
Failed asserting that null is identical to 0.

IntegersTest.php:8

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

Accessing unset array index can yield a "zero value" instead of null.

Running with ktest makes it easier to ensure that your code behaves identically in both PHP and KPHP.

Example - bench

There are 2 main ways to do benchmarking with bench subcommand:

  1. Run different benchmarks and see how they relate
  2. Run benchmarks by the same name and compare samples with benchstat

Let's assume that you have a function that concatenates 3 strings. You can write a benchmark for it:

<?php

// file "BenchmarkConcat3.php"

class BenchmarkConcat3 {
    private static $strings = [
        'foo',
        'bar',
        'baz',
    ];

    public function benchmarkConcat() {
        return self::$strings[0] . self::$strings[1] . self::$strings[2];
    }
}

This benchmark can be executed with a bench subcommand:

$ ktest bench BenchmarkConcat3.php
class BenchmarkConcat3
BenchmarkConcat3::benchmarkConcat	106500	372.0 ns/op
ok BenchmarkConcat3 147.153797ms

Suppose that somebody proposed to re-write this function with ob_start() claiming that it would make things faster.

First, we need to collect samples of the current implementation. We need at least 5 rounds, but usually the more - the better (don't get too crazy though, 10 is good enough in most cases).

$ ktest bench -count 5 Concat3Benchmark.php | tee old.txt

Now we have old implementation results, it's time to roll the a implementation:

<?php

class Concat3Benchmark {
    private static $strings = [
        'foo',
        'bar',
        'baz',
    ];

    public function benchmarkConcat() {
        ob_start();
        echo self::$strings[0];
        echo self::$strings[1];
        echo self::$strings[2];
        return ob_get_clean();
    }
}

Now we need to collect the new implementation results:

$ ktest bench -count 5 Concat3Benchmark.php | tee new.txt

When you have 2 sets of samples, it's possible to compare them with benchstat:

$ ktest benchstat old.txt new.txt
name    old time/op  new time/op  delta
Concat   372ns ± 2%   546ns ± 6%  +46.91%  (p=0.008 n=5+5)

As we can see, the new implementation is, in fact, almost 2 times slower!

TODO

  • Mocks

Limitations

  • Assert functions can't be used for objects (class instances)
  • No custom comparators for assert functions
Issues
  • Flag `kphp2cpp-binary` does not properly handle paths with `~`

    Flag `kphp2cpp-binary` does not properly handle paths with `~`

    Command

    ktest bench --kphp2cpp-binary=~/kphp/objs/bin/kphp2cpp Benchmark.php
    

    Actual behavior

    build error: fork/exec ~/kphp/objs/bin/kphp2cpp: no such file or directory
    

    Expected behavior

    No error as for:

    ktest bench --kphp2cpp-binary=/home/pmakhnev/kphp/objs/bin/kphp2cpp Benchmark.php
    
    bug 
    opened by i582 2
  • internal/bench: fixed ab args

    internal/bench: fixed ab args

    Now if you try to run the command with some arguments:

    ./ktest bench-ab file1.php file2.php --kphp2cpp-binary ./bin/kphp2cpp
    

    Then the --kphp2cpp-binary flag (and other if exists) will not be used to run the benchmark, resulting in errors.

    This PR fixes this behavior and leaving the default flag logic if none were provided.

    bug 
    opened by i582 0
  • internal/bench: add benchmem flag support

    internal/bench: add benchmem flag support

    ktest bench -benchmem enables allocations tracking. In addition to CPU time, allocations stats will be reported for every benchmark.

    BenchmarkArray::Literal	460840	63.0 ns/op	72 B/op	1 allocs/op
    

    B/op - number of bytes allocated per operation. allocs/op - number of memory allocations per operation.

    Note that both stats are calculated "on average". So if your benchmark allocates only sometimes, it will affect the stats. This is done on purpose.

    They can be compared with benchstat later.

    name            old time/op    new time/op    delta
    Array::Literal    63.2ns ± 2%    14.0ns ± 0%   -77.85%  (p=0.008 n=5+5)
    name            old alloc/op   new alloc/op   delta
    Array::Literal     72.0B ± 0%      0.0B       -100.00%  (p=0.008 n=5+5)
    name            old allocs/op  new allocs/op  delta
    Array::Literal      1.00 ± 0%      0.00       -100.00%  (p=0.008 n=5+5)
    
    opened by quasilyte 0
  • internal/bench: for php mode, print to stderr directly

    internal/bench: for php mode, print to stderr directly

    Instead of printing to stderr the entire output in the end, redirect benchmark stderr to the provided writer (output). This way we get proper progress reporting in bench-vs-php mode.

    opened by quasilyte 0
  • internal/phpunit: add FFI testing support

    internal/phpunit: add FFI testing support

    When --enable-ffi flag is passed to ktest phpunit, tests runner will compile classes with enabled FFI mode. It'll also copy (by creating a symlink) ./ffilibs folder in FFI mode.

    opened by quasilyte 0
  • added ability to run benchmarks from namespace

    added ability to run benchmarks from namespace

    For example:

    <?php
    
    namespace Test\Something;
    
    class BenchmarkTest {
      public function benchmarkSomething() {
        echo 1;
      }
    
      public function benchmarkSomethingElse() {
        echo 1;
      }
    }
    

    Fixed #8

    enhancement 
    opened by i582 0
  • Add flag to KPHP run command to disable unnecessary SQL output

    Add flag to KPHP run command to disable unnecessary SQL output

    Example output:

    [17683][2021-12-06 04:53:56.504367 net-mysql-client.cpp  362] ok packet expected in response to authentification from connection 8 (127.0.0.1:3306)
    [17683][2021-12-06 04:53:56.504582 net-mysql-client.cpp  362] ok packet expected in response to authentification from connection 9 (127.0.0.1:3306)
    [17683][2021-12-06 04:53:56.505048 net-mysql-client.cpp  362] ok packet expected in response to authentification from connection 10 (127.0.0.1:3306)
    
    enhancement 
    opened by i582 0
  • added output in Teamcity format for benchmarks

    added output in Teamcity format for benchmarks

    This is necessary so that when we run the ktest through PhpStorm, it can parse the names and locations of the benchmarks correctly, as well as know when the benchmark started and ended.

    enhancement 
    opened by i582 0
  • added two new flags for KPHP

    added two new flags for KPHP

    • include-dirs: a list of additional directories that are passed when compiling the script using KPHP

    • disable-kphp-autoload: disables autoload for KPHP, leaving it only available for PHP

    Both flags are required for proper testing within vkcom.

    Also, now, if the PHP benchmark fails, it will show the PHP output with an error.

    enhancement 
    opened by i582 0
Releases(v0.7.3)
Owner
VK.com
VK.com
A best practices Go source project with unit-test and integration test, also use skaffold & helm to automate CI & CD at local to optimize development cycle

Dependencies Docker Go 1.17 MySQL 8.0.25 Bootstrap Run chmod +x start.sh if start.sh script does not have privileged to run Run ./start.sh --bootstrap

Quang Nguyen 4 Apr 4, 2022
Open Source runtime scanner for Linux containers (LXD), It performs security audit checks based on CIS Linux containers Benchmark specification

lxd-probe Scan your Linux container runtime !! Lxd-Probe is an open source audit scanner who perform audit check on a linux container manager and outp

Chen Keinan 12 Dec 27, 2021
Kubelet-bench - Example Go-based e2e benchmark for various Kubelet operations without spinning up whole K8s cluster

kubelet-bench An example of Go based e2e benchmark for various Kubelet operation

Bartlomiej Plotka 3 Mar 17, 2022
Test-at-scale - TAS - An intelligent test execution platform for engineering teams to achieve high development velocity

Test At Scale Test Smarter, Release Faster with test-at-scale. Status Table of c

LambdaTest 133 May 7, 2022
sail is an operation framework based on Ansible/Helm. sail follows the principles of Infrastructure as Code (IaC), Operation as Code (OaC), and Everything as Code. So it is a tool for DevOps.

sail 中文文档 sail is an operation framework based on Ansible/Helm. sail follows the principles of Infrastructure as Code (IaC), Operation as Code (OaC),a

Bougou Nisou 10 Dec 16, 2021
Boxygen is a container as code framework that allows you to build container images from code

Boxygen is a container as code framework that allows you to build container images from code, allowing integration of container image builds into other tooling such as servers or CLI tooling.

nitric 5 Dec 13, 2021
A simple Go app and GitHub workflow that shows how to use GitHub Actions to test, build and deploy a Go app to Docker Hub

go-pipeline-demo A repository containing a simple Go app and GitHub workflow that shows how to use GitHub Actions to test, build and deploy a Go app t

Marat Bogatyrev 0 Nov 17, 2021
Topology-tester - Application to easily test microservice topologies and distributed tracing including K8s and Istio

Topology Tester The Topology Tester app allows you to quickly build a dynamic mi

Bas van Beek 1 Jan 14, 2022
Kubernetes-native framework for test definition and execution

████████ ███████ ███████ ████████ ██ ██ ██ ██ ██████ ███████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ █████

kubeshop 256 May 7, 2022
General-purpose actions for test and release in Go

go-actions This repository provides general-purpose actions for Go. setup This action runs actions/setup-go with actions/cache. For example, jobs: l

Hidetake Iwata 1 Nov 28, 2021
k6 extension to load test Apache Kafka with support for Avro messages and SASL Authentication

xk6-kafka This project is a k6 extension that can be used to load test Kafka, using a producer. Per each connection to Kafka, many messages can be sen

null 0 Dec 7, 2021
Aceptadora provides the boilerplate to orchestrate the containers for an acceptance test.

aceptadora Aceptadora provides the boilerplate to orchestrate the containers for an acceptance test. Aceptadora is a replacement for docker-compose in

Cabify 55 May 11, 2022
This is a K6 extension to be able to test using NATS protocol

xk6-nats This is a k6 extension using the xk6 system, that allows to use NATS protocol. ❗ This is a proof of concept, isn't supported by the k6 team,

Yeray Darias 3 Apr 18, 2022
A docker image for test redis HA

Guide 测试集群模式(Test Redis Cluster) redis-test --arch cluster 127.0.0.1:6379 127.0.0.1:6380 127.0.0.1:6381 测试哨兵模式(TEST Redis Sentinel) redis-test --arc

DevineLiu 1 Nov 30, 2021
used Terratest to write a test in GO for validating a Terraform module.

Terraform--Terragrant--Test used Terratest to write a test in GO for validating a Terraform module. will write a test for a Terraform module using Ter

EngineerAdnan-DEVOPS 0 Dec 4, 2021
Test Operator using operator-sdk 1.15

test-operator Test Operator using operator-sdk 1.15 operator-sdk init --domain rbt.com --repo github.com/ravitri/test-operator Writing kustomize manif

Ravi Trivedi 0 Dec 28, 2021
Output all versions of a local git repo, which could be used as test data for your ML program.

gitwalker Output all versions of a local git repo, which could be used as test data for your ML program. Notice This program is under development. Cur

zhuaiballl 0 Dec 27, 2021
Stockbit Test : Aldo Verrell Valiant

stockbit-test Stockbit Test : Aldo Verrell Valiant In this repository, there are answers to questions number 1,3, and 4. The answer to question number

null 0 Dec 28, 2021
Stockbit Test no. 2

movies_service Stockbit Test no. 2 This microservice is built with Clean Architecture. There are 2 routes: Search & Get movie detail by ID http://loca

null 0 Dec 28, 2021