February 8, 2017 Hans-Peter Martini h2p3m

Simplifying Debugging of UI Tests

Selenium is a great tool for running UI based tests. You can get automated browser tests up and running in a matter of hours. However, as easy as Selenium is to set up. Selenium tests can be unstable, sometimes they work and now and then they fail.

Understanding why a Selenium test failed can often be a painful and time-consuming process. We have created this feature to take the pain out, and drastically reduce the time you spend to figure the causes out.

Smart Display Recordings

For all of your test commands in the tests section of your .scrutinizer.yml, we record any activity that occurs on the build machine’s display. If your test fails, the recording is stored and you can use it for quickly debugging the failure. Let’s take a look:

https://d2hs8c246tsqgl.cloudfront.net/blog/simple-debugging-of-ui-tests/65b71e0d11e9afe7021c74d3f7b3c4db-1b.gif

The display recording and the console output of the running task are displayed in sync. As the video is played, we re-play the output as it happened when your tests were running. So when you see an error on the display, you can see which step was executing on the command line and vice-versa.

Recordings are automatically enabled for all repositories, and even if you heavily use the UI, they cause virtually no overhead. We observed a maximum of 3% added CPU usage. If your task succeeds, the recording will be discarded. If it fails, we encode it at the end of your run so that you can view it in your browser.

Live Display Stream

While an inspection is running in SSH debug mode, you can now also view a live stream of the display without having to go through any complicated set-up directly in the browser. Checkout the new “Display” tab:

https://d2hs8c246tsqgl.cloudfront.net/blog/simple-debugging-of-ui-tests/73466d029c7ecd280b9df67b3724e414-2a.png

We think this will make Selenium tests a lot more enjoyable.

Let us know what you think! :)


March 9, 2016 Johannes Schmitt schmittjoh

View Analysis Results inside GitHub

We are happy to announce the availability of our new Chrome extension. It boosts productivity by allowing you to now view analysis result directly inside GitHub. You do not need to leave the page, but you can go through the issues just when you manually review the code:

Inline Issue on GitHub

The extension is still in beta as we add some more features. You can check out our documentation for the features that are already available.

Happy & productive scrutinizing!


April 21, 2015 Johannes Schmitt schmittjoh

December 4, 2014 Johannes Schmitt schmittjoh

Composer, the GC performance improvement, and who else might be affected

Chances are you might have heard of sensational performance improvements in PHP’s package manager, composer, of 30-90% of original runtime by changing not even a single line of code.

I initially suggested this change to the composer team on December 1st, and after some positive reports, it made its way into the production version of composer soon after, and is since then available to all PHP users.

How was the issue found?

At Scrutinizer, we run a lot of CLI-based analysis tools, some of which are written in PHP, and some of these create a lot of objects f.e. during type-inference.

Very early in Scrutinizer’s development (maybe something like 1.5 years ago), we noticed that the analysis time of bigger packages was not increasing linearly, but rather expontentially. Of course, we used profilers like XHProf to track down the issue, but the results were inconclusive; time was spent randomly in different places.

After some research, we came to the conclusion that garbage collection might be responsible, and indeed after disabling it, the analysis of a bigger package like Symfony2 went down from over 50 minutes to less than 10 minutes (this was more than a year ago).

It was only until a few days ago, that we realized that composer did not contain this fix, and might benefit from this, too. When a performance improvement that Nils made had a significantly different impact depending on how many dependencies someone had. The rest of the story is history now :)

Is your app affected, too?

Chances are most likely, no. This particular bad behavior of PHP’s GC only manifests itself in long running CLI processes that create tens of thousands of objects. Web-requests are likely not affected.

If you profile your application, and time is randomly spent in different places from run to run. This is the usual pattern to look for. If you see this, only then try running with GC disabled.

There are also some initiatives now to provide better support for profilers to measure the time spent in GC which will make this even easier.

Is memory consumption increased when disabling GC?

The short, and counter-intuitive answer is no. You will not see a change in your memory consumption.

First, the call to gc_disable() does not turn off garbage collection entirely, it just disables one particular GC strategy that cleans up circular references.

Second, if you see this particular performance degradation, the garbage collection is not working anyway. It basically tries to clean-up your objects, only to recognize that it cannot clean them up, and it does that frequently. This is where the time is lost and the huge improvement is coming from when disabling it.

For a detailed explanation of the different GC strategies in PHP, check out this blog post from Anthony.

Which other tools might be affected?

Like said above, this is mostly something that affects CLI tools. One prime candidate that comes to mind is phpunit. If you have a big test-suite, try running it with GC disabled:

php -d zend.enable_gc=0 vendor/bin/phpunit

October 28, 2014 Johannes Schmitt schmittjoh

Security Audit of Request Data for PHP

The Open Web Application Security Project (OWASP) considered Injection attacks to be the most critical security risk in 2013. There are many different forms of injection attacks such as SQL injection, Path injection, Command injection, and many more. Injection attacks especially hurtful because they are easy to exploit, and can cause severe damage.

At Scrutinizer, we already notified you of SQL injection issues f.e. if you did not use parameter binding offered by PHP’s PDO abstraction, or libraries like Doctrine. However, there is a whole range of other PHP functions that are safe per-se, but can lead to severe security issues if passed arbitrary user input. file_get_contents for example is one of these functions. Malicious users can use it to gain access to sensitive credentials from your application.

Flagging each call to file_get_contents would be a very naive approach and not really helpful. However, at Scrutinizer we have invested into a solid foundation. Our PHP analyzer is almost like a compiler for PHP and uses advanced techniques such as data flow analysis, and abstract interpretation which distinguishes us from other competitors, or also existing open-source tools that only use AST-based approaches. It are these very analysis techniques that help us understand how data flows through your application.

In the most recent version of PHP analyzer, we have added a security analysis framework that specifically focuses on making sure your request data is not ending up in one of PHP’s sensitive functions, and that consequentially could make you vulnerable to injection attacks. Let’s take a look at an example:

use Symfony\Component\Yaml\Yaml;
use Symfony\Component\HttpFoundation\Request;

class Instance
{
    private $config;

    public function __construct($config)
    {
        $this->config = $config;
    }

    public function getParsedConfig()
    {
        return Yaml::parse($this->config);
    }
}

class MyController
{
    public function createAction(Request $request)
    {
        $instance = new Instance($request->request->get('config'));
        // ... save new instance, etc.
    }
}

In this example, we are assigning a YAML configuration from request data to a class property, and then save the new instance. Let’s see what Scrutinizer finds when analyzing this code:

File-Expansion Security Issue

This might come a bit unexpected. There is no problem with database persistence here, Doctrine takes care of escaping the content for us, but we still introduced a security issue by passing raw request data to the Yaml::parse() function which in turn calls file_get_contents(). Let’s take a look at it to see what is happening there:

class Yaml
{
    public static function parse($input, $exceptionOnInvalidType = false, $objectSupport = false)
    {
        // if input is a file, process it
        $file = '';
        if (strpos($input, "\n") === false && is_file($input)) {
            if (false === is_readable($input)) {
                throw new ParseException(sprintf('Unable to parse "%s" as the file is not readable.', $input));
            }

            $file = $input;
            $input = file_get_contents($file);
        }

        // ...
    }
}

This function has a feature that expands the input in case it is a valid file path, and replaces it with the contents of that path before parsing its actual contents. If someone were to pass a path such as ../app/config/parameters.yml, he might gain access to the parameters file of a Symfony2 application.

I would like to point out that we have not hard-coded the Yaml::parse function in PHP analyzer, but since we not only analyze your application, but also all your dependencies. We also assess the security relevance of the specific dependency version that your application uses. Analyzing your dependencies is even more important as you might not be as familiar with your dependencies’ code and consequentially also not aware of some of their features such as the file expansion shown above. This also allows us to compile a stack trace for you that aids in assessing the security issue more easily.

We are proud to now add this additional level of security to your PHP applications on Scrutinizer. Currently, we support tracking request data from Symfony2’s, and Zend’s request abstraction, as well as usage of PHP’s super globals like $_GET. If you are using another request data abstraction, just let us know by opening an issue.

Happy & secure coding!