Intermezzo about stability and compliance

When building a large scale high availability and high performance system you will need to scale horizontally in number of nodes, as compared to boosting a single node with higher performing hardware. You will also need to find components that does not limit the capabilities of underlying hardware. In a system with 1000 nodes an increase in performance of 30% means you need 230 less nodes also reducing data center cost, power consumption and maintenance cost. This does add up to a serious amount of time and money.

If you run a small scale solution quite often the 30% means little or nothing. Many will go for the “camera with the the megapixels” regardless of what they are going to use it for or the actual quality of the pictures.

Assuming your system is not overloaded what will decide the perceived quality of it will be that it actually works, that there is no down time, and of course that the integrity of the data isn’t compromised.

I wrote in a comment regarding benchmarks that I have trust issues with closed source components having worked with computer security for more than 15 years. I also wrote “Of course there are plenty of excellent products that aren’t open source, and not seldom they are more mature, stable and well written than the open source alternatives, but when they are on par I personally prefer to build on open source when possible since it gives me more control. But this is just my own preference, I’m not saying it applies to anybody else.”

G-WAN

Since I was benchmarking G-WAN and it was the only closed source web server I decided to do a black box test of the software. G-WAN has come out on top in some benchmarks, especially ones targeted for keep-alive requests of small files, and is promoted quite aggressively by the vendor as the only really scalable web server solution out there.

In my eyes Pierre deserves a lot of respect for, from what I can tell, making a huge effort in providing value for the community using G-WAN and pushing the limits for web server performance.

As another user “X4” pointed out, you can definitely not assume that a software that is open source is secure and without vulnerabilities. However commonly used open source components that are widely used are likely to be subject for reviews at some point. As a positive trend more and more commercial vendors are creating policies and guidelines for regularly doing source code reviews and security tests by third parties. Even huge corporations like Microsoft has improved massively over the last decade.

If you are a smaller commercial vendor or even a single developer writing closed source in many cases the author will be the only one reviewing the quality of the code. Being a small vendor you often are also under a lot of pressure to perform, being flexible, and keeping a high profile to keep up with competition.

Some observations

I’ll write about some findings that are more indications rather that serious vulnerabilities.

Sandbox

[root@wolf test]# curl localhost/..
Index of /..</pre>
<h2>Index of /..</h2>
<table width="94%">
<tbody>
<tr>
<th width="54%">name</th>
<th width="25%">last modified</th>
<th width="15%">size</th>
</tr>
<tr>
<td><a href="/../www">www</a></td>
<td>01-Jan-1970 00:00</td>
<td>1</td>
</tr>
<tr>
<td><a href="/../_logs">_logs</a></td>
<td>01-Jan-1970 00:00</td>
<td>1</td>
</tr>
<tr>
<td><a href="/../handlers">handlers</a></td>
<td>01-Jan-1970 00:00</td>
<td>1</td>
</tr>
</tbody>
</table>
<pre>

I haven’t found any serious implications of this since from what I can see you can’t traverse further down or up from this point. But from a more emotional point of view this does not give you a great feeling of security.

Parsing

[root@wolf test]# echo -e "GET /%31+%32=%33\r\n\r\n\r\n" | nc localhost 80
404 Not Found

</pre>
<h1>Not Found</h1>
<pre>
[...]
[pid 20618] stat64("/gwan/0.0.0.0_80/#0.0.0.0/www/1+2=332=%33", 0xf7595cc0) = -1 ENOENT (No such file or directory)

The parsing for HTTP v0.9 seems seriously broken, with ‘+’ not being replaced and the line not being terminated correctly after decoding.

Longer paths

[root@wolf gwan]# cd 0.0.0.0_80/#0.0.0.0/www/
[root@wolf www]# mkdir -p `perl -e "print 'a'x200,'/','a'x200,'/','a'x200"`
[root@wolf www]# (cd ../../../; strace ./gwan)
[...]
stat64("/gwan/0.0.0.0_80/#0.0.0.0/www/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/////////////////////////////////////////////////////////////////////", {st_mode=S_IFDIR|0755, st_size=1024, ...}) = 0
open("/gwan/0.0.0.0_80/#0.0.0.0/www/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/////////////////////////////////////////////////////////////////////", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 1036
getdents(1036, /* 1 entries */, 32768)  = 244
stat64("/gwan/0.0.0.0_80/#0.0.0.0/www/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/////////////////////////////////////////////////////////////////////", {st_mode=S_IFDIR|0755, st_size=1024, ...}) = 0
open("/gwan/0.0.0.0_80/#0.0.0.0/www/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/////////////////////////////////////////////////////////////////////", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 1037

The program is caught eating resources in an infinite loop since the length of the path is considered too long.

Optimizations

[root@wolf gwan]# echo test > 0.0.0.0_80/#0.0.0.0/www/test
[root@wolf gwan]# curl localhost/test
test
[...]
[pid 20753] accept(7, 0xf75552e0, [16]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 20753] epoll_wait(6, {{EPOLLIN, {u32=8, u64=8}}}, 1024, 1000) = 1
[pid 20753] read(8, "GET /test HTTP/1.1\r\nUser-Agent: "..., 2041) = 159
[pid 20753] stat64("/gwan/0.0.0.0_80/#0.0.0.0/www/test", {st_mode=S_IFREG|0644, st_size=5, ...}) = 0
[pid 20753] writev(8, [{"HTTP/1.1 200 OK\r\nServer: G-WAN\r\n"..., 274}, {"test\n", 5}], 2) = 279
[pid 20753] epoll_wait(6, {{EPOLLIN|0x2000, {u32=8, u64=8}}}, 1024, 1000) = 1
[pid 20753] epoll_ctl(6, EPOLL_CTL_DEL, 8, {EPOLLIN|EPOLLERR|EPOLLHUP|0x2000, {u32=8, u64=8}}) = 0
[pid 20753] shutdown(8, SHUT_WR)        = 0

If you pay attention you will find that there is no “open” syscall, which is related to the note above. What G-WAN actually does is to recurse and stat everyfile in the webroot, and open and cache files below a certain size.

This may seem like a great idea but in a large scale deployment you typically have a distributed setup, with distributed storage over network, and not seldom many files. Working with a CDN you are talking about many many millions of files at least. Obviously this would simply kill such an environment.

Compliance

[root@wolf gwan]# curl localhost/test?some=option
404 Not Found

</pre>
<h1>Not Found</h1>
<pre>
The requested URL was not found on this server.

[...]
[pid 20801] accept(7, 0xf753e2e0, [16]) = -1 EAGAIN (Resource temporarily unavailable)
[pid 20801] epoll_wait(6, {{EPOLLIN, {u32=8, u64=8}}}, 1024, 1000) = 1
[pid 20801] read(8, "GET /test?some=option HTTP/1.1\r\n"..., 2041) = 171
[pid 20801] stat64("/gwan/0.0.0.0_80/#0.0.0.0/www/test?some=option", 0xf753ccc0) = -1 ENOENT (No such file or directory)
[pid 20801] writev(8, [{"HTTP/1.1 404 Not Found\r\nServer: "..., 163}, {"[pid 20801] epoll_wait(6, {{EPOLLIN|0x2000, {u32=8, u64=8}}}, 1024, 1000) = 1
[pid 20801] epoll_ctl(6, EPOLL_CTL_DEL, 8, {EPOLLIN|EPOLLERR|EPOLLHUP|0x2000, {u32=8, u64=8}}) = 0
[pid 20801] shutdown(8, SHUT_WR)        = 0

It turns out G-WAN does not parse the query part of the URL at all. I suppose the argument would be this is a static file but the problem is that not seldom you use URL parameters to “tag” references to static files, for example with banner distribution, and here you simply end up being handed a 404.

Other things…

The above things were less serious indications. There were other very serious problems, things even ASLR won’t help you with, but I will contact the vendor first and give sufficient time to work with those before publishing detailed advisories about this.

Advertisements

2 thoughts on “Intermezzo about stability and compliance”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s