Cowboy web server application very slow

Question

I am currently playing around with minimal web servers, like Cowboy. I want to pass a number in the URL, load lines of a file, sort these lines and print the element in the middle to test IO and sorting. So the code loads the path like /123, makes a padded "00123" out of the number, loads the file "input00123.txt" and sorts its content and then returns something like "input00123.txt 0.50000".

At the sime time I have a test tool which makes 50 simultaneous requests, where only 2 get answered, the rest times out.

My handler looks like the following:

-module(toppage_handler).  
-export([init/3]).   
-export([handle/2]).   
-export([terminate/3]).    

init(_Transport, Req, []) ->
    {ok, Req, undefined}.

readlines(FileName) ->
    {ok, Device} = file:open(FileName, [read]),
    get_all_lines(Device, []).

get_all_lines(Device, Accum) ->
   case io:get_line(Device, "") of
       eof  -> file:close(Device), Accum;
       Line -> get_all_lines(Device, Accum ++ [Line])
   end.

handle(Req, State) ->
    {PathBin, _} = cowboy_req:path(Req),
    case PathBin of
       <<"/">> -> Output = <<"Hello, world!">>;
       _ -> PathNum = string:substr(binary_to_list(PathBin),2),
            Num = string:right(PathNum, 5, $0),
            Filename = string:concat("input",string:concat(Num, ".txt")),
            Filepath = string:concat("../data/",Filename),
            SortedLines = lists:sort(readlines(Filepath)),
            MiddleIndex = erlang:trunc(length(SortedLines)/2),
            MiddleElement = lists:nth(MiddleIndex, SortedLines),
            Output = iolist_to_binary(io_lib:format("~s\t~s",[Filename,MiddleElement]))
    end,
    {ok, ReqRes} = cowboy_req:reply(200, [], Output, Req),
    {ok, ReqRes, State}.

terminate(_Reason, _Req, _State) ->
    ok.

I am running this on Windows to compare it with .NET. Is there anything to make this more performant, like running the sorting/IO in threads or how can I improve it? Running with cygwin didn't change the result a lot, I got about 5-6 requests answered.

Thanks in advance!

Answer 1

The most glaring issue: get_all_lines is O(N^2) because list concatenation (++) is O(N). Erlang list type is a singly linked list. The typical approach here is to use "cons" operator, appending to the head of the list, and reverse accumulator at the end:

get_all_lines(Device, Accum) ->
   case io:get_line(Device, "") of
       eof  -> file:close(Device), lists:reverse(Accum);
       Line -> get_all_lines(Device, [Line | Accum])
   end.

Pass binary flag to file:open to use binaries instead of strings (which are just lists of characters in Erlang), they are much more memory and CPU-friendly.