I solved this problem. Do the following, install all packages except libgl1-mesa-glx:i386
:
$ sudo apt-get install git gnupg flex bison gperf build-essential \
zip curl libc6-dev libncurses5-dev:i386 x11proto-core-dev \
libx11-dev:i386 libreadline6-dev:i386 \
libgl1-mesa-dev g++-multilib mingw32 tofrodos \
python-markdown libxml2-utils xsltproc zlib1g-dev:i386
Install libglapi-mesa-lts-saucy:i386
instead of libgl1-mesa-glx:i386
:
$ sudo apt-get install libglapi-mesa-lts-saucy:i386
Now you have correct AOSP build environment and don't loose desktop environment.