wsdisplay color map not being retrieved correctly

Question

I've been trying to use wscons and wsdisplay on NetBSD 5.1.2 (using the VESA framebuffer implementation) recently and I've encountered a bit of a problem:

I can set color maps successfully and they look correct but getting color maps seems to return incorrect data, such that when I try to restore the original color map once the program has finished, all the colors are incorrect:

Here's a reduced program causing the problem (note that it must be run either as root or as the user logged in on the second virtual terminal (/dev/ttyE1)):

#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <dev/wscons/wsconsio.h>

int main(int argc, char **argv) {
    (void)argc, (void)argv;
    int tty = open("/dev/ttyE1", O_RDWR | O_EXCL);
    if(tty == -1) {
        perror("error opening tty");
        return EXIT_FAILURE;
    }
    struct wsdisplay_fbinfo fbinfo;
    if(ioctl(tty, WSDISPLAYIO_GINFO, &fbinfo) == -1) {
        perror("error retrieving framebuffer info");
        close(tty);
        return EXIT_FAILURE;
    }
    uint8_t *cmap_data = malloc(fbinfo.cmsize * 3);
    if(cmap_data == NULL) {
        perror("error allocating memory for color map data");
        close(tty);
        return EXIT_FAILURE;
    }
    struct wsdisplay_cmap cmap;
    cmap.index = 0;
    cmap.count = fbinfo.cmsize;
    cmap.red   = &cmap_data[fbinfo.cmsize * 0];
    cmap.green = &cmap_data[fbinfo.cmsize * 1];
    cmap.blue  = &cmap_data[fbinfo.cmsize * 2];
    if(ioctl(tty, WSDISPLAYIO_GETCMAP, &cmap) == -1) {
        perror("error getting color map");
        close(tty), free(cmap_data);
        return EXIT_FAILURE;
    }
    if(ioctl(tty, WSDISPLAYIO_PUTCMAP, &cmap) == -1) {
        perror("error putting color map");
        close(tty), free(cmap_data);
        return EXIT_FAILURE;
    }
    free(cmap_data);
    close(tty);
    return EXIT_SUCCESS;
}

What am I doing wrong and how can I make it retrieve and restore color maps correctly?

Answer 1

Cause

I looked into the problem more and it appears that some kernel memory is either uninitialized or getting corrupted. Specifically, sc_cmap_red, sc_cmap_green, and sc_cmap_blue of struct vesafb_softc (in vesafbvar.h on lines 89 to 91) contain incorrect data. This is somewhat surprising, as lines 719 to 722 of vesafb.c initialize it:

/* Fill in the softc colourmap arrays */
sc->sc_cmap_red[i / 3] = rasops_cmap[i + 0];
sc->sc_cmap_green[i / 3] = rasops_cmap[i + 1];
sc->sc_cmap_blue[i / 3] = rasops_cmap[i + 2];

It contains incorrect data even if I move that out of the if statement it's in, so it might be getting corrupted rather than being uninitialized.

The driver is able to get and set color maps correctly, however; it just cannot seem to get the initial one in struct vesafb_softc right.

Workaround

A simple solution would be to make a program re-set the default color map. As the above snippet indicated, it was supposed to get its initial colors from rasops_cmap, which happens to be defined on lines 55 to 122 in rasops.c:

/* ANSI colormap (R,G,B). Upper 8 are high-intensity */
const u_char rasops_cmap[256*3] = {
    /* ... */
};

With those colors, you can make a program that sets that as the current color map. I had to make a few changes so the cursor didn't disappear, but it mostly worked.

Better Solution

As I was looking around for more information, I had found this blog post. When I recompiled the kernel with genfb(4) rather than vesafb(4), the kernel hung at boot. It turns out this was because the bootloader I was using was not new enough to pass the required parameters to the kernel.

I happened to look at the NetBSD 6.0 changelog and noticed this entry:

• amd64, i386
  The bootloader has been enhanced to support framebuffer consoles using VESA BIOS extensions. These changes allow the x86 ports to work with the genfb(4) driver, and obsoletes the i386-only vesafb(4) driver. [jmcneill 20090216]

I downloaded NetBSD 6.0_BETA and booted it from the boot prompt like this:

> vesa 640x480x8
> boot netbsd

...and everything worked.

In short, using a newer version of NetBSD and ditching vesafb(4) solves the problem.