Linux 2.6.18 SCSI SPIN and SPOUT SG_IO ioctl's to a sg LTO5 device

Question

I am trying to take advantage of hardware encryption of LTO5 devices using ioctl's in my fuse file system. It should be quite simple, setup a io_hdr with a cdb:

0xB5, 0x20, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x34, 0x00, 0x00

and provide the key page as per the selected algorithm. I get back a return of 0 with a SCSI sense of all 0's when I execute the SG_IO ioctl.

Now, here is where it get weird. The data written to the device is never encrypted. I switch the tape to another drive and do SCSI reads and the data is all in clear text.

So the next step is perform a SCSI SPIN command to see if the SPOUT command took, even though all the returns say it did. Immediately after the SPOUT I send out a SPIN for the Security status page (0x20):

0xA2, 0x20, 0x00, 0x20, 0x00, 0x00, 0x00, 0x00, 0x00, 0xf8, 0x00, 0x00

Again, return value of 0 from the SG_IO ioctl and a sense of all zeros. However the page I get back is the SPOUT key page I put into the drive. I am sure this did not come out of the drive because it has the secret key that I sent which is definitely a SCSI spec no no. BTW I use a completely different buffer for both SCSI commands and memset the SPIN page to zero just for good measure. The sg driver IS providing this data in response to the SPIN.

Can anyone shed any light on this behavior?

Linux archive.xxxxx.xxx 2.6.18-274.7.1.el5 #1 SMP Thu Oct 20 16:21:01 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

sg3_utils-libs-1.25-5.el5

sgpio-1.2.0_10-2.el5

mt-st-0.9b-2.2.2

I am sending the ioctl commands to /dev/sg5:

[root@archive bin]# sg_inq /dev/sg5
standard INQUIRY:
  PQual=0  Device_type=1  RMB=1  version=0x06  [SPC-4]
  [AERC=0]  [TrmTsk=0]  NormACA=0  HiSUP=0  Resp_data_format=2
  SCCS=0  ACC=0  TPGS=0  3PC=0  Protect=1  BQue=0
  EncServ=0  MultiP=1 (VS=0)  [MChngr=0]  [ACKREQQ=0]  Addr16=0
  [RelAdr=0]  WBus16=0  Sync=0  Linked=0  [TranDis=0]  CmdQue=1
  [SPI: Clocking=0x0  QAS=0  IUS=0]
    length=70 (0x46)   Peripheral device type: tape
 Vendor identification: IBM     
 Product identification: ULTRIUM-HH5     
 Product revision level: BAKG
 Unit serial number: 106xxxxxxxxxx

I have discovered that the Linux ioctl system call is returning a SG_ERR_DID_ERROR [0x07] Internal error detected in the host adapter in the host_status member of the sg_io_hdr_t.

the setltokey.c code is also here: http://www.circlesoft.com/setltokey.c

/*
   setLTO4key: Set LTO4 Encryption Key

   Copyright (c) 2008  Andrew Schretter <schrett@math.duke.edu>
   Provided under GPL license

   added clear encryption,
     sense key and
     error printouts - Gerard J. Cerchio <gjpc@circlesoft.com>
*/

#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <scsi/sg.h>
#include <scsi/scsi.h>
#include <ctype.h>
#include <sys/ioctl.h>
#include <unistd.h>

#define SENSE_BUFF_LEN  96          /* from lto spec */

/*
 * here is a sample key - create a file with these HEX digits:
   4418AFCD046F2535B2E996123CE7DE3D418A15915A091C4BA12BDC85D4069A77
 */

/*
 * A good sg_io_hdr_t reference: http://tldp.org/HOWTO/SCSI-Generic-HOWTO/sg_io_hdr_t.html
 */

/* Print a hexadecimal dump of a block of data */
void hexdump(void *data, int size)
{
    unsigned char *p = data;
    unsigned char c;
    int n;
    char bytestr[4] = {0};
    char addrstr[10] = {0};
    char hexstr[ 16*3 + 5] = {0};
    char charstr[16*1 + 5] = {0};
    for(n=1;n<=size;n++) {
        if (n%16 == 1) {
            /* store address for this line */
            snprintf(addrstr, sizeof(addrstr), "%.4x",
               ((unsigned int)p-(unsigned int)data) );
        }

        c = *p;
        if (isalnum(c) == 0) {
            c = '.';
        }

        /* store hex str (for left side) */
        snprintf(bytestr, sizeof(bytestr), "%02X ", *p);
        strncat(hexstr, bytestr, sizeof(hexstr)-strlen(hexstr)-1);

        /* store char str (for right side) */
        snprintf(bytestr, sizeof(bytestr), "%c", c);
        strncat(charstr, bytestr, sizeof(charstr)-strlen(charstr)-1);

        if(n%16 == 0) {
            /* line completed */
            printf("  [%4.4s]  %-49.49s %s\n", addrstr, hexstr, charstr);
            hexstr[0] = 0;
            charstr[0] = 0;
        } else if(n%8 == 0) {
            /* half line: add whitespaces */
            strncat(hexstr, " ", sizeof(hexstr)-strlen(hexstr)-1);
        }
        p++; /* next byte */
    }

    if (strlen(hexstr) > 0) {
        /* print rest of buffer if not empty */
        printf("  [%4.4s]  %-49.49s %s\n", addrstr, hexstr, charstr);
    }
}

/* Send a SCSI command block and display the result. */
void do_read_command(int fd, char *desc, unsigned char *cmd, int len)
{
        unsigned char sense[SENSE_BUFF_LEN];
        memset( sense, 0, SENSE_BUFF_LEN );

        sg_io_hdr_t io;
        unsigned char buf[512];

        memset(buf, 0, sizeof(buf));

        memset(&io, 0, sizeof(io));
        io.interface_id = 'S';
        io.cmd_len = len;
        io.mx_sb_len = 0;
        io.dxfer_direction = SG_DXFER_FROM_DEV;
        io.dxfer_len = sizeof(buf);
        io.dxferp = buf;
        io.cmdp = cmd;

        printf("Command: %s\n", desc);
        hexdump(cmd, len);

        if (ioctl(fd, SG_IO, &io) < 0) {
                printf("Error: %s\n", strerror(errno));
                return;
        }

        if ( io.sb_len_wr ){
            printf("Sense\n");
            hexdump( sense, SENSE_BUFF_LEN );
        }
        else
            printf( "No Sense\n" );

        if ((io.info & SG_INFO_OK_MASK) != SG_INFO_OK) {
            printf("Failed with info 0x%02x  mask status 0x%02x  msg status 0x%02x  host status 0x%02x driver status 0x%02x\n", io.info, io.masked_status, io.msg_status, io.host_status, io.driver_status );
            return;
        }

        len = io.dxfer_len - io.resid;
        printf("Response: %d %s\n", len, (len == 1) ? "byte" : "bytes");
        hexdump(buf, len);
}

void do_write_command(int fd, char *desc, unsigned char *cmd, int len, char *data_desc, unsigned char *data, int datalen)
{
        unsigned char sense[SENSE_BUFF_LEN];
        memset( sense, 0, SENSE_BUFF_LEN );

        sg_io_hdr_t io;
        memset(&io, 0, sizeof(io));
        io.interface_id = 'S';
        io.cmd_len = len;
        io.mx_sb_len = SENSE_BUFF_LEN;
        io.dxfer_direction = SG_DXFER_TO_DEV;
        io.dxfer_len = datalen;
        io.dxferp = data;
        io.cmdp = cmd;
        io.sbp = sense;

        printf("Command: %s\n", desc);
        hexdump(cmd, len);
        printf("Data: %s\n", data_desc);
        hexdump(data, datalen);

        if (ioctl(fd, SG_IO, &io) < 0) {
                printf("Error: %s\n", strerror(errno));
                return;
        }

        if ( io.sb_len_wr ){
            printf("Sense\n");
            hexdump( sense, SENSE_BUFF_LEN );
        }
        else
            printf( "No Sense\n" );

        if ((io.info & SG_INFO_OK_MASK) != SG_INFO_OK) {
            printf("Failed with info 0x%02x  mask status 0x%02x  msg status 0x%02x  host status 0x%02x driver status 0x%02x\n", io.info, io.masked_status, io.msg_status, io.host_status, io.driver_status );
                return;
        }

        len = io.dxfer_len - io.resid;
        printf("Response: %d %s\n", len, (len == 1) ? "byte" : "bytes");

        //hexdump(buf, len);
}

struct {
        char *description;
        int len;
        unsigned char cmd[16];
} commands[] = {
        { "SCSI Inquiry", 6,
          { 0x12, 0x00, 0x00, 0x00, 0xFF, 0x00 } },
        { "SCSI SPOUT Set Encryption Key", 12,
          { 0xb5, 0x20, 0x00, 0x10, 0x00, 0x00,
            0x00, 0x00, 0x00, 0x34, 0x00, 0x00 } },
        { "SCSI SPIN Read Status", 12,
          { 0xa2, 0x20, 0x00, 0x20, 0x00, 0x00,
            0xff, 0xff, 0xff, 0xff, 0x00, 0x00 } },
        { NULL, 0, { 0 } },
};

struct {
        char *description;
        int len;
        unsigned char cmd[64];
} data[] = {
        { "SCSI SPOUT Send Encryption Key Page", 52,
          { 0x00, 0x10, 0x00, 0x30, 0x40, 0x00,
            0x02, 0x03, 0x01, 0x00,
            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
            0x00, 0x20,
          } },
        { "SCSI SPOUT Clear Encryption Mode Page", 52,
          { 0x00, 0x10, 0x00, 0x30, 0x40, 0x00,
            0x00, 0x00, 0x01, 0x00,
            0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
            0x00, 0x20,
          } },
        { NULL, 0, { 0 } },
};

int main(int argc, char **argv)
{
        FILE *fd2;
        int fd;
        int i = 0;

        if (argc < 2) {
                fprintf(stderr, "usage: %s /dev/sda < <keyfile.txt> | clear >\n", *argv);
                return 1;
        }

        if ((fd = open(argv[1], O_RDWR)) < 0) {
                perror(argv[1]);
                return 1;
        }

        if ((ioctl(fd, SG_GET_VERSION_NUM, &i) < 0) || (i < 30000)) {
                fprintf(stderr,"%s is not a sg device\n", argv[1]);
                close(fd);
                return 1;
        }

        printf("Opened %s\n", argv[1]);
        /* Send query command */
        do_read_command(fd, commands[0].description, commands[0].cmd, commands[0].len);

        if(argc > 2) {

        if ( strcasecmp( argv[2], "clear" ) == 0 ) {
            do_write_command(fd, commands[1].description, commands[1].cmd, commands[1].len, data[1].description, data[1].cmd, data[1].len);

        }
        else
        {
            if ((fd2 = fopen(argv[2], "r")) < 0) {
                perror(argv[2]);
                return 1;
                }

            for (i = 0; i < 32; i++) {
                if( fscanf(fd2, "%2x ", (unsigned int *) &data[0].cmd[i + 20]) != 1 ) {
                    fprintf(stderr, "Keyfile Error reading %s\n", argv[2]);
                    return 1;
                    }
                }
            fclose(fd2);
            /* Set Encryption key*/
            do_write_command(fd, commands[1].description, commands[1].cmd, commands[1].len, data[0].description, data[0].cmd, data[0].len);
            }
        }

        /* Query encryption status */
        do_read_command(fd, commands[2].description, commands[2].cmd, commands[2].len);
        close(fd);
        return 0;
}

Answer 1

I found the problem. The LSI 6Gb SAS SCSI HBA built into the Dell PowerEdge R410 is subtlety not compatible with IBM's ULTRIUM-HH5 LTO5 tape drives.

The HBA was failing on the both the SPIN and SPOUT commands. We assumed it was the version of RedHat we were using, which is older than Paul's Ubuntu 10.04. We performed the cryptography using our file system's built in facility and decided to re-visit the problem when we were able to upgrade to a newer RedHat.

The show stopper incompatibility was discovered during validation. The LSI HBA was generating extra SCSI MOVE MEDIA commands to the Qualstar RLS-8560 library when we loaded an expired cleaning cartridge into any of the drives. This error presented as an occasional sense key 2 ASC 3B ASQ 90 return to the host's single SCSI MOVE MEDIA command.

The solution is to install a ATTO ExpressSAS H680 6Gb/s SAS/SATA HBA, PCIe 2.0, 8 Port External SCSI adapter which is certified compatible with the IBM drives.

Answer 2

gjpc I am trying to accomplish the same thing, did you have make any progress on this issue?

Update: I just found out that your program is working. I tested it with a HP drive it uploaded the key correctly (blue led on). I was able to write a tape, uploaded another key and I could not read the tape, after changing to the correct key I could read my data from tape again. Thanks for sharing your code.