read() from a HDD with O_DIRECT fails with 22 (EINVAL, Invalid Argument)

Question

I want to make a basic read() from a SATA HDD /dev/sdd. A write() seems to work. Also read() and write() works without the O_DIRECT Flag. I've read, that it has to be aligned to the blocksize. So I used this to get the blocksize:

root$ blockdev --getsize /dev/sdd
488397168

root$ blockdev --getsize64 /dev/sdd
250059350016

root$ python -c "print 250059350016.0/488397168"
512.0

As you can see I have root. The HDD is connected via a PCIe SATA Card and lspci -vv shows me, that it uses the basic ahci (drivers/ata/ahci.c) driver. I work with the 3.2.0 Linux Kernel on a 64 bit Power Architecture.

Here is my code:

#define _GNU_SOURCE

#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <stdlib.h> 
#include <unistd.h> 
#include <errno.h>

int main() {

    int r;
    char *buffer, *tmp_buffer;
    // alloc more than necesarry to align the real buffer
    tmp_buffer = malloc(2*512*sizeof(char)); 
    long align = (unsigned long)tmp_buffer%512;
    printf("tmp_buffer is at: %x \% 512 = %d\n",tmp_buffer,align);

    buffer = tmp_buffer+(512-align);
    printf("buffer is at: %x \% 512 = %d\n",buffer,(unsigned long)buffer%512);

    memset(buffer,0,sizeof(512));

    // OPEN
    int fd = open("/dev/sdd",O_DIRECT | O_RDWR | O_SYNC);
    if(fd!=3) printf("fd = %d\n",fd);

    // READ
    printf("try to read and then dump buffer:\n");

    r = read(fd,buffer,sizeof(512));
    if(r == -1) printf("Error: %s\n",strerror(errno));
    else {
        // DUMP BUFFER
        int i;
        for(i=0; i<sizeof(512); i++)
            printf("%c",buffer[i]);
    }
    printf("\n");
    return 0;
}

The output is:

tmp_buffer is at: 1cc80010 % 512 = 16
buffer is at: 1cc80200 % 512 = 0
try to read and then dump buffer:
Error: Invalid argument

edit: I have updated my source as suggested by Brett Hale's answer. Unfortunately I still get the error. Is my way to find out the blocksize ok? Have I done the aligning right?

Thank you very much for reading,
Fabian

Answer 1

Direct DMA transfer typically requires the buffer to be aligned. From man:

The O_DIRECT flag may impose alignment restrictions on the length and address of userspace buffers and the file offset of I/Os. ... Under Linux 2.6, alignment to 512-byte boundaries suffices.

So char buffer[512]; might need to be aligned to a 512-byte address.

It may not be possible to achieve this alignment on the stack, so something like:

static char buffer[512] __attribute__ ((__aligned__ (512)));

may work. Or maybe this alignment will work on the stack. Alternatively, if you're using x86, you could use the <mm_malloc.h> intrinsic support functions: _mm_malloc and _mm_free.

Answer 2

BTW, the alignment always need not be in multiples of 512 bytes. it depends on the device block size. You should find it using ioctl with the BLKSSZGET. If the read is not aligned to this value while using O_DIRECT, the read() will fail with EINVAL.