Code runs in SLIME+SBCL but not plain SBCL

Question

I've been trying to build a lispy interface to the CFFI bindings (https://gitorious.org/dh-misc/hdf5/source/cb616fd619a387e3cdc927994b9ad12b6b514236:) but I ran into a situation where code runs correctly in SLIME which has an SBCL instance as the backend, but won't run whenever I run the code just in SBCL.

So, I created a test case file which demonstrates the error:

(asdf:load-system :cffi)
;;(asdf:operate 'asdf:load-op :cffi)

(defpackage :hdf5test
  (:use :cl :cffi)
  (:export :test))

(in-package :hdf5test)

(define-foreign-library hdf5
    (t (:default "libhdf5")))

(use-foreign-library hdf5)

;; hdf types:

(defctype size-t :uint)
(defctype hid-t :int)
(defctype herr-t :int)
(defctype hsize-t :uint64)

;; hdf constants:

;; H5S_UNLIMITED: 2^64-1
(defconstant +H5S-UNLIMITED+ 18446744073709551615)

;; H5F_ACC_TRUNC
(defconstant +H5F-ACC-TRUNC+ 2) ;; we'll see if it works

;; H5P_DEFAULT
(defconstant +H5P-DEFAULT+ 0)

;; H5T types:

(defconstant +H5P-DATASET-CREATE+ 150994953)
(defconstant +H5T-NATIVE-INT+ 50331660)

;; hdf functions:

;; H5Screate_simple
(defcfun "H5Screate_simple" hid-t
  (rank :int)
  (current-dims :pointer) ; const hsize_t*
  (maximum-dims :pointer)) ; cons hsize_t*

;; H5Fcreate
(defcfun "H5Fcreate" hid-t
  (filename :string)
  (flags :uint)
  (fcpl-id hid-t)
  (fapl-id hid-t))

;; H5Pcreate
(defcfun "H5Pcreate" hid-t
  (cls-id hid-t))

;; H5Pset_chunk
(defcfun "H5Pset_chunk" herr-t
  (plist hid-t)
  (ndims :int)
  (dim :pointer)) ;; const hsize_t*

;; H5Pset_deflate
(defcfun "H5Pset_deflate" herr-t
  (plist-id hid-t)
  (level :uint))

;; H5Dcreate1
(defcfun "H5Dcreate1" hid-t
  (loc-id hid-t)
  (name :string)
  (type-id hid-t)
  (space-id hid-t)
  (dcpl-id hid-t))

;; H5Dclose
(defcfun "H5Dclose" herr-t
  (dataset-id hid-t))

;; H5Dwrite
(defcfun "H5Dwrite" herr-t
  (datset-id hid-t)
  (mem-type-id hid-t)
  (mem-space-id hid-t)
  (file-space-id hid-t)
  (xfer-plist-id hid-t)
  (buf :pointer))

;; H5Fclose
(defcfun "H5Fclose" herr-t
  (file-id hid-t))

;; H5Sclose
(defcfun "H5Sclose" herr-t
  (space-id hid-t))

(defparameter *rank* 1)

(defun test (filename)
  (with-foreign-string (dataset-name "dataset")
    (with-foreign-objects ((dim :int 1)
               (dataspace-maxdim :uint64 1)
               (memspace-maxdim :uint64 1)
               (chunkdim :int 1)
               (dataspace 'hid-t)
               (dataset 'hid-t)
               (memspace 'hid-t)
               (cparms 'hid-t))
      (setf (mem-aref dim :int 0) 5)
      (format t "dim: ~a~%" (mem-aref dim :int 0))
      ;;(setf (mem-aref maxdim :int 0) -1)
      (setf (mem-aref dataspace-maxdim :uint64 0) +H5S-UNLIMITED+)
      (setf (mem-aref memspace-maxdim :uint64 0) 5)
      (setf (mem-aref chunkdim :int 0) 1)
      (format t "dataspace-maxdim: ~a~%" (mem-aref dataspace-maxdim :uint64 0))
      (format t "memspace-maxdim: ~a~%" (mem-aref memspace-maxdim :uint64 0))
      ;;(with-open-hdf-file (file filename :direction :output :if-exists :supersede)
      (let ((file (h5fcreate filename +H5F-ACC-TRUNC+ +H5P-DEFAULT+ +H5P-DEFAULT+)))
    (setf cparms (h5pcreate +H5P-DATASET-CREATE+))
    (h5pset-chunk cparms *rank* chunkdim)
    (setf dataspace (h5screate-simple *rank* dim dataspace-maxdim))
    (setf dataset (h5dcreate1
               file
               dataset-name
               +H5T-NATIVE-INT+
               dataspace
               cparms))
    (format t "dataspace: ~a~%" dataspace)
    (format t "dataset: ~a~%" dataset)
    (setf memspace (h5screate-simple *rank* dim memspace-maxdim))
    (with-foreign-object (data :int 5)
      (loop for i from 0 to 4 do (setf (mem-aref data :int i) (* i i)))
      (h5dwrite dataset +H5T-NATIVE-INT+ memspace dataspace +H5P-DEFAULT+ data))
    (h5dclose dataset)
    (h5sclose memspace)
    (h5sclose dataspace)
    (h5fclose file)))))

The output I get from running (hdf5test:test "test.h5") in SLIME+SBCL is

dim: 5
dataspace-maxdim: 18446744073709551615
memspace-maxdim: 5
dataspace: 67108866
dataset: 83886080
0

The output I get from running (hdf5test:test "test.h5") in just SBCL is

dim: 5
dataspace-maxdim: 18446744073709551615
memspace-maxdim: 5
dataspace: 67108866
dataset: 83886080
HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0:
  #000: H5S.c line 1388 in H5Screate_simple(): maxdims is smaller than dims
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0:
  #000: H5Dio.c line 233 in H5Dwrite(): not a data space
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.10-patch1) thread 0:
  #000: H5S.c line 405 in H5Sclose(): not a dataspace
    major: Invalid arguments to routine
    minor: Inappropriate type
0

So you can see it's something to do with how the array is being passed to the hdf functions, but I have no clue why SLIME+SBCL would handle this but not SBCL.

I've also tried the exact same code with CLISP and it works fine, no issues, so it seems to be an SBCL issue.

Any thoughts on this?

EDIT: I thought I should add to the main post that the resulting files truly are different in each case. In SLIME+SBCL or CLISP the file contains a finite dataset with squared integers inside (no reason really, just a test). But, with plain SBCL the data file is left incomplete; if you try to view the contents with h5dump it is an unending trial of zeros (that's how it handles incomplete datasets).

Answer 1

Like @nixeagle said, slime seems to hide the error messages originating in the hdf5 library. Along these lines I'd wager that passing the results from SBCL to emacs in slime is what allows the file to be written.

Now the following needs to be taken with a few grains of salt, as I don't really know anything about hdf5 or cffi and am now just getting back into common lisp, but things started working consistently in both slime and sbcl on my x86_64 linux box, once I replaced all those :int types with :uint64, which seems to make sense, as the declarations resolve to that type anyways.

your code in sbcl:

* (load "temp.lisp")
T
* (hdf5test:test "test2.h5")
dim: 5
dataspace-maxdim: 18446744073709551615
memspace-maxdim: 5
dataspace: 67108866
dataset: 83886080
HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
  #000: H5S.c line 1388 in H5Screate_simple(): maxdims is smaller than dims
    major: Invalid arguments to routine
    minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
  #000: H5Dio.c line 231 in H5Dwrite(): can't prepare for writing data
    major: Dataset
    minor: Write failed
  #001: H5Dio.c line 332 in H5D__pre_write(): not a data space
    major: Invalid arguments to routine
    minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.12) thread 0:
  #000: H5S.c line 405 in H5Sclose(): not a dataspace
    major: Invalid arguments to routine
    minor: Inappropriate type
0

part with changes:

    (with-foreign-objects ((dim :uint64 1)
               (dataspace-maxdim :uint64 1)
               (memspace-maxdim :uint64 1)
               (chunkdim :uint64 1)
               (dataspace 'hid-t)
               (dataset 'hid-t)
               (memspace 'hid-t)
               (cparms 'hid-t))
      (setf (mem-aref dim :uint64 0) 5)
      (format t "dim: ~a~%" (mem-aref dim :uint64 0))
      ;;(setf (mem-aref maxdim :int 0) -1)
      (setf (mem-aref dataspace-maxdim :uint64 0) +H5S-UNLIMITED+)
      (setf (mem-aref memspace-maxdim :uint64 0) 5)
      (setf (mem-aref chunkdim :uint64 0) 1)
      (format t "dataspace-maxdim: ~a~%" (mem-aref dataspace-maxdim :uint64 0))
      (format t "memspace-maxdim: ~a~%" (mem-aref memspace-maxdim :uint64 0))

changed code in sbcl:

* (load "temp.lisp")
T
* (hdf5test:test "test2.h5")
dim: 5
dataspace-maxdim: 18446744073709551615
memspace-maxdim: 5
dataspace: 67108866
dataset: 83886080
0

result files:

% h5dump test.h5 
HDF5 "test.h5" {
GROUP "/" {
   DATASET "dataset" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 5 ) / ( H5S_UNLIMITED ) }
      DATA {
      (0): 0, 1, 4, 9, 16
      }
   }
}
}
% h5dump test2.h5
HDF5 "test2.h5" {
GROUP "/" {
   DATASET "dataset" {
      DATATYPE  H5T_STD_I32LE
      DATASPACE  SIMPLE { ( 5 ) / ( H5S_UNLIMITED ) }
      DATA {
      (0): 0, 1, 4, 9, 16
      }
   }
}
}