Reading raw data, using memcpy

Support
1

Hi All,

I am trying to read a volume (1024x2048x2048) from a raw data to render. Initially, I tried VTK image reader, although it’s only able to read (1024x2048x1024) portion of the whole data, I am not sure what is the reason. I get this warning:

warning
warning1330×346 13.8 KB

I have checked the data size and am able to open the whole volume in MATLAB.

Then I tried an alternative way to read the raw data and just allocate it to a vtkImageData using memcpy, although nothing is copied into vtkImageData and the rendered volume is consequently just blank. Here’s my code:

FILE* f = fopen("path", "rb");

// extract image height and width from header
int width = 2048;
int height = 2048;
int depth = 1024;

int size = depth * width * height;
unsigned char* data = new unsigned char[size];

if (f) {

    fread(data, sizeof(unsigned char), size, f);
    fclose(f);
}

vtkSmartPointer<vtkImageData> test = vtkSmartPointer<vtkImageData>::New();

test->SetSpacing(0.0039, 0.0049, 0.0049);
test->SetOrigin(0, 0, 0);
test->SetDimensions(1024,2048,2048);
test->AllocateScalars(VTK_UNSIGNED_CHAR, 1);
size_t byte_count = 1024 * 2048 * 2048 * sizeof(unsigned char);
if (data!=nullptr){
    memcpy(test->GetScalarPointer(), data, byte_count);
}
test->Modified();

Any help would be much appreciated.

Parisa

2

You get this error because the std::istream flags std::ios::fail for some reason. You either get a failbit or a badbit (see here)

The error happens when std::istream::seekg is called. I think the most likely reason why it fails is if you would overflow its parameter streamStart.

What does your machine return for sizeof(long)? If it returns 4 you’re for sure overflowing.

Edit: The faulty code lives in vtkImageReader2::SeekFile if you want to look at the source code.

3

This is 2³¹ elements which is 1 beyond the maximum for a 32bit signed integer. Given your F:\ path, I would guess that this is Windows which has a 32bit long. Which means that the image reader just doesn’t support files larger than 2GB on Windows today (nevermind the 8.2.0 you seem to be using).

1 Like
4

Thank you both for your help.

You are right I am using windows. I was not aware of the limitation of image reader for files larger than 2G. Is there any work around for this issue beside down sampling the volume?

5

As of today, probably not an easy one… We’re going to try to fix this issue for the next release.

You could run this on a linux box, which had an 8 bytes long definition. Alternatively, you could also split your file into multiple files? You wouldn’t necessarily need to do many copies. If you’re storing your data inside a vtkImageData, you can have multiple vtkImageData inside a vtkPartitionedDataSet. So you don’t need to do copies from each sub image into the final big image.

6

Oh you’re using VTK 8.2.0… I don’t think vtkPartitonedDataSet exist in this release… So maybe the best solution would be to read multiple files, and then create you big image by copying each buffer. Coordinate ordering is done (x → y → z) in VTK, so you should split on the z axis so you can directly do calls to std::copy into your final image.

7

I followed your recommendation to read multiple files and copy each to the bigger image data and it’s working great. Thank you.

8

Being adressed here: https://gitlab.kitware.com/vtk/vtk/-/merge_requests/9583#note_1260437

@toddy

1 Like