Memory problem when reading large files

Support
1

I have a certain problem reading large files that previously have been generated in a vtk application (VTK-7.1.1 64 bits). In this particular case I generated a model with 100 million points in double precision and a scalar part in single precision. No cell info, just a plain point and scalar set.

This dataset can easily be saved with the vtkDataWriter. But when trying to read it back using vtkDataReader I get this error message:
ERROR: In D:\VTK-7.1.1\IO\Legacy\vtkDataReader.cxx, line 430\nvtkDataReader (0000029EDEF98B90): Unable to open file.

Is there anyone that can explain this and hopefully give a solution to the problem. If I reduce that file size to 50 millions it all goes fine so maybe there is a memory limit, but I can not find any such information.

Thanks, b.r.
Oleg

2

Does it work well if you use current VTK version and/or non-legacy file format?

3

Since this is on Windows, is your data file > 2.1 GB? I suspect so, and I suspect VTK 7.1.1 does not make use of the versions of file IO functions needed on Windows such as _fseeki64. Please file a bug report at https://gitlab.kitware.com/vtk/vtk/issues

4

I use VTK 7.1.1 and doubt that there is a correction in VTK 8. I have not seen any info on that in the release note. Maybe the XML format will handle it, but I have not tried yet.

5

The file size is definitely > 2.1 GB. I just generated a file of size 15 GB in the program and saved it to disk. No problem until I tried to read it. These file sizes are realistic in my application (sonar data, seismic data). I will post a bug report as you recommend.

6

XML should work.

7

Well actually, XML has the same problem. Here is my comment in the GitLab page:
I experienced the same problem also using XML format. This actually has been reported 4 month ago by Lutz Hofmann. So it should get some priority and get fixed.

8

If this is important for you then the best you can do is to send a pull request with the fix.

Even if you cannot fix the issue, just providing detailed information about where exactly the issue occurs and why, would increase the chance that somebody would spend time with this.

You can also buy engineering time from Kitware to quickly fix any problems that are important for you.

9

I posted this comment on GitLab:
It is a little unclear why the read error is present when reading large files. The error message is: “ERROR: In D:\VTK-7.1.1\IO\Legacy\vtkDataReader.cxx, line 430\nvtkDataReader (0000024508DD6DC0): Unable to open file: d77m.vtk\n\n” If one follows the error message into the vtkDataReader it says:

**
// first make sure the file exists, this prevents an empty file from // being created on older compilers struct stat fs; if (stat(this->FileName, &fs) != 0) { // The next line is line 430 vtkErrorMacro(<< "Unable to open file: "<< this->FileName); this->SetErrorCode( vtkErrorCode::CannotOpenFileError ); return 0; } ** The test: if (stat(this->FileName, &fs) != 0) uses the file #include <sys/stat.h> which assumedly tests 32 vs 64 bits. Maybe Kitware has not trimmed the reader properly when reading into 64 bits addressing?

10

stat() method failure is a good hint. You need to do extra steps to make this method work for large files (such as define _FILE_OFFSET_BITS or use stat64).

Have you built VTK yourself?
Have you enabled VTK_REQUIRE_LARGE_FILE_SUPPORT?

Please don’t report any errors related to VTK 7 - they may have been already fixed. Test with latest VTK release or master version instead.