# Memory problem when reading large files
**URL:** https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511
**Category:** Support
**Created:** [March 18, 2019, 1:24am UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511 "2019-03-18T01:24:42Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Oleg](https://discourse.vtk.org/u/Oleg)
#### Post date: [March 18, 2019, 1:24am UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/1 "2019-03-18T01:24:42Z")
I have a certain problem reading large files that previously have been generated in a vtk application (VTK-7.1.1 64 bits). In this particular case I generated a model with 100 million points in double precision and a scalar part in single precision. No cell info, just a plain point and scalar set.
This dataset can easily be saved with the vtkDataWriter. But when trying to read it back using vtkDataReader I get this error message:
**ERROR: In D:\VTK-7.1.1\IO\Legacy\vtkDataReader.cxx, line 430\nvtkDataReader (0000029EDEF98B90): Unable to open file.**
Is there anyone that can explain this and hopefully give a solution to the problem. If I reduce that file size to 50 millions it all goes fine so maybe there is a memory limit, but I can not find any such information.
Thanks, b.r.
Oleg
---
### Author:  [@lassoan](https://discourse.vtk.org/u/lassoan)
#### Post date: [March 18, 2019, 2:36am UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/2 "2019-03-18T02:36:50Z")
Does it work well if you use current VTK version and/or non-legacy file format?
---
### Author:  [@cory.quammen](https://discourse.vtk.org/u/cory.quammen)
#### Post date: [March 18, 2019, 3:00pm UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/3 "2019-03-18T15:00:32Z")
Since this is on Windows, is your data file \> 2.1 GB? I suspect so, and I suspect VTK 7.1.1 does not make use of the versions of file IO functions needed on Windows such as `_fseeki64`. Please file a bug report at [https://gitlab.kitware.com/vtk/vtk/issues](https://gitlab.kitware.com/vtk/vtk/issues)
---
### Author:  [@Oleg](https://discourse.vtk.org/u/Oleg)
#### Post date: [March 18, 2019, 4:58pm UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/4 "2019-03-18T16:58:00Z")
I use VTK 7.1.1 and doubt that there is a correction in VTK 8. I have not seen any info on that in the release note. Maybe the XML format will handle it, but I have not tried yet.
---
### Author:  [@Oleg](https://discourse.vtk.org/u/Oleg)
#### Post date: [March 18, 2019, 5:02pm UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/5 "2019-03-18T17:02:34Z")
The file size is definitely \> 2.1 GB. I just generated a file of size 15 GB in the program and saved it to disk. No problem until I tried to read it. These file sizes are realistic in my application (sonar data, seismic data). I will post a bug report as you recommend.
---
### Author:  [@cory.quammen](https://discourse.vtk.org/u/cory.quammen)
#### Post date: [March 19, 2019, 2:32am UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/6 "2019-03-19T02:32:44Z")
XML should work.
---
### Author:  [@Oleg](https://discourse.vtk.org/u/Oleg)
#### Post date: [March 20, 2019, 7:40pm UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/7 "2019-03-20T19:40:12Z")
Well actually, XML has the same problem. Here is my comment in the GitLab page:
I experienced the same problem also using XML format. This actually has been reported 4 month ago by Lutz Hofmann. So it should get some priority and get fixed.
---
### Author:  [@lassoan](https://discourse.vtk.org/u/lassoan)
#### Post date: [March 20, 2019, 7:55pm UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/8 "2019-03-20T19:55:26Z")
> [@Oleg](#):
>
> So it should get some priority and get fixed.
If this is important for you then the best you can do is to send a pull request with the fix.
Even if you cannot fix the issue, just providing detailed information about where exactly the issue occurs and why, would increase the chance that somebody would spend time with this.
You can also buy engineering time from Kitware to quickly fix any problems that are important for you.
---
### Author:  [@Oleg](https://discourse.vtk.org/u/Oleg)
#### Post date: [March 21, 2019, 11:20am UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/9 "2019-03-21T11:20:26Z")
I posted this comment on GitLab:
It is a little unclear why the read error is present when reading large files. The error message is: “ERROR: In D:\VTK-7.1.1\IO\Legacy\vtkDataReader.cxx, line 430\nvtkDataReader (0000024508DD6DC0): Unable to open file: d77m.vtk\n\n” If one follows the error message into the vtkDataReader it says:
\*\*
// first make sure the file exists, this prevents an empty file from // being created on older compilers struct stat fs; if (stat(this-\>FileName, &fs) != 0) { // The next line is line 430 vtkErrorMacro(\<\< "Unable to open file: "\<\< this-\>FileName); this-\>SetErrorCode( vtkErrorCode::CannotOpenFileError ); return 0; } \*\* The test: if (stat(this-\>FileName, &fs) != 0) uses the file #include \ which assumedly tests 32 vs 64 bits. Maybe Kitware has not trimmed the reader properly when reading into 64 bits addressing?
---
### Author:  [@lassoan](https://discourse.vtk.org/u/lassoan)
#### Post date: [March 21, 2019, 4:02pm UTC](https://discourse.vtk.org/t/memory-problem-when-reading-large-files/511/10 "2019-03-21T16:02:02Z")
`stat()` method failure is a good hint. You need to do extra steps to make this method work for large files (such as define \_FILE\_OFFSET\_BITS or use stat64).
Have you built VTK yourself?
Have you enabled VTK\_REQUIRE\_LARGE\_FILE\_SUPPORT?
Please don’t report any errors related to VTK 7 - they may have been already fixed. Test with latest VTK release or master version instead.