1

I've reviewed a number of answers for compiling code to make it read/write files beyond the 4GB max allowed by 32bit file offsets. I'm not having luck with any of the "simple" solutions that I'm finding.

The gist is I've got Ubuntu Server 11.10 running on a small laptop (32 bit Intel architecture). I'm trying to read a unicode file that is 343868522737 bytes in size (0x50102940f1). The Ubuntu machine keeps thinking it is much smaller (0x102940f1) which turns out to be only the lower 32 bits of a true 64bit sized file.

I wrote a small program that I've compiled on a MacOS and on the Ubuntu box. The Mac seems to behave correctly, the Ubuntu box does not.

The small program is below. While I've commented out a block of code, that is really only necessary for the Mac. Ubuntu environment will compile both blocks of code fine - and generates exactly the same response for both blocks.

// Necessary for Ubuntu build?
#define _LARGEFILE_SOURCE
#define _LARGEFILE64_SOURCE
#define _FILE_OFFSET_BITS 64
//#include <features.h>
// finish Ubuntu

#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <string.h>
#include <limits.h>
#include <wchar.h>
#include <wctype.h>
#include <locale.h>

// Ubuntu version
off64_t fileMaxLen(FILE* fin) {
    off64_t fmaxlen = 0;
    if( (fin!=NULL) && (fseeko64(fin,0,SEEK_END) == 0) ) {
        fmaxlen = ftello64(fin);
        fprintf(stdout,"fileMaxLen(): file length is: %#lx \n",(long unsigned int)fmaxlen);
        fseeko64(fin,0,SEEK_SET);
    }
}

// Mac OS version
//off_t fileMaxLen(FILE* fin) {
//    off_t fmaxlen = 0;
//    if( (fin!=NULL) && (fseeko(fin,0,SEEK_END) == 0) ) {
//        fmaxlen = ftello(fin);
//        fprintf(stdout,"fileMaxLen(): file length is: %#lx \n",(long unsigned int)fmaxlen);
//        fseeko(fin,0,SEEK_SET);
//    }
//}

main(int argc, char* argv[]) {
    char fname[255];
    char *locale;
    FILE* f = NULL;

    locale = setlocale(LC_ALL, "");

    if( argc>=2 ) {
        // get the file for segmenting
        memset(fname, '\0', 255);
        sprintf(fname,"%s",argv[1]);
        fprintf(stdout,"Opening: %s\n",fname);
        f = fopen(fname,"r");
        fileMaxLen(f);
        fprintf(stdout,"Done!\n");
    } else {
        fprintf(stdout,"Need a filename\n");
    }
}

Save the snippet as file_test.c, then the compile is really simple.

gcc file_test.c

Then run the a.out

Any suggestions for getting this code to recognize files beyond that 32bit boundary? At this point I'm simply stumped.

3

According to this, the size of long unsigned int on 32-bit Unix is 4 bytes - can it be that the value is trimmed where you're casting fmaxlen to (long unsigned int)?

  • This was helpful - thank you. Yes, it looks like that cast was doing the truncation. Changing the cast to (long long unsigned int) and also changing the format string to %#llx got the Ubuntu version to generate the right result. – dwmc Jan 12 '12 at 7:44
1

You can find documentation for the macros controlling this behaviour with man feature_test_macros.

From that documentation, you should only need to set _FILE_OFFSET_BITS to 64. That should redefine off_t and functions like fseeko, ftello to 64-bit safe versions (this is a no-op on 64-bit systems, and redirects the symbols to the 64 suffixed versions on 32-bit systems). This is preferable to using the 64 suffixed functions directly.

And as Sergey said, casting the off_t value down to a long unsigned int will lose information on 32-bit systems. I don't think there is a standard printf format code for off_t, so you are probably best off casting the value to unsigned long long int and using the %#llx as the format code. That should be warning free on both word sizes and not lose information.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.