I've run into an issue with althttpd serving a 3.1GB iso file. To reproduce:

In a directory with a 3+ GB file (not sure what sizes will trigger it), start althttpd with:

./althttpd --popup

Trying to download the file with Firefox on localhost, the download size is unknown and ends prematurely at 2GB. Looking at the response headers with developer tools I see: Content-length: -953595904

Trying with wget gives me:

wget: content-length -953595904 is garbage

Making this change fixes both Firefox and wget at least for knowing the file size:

--- althttpd.c.orig
+++ althttpd.c
@@ -2650,7 +2650,7 @@
   if( zEncoding ){
     nOut += althttpd_printf("Content-encoding: %s" CRLF, zEncoding);
   }
-  nOut += althttpd_printf("Content-length: %d" CRLF CRLF,(int)pStat->st_size);
+  nOut += althttpd_printf("Content-length: %jd" CRLF CRLF,(intmax_t)pStat->st_size);
   fflush(stdout);
   if( strcmp(zMethod,"HEAD")==0 ){
     MakeLogEntry(0, 2); /* LOG: Normal HEAD reply */

But the transfer still fails in both:

~/tmp/del $ time wget http://localhost:8080/doors.iso
Connecting to localhost:8080 ([::1]:8080)
saving to 'doors.iso'
doors.iso             51% |*************************************************************************                                                                     | 1656M  0:00:02 ETAwget: connection closed prematurely
Command exited with non-zero status 1
real	0m 3.97s
user	0m 0.02s
sys	0m 1.49s

wget consistently dies right around the 4s mark. Firefox runs longer but download speed is lower and stalls at 2.0GB and never completes. I'll look at this more tomorrow but wanted to post preliminary findings.

I've run into an issue with althttpd serving a 3.1GB iso file.

It looks like it's caused by SendFile() using a 32-bit size:

#ifdef linux
  if( 2!=useHttps ){
    off_t offset = rangeStart;
    nOut += sendfile(fileno(stdout), fileno(in), &offset, pStat->st_size);
  }else
#endif
  {
    xferBytes(in, stdout, (int)pStat->st_size, rangeStart); // HERE: args 3/4 are 'int'
  }

The 2!=useHttps is checking for transport modes other than the built-in SSL (i.e. http or an external https via stunnel4 or similar). To my eyes, without having actually tried it, my guess is that you're running in SSL mode and that xferBytes() is overflowing your content size. My recommended patch, which i'll leave to you because you have it ready to reproduce:

Change xferBytes() to take long long (edit: it looks like size_t or ssize_t would be a better choice) as its 3rd and 4th arguments, then change any calls to it to ensure that they're using long long.

It's kinda barbaric to use long long given the fixed-size integer types C99 provides, but althttpd still needs to compile on C89. In C99 you'd include <stdint.h> and use int64_t. stdint.h is actually available on most compilers, even when using -std=c89, but including it in this particular project may introduce build regressions.

Edit: this patch might be sufficient, depending on the data types of the arguments where it's called:

Edit: not true - its first inner loop would need to be adjusted to ensure that it doesn't underflow 0. After some reading, it looks like intmax_t is the right type, but AFAIK it's from C99, not C89.

Index: althttpd.c
==================================================================
--- althttpd.c
+++ althttpd.c
@@ -2485,11 +2485,11 @@
 ** according to the number of bytes transferred.
 **
 ** When running in built-in TLS mode the 2nd argument is ignored and
 ** output is instead sent via the TLS connection.
 */
-static void xferBytes(FILE *in, FILE *out, int nXfer, int nSkip){
+static void xferBytes(FILE *in, FILE *out, size_t nXfer, size_t nSkip){
   size_t n;
   size_t got;
   char zBuf[16384];
   while( nSkip>0 ){
     n = nSkip;

xferBytes() bytes has always taken an int but does not rely on the values going negative, so its parameters can be unsigned.

After some reading, it looks like intmax_t is the right type, but AFAIK it's from C99, not C89.

The alternative would be POSIX's ssize_t, but there's no guaranty (from POSIX) that it will be > 32 bits. It will be on any modern system, though.

My reading is that (A) changing xferBytes()'s 3rd and 4th paramters to ssize_t and (B) editing the 2 callers of it to use ssize_t instead of int (throughout their bodies, not just at the call point) would fix it. It's very possible that only SendFile() needs fixing, as it's highly unlikely that one will produce 2GB+ of output from a non-file sources (e.g. CGI output).

This is completely untested but might give you a starting point:

Index: althttpd.c
==================================================================
--- althttpd.c
+++ althttpd.c
@@ -491,12 +491,12 @@
 static struct rusage priorChild; /* Previously report CHILD time */
 /*static struct timespec tsSelf;*/
 static int mxAge = 120;          /* Cache-control max-age */
 static char *default_path = "/bin:/usr/bin";  /* Default PATH variable */
 static char *zScgi = 0;          /* Value of the SCGI env variable */
-static int rangeStart = 0;       /* Start of a Range: request */
-static int rangeEnd = 0;         /* End of a Range: request */
+static ssize_t rangeStart = 0;       /* Start of a Range: request */
+static ssize_t rangeEnd = 0;         /* End of a Range: request */
 static int maxCpu = MAX_CPU;     /* Maximum CPU time per process */
 static int enableSAB = 0;        /* Add reply header to enable
                                  ** SharedArrayBuffer */
 static int inSignalHandler = 0;  /* True if running a signal handler */
 static int isExiting = 0;        /* True when althttpd_exit() has been called */
@@ -2485,11 +2485,11 @@
 ** according to the number of bytes transferred.
 **
 ** When running in built-in TLS mode the 2nd argument is ignored and
 ** output is instead sent via the TLS connection.
 */
-static void xferBytes(FILE *in, FILE *out, int nXfer, int nSkip){
+static void xferBytes(FILE *in, FILE *out, ssize_t nXfer, ssize_t nSkip){
   size_t n;
   size_t got;
   char zBuf[16384];
   while( nSkip>0 ){
     n = nSkip;
@@ -2586,12 +2586,13 @@
   if( rangeEnd>0 && rangeStart<pStat->st_size ){
     StartResponse("206 Partial Content");
     if( rangeEnd>=pStat->st_size ){
       rangeEnd = pStat->st_size-1;
     }
-    nOut += althttpd_printf("Content-Range: bytes %d-%d/%d" CRLF,
-                    rangeStart, rangeEnd, (int)pStat->st_size);
+    nOut += althttpd_printf("Content-Range: bytes %lld-%lld/%llu" CRLF,
+                            (long long)rangeStart, (long long)rangeEnd,
+                            (unsigned long long)pStat->st_size);
     pStat->st_size = rangeEnd + 1 - rangeStart;
   }else{
     StartResponse("200 OK");
     rangeStart = 0;
   }
@@ -2732,12 +2733,13 @@
   if( rangeEnd>0 && seenContentLength && rangeStart<contentLength ){
     StartResponse("206 Partial Content");
     if( rangeEnd>=contentLength ){
       rangeEnd = contentLength-1;
     }
-    nOut += althttpd_printf("Content-Range: bytes %d-%d/%d" CRLF,
-                            rangeStart, rangeEnd, contentLength);
+    nOut += althttpd_printf("Content-Range: bytes %lld-%lld/%llu" CRLF,
+                            (long long)rangeStart, (long long)rangeEnd,
+                            (unsigned long long)contentLength);
     contentLength = rangeEnd + 1 - rangeStart;
   }else{
     StartResponse("200 OK");
   }
   if( nRes>0 ){

Forewarning: insofar as i can find, sstream_t is not actually guaranteed to be signed. Its name ostensibly means "signed size type" but the definition of it, and common use of it, is essentially "size_t with use of -1 as an error code." Ergo: it may be necessary and/or prudent to use either intmax_t (requires C99 and presumably stdint.h) or long long (inelegant, IMO, but undeniably portable). (Sidebar: i keep harping about C89 compatibility but long long is actually not in C89. It is supported by every known compiler, though, except when building in pedantically-correct C89 mode (-std=c89 -pedantic will do the trick).)

Thanks Stephen, I've been testing for awhile this morning and the above patch by itself isn't enough (you said it was a starting point...) and the only completely successful transfer I've managed to get is by disabling the use of the sendfile() system call in the SendFile() function by changing:

#ifdef linux
to
#ifndef linux

which then uses xferBytes() instead. The man page for sendfile() has this note:

  sendfile() will transfer at most 0x7ffff000 (2,147,479,552) bytes,
   returning the number of bytes actually transferred.  (This is true
   on both 32-bit and 64-bit systems.)

So maybe some logic to toggle use of sendfile() and xferBytes() based on file size is needed?

#ifndef linux

We should be able to gate that behind a mulibc-specific flags if we can find a reliable one. mulibc very specifically makes it difficult to detect at compile-time:

https://catfox.life/2022/04/16/the-musl-preprocessor-debate/

Another alternative would be to add a CLI flag, like -no-sendfile.

Edit: after finishing reading your post before prematurely sending, i think adding a CLI flag makes the most sense. Most of us aren't serving multi-gig files with it, so would benefit from the default of using sendfile(). i wasn't aware of that sendfile() limitation. We'd still need to fix the internal use of int, in any case.

I was thinking more along the lines of just checking the file size in this section?

Even better:).

if( 2!=useHttps && (unsigned long long)pStat->st_size <= 2147479552 )

With my apologies for the delay, this check is now in /info/2ecbd4a08d95528b.

I’m a bit late to this thread, but wouldn’t the correct approach to be to use sendfile in a loop until all bytes are sent?

This is no different than read (and write), which may always read (or write) fewer bytes than requested. (And in the Linux case, never more than 0x7ffff000 bytes in any one call, which fits in a 32-bit signed int.)

I’d offer sample code, but I’m on my phone in an airport. :)

Mark.

... wouldn’t the correct approach to be to use sendfile in a loop until all bytes are sent?

Also good - it wasn't clear to me that sendfile() could be used that way. Given what an outlier case this is for althttpd¹ i'm having a hard time justifying reworking it. That's not to say it won't ever happen, but not today.

1. ^{^} this is the only known thread about someone running into the 2gb limitation.

I missed this at the time, thanks for the fix!

Stephen, having missed this commit at the time, I didn't test it. I made more changes than this when I was looking at it before and indeed testing tonight with latest check-in still fails for files over 2GB. Looking back at the changes I made last time and adjusting for the current check-in, this patch seems to work:

--- althttpd.c.orig
+++ althttpd.c
@@ -506,8 +506,8 @@
 static int mxAge = 120;          /* Cache-control max-age */
 static char *default_path = "/bin:/usr/bin";  /* Default PATH variable */
 static char *zScgi = 0;          /* Value of the SCGI env variable */
-static int rangeStart = 0;       /* Start of a Range: request */
-static int rangeEnd = 0;         /* End of a Range: request */
+static ssize_t rangeStart = 0;       /* Start of a Range: request */
+static ssize_t rangeEnd = 0;         /* End of a Range: request */
 static int maxCpu = MAX_CPU;     /* Maximum CPU time per process */
 static int enableSAB = 0;        /* Add reply header to enable
                                  ** SharedArrayBuffer */
@@ -2528,7 +2528,7 @@
 ** When running in built-in TLS mode the 2nd argument is ignored and
 ** output is instead sent via the TLS connection.
 */
-static void xferBytes(FILE *in, FILE *out, int nXfer, int nSkip){
+static void xferBytes(FILE *in, FILE *out, ssize_t nXfer, ssize_t nSkip){
   size_t n;
   size_t got;
   char zBuf[16384];
@@ -2629,8 +2629,9 @@
     if( rangeEnd>=pStat->st_size ){
       rangeEnd = pStat->st_size-1;
     }
-    nOut += althttpd_printf("Content-Range: bytes %d-%d/%d" CRLF,
-                    rangeStart, rangeEnd, (int)pStat->st_size);
+    nOut += althttpd_printf("Content-Range: bytes %lld-%lld/%llu" CRLF,
+                            (long long)rangeStart, (long long)rangeEnd,
+                            (unsigned long long)pStat->st_size);
     pStat->st_size = rangeEnd + 1 - rangeStart;
   }else{
     StartResponse("200 OK");
@@ -2650,7 +2651,7 @@
   if( zEncoding ){
     nOut += althttpd_printf("Content-encoding: %s" CRLF, zEncoding);
   }
-  nOut += althttpd_printf("Content-length: %d" CRLF CRLF,(int)pStat->st_size);
+  nOut += althttpd_printf("Content-length: %lld" CRLF CRLF,(unsigned long long)pStat->st_size);
   fflush(stdout);
   if( strcmp(zMethod,"HEAD")==0 ){
     MakeLogEntry(0, 2); /* LOG: Normal HEAD reply */
@@ -2667,7 +2668,7 @@
   }else
 #endif
   {
-    xferBytes(in, stdout, (int)pStat->st_size, rangeStart);
+    xferBytes(in, stdout, (unsigned long long)pStat->st_size, rangeStart);
   }
   fclose(in);
   return 0;
@@ -2781,8 +2782,9 @@
     if( rangeEnd>=contentLength ){
       rangeEnd = contentLength-1;
     }
-    nOut += althttpd_printf("Content-Range: bytes %d-%d/%d" CRLF,
-                            rangeStart, rangeEnd, contentLength);
+    nOut += althttpd_printf("Content-Range: bytes %lld-%lld/%llu" CRLF,
+                            (long long)rangeStart, (long long)rangeEnd,
+                            (unsigned long long)contentLength);
     contentLength = rangeEnd + 1 - rangeStart;
   }else{
     StartResponse("200 OK");

I made more changes than this when I was looking at it before and indeed testing tonight with latest check-in still fails for files over 2GB. Looking back at the changes I made last time and adjusting for the current check-in, this patch seems to work:

With my sincere apologies for the delay in getting to this: a variation of that patch is now in /timeline?r=gt-2gb, extended to apply the larger number sizes to the nIn and nOut globals, necessary for logging of sizes >2gb.

That's currently running on my server, and you are invited to try it out :).

Tested and working for me. Thanks again Stephan!

Probably not related, but a few years ago i had issues downloading large files (which were < 3GB) with althttpd: https://sqlite.org/althttpd/info/246e96a260ea6d90