How LUXCM Handles Large-Site Exports

Table of Contents

Split Exports for Large Sites — The Principles Behind Upload Limits and Part Splitting

I treat split exports as a core feature, not an optional add-on.

I have seen firsthand that once a site grows beyond a certain point, uploading the export file to the import screen can become impossible without it.

In this article, I explain why that happens and how I solved it, using the actual validation logic implemented in the code.

Upload limits are a safeguard for shared resources, not an arbitrary inconvenience

I often meet people who initially assume, “Surely modern hosting can handle a large file upload.”

Bluehost’s guide to upload limits on shared hosting explains precisely why that assumption is wrong. The actual limit “varies by host and plan, but is typically somewhere between a few megabytes and 128MB.”

Those limits are not arbitrary. In a shared hosting environment, where multiple sites use the same server resources, “one unlimited upload can consume enough of the resource pool to slow down or take down every other site on the server.”

The conservative limits on upload_max_filesize, post_max_size, and memory_limit are not a WordPress restriction but a built-in resource-protection mechanism for shared hosting environments. That is the conclusion of this article.

Reading this reinforced my view that telling users to “ask their hosting provider to raise the limit” is not a fundamental solution.

This is not about hosting companies being unhelpful. It is a deliberate design choice intended to protect other sites sharing the same server. So rather than trying to bypass the limit, I chose to break the file into smaller pieces so the process could finish within that limit.

Why the default is 19MB and the adjustment range is 1–500MB

On the export screen, the default maximum size for each part is 19MB. With the “Advanced” option enabled, it can be adjusted anywhere from 1MB to 500MB.

I did not choose the default arbitrarily. Given the real-world range of “a few megabytes to 128MB” cited in Bluehost’s article, 19MB is a value that can pass safely in most hosting environments without requiring configuration changes.
Conversely, I made the range broad—from 1MB to 500MB—to accommodate users whose server upload limit has already been raised, as well as those working in unusually restrictive environments.

This acknowledges that there is no single setting that works for everyone, while allowing users to adjust it for their own hosting environment.

Why the sequence number comes before the filename

I changed the naming scheme for split files from one that placed the sequence number at the end, such as NEWS-LUX-20260825-074327.wpmpx.part1of2, to one that places it at the beginning: part1of2-NEWS-LUX-20260825-074327.wpmpx.

It may seem minor, but this decision came from a real usability problem. When multiple parts are listed in a folder, the sequence number is the only thing that differs, while the rest of the filename is long and nearly identical. If that number is buried at the very end, it is difficult to distinguish the files at a glance in a file browser or upload window.

Moving the sequence number to the front makes the order immediately visible from the first characters when multiple parts are listed together. However, I kept the parser compatible with both patterns so it would continue to recognize files created by older versions, since some users may already have parts in the original format.

Reassembly is not a matter of “joining everything first and seeing what happens”

When users select and upload the parts together on the import screen, I do not merge them immediately. I run four checks in sequence.

First, I check whether any non-part files have been included. If a filename does not match the partN of M- pattern, the system immediately returns the error, “A non-part file has been included.”

Second, I check whether parts from different export jobs have been mixed together. All parts must share the same base filename and the same total number of parts. If they do not, the system reports, “Parts from different export sets have been mixed.” This prevents users from accidentally combining files exported yesterday with files exported today.

Third, I check for duplicate sequence numbers. For example, if part2of5 is uploaded twice while part3of5 is missing, the system reports, “A part with the same sequence number has been uploaded more than once.”

Even after all three checks pass, the system performs one final validation: it confirms that the number of parts actually collected matches the declared total. If parts are missing, it reports the exact numbers: “Parts are missing (uploaded: N, required: M). Please upload all the part files created on the export screen.”

If any of these four validations fails, I do not silently continue. I decided it is better to stop immediately and explain exactly what is wrong than to attempt an import with an incomplete or otherwise invalid set of files.

It is important to be honest that this is not a true “resumable upload”

I also want to distinguish this splitting approach clearly from the “chunked uploads” described in Filestack’s guide to handling large file uploads.

That guide highlights the problems that arise when an entire file is uploaded in a single request: timeouts, when “the server’s execution time limit prevents an upload from completing over a slow or unreliable network”; the cost of retries, because a dropped connection means the upload “has to start over from the beginning”; and the risk that “a shared server processing large files in memory can consume all available resources and eventually crash.” The true chunk-upload protocols it recommends, such as tus and Resumable.js, divide one file into byte-level chunks and can resume from the precise point where the transfer was interrupted.

My approach is different.

Each .wpmpx part is an independent, self-contained file, and users upload these parts one at a time through separate, complete HTTP upload requests.

It is not a true resumable upload that picks up from the exact point where a byte stream was interrupted. Even so, this approach addresses a substantial portion of the problems identified in Filestack’s guide.

Because each part is smaller, the risk of hitting the server’s execution-time limit is reduced. And if one part fails during upload, there is no need to resend a several-hundred-megabyte file from the beginning; only that individual part needs to be uploaded again.

The reassembly step also avoids loading all the parts into memory at once. Instead, it reads and writes them sequentially with stream_copy_to_stream, keeping memory usage during reassembly constant regardless of the number or total size of the parts. I do not want to overstate this as a “fully resumable upload.”

Instead, I describe exactly what it does: divide the file into multiple self-contained parts, validate those parts rigorously, and join them back together in sequence without placing a heavy memory burden on the server.

To sum up

Split exports were designed around the reality of upload limits that vary from one hosting provider to another, but exist for understandable reasons.

The default value and adjustment range were chosen with the real-world distribution of hosting environments in mind, while the decision to move the sequence number in the filename came directly from a practical usability problem.

The most important part is the reassembly stage: mixed files, different export sets, duplicate sequence numbers, and missing parts are each validated separately so the user knows exactly what went wrong, while validated parts are joined sequentially using a streaming approach without adding memory pressure.

I am not hiding the fact that this is not a true resumable upload. But I believe this design alone can eliminate the frustrating situation in which someone trying to migrate a large site is stopped by an upload limit.

References

LUX Content Migration
SOHO SEO
DecoBlocks