Guides
importOK is designed to process and validate data entirely in the browser, ensuring a seamless user experience. However, several factors can impact performance, including the number of rows in the uploaded file.
There's no strict limitation on file size since processing happens within the browser. However, larger files require more time and computing power to parse. In our experience, Excel files with numerous references and functions tend to import more slowly.
Data transformation and validation run in the background to keep the main thread responsive. The first few records are parsed and validated before reaching the review step, allowing us to provide initial feedback quickly. As validation continues in the background, errors and feedback are updated dynamically.
For async transformers and validators that involve API calls, performance can be impacted by the number of requests made. To optimize this, we require all async validators to be pure functions (i.e., always returning the same output for identical inputs, with no side effects). This allows us to cache validation results, significantly reducing API calls and improving overall efficiency.
Displaying hundreds or thousands of rows in the importOK editor can be resource-intensive. To address this, we use virtual scrolling, a technique that renders only the visible rows while maintaining the correct overall structure. This differs from infinite scrolling, as all items are virtually accounted for without rendering everything at once.
Additionally, when a user modifies a value, only that row is reprocessed for validation, ensuring real-time feedback while keeping performance smooth.
When working with large files, it's best to avoid sending records to your API one by one. Not only would this be extremely slow, but it could also quickly hit your API rate limits.
To optimize this, we recommend using batch processing—sending records in subsets rather than individually. This significantly reduces the number of API calls while also allowing us to provide better progress feedback to the user. For example, with a 2M-row file, you could make just 200 API calls by processing 10K records per request.
Alternatively, if your server can handle it without file size limitations, you can also pass all records at once for even faster processing.
At importOK, we take performance seriously. Running the entire import and validation process in the browser presents challenges, but we continuously refine our approach based on real-world usage. We believe our solution strikes the right balance—providing real-time feedback while ensuring efficient processing.
For extremely large files (e.g., 2 million records), performance can become a challenge, especially if a high number of errors are expected. A 0.5% error rate would require the user to manually review around 10,000 records, which can be cumbersome in any system. If feasible, we recommend splitting large files before processing, allowing users to handle imports in smaller, more manageable batches.