Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export large collections with chunk and relation #319

Open
zotopteam opened this issue Mar 4, 2023 · 0 comments
Open

Export large collections with chunk and relation #319

zotopteam opened this issue Mar 4, 2023 · 0 comments

Comments

@zotopteam
Copy link

zotopteam commented Mar 4, 2023

This is described in the documentation

Export large collections with chunk

Export rows one by one to avoid memory_limit issues [using yield](https://www.php.net/manual/en/language.generators.syntax.php):

function usersGenerator() {
    foreach (User::cursor() as $user) {
        yield $user;
    }
}

// Export consumes only a few MB, even with 10M+ rows.
(new FastExcel(usersGenerator()))->export('test.xlsx');

Problems and solutions

There is no problem with this code, but the problem occurs when exporting complex data. If an export has relation data, there will be problems using cursors (cursors do not support relation, see laravel official documents for details). This will lead to a large number of queries. At this time A better way is to use lazy (lazy also uses yield),

For example, if we want to export the user's group name and school name at the same time, using cursor, each school and group will have to be queried separately. with does not work. There is no problem with using lazy instead, and the speed is much faster.

Export rows one by one to avoid memory_limit issues [using yield](https://www.php.net/manual/en/language.generators.syntax.php):

function usersGenerator() {
    foreach (User::with(['group','school'])->lazy() as $user) {
        yield $user;
    }
}

// Export consumes only a few MB, even with 10M+ rows.
(new FastExcel(usersGenerator()))->export('test.xlsx');

But what needs to be careful is the coexistence of lazy and limit. When using lazy, limit does not take effect.

The above are the problems I encountered in use and the solutions I found. Please refer to them. If there are any problems, please remind me

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant