You do not need to set paged query arguments if WP is already doing this. It happens automatically when the url includes /page/XX/
I don’t completely understand your query or why you would do something different on pages>2 verses page 1. but here are some things
$query->set( 'posts_per_page', -1 );
cause there to be no pages and all of the paged stuff you are setting is ignored.
You want your query for all pages to be identical or you will get different results.
