A quick idea this morning.
What if there is fast way to make a photo super high resolution?
Splitting the photo into a larger grid with photo pixels and additional transparent pixels should do the trick. Just use a GAN network to imagine the extra pixels to real looking color pixels. An extra pixel could be stored in a separated pixel matrix.
Then this could work for other GAN problems. Just use the model on a small downscaled photo and then apply pixel separation and GAN imagination to increase the resolution. Should be much faster than to do it all at once for a large image.