Why are Pre-rendered HTML or SSG pages given an entry in the _redirects file?

[moop-moop]

The Netlify platform does file shadowing, where the static file is delivered first regardless:
https://docs.netlify.com/routing/redirects/rewrites-proxies/#shadowing

Example in _redirects:
/health/en/kids/emily-story/article /health/en/kids/emily-story/article.html 200

I ask, because we are pre-rendering + SSG many thousands (15,000+) of pages. So the _headers files has an entry for each. Also, the console log prints a line for each during build.

[FinnWoelm]

Hey @moop-moop,

You are totally right about these redirect being unnecessary. I will change that this weekend 😊

Do you have a suggestion for how to improve the build output? I do want to keep the build output, so that people know what's happening. Maybe we can limit each build step to print max 50 files. And passing the option --debug to next-on-netlify could be used to print all files, if desired?

Thanks for taking the time to post this issue!

[moop-moop]

Hey @moop-moop,

You are totally right about these redirect being unnecessary. I will change that this weekend 😊

If redirect rules is not present, the logging won't be so long. 😉

Do you have a suggestion for how to improve the build output?
I do want to keep the build output, so that people know what's happening. Maybe we can limit each build step to print max 50 files. And passing the option --debug to next-on-netlify could be used to print all files, if desired?

Excellent idea!, but I would limit any list to maybe 5 and say "X more entries.". next build sort of does that....

Thanks for taking the time to post this issue!

No... thank you! 🙇
Awesome package and a great idea by-the-way! We will be using on some large healthcare sites.
I'll be sure to post anything that arises as we implement.

[moop-moop]

Hey @moop-moop,

You are totally right about these redirect being unnecessary. I will change that this weekend 😊

Do you have a suggestion for how to improve the build output? I do want to keep the build output, so that people know what's happening. Maybe we can limit each build step to print max 50 files. And passing the option --debug to next-on-netlify could be used to print all files, if desired?

Ahhh, my lists are so long, I just noticed the log out of the file copying too, not just the rewrite rules ....

🔥 Setting up SSG pages
5:54:19 PM:    1. Copying pre-rendered SSG pages to out_publish/

I still think 5 is enough, if there is an option for all. Whatever makes my 36,000 line log smaller helps! 😄

Thanks for taking the time to post this issue!

[FinnWoelm]

Hey @moop-moop,

I removed all redirects for SSG & HTML pages that are using static routing. I also added a --max-log-lines option that defaults to 50 and can be used to limit the number of lines shown in each step of the build process (i.e., 50 SSR pages, 50 SSG pages, 50 redirects, ...). If you prefer 5 lines of output, just run next-on-netlify --max-log-lines 5.

Unfortunately, I encountered a small issue with locally previewing using netlify-cli. When you have a file (e.g., test.html) and a directory with the same name (e.g., /test), you get a 403 forbidden error during local preview. It looks like the local server used by netlify-cli serves the directory rather than the file. Because directory listings are disabled, the server responds with forbidden. Everything works correctly when deployed on Netlify.

When having a redirect for every page in the _redirects file, this error does not occur. So with the changes I made today, I would essentially be adding a bug to next-on-netlify when used for local previewing. Admittedly, it's a very small bug, but I'd rather file the issue on the netlify-cli repo and get the local preview behavior fixed, so that it matches the behavior when deployed on Netlify. Until then, I'm keeping the changes on the reduce-redirects branch.

The code on the reduce-redirects branch is fully tested and works perfectly, aside from this minor issue in local preview. If you don't mind using a branch until the issue is resolved, you can do it like so:

npm install finnWoelm/next-on-netlify#reduce-redirects --save

I will keep the branch up for at least one month after it gets merged into master. Hopefully that works for you as a temporary solution.

I'll keep you posted on the issue. Hopefully it gets resolved quickly, so that we can merge this into master!

[moop-moop]

@FinnWoelm
I understand. I have never tested that behavior with netlify dev or a live deploy for that matter. I don't see any Netlify documentation on what the behavior should be. There is also the Asset Optimization -> Pretty URLs feature, again which I have no idea if netlify dev employs or if it would impact testing for next-on-netlify

[moop-moop]

Hey @moop-moop,

I removed all redirects for SSG & HTML pages that are using static routing. I also added a --max-log-lines option that defaults to 50 and can be used to limit the number of lines shown in each step of the build process (i.e., 50 SSR pages, 50 SSG pages, 50 redirects, ...). If you prefer 5 lines of output, just run next-on-netlify --max-log-lines 5.

Unfortunately, I am not sure https://www.npmjs.com/package/log-update is compatible with the Netlify logging. Locally it worked fine, but with next-on-netlify --max-log-lines 5 on Netlify propper, I saw this:

5:26:43 PM: 🔥 Copying pre-rendered SSG pages to out_publish/ and JSON data to out_publish/_next/data/
5:26:43 PM:    /healthwise/d04376a1
5:26:43 PM:    /healthwise/d05354a1
5:26:43 PM:    /healthwise/d08284a1
5:26:43 PM:    /healthwise/d04727a1
5:26:43 PM:    /healthwise/d08578a1
5:26:43 PM:    + 1 more (run next-on-netlify with --max-log-lines XX to show more or fewer l
5:26:43 PM: ines)
5:26:43 PM:    + 2 more (run next-on-netlify with --max-log-lines XX to show more or fewer l
5:26:43 PM: ines)
5:26:43 PM:    + 3 more (run next-on-netlify with --max-log-lines XX to show more or fewer l
5:26:43 PM: ines)
5:26:43 PM:    + 4 more (run next-on-netlify with --max-log-lines XX to show more or fewer l
5:26:43 PM: ines)
5:26:43 PM:    + 5 more (run next-on-netlify with --max-log-lines XX to show more or fewer l
5:26:43 PM: ines)
5:26:43 PM:    + 6 more (run next-on-netlify with --max-log-lines XX to show more or fewer l
5:26:43 PM: ines)

Unfortunately, I encountered a small issue with locally previewing using netlify-cli. When you have a file (e.g., test.html) and a directory with the same name (e.g., /test), you get a 403 forbidden error during local preview. It looks like the local server used by netlify-cli serves the directory rather than the file. Because directory listings are disabled, the server responds with forbidden. Everything works correctly when deployed on Netlify.

When having a redirect for every page in the _redirects file, this error does not occur. So with the changes I made today, I would essentially be adding a bug to next-on-netlify when used for local previewing. Admittedly, it's a very small bug, but I'd rather file the issue on the netlify-cli repo and get the local preview behavior fixed, so that it matches the behavior when deployed on Netlify. Until then, I'm keeping the changes on the reduce-redirects branch.

The code on the reduce-redirects branch is fully tested and works perfectly, aside from this minor issue in local preview. If you don't mind using a branch until the issue is resolved, you can do it like so:

npm install finnWoelm/next-on-netlify#reduce-redirects --save

I will keep the branch up for at least one month after it gets merged into master. Hopefully that works for you as a temporary solution.

I'll keep you posted on the issue. Hopefully it gets resolved quickly, so that we can merge this into master!

[FinnWoelm]

Oh, you are absolutely right. I wonder if there is a way to have an updating console.log on Netlify. I will look into it and report back asap!

[moop-moop]

@FinnWoelm

Netlify support/engineering provided me with the following information about large numbers of redirects/rewrites:

We do recommend a max of 10k redirects, and under 1000 if you can. The reason for this is that we parse the entire redirects list looking for a match on any new (uncached) content, on every page load.
...
Will that many break your site? Probably not; I've seen a customer use around that many redirects before. Will it slow your site loads down? Potentially by a few hundred milliseconds for every asset so...probably worth trying to optimize.

What do you think of another special argument/flag to not write out static file redirects?

This would allow most users to ignore, and not notice the behavior difference (bug) you noticed in netlify dev and Netlify proper.
In my case, I would use with the understanding that netlify dev behavior with static files may behave differently that Netlify proper.

Thanks for listening!

[moop-moop]

@FinnWoelm

Netlify support/engineering provided me with the following information about large numbers of redirects/rewrites:

We do recommend a max of 10k redirects, and under 1000 if you can. The reason for this is that we parse the entire redirects list looking for a match on any new (uncached) content, on every page load.
...
Will that many break your site? Probably not; I've seen a customer use around that many redirects before. Will it slow your site loads down? Potentially by a few hundred milliseconds for every asset so...probably worth trying to optimize.

What do you think of another special argument/flag to not write out static file redirects?

This would allow most users to ignore, and not notice the behavior difference (bug) you noticed in netlify dev and Netlify proper.
In my case, I would use with the understanding that netlify dev behavior with static files may behave differently that Netlify proper.

Thanks for listening!

I'll have 16-17k static files in one build for reference.

[FinnWoelm]

Hi @moop-moop,

Unfortunately, I am not sure https://www.npmjs.com/package/log-update is compatible with the Netlify logging.

This is now fixed in the reduce-redirects branch! log-update is gone and next-on-netlify now uses a simple console.log at the end of each build step to indicate the number of hidden items, if any.

What do you think of another special argument/flag to not write out static file redirects?

I just filed an issue with Netlify about the inconsistency between netlify dev and deployed Netlify: netlify/cli#1017

Based on previous interactions with the Netlify team, my sense is that they will be eager to resolve this difference and make netlify dev match the behavior of deployed Netlify. As soon as that is done, reduce-redirects will be merged into master and become the default behavior for anyone using next-on-netlify.

If the Netlify team does not resolve the difference, then I think your idea of a special argument is a great idea. Something like --no-static-redirects.

How does that sound to you? 😊
Finn

[FinnWoelm]

Hey @moop-moop,

The netlify dev issue that was blocking this branch from merging into master was just resolved: netlify/cli#1017

Once the updated version of netlify-cli is released, I will merge this branch into master 🎉

Thanks for your patience!

[moop-moop]

Hey @moop-moop,

The netlify dev issue that was blocking this branch from merging into master was just resolved: netlify/cli#1017

Once the updated version of netlify-cli is released, I will merge this branch into master 🎉

Thanks for your patience!

Thank you for all your work and effort! 🥇