Restructure the documentation.

[pooja]

The documentation site structure today is not ideal. It's hard for users to know exactly where to go. While this is not so bad today since the docs site is small, it will be harder to change as the site grows (all changed links will need to be manually updated). We should invest some time in aligning on a site structure that we feel good about committing to for the next 2-3 months, and then implement it/migrate over to it asap.

[cwaring]

Completely agree! Organising content with logical grouping, scope for extension and long-term sensible url structures (ensuring we minimise breaking inbound urls) is critical in providing a good learning experience. This should be a topic of its own design sprint so we can always work towards this goal.

[pooja]

Awesome! As part of this, we should also think about how to incorporate lotus docs into the docs site.

[terichadbourne]

Links from filecoin.io to a variety of docs site articles are being broken as the docs site is restructured. I found a couple in response to this thread but there are likely more: https://filecoinproject.slack.com/archives/CEHTVSEG6/p1595015726199500

Any way to programmatically scan the Filecoin website for links that no longer exist on the docs site? Or set up redirects when an article is moved within the docs structure?

From Johnny who also noticed the Slack thread: "Might be worth pinging #infra to set up a redirect for this link. It takes infra 2 seconds to add, and it's handy in case anyone finds that link again (old IPFS version or something)."

[brunolm]

#	Broken link	Link Text	Page where found	Server response
1	https://docs.filecoin.io/introduction/new-to-web3.html	this guide	url	404
2	https://docs.filecoin.io/introduction/what-is-filecoin.html	What is Filecoin?	url	404
3	https://docs.filecoin.io/introduction/why-filecoin.html	Why Filecoin?	url	404
4	https://docs.filecoin.io/introduction/ipfs-and-filecoin.html	IPFS and Filecoin	url	404
5	https://docs.filecoin.io/introduction/filecoin-compared-to.html	Filecoin compared to...	url	404
6	https://docs.filecoin.io/community/chat-and-discussion-forums.html	Filecoin’s chat and forums	url	404
7	https://docs.filecoin.io/build/core-products/filecoin-backed-pinning-services.html	FPS	url	404
8	https://docs.filecoin.io/build/start-building/interacting-with-the-network.html	Interacting with the network	url	404
9	https://docs.filecoin.io/build/core-products/powergate.html	Powergate	url	404
10	https://docs.filecoin.io/build/core-products/protocol-implementations.html	Filecoin protocol implementations	url	404
11	https://docs.filecoin.io/build/developer-tools/wallets-signing-tools-api-clients.html	Wallets signing tools and API clients	url	404
12	https://docs.filecoin.io/build/examples/meme-marketplace/overview.html	Meme Marketplace	url	404
13	https://docs.filecoin.io/build/examples/simple-pinning-service/overview.html	Simple Pinning Service	url	404
14	https://docs.filecoin.io/build/examples/slate/overview.html	Slate	url	404
15	https://docs.filecoin.io/build/examples/network-inspector/overview.html	Network Inspector	url	404
16	https://docs.filecoin.io/community/contribute/ways-to-contribute.html	ways to contribute	url	404
17	https://docs.filecoin.io/community/social-media/social-media.html	how to find them all	url	404
18	https://docs.filecoin.io/project/related-projects.html	Filecoin-related projects	url	404
19	https://docs.filecoin.io/how-to/store/large-files.html	Very Large Files	url	404
20	https://docs.filecoin.io/community/contribute/grammar-formatting-and-style.html	Style and formatting guide	url	404
21	https://docs.filecoin.io/community/contribute/writing-guide.html	Writing guide	url	404
22	https://docs.filecoin.io/community/contribute/contribution-tutorial.html	contribution tutorial	url	404
23	https://docs.filecoin.io/build/examples/network-inspector/step-1-start-lotus-devnet-and-go-ipfs.html	Step 1 - Start lotus-devnet and go-ipfs	url	404
24	https://docs.filecoin.io/build/examples/meme-marketplace/step-1-blockchain-and-contracts-setup.html	Step 1	url	404
25	https://docs.filecoin.io/build/examples/meme-marketplace/step-5-connecting-app-with-blockchain.html	Step 5	url	404
26	https://docs.filecoin.io/build/examples/meme-marketplace/step-5-connecting-app-with-blockchain.md/	Step 5	url	404
27	https://slate.host/system	React Component Library	url	404
28	https://localhost:3000/	https://localhost:3000	url	irrelevant
29	https://docs.textile.io/hub/app-apis/	A user group key	url	404
30	https://www.npmjs.com/package/koa-WebSocket	koa-WebSocket	url	404
31	https://www.npmjs.com/package/filecoin-shipyard/lotus-client-provider-browser	@filecoin-shipyard/lotus-client-provider-browser	url	404

1-25 are redirecting to the correct page, but they're responding with status code 404. Ideally they'd respond with 301. Just updating those links to point to the new URLs would be even better.

26, 27, 29 - broken links

30, 31 - correct links for NPM are:

https://www.npmjs.com/package/koa-websocket
https://www.npmjs.com/package/@filecoin-shipyard/lotus-client-provider-browser

I also found some links to assets that are responding 404

Broken link	Page where found
https://docs.filecoin.io/build/favicon.ico	url
https://docs.filecoin.io/build/favicon-16x16.png	url
https://docs.filecoin.io/build/favicon-32x32.png	url
https://docs.filecoin.io/build/manifest.json	url
https://docs.filecoin.io/build/apple-touch-icon.png	url
https://docs.filecoin.io/build/safari-pinned-tab.svg	url
https://docs.filecoin.io/build/examples/favicon.ico	url
https://docs.filecoin.io/build/examples/favicon-16x16.png	url
https://docs.filecoin.io/build/examples/favicon-32x32.png	url
https://docs.filecoin.io/build/examples/manifest.json	url
https://docs.filecoin.io/build/examples/apple-touch-icon.png	url
https://docs.filecoin.io/build/examples/safari-pinned-tab.svg	url
https://docs.filecoin.io/build/examples/meme-marketplace/favicon.ico	url
https://docs.filecoin.io/build/examples/meme-marketplace/favicon-16x16.png	url
https://docs.filecoin.io/build/examples/meme-marketplace/favicon-32x32.png	url
https://docs.filecoin.io/build/examples/meme-marketplace/manifest.json	url
https://docs.filecoin.io/build/examples/meme-marketplace/apple-touch-icon.png	url
https://docs.filecoin.io/build/examples/meme-marketplace/safari-pinned-tab.svg	url

[cwaring]

Hey @brunolm and welcome! 👋

The first chunk of 404s are caused by this upstream issue in VuePress, we recently introduced local 404 checking for internal content urls #119, which requires the links to point to the .md version of the file and unfortunately it seems that the markdown link generator is hardcoding the .html suffix.

A couple of solutions to this:

1. Patch VuePress upstream to fix this issue so that it interacts with the clean url plugin better.
2. Revert back to using absolute path links in our docs and then do 404 checking async via another external crawler (could be more comprehensive).
3. Remove the clean urls plugin and default to .html extensions for our docs.

The later broken links are related to how your browser refetches these metadata files on route change. These paths are relative in order to be compatible with IPFS gateway links (gateway.ipfs.io/ipfs/hash/) and if you open the page in a new window you will see the paths are correct on first load. I would like to find a better solution to this but it's a lower priority, but as we are currently not hosting these docs on IPFS (due to required server-side redirects) I could disable this for now.

[cwaring]

@brunolm also to add; the .html links while not ideal doesn't break the site in practice due to the client-side routing. SEO should also be fine because all the content is linked to from the sidebar tree and via an auto-generated sitemap.xml. However if you have any suggestions for how to implement more robust 404 checking to our PR process I would be keen to push that upgrade 😊

[cwaring]

Completed in #367