You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+20-14
Original file line number
Diff line number
Diff line change
@@ -31,7 +31,8 @@ module.exports = {
31
31
32
32
## Options
33
33
34
-
This plugin uses [`generate-robotstxt`](https://github.com/itgalaxy/generate-robotstxt#usage) to generate content of `robots.txt` and it has the following options:
34
+
This plugin uses [`generate-robotstxt`](https://github.com/itgalaxy/generate-robotstxt#usage) to generate content
The `env` key will be taken from `process.env.GATSBY_ACTIVE_ENV` first (see [Gatsby Environment Variables](https://www.gatsbyjs.org/docs/environment-variables/) for more information on this variable), falling back to `process.env.NODE_ENV`. When this is not available then it defaults to `development`.
88
+
The `env` key will be taken from `process.env.GATSBY_ACTIVE_ENV` first (
89
+
see [Gatsby Environment Variables](https://www.gatsbyjs.org/docs/environment-variables/) for more information on this
90
+
variable), falling back to `process.env.NODE_ENV`. When this is not available then it defaults to `development`.
89
91
90
92
You can resolve the `env` key by using `resolveEnv` function:
91
93
@@ -102,10 +104,10 @@ module.exports = {
102
104
resolveEnv: () =>process.env.GATSBY_ENV,
103
105
env: {
104
106
development: {
105
-
policy: [{userAgent:'*', disallow: ['/']}]
107
+
policy: [{userAgent:'*', disallow: ['/']}]
106
108
},
107
109
production: {
108
-
policy: [{userAgent:'*', allow:'/'}]
110
+
policy: [{userAgent:'*', allow:'/'}]
109
111
}
110
112
}
111
113
}
@@ -139,13 +141,15 @@ module.exports = {
139
141
module.exports= {
140
142
host:'https://www.example.com',
141
143
sitemap:'https://www.example.com/sitemap.xml',
142
-
policy: [{userAgent:'*'}]
144
+
policy: [{userAgent:'*'}]
143
145
};
144
146
```
145
147
146
148
#### Netlify
147
149
148
-
If you would like to disable crawlers for [deploy-previews](https://www.netlify.com/blog/2016/07/20/introducing-deploy-previews-in-netlify/) you can use the following snippet:
150
+
If you would like to disable crawlers
151
+
for [deploy-previews](https://www.netlify.com/blog/2016/07/20/introducing-deploy-previews-in-netlify/) you can use the
152
+
following snippet:
149
153
150
154
`gatsby-config.js`
151
155
@@ -170,15 +174,15 @@ module.exports = {
170
174
resolveEnv: () =>NETLIFY_ENV,
171
175
env: {
172
176
production: {
173
-
policy: [{userAgent:'*'}]
177
+
policy: [{userAgent:'*'}]
174
178
},
175
179
'branch-deploy': {
176
-
policy: [{userAgent:'*', disallow: ['/']}],
180
+
policy: [{userAgent:'*', disallow: ['/']}],
177
181
sitemap:null,
178
182
host:null
179
183
},
180
184
'deploy-preview': {
181
-
policy: [{userAgent:'*', disallow: ['/']}],
185
+
policy: [{userAgent:'*', disallow: ['/']}],
182
186
sitemap:null,
183
187
host:null
184
188
}
@@ -191,7 +195,9 @@ module.exports = {
191
195
192
196
### `query`-option
193
197
194
-
By default the site URL will come from the Gatsby node `site.siteMeta.siteUrl`. Like in [Gatsby's sitemap plugin](https://www.gatsbyjs.org/packages/gatsby-plugin-sitemap/) an optional GraphQL query can be used to provide a different value from another data source as long as it returns the same shape:
198
+
By default the site URL will come from the Gatsby node `site.siteMeta.siteUrl`. Like
199
+
in [Gatsby's sitemap plugin](https://www.gatsbyjs.org/packages/gatsby-plugin-sitemap/) an optional GraphQL query can be
200
+
used to provide a different value from another data source as long as it returns the same shape:
0 commit comments