You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: packages/gatsby-source-filesystem/README.md
+47-29Lines changed: 47 additions & 29 deletions
Original file line number
Diff line number
Diff line change
@@ -1,35 +1,28 @@
1
1
# gatsby-source-filesystem
2
2
3
-
A Gatsby source plugin for sourcing data into your Gatsby application
4
-
from your local filesystem.
3
+
A Gatsby source plugin for sourcing data into your Gatsby application from your local filesystem.
5
4
6
-
The plugin creates `File` nodes from files. The various "transformer"
7
-
plugins can transform `File` nodes into various other types of data e.g.
8
-
`gatsby-transformer-json` transforms JSON files into JSON data nodes and
9
-
`gatsby-transformer-remark` transforms markdown files into `MarkdownRemark`
10
-
nodes from which you can query an HTML representation of the markdown.
5
+
The plugin creates `File` nodes from files. The various "transformer" plugins can transform `File` nodes into various other types of data e.g. [`gatsby-transformer-json`](https://www.gatsbyjs.com/plugins/gatsby-transformer-json/) transforms JSON files into JSON data nodes and [`gatsby-transformer-remark`](https://www.gatsbyjs.com/plugins/gatsby-transformer-remark/) transforms markdown files into `MarkdownRemark` nodes from which you can query an HTML representation of the markdown.
11
6
12
7
## Install
13
8
14
-
`npm install gatsby-source-filesystem`
9
+
```shell
10
+
npm install gatsby-source-filesystem
11
+
```
15
12
16
13
## How to use
17
14
18
-
```javascript
19
-
// In your gatsby-config.js
15
+
You can have multiple instances of this plugin to read source nodes from different locations on your filesystem. Be sure to give each instance a unique `name`.
16
+
17
+
```js:title=gatsby-config.js
20
18
module.exports= {
21
19
plugins: [
22
-
// You can have multiple instances of this plugin
23
-
// to read source nodes from different locations on your
24
-
// filesystem.
25
-
//
26
-
// The following sets up the Jekyll pattern of having a
27
-
// "pages" directory for Markdown files and a "data" directory
28
-
// for `.json`, `.yaml`, `.csv`.
29
20
{
30
21
resolve:`gatsby-source-filesystem`,
31
22
options: {
23
+
// The unique name for each instance
32
24
name:`pages`,
25
+
// Path to the directory
33
26
path:`${__dirname}/src/pages/`,
34
27
},
35
28
},
@@ -38,7 +31,10 @@ module.exports = {
38
31
options: {
39
32
name:`data`,
40
33
path:`${__dirname}/src/data/`,
41
-
ignore: [`**/\.*`], // ignore files starting with a dot
34
+
// Ignore files starting with a dot
35
+
ignore: [`**/\.*`],
36
+
// Use "mtime" and "inode" to fingerprint files (to check if file has changed)
37
+
fastHash:true,
42
38
},
43
39
},
44
40
],
@@ -47,9 +43,23 @@ module.exports = {
47
43
48
44
## Options
49
45
50
-
In addition to the name and path parameters you may pass an optional `ignore` array of file globs to ignore.
46
+
### name
47
+
48
+
**Required**
49
+
50
+
A unique name for the `gatsby-source-filesytem` instance. This name will also be a key on the `File` node called `sourceInstanceName`. You can use this e.g. for filtering.
51
+
52
+
### path
53
+
54
+
**Required**
55
+
56
+
Path to the folder that should be sourced. Ideally an absolute path.
51
57
52
-
They will be added to the following default list:
58
+
### ignore
59
+
60
+
**Optional**
61
+
62
+
Array of file globs to ignore. They will be added to the following default list:
53
63
54
64
```text
55
65
**/*.un~
@@ -62,8 +72,24 @@ They will be added to the following default list:
62
72
../**/dist/**
63
73
```
64
74
75
+
### fastHash
76
+
77
+
**Optional**
78
+
79
+
By default, `gatsby-source-filesystem` creates an MD5 hash of each file to determine if it has changed between sourcing. However, on sites with many large files this can lead to a significant slowdown. Thus you can enable the `fastHash` setting to use an alternative hashing mechanism.
80
+
81
+
`fastHash` uses the `mtime` and `inode` to fingerprint the files. On a modern OS this can be considered a robust solution to determine if a file has changed, however on older systems it can be unreliable. Therefore it's not enabled by default.
82
+
83
+
### Environment variables
84
+
65
85
To prevent concurrent requests overload of `processRemoteNode`, you can adjust the `200` default concurrent downloads, with `GATSBY_CONCURRENT_DOWNLOAD` environment variable.
66
86
87
+
In case that due to spotty network, or slow connection, some remote files fail to download. Even after multiple retries and adjusting concurrent downloads, you can adjust timeout and retry settings with these environment variables:
88
+
89
+
-`GATSBY_STALL_RETRY_LIMIT`, default: `3`
90
+
-`GATSBY_STALL_TIMEOUT`, default: `30000`
91
+
-`GATSBY_CONNECTION_TIMEOUT`, default: `30000`
92
+
67
93
## How to query
68
94
69
95
You can query file nodes like the following:
@@ -263,7 +289,7 @@ The `createFileNodeFromBuffer` helper accepts a `Buffer`, caches its contents to
263
289
264
290
The name of the file can be passed to the `createFileNodeFromBuffer` helper. If no name is given, the content hash will be used to determine the name.
265
291
266
-
## Example usage
292
+
####Example usage
267
293
268
294
The following example is adapted from the source of [`gatsby-source-mysql`](https://github.com/malcolm-kee/gatsby-source-mysql):
In case that due to spotty network, or slow connection, some remote files fail to download. Even after multiple retries and adjusting concurrent downloads, you can adjust timeout and retry settings with these environment variables:
0 commit comments