Merge pull request #293 from cmu-delphi/docs/survey-signup

capnrefsmmat · web-flow · commit d7add864ba6d · 2020-11-20T18:22:36.000-05:00
Document next survey wave and how to get involved
diff --git a/docs/symptom-survey/coding.md b/docs/symptom-survey/coding.md
@@ -1,7 +1,7 @@
 ---
 title: Questions and Coding
 parent: COVID Symptom Survey
-nav_order: 3
+nav_order: 5
 ---
 
 # Questions and Coding
@@ -268,3 +268,68 @@ your protocol.
   your household’s finances?") has been removed and replaced with item C15, for
   consistency with the international version of the survey.
 * Item D1b ("Are you currently pregnant?") has been removed.
+
+## Wave 5
+
+Wave 5 will be deployed in late November, 2020. It is available in English, as
+well as
+
+* Simplified Chinese
+* English (UK)
+* Spanish (Latin America)
+* Spanish
+* French
+* Brazilian Portuguese
+* Vietnamese
+
+**Draft** files:
+
+* [Survey text and coding](waves/Survey_of_COVID-Like_Illness_-_Wave_5.pdf)
+  (PDF)
+
+### Summary of Changes
+
+Wave 5 contains minor changes to the survey instrument and a few new items.
+Please review the changes carefully when you use responses from multiple waves
+of this survey.
+
+Note that this changelog is a draft; as the survey instrument is finalized, this
+changelog will be updated to include all final changes.
+
+#### Consent Text
+
+The survey consent text has been altered to encourage respondents to answer the
+survey, even if they have already taken it before:
+
+> We encourage you to complete the survey each time you are invited, even if you
+> have participated before. Completing the survey again will help us understand
+> how the situation is changing.
+
+#### New Items
+
+* Item C16 asks respondents to estimate how many people are wearing masks in
+  their community.
+* Item C7 was previously asked in Waves 1-3; it asks respondents the extent they
+  are avoiding other people.
+* Item C17 asks whether respondents have received a flu vaccine. The time frame
+  and responses are adapted to specify the current seasonal flu vaccine. A more
+  general version inquiry about the flu vaccine appeared in Wave 1-3 as item C2.
+* Items E1-E3 ask about household children and their education during the
+  pandemic. These items appear for respondents who indicate there is a child in
+  their household under the age of 18. E1 asks respondents to indicate the
+  current grade level(s) the child(ren) in their household. Item E2 asks the
+  respondents if the child(ren) are attending in-person classes part time or
+  full time. Item E3 asks respondents what measures are applied to prevent the
+  spread of COVID-19 when the child(ren) attend in-person classes (e.g.
+  mandatory, mask wearing, closed communal areas).
+
+#### Changed Items
+
+* Item B2 now includes headaches and changes in sleep as symptoms.
+* Item D8 now includes the option of Master’s degree (unfortunately omitted in
+  Wave 4) and has examples of professional degree for clarification.
+
+#### Removed Items
+
+* There are no items from Wave 4 that were removed in the Wave 5 version of
+  this survey.
diff --git a/docs/symptom-survey/collaboration-revision.md b/docs/symptom-survey/collaboration-revision.md
@@ -0,0 +1,32 @@
+---
+title: Collaboration and Survey Revision
+parent: COVID Symptom Survey
+nav_order: 1
+---
+
+# Collaboration and Survey Revision
+
+Delphi continues to revise the COVID-19 Symptom Survey instruments in order to
+prioritize items that have the greatest utility for the response to the COVID-19
+pandemic. We conduct revisions in collaboration with data users, fellow
+researchers, and public health officials, to ensure the survey data best serves
+public health and research goals.
+
+## Survey Revisions
+
+If there is a revision or question you would like us to consider, please fill
+out [this form requesting details about your
+proposal](https://forms.gle/q6NS8fPJJofKQ9mM8). This request can be submitted by
+researchers regardless of whether they have a signed Data Use Agreement for the
+individual responses to the COVID Symptom Survey.
+
+## Collaboration Meetings
+
+Collaboration in this ongoing effort is our priority. Delphi hosts a
+collaboration meeting the first Friday of each month at 2–3pm ET. The meeting is
+a chance to announce upcoming changes to the survey, have a discussion and get
+input about the instrument, share preliminary findings and network with other
+researchers.
+
+If you're interested in joining, contact us at
+<delphi-survey-info@lists.andrew.cmu.edu>.
diff --git a/docs/symptom-survey/data-access.md b/docs/symptom-survey/data-access.md
@@ -0,0 +1,22 @@
+---
+title: Getting Data Access
+parent: COVID Symptom Survey
+nav_order: 0
+---
+
+# Getting Data Access
+
+The Delphi Research Group at Carnegie Mellon University (CMU), in partnership
+with Facebook, has conducted the COVID Symptom Survey to better understand the
+spread of COVID-19 and its effects on public health and well-being. This may
+help improve our local and national responses to the pandemic and our
+understanding of how it has affected society.
+
+De-identified data can be made available to researchers associated with
+universities or non-profit organizations. To request access to the data please
+submit the information requested in [Facebook's page on obtaining data
+access](https://dataforgood.fb.com/docs/covid-19-symptom-survey-request-for-data-access/),
+which sets out the basic conditions and provides a form to request access. An
+[international version of the COVID Symptom Survey](https://covidmap.umd.edu/)
+is conducted by the University of Maryland (UMD) and access can be requested
+through the same form.
diff --git a/docs/symptom-survey/index.md b/docs/symptom-survey/index.md
@@ -19,7 +19,8 @@ as the [`fb-survey` data source](../api/covidcast-signals/fb-survey.md).
 This documentation is for users who have a signed Data Use Agreement to receive
 individual response data from the survey. It describes the survey items, data
 coding, data distribution, and the survey weights computed by Facebook. If you
-are a researcher and would like to get access to the data, check [Facebook's
-page on obtaining data
-access](https://dataforgood.fb.com/docs/covid-19-symptom-survey-request-for-data-access/),
-which sets out the basic conditions and provides a form to request access.
+are a researcher and would like to get access to the data, see our page on
+getting [data access](data-access.md).
+
+If you have questions about the survey or getting access to data, contact us at
+<delphi-survey-info@lists.andrew.cmu.edu>.
diff --git a/docs/symptom-survey/server-access.md b/docs/symptom-survey/server-access.md
@@ -1,7 +1,7 @@
 ---
 title: SFTP Server Access
 parent: COVID Symptom Survey
-nav_order: 4
+nav_order: 2
 ---
 
 # SFTP Server Access
diff --git a/docs/symptom-survey/survey-files.md b/docs/symptom-survey/survey-files.md
@@ -1,13 +1,13 @@
 ---
 title: Response Files
 parent: COVID Symptom Survey
-nav_order: 1
+nav_order: 3
 ---
 
 # Response Files
 {: .no_toc}
 
-Users with access to the [COVID symptom survey](./index.md) individual response
+Users with access to the [COVID Symptom Survey](./index.md) individual response
 data should have received SFTP credentials for a private server where the data
 are stored. To connect to the server, see the [server access documentation](server-access.md).
 This documentation describes the survey data available on that server.
@@ -41,12 +41,23 @@ day the survey response was started, in the Pacific time zone (UTC -
 7). `recorded` refers to the day survey data was retrieved; see the [lag
 policy](#lag-policy) for more details.
 
-Every day, we write response files for *all* days of data, with today's
+Every day, we write response files for all recent days of data, with today's
 `recorded` date. You need only load the most recent set of `recorded` files to
 obtain all survey responses; the older versions are available to track any
 changes in file formats or slight changes from late-arriving responses, as
 described in the [lag policy below](#lag-policy).
 
+## Loading Data Files
+
+As described above, one day of data may be reissued several times, if responses
+arrive late, file formats are changed, or errors in data processing are
+corrected. You need only load the latest version of each file.
+
+For data users who use R to load and process data, we provide a [`get_survey_df`
+function](survey-utils.R) to read a directory of CSV files (such as those
+provided on the SFTP server), select the correct files, and read them into a
+single data frame for use.
+
 ## Conditions Responses are Recorded
 
 The survey was configured to record responses under two sets of circumstances:
diff --git a/docs/symptom-survey/survey-utils.R b/docs/symptom-survey/survey-utils.R
@@ -0,0 +1,86 @@
+library(readr)
+library(purrr)
+library(dplyr)
+
+#' Fetch all survey data in a chosen directory.
+#'
+#' There can be multiple data files for a single day of survey responses, for
+#' example if the data is reissued when late-arriving surveys are recorded.
+#' Each file contains *all* data recorded for that date, so only the later files
+#' are needed.
+#'
+#' This function extracts the date from each file, determines which files are
+#' reissued data, and produces a single data frame representing the most recent
+#' data available for each day. It can read gzip-compressed CSV files, such as
+#' those on the SFTP site, using `readr::read_csv`.
+#'
+#' This function handles column types correctly for surveys up to Wave 4.
+#'
+#' @param directory Directory in which to look for survey CSV files, relative to
+#'   the current working directory.
+#' @param pattern Regular expression indicating which files in that directory to
+#'   open. By default, selects all `.csv.gz` files, such as those downloaded
+#'   from the SFTP site.
+#' @return A single data frame containing all survey responses. Note that this
+#'   data frame may have millions of rows and use gigabytes of memory, if this
+#'   function is run on *all* survey responses.
+get_survey_df <- function(directory, pattern = "*.csv.gz$") {
+  files <- list.files(directory, pattern = pattern)
+
+  files <- map_dfr(files, get_file_properties)
+
+  latest_files <- files %>%
+    group_by(date) %>%
+    filter(recorded == max(recorded)) %>%
+    ungroup() %>%
+    pull(filename)
+
+  big_df <- map_dfr(
+    latest_files,
+    function(f) {
+      # stop readr from thinking commas = thousand separators,
+      # and from inferring column types incorrectly
+      read_csv(file.path(directory, f), locale = locale(grouping_mark = ""),
+               col_types = cols(
+                 A2b = col_number(),
+                 A3 = col_character(),
+                 A4 = col_number(),
+                 B2 = col_character(),
+                 B2_14_TEXT = col_character(),
+                 B2c = col_character(),
+                 B2c_14_TEXT = col_character(),
+                 B4 = col_number(),
+                 B5 = col_number(),
+                 B7 = col_character(),
+                 B10b = col_character(),
+                 B12a = col_character(),
+                 C1 = col_character(),
+                 C3 = col_number(),
+                 C4 = col_number(),
+                 C5 = col_number(),
+                 C7 = col_number(),
+                 C13 = col_character(),
+                 C13a = col_character(),
+                 D1_4_TEXT = col_character(),
+                 fips = col_character(),
+                 UserLanguage = col_character(),
+                 StartDatetime = col_character(),
+                 EndDatetime = col_character(),
+                 .default = col_number()))
+    }
+  )
+  return(big_df)
+}
+
+## Helper function to extract dates from each file's filename.
+get_file_properties <- function(filename) {
+  short <- strsplit(filename, ".", fixed = TRUE)[[1]][1]
+  parts <- strsplit(short, "_", fixed = TRUE)[[1]]
+
+  filedate <- as.Date(paste(parts[3:5], collapse = "-"))
+  recordeddate <- as.Date(paste(parts[7:9], collapse = "-"))
+
+  return(data.frame(filename = filename,
+                    date = filedate,
+                    recorded = recordeddate))
+}
diff --git a/docs/symptom-survey/waves/Survey_of_COVID-Like_Illness_-_Wave_5.pdf b/docs/symptom-survey/waves/Survey_of_COVID-Like_Illness_-_Wave_5.pdf
diff --git a/docs/symptom-survey/weights.md b/docs/symptom-survey/weights.md
@@ -1,7 +1,7 @@
 ---
 title: Survey Weights
 parent: COVID Symptom Survey
-nav_order: 2
+nav_order: 4
 ---
 
 # Survey Weights
