sqlpage.run_sql

CHANGELOG.md

@@ -2,9 +2,15 @@
 
 ## 0.20.0 (unreleased)
 
+ - **file inclusion**. This is a long awaited feature that allows you to include the contents of one file in another. This is useful to factorize common parts of your website, such as the header, or the authentication logic. There is a new [`sqlpage.run_sql`](https://sql.ophir.dev/functions.sql?function=run_sql#function) function that runs a given SQL file and returns its result as a JSON array. Combined with the existing [`dynamic`](https://sql.ophir.dev/documentation.sql?component=dynamic#component) component, this allows you to include the content of a file in another, like this:
+  ```sql
+  select 'dynamic' as component, sqlpage.run_sql('header.sql') as properties;
+  ```
+ - **more powerful *dynamic* component**: the [`dynamic`](https://sql.ophir.dev/documentation.sql?component=dynamic#component) component can now be used to generate the special *header* components too, such as the `redirect`, `cookie`, `authentication`, `http_header` and `json` components. The *shell* component used to be allowed in dynamic components, but only if they were not nested (a dynamic component inside another one). This limitation is now lifted. This is particularly useful in combination with the new file inclusion feature, to factorize common parts of your website. There used to be a limited to how deeply nested dynamic components could be, but this limitation is now lifted too.
  - Add an `id` attribute to form fields in the [form](https://sql.ophir.dev/documentation.sql?component=form#component) component. This allows you to easily reference form fields in custom javascript code.
  - New [`rss`](https://sql.ophir.dev/documentation.sql?component=rss#component) component to create RSS feeds, including **podcast feeds**. You can now create and manage your podcast feed entirely in SQL, and distribute it to all podcast directories such as Apple Podcasts, Spotify, and Google Podcasts.
  - Better error handling in template rendering. Many template helpers now display a more precise error message when they fail to execute. This makes it easier to debug errors when you [develop your own custom components](https://sql.ophir.dev/custom_components.sql).
+ - better error messages when an error occurs when defining a variable with `SET`. SQLPage now displays the query that caused the error, and the name of the variable that was being defined.
  - Updated SQL parser to [v0.44](https://github.com/sqlparser-rs/sqlparser-rs/blob/main/CHANGELOG.md#0440-2024-03-02)
    - support [EXECUTE ... USING](https://www.postgresql.org/docs/current/plpgsql-statements.html#PLPGSQL-STATEMENTS-EXECUTING-DYN) in PostgreSQL
    - support `INSERT INTO ... SELECT ... RETURNING`, which allows you to insert data into a table, and easily pass values from the inserted row to a SQLPage component. [postgres docs](https://www.postgresql.org/docs/current/dml-returning.html), [mysql docs](https://mariadb.com/kb/en/insertreturning/), [sqlite docs](https://sqlite.org/lang_returning.html)

examples/official-site/sqlpage/migrations/38_run_sql.sql

@@ -0,0 +1,48 @@
+INSERT INTO sqlpage_functions (
+        "name",
+        "introduced_in_version",
+        "icon",
+        "description_md"
+    )
+VALUES (
+        'run_sql',
+        '0.20.0',
+        'login',
+        'Executes another SQL file and returns its result as a JSON array.
+
+### Example
+
+#### Include a common header in all your pages
+
+It is common to want to run the same SQL queries at the beginning of all your pages,
+to check if an user is logged in, render a header, etc.
+You can create a file called `common_header.sql`, and use the `dynamic` component with the `run_sql` function to include it in all your pages.
+
+```sql
+select ''dynamic'' as component, sqlpage.run_sql(''common_header.sql'') as properties;
+```
+
+#### Notes
+
+ - **recursion**: you can use `run_sql` to include a file that itself includes another file, and so on. However, be careful to avoid infinite loops. SQLPage will throw an error if the inclusion depth is superior to 8.
+ - **security**: be careful when using `run_sql` to include files. Never use `run_sql` with a user-provided parameter. Never run a file uploaded by a user, or a file that is not under your control.
+ - **variables**: the included file will have access to the same variables (URL parameters, POST variables, etc.)
+   as the calling file. It will not have access to uploaded files.
+   If the included file changes the value of a variable or creates a new variable, the change will not be visible in the calling file.
+'
+    );
+INSERT INTO sqlpage_function_parameters (
+        "function",
+        "index",
+        "name",
+        "description_md",
+        "type"
+    )
+VALUES (
+        'run_sql',
+        1,
+        'file',
+        'Path to the SQL file to execute, can be absolute, or relative to the web root (the root folder of your website sql files).
+        In-database files, from the sqlpage_files(path, contents, last_modified) table are supported.',
+        'TEXT'
+    );

src/dynamic_component.rs

@@ -0,0 +1,166 @@
+use anyhow::{self, Context as _};
+use serde_json::Value as JsonValue;
+
+use crate::webserver::database::DbItem;
+
+pub fn parse_dynamic_rows(row: DbItem) -> impl Iterator<Item = DbItem> {
+    DynamicComponentIterator {
+        stack: vec![],
+        db_item: Some(row),
+    }
+}
+
+struct DynamicComponentIterator {
+    stack: Vec<anyhow::Result<JsonValue>>,
+    db_item: Option<DbItem>,
+}
+
+impl Iterator for DynamicComponentIterator {
+    type Item = DbItem;
+
+    fn next(&mut self) -> Option<Self::Item> {
+        if let Some(db_item) = self.db_item.take() {
+            if let DbItem::Row(mut row) = db_item {
+                if let Some(properties) = extract_dynamic_properties(&mut row) {
+                    self.stack = dynamic_properties_to_vec(properties);
+                } else {
+                    // Most common case: just a regular row. We allocated nothing.
+                    return Some(DbItem::Row(row));
+                }
+            } else {
+                return Some(db_item);
+            }
+        }
+        expand_dynamic_stack(&mut self.stack);
+        self.stack.pop().map(|result| match result {
+            Ok(row) => DbItem::Row(row),
+            Err(err) => DbItem::Error(err),
+        })
+    }
+}
+
+fn expand_dynamic_stack(stack: &mut Vec<anyhow::Result<JsonValue>>) {
+    while let Some(Ok(mut next)) = stack.pop() {
+        if let Some(properties) = extract_dynamic_properties(&mut next) {
+            stack.extend(dynamic_properties_to_vec(properties));
+        } else {
+            stack.push(Ok(next));
+            return;
+        }
+    }
+}
+
+/// if row.component == 'dynamic', return Some(row.properties), otherwise return None
+#[inline]
+fn extract_dynamic_properties(data: &mut JsonValue) -> Option<JsonValue> {
+    let component = data.get("component").and_then(|v| v.as_str());
+    if component == Some("dynamic") {
+        let properties = data.get_mut("properties").map(JsonValue::take);
+        Some(properties.unwrap_or_default())
+    } else {
+        None
+    }
+}
+
+/// reverse the order of the vec returned by `dynamic_properties_to_result_vec`,
+/// and wrap each element in a Result
+fn dynamic_properties_to_vec(properties_obj: JsonValue) -> Vec<anyhow::Result<JsonValue>> {
+    dynamic_properties_to_result_vec(properties_obj).map_or_else(
+        |err| vec![Err(err)],
+        |vec| vec.into_iter().rev().map(Ok).collect::<Vec<_>>(),
+    )
+}
+
+/// if properties is a string, parse it as JSON and return a vec with the parsed value
+/// if properties is an array, return it as is
+/// if properties is an object, return it as a single element vec
+/// otherwise, return an error
+fn dynamic_properties_to_result_vec(
+    mut properties_obj: JsonValue,
+) -> anyhow::Result<Vec<JsonValue>> {
+    if let JsonValue::String(s) = properties_obj {
+        properties_obj = serde_json::from_str::<JsonValue>(&s).with_context(|| {
+            format!(
+                "Unable to parse the 'properties' property of the dynamic component as JSON.\n\
+                    Invalid json: {s}"
+            )
+        })?;
+    }
+    match properties_obj {
+        obj @ JsonValue::Object(_) => Ok(vec![obj]),
+        JsonValue::Array(values) => Ok(values),
+        other => anyhow::bail!(
+            "Dynamic component expected properties of type array or object, got {other} instead."
+        ),
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_dynamic_properties_to_result_vec() {
+        let mut properties = JsonValue::String(r#"{"a": 1}"#.to_string());
+        assert_eq!(
+            dynamic_properties_to_result_vec(properties.clone()).unwrap(),
+            vec![JsonValue::Object(
+                serde_json::from_str(r#"{"a": 1}"#).unwrap()
+            )]
+        );
+
+        properties = JsonValue::Array(vec![JsonValue::String(r#"{"a": 1}"#.to_string())]);
+        assert_eq!(
+            dynamic_properties_to_result_vec(properties.clone()).unwrap(),
+            vec![JsonValue::String(r#"{"a": 1}"#.to_string())]
+        );
+
+        properties = JsonValue::Object(serde_json::from_str(r#"{"a": 1}"#).unwrap());
+        assert_eq!(
+            dynamic_properties_to_result_vec(properties.clone()).unwrap(),
+            vec![JsonValue::Object(
+                serde_json::from_str(r#"{"a": 1}"#).unwrap()
+            )]
+        );
+
+        properties = JsonValue::Null;
+        assert!(dynamic_properties_to_result_vec(properties).is_err());
+    }
+
+    #[test]
+    fn test_dynamic_properties_to_vec() {
+        let properties = JsonValue::String(r#"{"a": 1}"#.to_string());
+        assert_eq!(
+            dynamic_properties_to_vec(properties.clone())
+                .first()
+                .unwrap()
+                .as_ref()
+                .unwrap(),
+            &serde_json::json!({"a": 1})
+        );
+    }
+
+    #[test]
+    fn test_parse_dynamic_rows() {
+        let row = DbItem::Row(serde_json::json!({
+            "component": "dynamic",
+            "properties": [
+                {"a": 1},
+                {"component": "dynamic", "properties": {"nested": 2}},
+            ]
+        }));
+        let iter = parse_dynamic_rows(row)
+            .map(|item| match item {
+                DbItem::Row(row) => row,
+                x => panic!("Expected a row, got {x:?}"),
+            })
+            .collect::<Vec<_>>();
+        assert_eq!(
+            iter,
+            vec![
+                serde_json::json!({"a": 1}),
+                serde_json::json!({"nested": 2}),
+            ]
+        );
+    }
+}

src/file_cache.rs

@@ -5,7 +5,7 @@ use anyhow::Context;
 use async_trait::async_trait;
 use chrono::{DateTime, TimeZone, Utc};
 use std::collections::HashMap;
-use std::path::PathBuf;
+use std::path::{Path, PathBuf};
 use std::sync::atomic::{
     AtomicU64,
     Ordering::{Acquire, Release},
@@ -97,7 +97,7 @@ impl<T: AsyncFromStrWithState> FileCache<T> {
 
     /// Gets a file from the cache, or loads it from the file system if it's not there
     /// This is a privileged operation; it should not be used for user-provided paths
-    pub async fn get(&self, app_state: &AppState, path: &PathBuf) -> anyhow::Result<Arc<T>> {
+    pub async fn get(&self, app_state: &AppState, path: &Path) -> anyhow::Result<Arc<T>> {
         self.get_with_privilege(app_state, path, true).await
     }
 
@@ -107,7 +107,7 @@ impl<T: AsyncFromStrWithState> FileCache<T> {
     pub async fn get_with_privilege(
         &self,
         app_state: &AppState,
-        path: &PathBuf,
+        path: &Path,
         privileged: bool,
     ) -> anyhow::Result<Arc<T>> {
         log::trace!("Attempting to get from cache {:?}", path);
@@ -164,7 +164,7 @@ impl<T: AsyncFromStrWithState> FileCache<T> {
             Ok(value) => {
                 let new_val = Arc::clone(&value.content);
                 log::trace!("Writing to cache {:?}", path);
-                self.cache.write().await.insert(path.clone(), value);
+                self.cache.write().await.insert(PathBuf::from(path), value);
                 log::trace!("Done writing to cache {:?}", path);
                 log::trace!("{:?} loaded in cache", path);
                 Ok(new_val)

src/lib.rs

@@ -4,6 +4,7 @@
 extern crate core;
 
 pub mod app_config;
+pub mod dynamic_component;
 pub mod file_cache;
 pub mod filesystem;
 pub mod render;

src/render.rs

@@ -274,15 +274,12 @@ pub struct RenderContext<W: std::io::Write> {
     pub writer: W,
     current_component: Option<SplitTemplateRenderer>,
     shell_renderer: SplitTemplateRenderer,
-    recursion_depth: usize,
     current_statement: usize,
 }
 
 const DEFAULT_COMPONENT: &str = "table";
 const PAGE_SHELL_COMPONENT: &str = "shell";
 const FRAGMENT_SHELL_COMPONENT: &str = "shell-empty";
-const DYNAMIC_COMPONENT: &str = "dynamic";
-const MAX_RECURSION_DEPTH: usize = 256;
 
 impl<W: std::io::Write> RenderContext<W> {
     pub async fn new(
@@ -293,12 +290,7 @@ impl<W: std::io::Write> RenderContext<W> {
     ) -> anyhow::Result<RenderContext<W>> {
         log::debug!("Creating the shell component for the page");
 
-        let mut initial_rows =
-            if get_object_str(&initial_row, "component") == Some(DYNAMIC_COMPONENT) {
-                Self::extract_dynamic_properties(&initial_row)?
-            } else {
-                vec![Cow::Borrowed(&initial_row)]
-            };
+        let mut initial_rows = vec![Cow::Borrowed(&initial_row)];
 
         if !initial_rows
             .first()
@@ -336,7 +328,6 @@ impl<W: std::io::Write> RenderContext<W> {
             writer,
             current_component: None,
             shell_renderer,
-            recursion_depth: 0,
             current_statement: 1,
         };
 
@@ -367,11 +358,6 @@ impl<W: std::io::Write> RenderContext<W> {
         let new_component = get_object_str(data, "component");
         let current_component = self.current_component().await?.name();
         match (current_component, new_component) {
-            (_current_component, Some(DYNAMIC_COMPONENT)) => {
-                self.render_dynamic(data).await.with_context(|| {
-                    format!("Unable to render dynamic component with properties {data}")
-                })?;
-            }
             (
                 _,
                 Some(
@@ -398,41 +384,6 @@ impl<W: std::io::Write> RenderContext<W> {
         Ok(())
     }
 
-    fn extract_dynamic_properties(data: &Value) -> anyhow::Result<Vec<Cow<'_, JsonValue>>> {
-        let properties_key = "properties";
-        let properties_obj = data
-            .get(properties_key)
-            .with_context(|| format!("Missing '{properties_key}' key."))?;
-        Ok(match properties_obj {
-            Value::String(s) => match serde_json::from_str::<JsonValue>(s)
-                .with_context(|| "parsing json properties")?
-            {
-                Value::Array(values) => values.into_iter().map(Cow::Owned).collect(),
-                obj @ Value::Object(_) => vec![Cow::Owned(obj)],
-                other => bail!(
-                    "Expected properties string to parse as array or object, got {other} instead."
-                ),
-            },
-            obj @ Value::Object(_) => vec![Cow::Borrowed(obj)],
-            Value::Array(values) => values.iter().map(Cow::Borrowed).collect(),
-            other => bail!("Expected properties of type array or object, got {other} instead."),
-        })
-    }
-
-    async fn render_dynamic(&mut self, data: &Value) -> anyhow::Result<()> {
-        anyhow::ensure!(
-            self.recursion_depth <= MAX_RECURSION_DEPTH,
-            "Maximum recursion depth exceeded in the dynamic component."
-        );
-        for dynamic_row_obj in Self::extract_dynamic_properties(data)? {
-            self.recursion_depth += 1;
-            let res = self.handle_row(&dynamic_row_obj).await;
-            self.recursion_depth -= 1;
-            res?;
-        }
-        Ok(())
-    }
-
     #[allow(clippy::unused_async)]
     pub async fn finish_query(&mut self) -> anyhow::Result<()> {
         log::debug!("-> Query {} finished", self.current_statement);