Merge pull request #11 from rkkautsar/refactor

rkkautsar · web-flow · commit bacd8ee658f1 · 2019-02-21T16:10:11.000+01:00
Add basic documentation with mkdocs
diff --git a/Pipfile b/Pipfile
@@ -4,6 +4,7 @@ url = "https://pypi.org/simple"
 verify_ssl = true
 
 [dev-packages]
+mkdocs = "*"
 
 [packages]
 lxml = "*"
diff --git a/Pipfile.lock b/Pipfile.lock
diff --git a/benchmarks/examples/clasp/runscript.yml b/benchmarks/examples/clasp/runscript.yml
@@ -18,7 +18,7 @@ configs:
 systems:
   - name: clasp
     version: 1.3.2
-    measures: .resultparsers.clasp
+    measures: resultparsers.clasp
     config: seq-generic
     settings:
       - name: default
@@ -44,7 +44,7 @@ systems:
         cmdline: '--stats --restarts=no 1'
   - name: claspar
     version: 2.1.0
-    measures: .resultparsers.claspar
+    measures: resultparsers.claspar
     config: pbs-generic
     settings:
       - name: one-as
diff --git a/docs/_old/README b/docs/_old/README
diff --git a/docs/export-import/index.md b/docs/export-import/index.md
@@ -0,0 +1,11 @@
+# Exporting a benchmark
+
+```sh
+./bexport my.benchmark.tgz benchmarks/my-benchmark
+```
+
+# Importing a benchmark
+
+```
+./bexport imported.benchmark.tgz benchmarks/imported_benchmark
+```
diff --git a/docs/how-to/evaluating-results.md b/docs/how-to/evaluating-results.md
@@ -0,0 +1,8 @@
+# Evaluating the results
+
+```sh
+# pipenv shell
+./beval benchmarks/.../runscript.yml > evaluation.xml
+```
+
+This script will collect statistics from the runs like its time and memory usage, errors etc according to the [result parser](../result-parser.md) defined in the system measures.
diff --git a/docs/how-to/generating-scripts.md b/docs/how-to/generating-scripts.md
@@ -0,0 +1,20 @@
+# Generating Scripts
+
+```sh
+# pipenv shell
+./bgen benchmarks/.../runscript.yml
+```
+
+After running the command, shortly the scripts will be generated in `$(base_dir)/$(output_dir)/$(project)/$(machine)` according to the runscript.
+
+The structure of the scripts generated will depend on the job, but generally the structure will be like this:
+
+```
+[base_dir]/[output_dir]/[project]/[machine]/results/
+└── start.py
+└── $(benchmark)
+    ├── $(system)-$(system)version]-$(system)setting]-n$(system)setting.proc]
+    │   ├── $(instance_file_name)
+    │   │   └── run$(run_number)
+    │   │       └── start.sh
+```
diff --git a/docs/how-to/running-benchmark.md b/docs/how-to/running-benchmark.md
@@ -0,0 +1,7 @@
+# Running the benchmark
+
+Depending on the job, there should be a single entry point for running the benchmark in `$(base_dir)/$(output_dir)`. For example, there will be a `start.py` for a [Sequential Job](#). Just run this script to run the whole benchmark.
+
+```sh
+./benchmarks/.../output/.../start.py
+```
diff --git a/docs/how-to/summarize-evaluation.md b/docs/how-to/summarize-evaluation.md
@@ -0,0 +1,5 @@
+# Summarizing evaluations
+
+```
+./bconv -c < evaluation.xml > result.csv
+```
diff --git a/docs/how-to/writing-runscript.md b/docs/how-to/writing-runscript.md
@@ -0,0 +1,43 @@
+# Writing a runscript
+
+A runscript is a yaml script that defines the benchmark. The runscript will be validated with [pykwalify](https://pykwalify.readthedocs.io/en/master/). The schema can be seen in `src/benchmarktool/runscript/schema.yml`. An example runscript can be seen in `benchmarks/examples/*/runscript.yml`. Here's an overview of the keys:
+
+## base_dir
+
+The base directory of the benchmark. Example: `benchmarks/examples/clasp`.
+
+## output_dir
+
+The directory to where the output (scripts and results) are generated. The path is relative to [base_dir](#base_dir). Example: `output`.
+
+## machines
+
+Description such as CPU and memory capabilities of the machine used for the benchmarks. Currently not used.
+
+## configs
+
+Description of the configuration to run the systems, i.e. template for the run script (might be shell scripts, etc).
+
+## systems
+
+Description of the system (tools/solvers) on which the benchmark instances will be run against.
+
+### system.measures
+
+Module path to the python function of the [result parser](../result-parser.md) that will be used to measure the result of this system, relative to [base_dir](#base_dir). For example, a value of `resultparser.sudokuresultparser` will use the function `sudokuresultparser` defined in `[base_dir]/resultparser.py`.
+
+### system.settings
+
+Description of the system's various settings, such as the `cmdline` options, `tag`, etc.
+
+## jobs
+
+Description of various [jobs](../jobs/index.md), including its type and resource limits.
+
+## benchmarks
+
+Description of various benchmark instances through specifications. Currently the specification type can be either `folder` or `files`.
+
+## projects
+
+Description of project, which can consists of many becnhmark runs by tags (runtags), or by manual specifications (runspecs).
diff --git a/docs/index.md b/docs/index.md
@@ -0,0 +1,49 @@
+# Welcome to benchmark-tool
+
+## Installation
+
+Clone the repository:
+
+```bash
+git clone git@github.com:daajoe/benchmark-tool.git
+```
+
+Installing the dependencies can be done with [Pipenv](https://github.com/pypa/pipenv) (preferred)
+or pip + virtualenv.
+
+### Pipenv
+
+```bash
+pipenv install
+```
+
+### pip + virtualenv
+
+```bash
+python3 -m pip install -r requirements.txt
+```
+
+## How to use
+
+1. [Writing a runscript](how-to/writing-runscript.md)
+2. [Generate scripts with `./bgen`](how-to/generating-scripts.md)
+3. [Running the benchmarks](how-to/running-benchmark.md)
+4. [Evaluating the results with `./beval`](how-to/evaluating-results.md)
+
+## Project layout
+
+```
+.
+├── benchmarks/         # benchmark directory
+│   └── examples/       # example benchmarks
+├── docs/               # this documentation
+├── external-tools/     # various third-party tools
+├── src/                # main source code
+│   ├── benchmarktool/  # main module
+├── utils/              # general utilities
+├── bgen*               # script generation
+├── beval*              # evaluate benchmark results
+├── bconv*              # summarize benchmark evaluation
+├── bexport*            # export benchmarks
+├── bimport*            # import benchmarks
+```
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -0,0 +1,16 @@
+site_name: benchmark-tool
+theme:
+  name: readthedocs
+  highlightjs: true
+  hljs_languages:
+    - yaml
+    - python
+repo_url: https://github.com/daajoe/benchmark-tool/
+nav:
+  - Home: 'index.md'
+  - How to run your benchmark:
+      - '1 - Writing a runscript': 'how-to/writing-runscript.md'
+      - '2 - Generating scripts': 'how-to/generating-scripts.md'
+      - '3 - Running the benchmark': 'how-to/running-benchmark.md'
+      - '4 - Evaluating the results': 'how-to/evaluating-results.md'
+  - Export / Import: 'export-import/index.md'
diff --git a/requirements.txt b/requirements.txt
@@ -0,0 +1,5 @@
+lxml=="4.3.1"
+Jinja2=="2.10"
+PyYAML=="3.13"
+pykwalify=="1.7.0"
+dotmap=="1.3.4"

-Original file line number
+Diff line change
@@ @@ -0,0 +1,5 @@ @@
 +# Summarizing evaluations
++
 +```
 +./bconv -c < evaluation.xml > result.csv
 +```