Skip to content

Commit 5125314

Browse files
rgommersrgommers
authored
Merge pull request #218 from shaloo/sfork-217-depgraph
Enhanced scripts to generate numpy dependency graph for various pkg
2 parents 4a5da4b + 140823a commit 5125314

File tree

10 files changed

+507
-168
lines changed

10 files changed

+507
-168
lines changed

‎content/en/case-studies/gw-discov.md

+5
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,11 @@ speed. Here are some examples:
119119
providing object based interfaces to utilities, tools and methods for
120120
studying data from gravitational-wave detectors.
121121

122+
{{< figure src="/images/content_images/cs/gwpy-numpy-dep-graph.png" class="fig-center" alt="gwpy-numpy depgraph" caption="**Dependency graph showing how GwPy package depends on NumPy**" >}}
123+
124+
----
125+
126+
{{< figure src="/images/content_images/cs/PyCBC-numpy-dep-graph.png" class="fig-center" alt="PyCBC-numpy depgraph" caption="**Dependency graph showing how PyCBC package depends on NumPy**" >}}
122127

123128
## Summary
124129

‎scripts/fetch-ehtim.sh

-78
This file was deleted.

‎scripts/numpy-ehtim-dep-graph.sh

-90
This file was deleted.

‎scripts/pkgdepchart/README.md

+138
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# NumPy Dependency Chart generator Scripts
2+
3+
## Overview
4+
5+
Typically, you only need to run the top level script **redraw-numpy-dep-charts.sh.**
6+
7+
Here is how these scripts are organized:
8+
9+
* **Top level script:** redraw-numpy-dep-charts.sh
10+
* **Helper scripts:** These scripts are used by **redraw-numpy-dep-charts.sh**
11+
and should be run in the suggested order below:
12+
13+
1. fetch-numpydeppkg.sh
14+
2. gen-numpy-dep-graph.sh
15+
3. cleanup-numpydeppkg.sh
16+
17+
If you run the helper scripts, make sure you run them in this order to generate graphs in specified output directory.
18+
19+
Usage details of each script are listed in the section below.
20+
21+
----
22+
23+
## Script: redraw-numpy-dep-charts.sh
24+
25+
### Use Case:
26+
27+
NumPy case studies refers to ehtim, gwpy and PyCBC packages. For each of these,
28+
we have created NumPy dependency graphs. You can redraw all these three graphs
29+
in one go (obtain 3 png files) in the specified output directory. If you wish
30+
to add a new package and also refresh existing charts, you can edit the script
31+
and update three arrays:
32+
33+
* *pkgarray:* List of packages for which numpy dependency chart is to be generated
34+
* *pkgurl:* List of github urls from where pkg sources can be git cloned
35+
* *pkgsetup:* Dirname (not path) of the top level dir in pkg source that contains
36+
requirements.txt file used to pip install the package in a virtualenv.
37+
The package is uninstalled as part of cleanup after graphs is generated.
38+
39+
#### Usage
40+
41+
redraw-numpy-dep-charts.sh **outputdir**
42+
43+
*outputdir:* The output directory is an optional parameter.
44+
45+
If no output directory is specified, the graphs (.png files) of all the packages
46+
listed in pkgarray variable are generated in the current directory from where
47+
the script is run.
48+
49+
**Pre-requisites:**
50+
51+
The following software packages must be installed on the system where you are running this bash script: git, python3, pip3, virtualenv.
52+
53+
----
54+
55+
## Script: fetch-numpydeppkg.sh
56+
57+
### Use Case:
58+
59+
This script is one of the helper scripts used by **redraw-numpy-dep-charts.sh**
60+
script. Its sole purpose is to download the package for which numpy dependency
61+
graph is to be generated and install it in a virtual environment, setup the
62+
directory structure which can be used by other helper script
63+
**gen-numpy-dep-graph.sh** to generate the graph. This script is paired with
64+
**cleanup-numpydeppkg.sh** script for uninstalling and removing scratch data.
65+
66+
#### Usage
67+
68+
fetch-numpydeppkg.sh **package_name scratchdir git_url reqfile_parentdir**
69+
70+
*package_name:* Name of the package (as in pip registry) that will be fetched.
71+
72+
*scratchdir:* local directory where the package contents will be fetched.
73+
74+
*git_url:* Full git url required for git cloning.
75+
76+
*reqfile_parentdir:* This is the name (not path) of the root directory of package that contains requirements.txt file
77+
78+
----
79+
80+
## Script: gen-numpy-dep-graph.sh
81+
82+
### Use Case:
83+
84+
This is a helper script used by **redraw-numpy-dep-charts.sh** script. Its sole
85+
purpose is to use the pre-installed pkg in a local virtual environment by
86+
the script **fetch-numpydeppkg.sh** (or manually installed locally) and use them
87+
to run graphviz, dot and other utilities for creating NumPy dependency chart
88+
as png files for the pkg. It assumes that packages is pre-installed and does
89+
not clean up any pre-installed packages. In case a user has the package
90+
pre-installed and would like to simply generate NumPy dependency chart, this script can be used.
91+
92+
#### Usage:
93+
94+
gen-numpy-dep-graph.sh **package_name scratchdir output_graphdir highlight_color (optional)**
95+
96+
*package_name:* Name of the package (as in pip registry) for which NumPy dependency graph is to be generated.
97+
98+
*scratchdir:* local directory where the package contents reside, the same directory as input to the fetch-numpydeppkg.sh where package_name was fetched by that script.
99+
100+
*output_graphdir:* directory pathname where generated graphs in png format will be stored.
101+
102+
highlight_color:* color understood by dot language. Default is cyan. Other
103+
acceptable colors are red, blue. For a complete list see:
104+
105+
* https://www.graphviz.org/doc/info/lang.html
106+
* https://www.graphviz.org/doc/info/colors.html
107+
108+
**Pre-requisite:**
109+
110+
Please note, **gen-numpy-dep-graph** script needs the virtualenv directory
111+
that was activated and used for package deployment a priori say by
112+
**fetch-numpydeppkg.sh** script or manually by user. It is assumed that the
113+
scratchdir comprises of two top level directories - virtualenv dir and another
114+
dir with package_name sources that are git cloned or a local copy of the same.
115+
116+
----
117+
118+
##Script: cleanup-numpydeppkg.sh
119+
120+
### Use Case:
121+
122+
This is a helper script used by **redraw-numpy-dep-charts.sh** script. Its sole
123+
purpose is to cleanup all the packages that were installed by
124+
**fetch-numpydeppkg.sh** script executed prior to running cleanup script. It
125+
also removes the scratch directory contents related to virtualenv setup and
126+
package sources that were installed by **fetch-numpydeppkg.sh** script. If you
127+
have not used **fetch-numpydeppkg** script then you do not need to run the cleanup script.
128+
129+
#### Usage:
130+
131+
*package_name:* Name of the package (as in pip registry) that will be uninstalled.
132+
133+
*scratchdir:* local directory where the package contents were earlier fetched by
134+
**fetch-numpydeppkg.sh** script.
135+
136+
*reqfile_parentdir:* This is the name (not path) of the root directory of
137+
package that contains requirements.txt file. This is needed to uninstall all
138+
package dependencies that were installed as part of fetching and setting up of package by **fetch-numpydeppkg.sh** script.

‎scripts/pkgdepchart/cleanup-numpydeppkg.sh

+67
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
#!/bin/bash
2+
# Script to cleanup and uninstall
3+
# python package fetched using fetch-numpydeppkg.sh script
4+
#
5+
6+
set -ue
7+
8+
cleanup_pkg() {
9+
10+
pkgname=$1
11+
workingdir=$2
12+
setupdir=$3
13+
14+
if [ -z "$pkgname" ] || [ -z "$workingdir" ] || [ -z "$setupdir" ]; then
15+
display_help
16+
exit
17+
fi
18+
19+
ygraphviz=`$workingdir/env-${pkgname}/bin/pip3 list | grep graphviz`
20+
if ! [ -z "$ygraphviz" ];
21+
then
22+
$workingdir/env-${pkgname}/bin/pip3 uninstall -y graphviz
23+
fi
24+
25+
ypipdeptree=`$workingdir/env-${pkgname}/bin/pip3 list | grep pipdeptree`
26+
if ! [ -z "$ypipdeptree" ];
27+
then
28+
$workingdir/env-${pkgname}/bin/pip3 uninstall -y pipdeptree
29+
fi
30+
31+
$workingdir/env-${pkgname}/bin/pip3 uninstall -y -r $workingdir/numpy_${pkgname}_dep/$setupdir/requirements.txt
32+
33+
\rm -rf $workingdir/env-${pkgname}
34+
\rm -rf $workingdir/numpy_${pkgname}_dep
35+
36+
return 0
37+
}
38+
39+
display_help() {
40+
echo "-----------------------------------------------------------------------"
41+
echo "Usage:"
42+
echo ""
43+
echo "./cleanup-numpydeppkg.sh pkgname workingdir setupdirname"
44+
echo ""
45+
echo "where:"
46+
echo ""
47+
echo "pkgname - the name of package in pypi directory"
48+
echo "workingdir - the directory where latest sources are located"
49+
echo "setupdirname - the name of top level source directory that contains
50+
setup.py for the package."
51+
echo ""
52+
echo "For example, to download ehtim use the following command:"
53+
echo ""
54+
echo "./cleanup-numpydeppkg.sh <pkgname> <working-dir> <setupdirname>"
55+
echo ""
56+
echo "-----------------------------------------------------------------------"
57+
}
58+
59+
if [[ ( $# -gt 3) || ( -z "$1" )|| ( -z "$2" ) || ( -z "$3" ) || ("$1" == "-h" ) || ("$1" == "--help") ]];
60+
then
61+
display_help
62+
exit
63+
fi
64+
65+
cleanup_pkg $1 $2 $3
66+
67+
echo Done!

0 commit comments

Comments
 (0)