Skip to content

Cython cache diff compare #13526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

Cython cache diff compare #13526

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
6cc3d6e
Diff python pyx files as a test if they have changed
nparley Jun 28, 2016
aaf4dd2
Merge branch 'upstream' into pyx-diff
nparley Jun 28, 2016
30bd9da
Check for the case where less cython file has been deleted
nparley Jun 28, 2016
dca6e61
Test commit
nparley Jun 28, 2016
79fe0f0
Remove do_not_merge
nparley Jun 28, 2016
81cbb7b
Reset pyx changes
nparley Jul 1, 2016
2dc2ce2
Merge branch 'upstream' into pyx-diff
nparley Jul 1, 2016
8b4ad0b
Remove test commit
nparley Jul 1, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 48 additions & 18 deletions ci/prep_cython_cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,43 +1,73 @@
#!/bin/bash

ls "$HOME/.cache/"

PYX_CACHE_DIR="$HOME/.cache/pyxfiles"
pyx_file_list=`find ${TRAVIS_BUILD_DIR} -name "*.pyx"`
pyx_cache_file_list=`find ${PYX_CACHE_DIR} -name "*.pyx"`

CACHE_File="$HOME/.cache/cython_files.tar"

# Clear the cython cache 0 = NO, 1 = YES
clear_cache=0

pyx_files=`echo "$pyx_file_list" | wc -l`
pyx_cache_files=`echo "$pyx_cache_file_list" | wc -l`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will not work. you would need to compare .pyx file sizes I think to make this reliable. Just adding / removing a file is not good enough at all. Most often we have a change in actual contents.

Copy link
Contributor Author

@nparley nparley Jun 29, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The actually checking is done below. diff -u $i $PYX_CACHE_DIR${i} for each pyx file . This code is only here to check no pyx files have been deleted in full.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh i c.


if [[ pyx_files -ne pyx_cache_files ]]
then
echo "Different number of pyx files"
clear_cache=1
fi

home_dir=$(pwd)

if [ -f "$CACHE_File" ] && [ "$USE_CACHE" ]; then
if [ -f "$CACHE_File" ] && [ "$USE_CACHE" ] && [ -d "$PYX_CACHE_DIR" ]; then

echo "Cache available - checking pyx diff"

for i in ${pyx_file_list}
do
diff=`diff -u $i $PYX_CACHE_DIR${i}`
if [[ $? -eq 2 ]]
then
echo "${i##*/} can't be diffed; probably not in cache"
clear_cache=1
fi
if [[ ! -z $diff ]]
then
echo "${i##*/} has changed:"
echo $diff
clear_cache=1
fi
done

echo "Cache available"
clear_cache=1
# did the last commit change cython files?
# go back 2 commits
if [ "$TRAVIS_PULL_REQUEST" == "false" ]
then
echo "Not a PR: checking for cython files changes from last 2 commits"
git diff HEAD~2 --numstat | grep -E "pyx|pxd"
retval=$(git diff HEAD~2 --numstat | grep -E "pyx|pxd"| wc -l)
echo "Not a PR"
# Uncomment next 2 lines to turn off cython caching not in a PR
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you meaning to leave in these comments?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about it; as a quick way of turning off caching of cython files in the future if something goes wrong. But happy to take them out also.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok that's fine then.

# echo "Non PR cython caching is disabled"
# clear_cache=1
else
echo "PR: checking for any cython file changes from last 5 commits"
git diff PR_HEAD~5 --numstat | grep -E "pyx|pxd"
retval=$(git diff PR_HEAD~5 --numstat | grep -E "pyx|pxd"| wc -l)
echo "Forcing cython rebuild due to possibility of history rewritting in PR"
retval=-1
echo "In a PR"
# Uncomment next 2 lines to turn off cython caching in a PR
# echo "PR cython caching is disabled"
# clear_cache=1
fi
echo "number of cython files changed: $retval"

fi

if [ $clear_cache -eq 1 ] && [ $retval -eq 0 ] && [ "$USE_CACHE" ]
if [ $clear_cache -eq 0 ] && [ "$USE_CACHE" ]
then
# nope, reuse cython files
# No and use_cache is set
echo "Will reuse cached cython file"
cd /
tar xvmf $CACHE_File
cd $home_dir
else
echo "Rebuilding cythonized files"
echo "Use cache = $USE_CACHE"
echo "Clear cache = $clear_cache"
echo "Use cache (Blank if not set) = $USE_CACHE"
echo "Clear cache (1=YES) = $clear_cache"
fi


Expand Down
12 changes: 9 additions & 3 deletions ci/submit_cython_cache.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,23 @@
#!/bin/bash

CACHE_File="$HOME/.cache/cython_files.tar"
PYX_CACHE_DIR="$HOME/.cache/pyxfiles"
pyx_file_list=`find ${TRAVIS_BUILD_DIR} -name "*.pyx"`

rm -rf $CACHE_File
rm -rf $PYX_CACHE_DIR

home_dir=$(pwd)

pyx_files=`find ${TRAVIS_BUILD_DIR} -name "*.pyx"`
mkdir $PYX_CACHE_DIR
rsync -Rv $pyx_file_list $PYX_CACHE_DIR

echo "pyx files:"
echo $pyx_files
echo $pyx_file_list

tar cf ${CACHE_File} --files-from /dev/null

for i in ${pyx_files}
for i in ${pyx_file_list}
do
f=${i%.pyx}
ls $f.{c,cpp} | tar rf ${CACHE_File} -T -
Expand Down