Threading.Lock not effective across processes

[c-simpson]

TL;DR: file_uploader uses Threading.Lock - is issue specifically thread safety or simultaneous execution?

In investigating #348, I looked at determine_upload_type() which uses a Threading.Lock to protect a critical section. I added the logging statements seen here:

                    current_app.logger.info("Acquiring lock for " + file.filename + " >>>>>>>>>>>>>>>>>>>>>>>>>>>")
                with lock:
                    current_app.logger.info("Got lock  for " + file.filename + " <<<<<<<<<<<<<<<<<<<<<<<<<<<<<")
                    found_sources += 1
                    filename = secure_filename(file.filename)
                    now = time.gmtime()
                    now_date = time.strftime("%Y-%m-%d--%H-%M-%S", now)
                    current_app.logger.info("  -File: " + filename + " Matches files type: " + src_type)
                    df.to_csv(os.path.join(destination_path, src_type + '-' + now_date + '.csv'))
                    clean_current_folder(destination_path + '/current', src_type)
                    df.to_csv(os.path.join(destination_path + '/current', src_type + '-' + now_date + '.csv'))
                    current_app.logger.info("  -Uploaded successfully as : " + src_type + '-' + now_date + '.' + file_extension)
                    flash(src_type + " {0} ".format(SUCCESS_MSG), 'info')
                    current_app.logger.info("Releasing lock for " + file.filename + "  >>>>>>>>>>>>>>>>>>>>>>>>>>>")
    if found_sources == 0:

as the lock does not seem to prevent parallel execution from two browser tabs (Date and HH:MM removed, leaving seconds):

:12,955] INFO in file_uploader: Acquiring lock for Salesforce_PAWS Donation (all Time) (excel).xlsx >>>>>>>>>>>>>>>>>>>>>>>>>>>
:12,955] INFO in file_uploader: Got lock  for Salesforce_PAWS Donation (all Time) (excel).xlsx <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
:12,956] INFO in file_uploader:   -File: Salesforce_PAWS_Donation_all_Time_excel.xlsx Matches files type: salesforcedonations

:14,119] INFO in file_uploader: Start uploading file: shelterluv_people.csv
:14,501] INFO in file_uploader: Acquiring lock for shelterluv_people.csv >>>>>>>>>>>>>>>>>>>>>>>>>>>
:14,501] INFO in file_uploader: Got lock  for shelterluv_people.csv <<<<<<<<<<<<<<<<<<<<<<<<<<<<<
:15,384] INFO in file_uploader:   -Uploaded successfully as : shelterluvpeople-2021-06-03--23-49-14.csv
:15,384] INFO in file_uploader: Releasing lock for shelterluv_people.csv  >>>>>>>>>>>>>>>>>>>>>>>>>>>

:20,109] INFO in file_uploader:   -Uploaded successfully as : salesforcedonations-2021-06-03--23-49-12.xlsx
:20,110] INFO in file_uploader: Releasing lock for Salesforce_PAWS Donation (all Time) (excel).xlsx  >>>>>>>>>>>>>>>>>>>>>>>>>>>

I think this is because uWSGI is handling requests on a per-process basis:

paws-compose-server | *** uWSGI is running in multiple interpreter mode ***
paws-compose-server | spawned uWSGI master process (pid: 13)
paws-compose-server | spawned uWSGI worker 1 (pid: 22, cores: 4)
paws-compose-server | spawned uWSGI worker 2 (pid: 26, cores: 4)

Does it matter? If everything works properly, it may not matter. Is there a danger if both users are uploading the same file?

If we need to synchronize across processes, we could use the db.

[carlos-dominguez]

this would be fixed by #480, wherein transaction-level locking renders this moot