Skip to content

AWS lambda deploy fails using a private python repo #563

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
dtosato opened this issue Nov 18, 2020 · 2 comments
Open

AWS lambda deploy fails using a private python repo #563

dtosato opened this issue Nov 18, 2020 · 2 comments

Comments

@dtosato
Copy link

dtosato commented Nov 18, 2020

Hello,
I am trying to deploy a python lambda function which depends on a package into a private repo, but I get the error below. A GitLab runner executes the sls deploy using poetry to install the dependencies and packagr to get the private package.

serverless.yml 
service: premani-rg270-dukane-iqtc-adt

custom:
  region: us-west-2
  stage: stage
  duration_threshold: 20
  pythonRequirements:
    slim: true
  alerts:
    dashboards: false
    topics:
      alarm:
        topic: arn:xxx
        notifications:
          - protocol: email
            endpoint: [email protected]
    definitions:
      functionErrors:
        treatMissingData: notBreaching
      functionDuration:
        treatMissingData: notBreaching
    alarms:
      - functionErrors
      - functionDuration

provider:
  name: aws
  runtime: python3.8
  region: ${self:custom.region}
  stage: ${self:custom.stage}
  memorySize: 512
  timeout: 30
  tracing:
    lambda: true
  iamRoleStatements:
    - Effect: "Allow"
      Action:
        - "ses:SendEmail"
        - "ses:SendRawEmail"
        - "sagemaker:InvokeEndpoint"
        - "sagemaker:DescribeEndpoint"
      Resource: "*"

plugins:
  - serverless-python-requirements
  - serverless-plugin-aws-alerts

functions:
  premani-rg270-dukane-iqtc-adt:
    handler: lambda_function.lambda_handler
    name: premani-rg270-dukane-iqtc-adt-${self:custom.stage}
    description: Trigger anomaly detection for RG270 ultrasound generator Dukane IQT C-series a dis.400-2256 June 2013 version in ${self:custom.stage} env.
    alarms:
      - name: functionErrors
      - name: functionDuration
        type: anomalyDetection
        comparisonOperator: GreaterThanUpperThreshold
        threshold: ${self:custom.duration_threshold}
    events:
      - schedule:
          rate: cron(5 * * * ? *)
sls deploy output 
Serverless: Generating requirements.txt from pyproject.toml...
Serverless: Parsed requirements.txt from pyproject.toml in /builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/.serverless/requirements.txt...
Serverless: Installing requirements from /root/.cache/serverless-python-requirements/bcd1f00a0181384da43ae6dd50e0d4ed234ff7c1e88ac99f66060683a2e854e4_slspyc/requirements.txt ...
Serverless: Using download cache directory /root/.cache/serverless-python-requirements/downloadCacheslspyc
Serverless: Running ...
 
  Error --------------------------------------------------
 
  Error: STDOUT: Looking in indexes: https://pypi.org/simple, https://api.packagr.app/Wk3Sg2IP1
  Collecting boto3==1.16.20
    Downloading boto3-1.16.20-py2.py3-none-any.whl (129 kB)
  Collecting botocore==1.19.20
    Downloading botocore-1.19.20-py2.py3-none-any.whl (6.8 MB)
  Collecting certifi==2020.11.8
    Downloading certifi-2020.11.8-py2.py3-none-any.whl (155 kB)
  Collecting chardet==3.0.4
    Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
  Collecting cycler==0.10.0
    Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
  User for api.packagr.app: 
  
  STDERR: ERROR: Exception:
  Traceback (most recent call last):
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 228, in _main
      status = self.run(options, args)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 182, in wrapper
      return func(self, options, args)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 323, in run
      requirement_set = resolver.resolve(
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 183, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 388, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 339, in _get_abstract_dist_for
      self._populate_link(req)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 305, in _populate_link
      req.link = self._find_requirement_link(req)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/resolution/legacy/resolver.py", line 270, in _find_requirement_link
      best_candidate = self.finder.find_requirement(req, upgrade)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 900, in find_requirement
      best_candidate_result = self.find_best_candidate(
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 883, in find_best_candidate
      candidates = self.find_all_candidates(project_name)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 827, in find_all_candidates
      package_links = self.process_project_url(
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/package_finder.py", line 791, in process_project_url
      html_page = self._link_collector.fetch_page(project_url)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 629, in fetch_page
      return _get_html_page(location, session=self.session)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 441, in _get_html_page
      resp = _get_html_response(url, session=session)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/index/collector.py", line 138, in _get_html_response
      resp = session.get(
    File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 543, in get
      return self.request('GET', url, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/session.py", line 421, in request
      return super(PipSession, self).request(method, url, *args, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 530, in request
      resp = self.send(prep, **send_kwargs)
    File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/sessions.py", line 650, in send
      r = dispatch_hook('response', hooks, r, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/pip/_vendor/requests/hooks.py", line 31, in dispatch_hook
      _hook_data = hook(hook_data, **kwargs)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/auth.py", line 256, in handle_401
      username, password, save = self._prompt_for_password(parsed.netloc)
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/network/auth.py", line 226, in _prompt_for_password
      username = ask_input("User for {}: ".format(netloc))
    File "/usr/local/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 259, in ask_input
      return input(message)
  EOFError: EOF when reading a line
  
      at /builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/serverless-python-requirements/lib/pip.js:325:13
      at Array.forEach (<anonymous>)
      at installRequirements (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/serverless-python-requirements/lib/pip.js:312:28)
      at installRequirementsIfNeeded (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/serverless-python-requirements/lib/pip.js:556:3)
      at ServerlessPythonRequirements.installAllRequirements (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/serverless-python-requirements/lib/pip.js:635:29)
      at ServerlessPythonRequirements.tryCatcher (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/bluebird/js/release/util.js:16:23)
      at Promise._settlePromiseFromHandler (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/bluebird/js/release/promise.js:547:31)
      at Promise._settlePromise (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/bluebird/js/release/promise.js:604:18)
      at Promise._settlePromise0 (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/bluebird/js/release/promise.js:649:10)
      at Promise._settlePromises (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/bluebird/js/release/promise.js:729:18)
      at _drainQueueStep (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/bluebird/js/release/async.js:93:12)
      at _drainQueue (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/bluebird/js/release/async.js:86:9)
      at Async._drainQueues (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/bluebird/js/release/async.js:102:5)
      at Immediate.Async.drainQueues [as _onImmediate] (/builds/galdi/premani/premani-rg270-dukane-iqtc-anomaly-detection-trigger/node_modules/bluebird/js/release/async.js:15:14)
      at processImmediate (internal/timers.js:461:21)

Installed version

     Operating System:          linux
     Node Version:              12.19.1
     Framework Version:         2.11.1
     Plugin Version:            4.1.2
     SDK Version:               2.3.2
     Components Version:        3.3.1
@dtosato dtosato mentioned this issue Nov 19, 2020
Open
@brycehemme
Copy link

This is likely caused by an issue with the poetry export command in the current released version not using the --with-credentials parameter when it generates your requirements.txt file from the pyproject.toml file. This is fixed in master but since there hasn't been a release since February you likely aren't running with this update. You have a few options to resolve your issue.
Options:

  1. change to reference GitHub when you install serverless-python-requirements in your package.json
  2. generate the requirements.txt file yourself before running serverless and change your serverless config so usePoetry = false. The command to generate the requirements file is poetry export -f requirements.txt -o requirements.txt --without-hashes --with-credentials.

@LironEr LironEr mentioned this issue Mar 8, 2021
Closed
@iainvm
Copy link

iainvm commented Jun 22, 2021

v5.1.1 seems to now have the --with-credentials so this issue might be resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@iainvm @brycehemme @dtosato