by usingin #1546

saisubramani · 2020-06-04T09:35:20Z

Describe the bug
i am trying to deploy a pretrained keras model in amazon sage maker,

To reproduce

I converted the keras model into tensorflow model format as sagemkaer supported one.
[Link: https://aws.amazon.com/blogs/machine-learning/deploy-trained-keras-or-tensorflow-models-using-amazon-sagemaker/]
I created a code folder and added inference.py and requirements.txt file
The inference script as mentioned in this link, i created with some changes for my model.
Link : https://github.com/aws/sagemaker-tensorflow-serving-container#prepost-processing

`# inference.py

import tensorflow as tf
import numpy as np
import requests
import base64
from PIL import Image
import io
from collections import namedtuple
import cv2
import json
import os
 
JPEG_CONTENT_TYPE = 'application/x-image'
JSON_CONTENT_TYPE = 'application/json'

Context = namedtuple

def input_handler(data, context):
    #if context.request_content_type == 'application/x-image':
    if context== 'application/x-image':
        #payload = data.read()
        #encoded_image = base64.b64encode(payload).decode('utf-8')
        image = Image.open(io.BytesIO(data))
        result_image  =cv2.cvtColor(np.array(image), cv2.COLOR_BGR2RGB)
        img_cpy=cv2.resize(result_image, (300,300))
        img_cpy=img_cpy/255.
        img_cpy=np.expand_dims(img_cpy,0)
        print(type(img_cpy))
        encoded_image = base64.b64encode(img_cpy).decode('utf-8')
        print(type(encoded_image))
        instance = [{"b64": encoded_image}]
        return json.dumps({"instances": instance})
    else:
        _return_error(415, 'Unsupported content type "{}"'.format(
            context.request_content_type or 'Unknown'))


def output_handler(data, context):

	response = json.loads(data.content.decode('utf-8'))
    pred = response.get('predictions')
    prediction=np.argmax(pred)
    prediction = str(prediction)
    return json.dumps({'classification_id':prediction})


def _return_error(code, message):
    raise ValueError('Error: {}, {}'.format(str(code), message))

i zipped (tared) the model folder into tar.gz format and uploaded in s3 bucket.
Then i open a sagemaker notebooke instance to deply, i uploaded the code folder in the notebook instance also then i started to deploy it.
It show error :

2020-06-03T15:21:54.839+05:30 | /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)])

2020-06-03T15:21:54.840+05:30 | Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker worker.init_process() File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 203, in init_process super(GeventWorker, self).init_process() File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 129, in init_process self.load_wsgi() File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi self.wsgi = self.app.wsgi() File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 67, in wsgi self.callable = self.load() File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load return self.load_wsgiapp() File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp return util.import_app(self.app_uri) File "/usr/local/lib/python3.5/dist-packages/gunicorn/util.py", line 350, in import_app import(module) File "/sagemaker/python_service.py", line 239, in resources.add_routes(app) File "/sagemaker/python_service.py", line 227, in add_routes invocation_resource = InvocationResource() File "/sagemaker/python_service.py", line 48, in init self._handler, self._input_handler, self._output_handler = self._import_handlers() File "/sagemaker/python_service.py", line 68, in _import_handlers spec.loader.exec_module(inference) File "/opt/ml/model/code/inference.py", line 8, in import cv2 File "/usr/local/lib/python3.5/dist-packages/cv2/init.py", line 5, in from .cv2 import *

*Expected behavior

i need to check the inference, when i pass the image by using invoke endpoint api. it should return the json response.

System information
A description of your system. Please provide:

SageMaker Python SDK version:
Framework name (eg. PyTorch) or algorithm (eg. KMeans): tensorflow backed keras classification
Framework version:1.14 or 1.12
Python version:3
CPU or GPU:GPU
Custom Docker image (Y/N):N

The text was updated successfully, but these errors were encountered:

nadiaya · 2020-06-09T00:55:58Z

OpenCV is not installed by default in the container 1.14 gpu container.

Are you install it using requirements.txt? Can you share that file?

Also it looks like the actual error message is cut off. Can you share the full stack with the error message?

saisubramani · 2020-06-09T05:23:48Z

OpenCV is not installed by default in the container 1.14 gpu container.

Are you install it using requirements.txt? Can you share that file?

Also it looks like the actual error message is cut off. Can you share the full stack with the error message?

hi,Thanks for the reply @nadiaya
I had installed opencv-python in requirements.txt file.
Below mentioned is the packages which we installed using the requirements.txt file.

pillow
numpy
tensorflow-gpu
Keras
opencv-python
opencv-contrib-python

Full Cloud watch error:

2020-06-03T15:21:54.839+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
	
2020-06-03T15:21:54.839+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
	
2020-06-03T15:21:54.839+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
	
2020-06-03T15:21:54.839+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
	
2020-06-03T15:21:54.839+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
	
2020-06-03T15:21:54.839+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
	
2020-06-03T15:21:54.839+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
	
2020-06-03T15:21:54.840+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
	
2020-06-03T15:21:54.840+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
	
2020-06-03T15:21:54.840+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
	
2020-06-03T15:21:54.840+05:30
	
[2020-06-03 09:51:53 +0000] [1917] [ERROR] Exception in worker process
	
2020-06-03T15:21:54.840+05:30
	
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 203, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/sagemaker/python_service.py", line 239, in <module>
    resources.add_routes(app)
  File "/sagemaker/python_service.py", line 227, in add_routes
    invocation_resource = InvocationResource()
  File "/sagemaker/python_service.py", line 48, in __init__
    self._handler, self._input_handler, self._output_handler = self._import_handlers()
  File "/sagemaker/python_service.py", line 68, in _import_handlers
    spec.loader.exec_module(inference)
  File "/opt/ml/model/code/inference.py", line 8, in <module>
    import cv2
  File "/usr/local/lib/python3.5/dist-packages/cv2/__init__.py", line 5, in <module>
    from .cv2 import *
	
2020-06-03T15:21:54.841+05:30
	
ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory
	
2020-06-03T15:21:54.841+05:30
	
[2020-06-03 09:51:53 +0000] [1917] [INFO] Worker exiting (pid: 1917)
	
2020-06-03T15:21:54.841+05:30
	
[2020-06-03 09:51:54 +0000] [1914] [INFO] Shutting down: Master
	
2020-06-03T15:21:54.841+05:30
	
[2020-06-03 09:51:54 +0000] [1914] [INFO] Reason: Worker failed to boot.
	
2020-06-03T15:21:54.841+05:30
	
WARNING:__main__:unexpected gunicorn exit (status: 768). restarting.
	
2020-06-03T15:21:54.841+05:30
	
INFO:__main__:gunicorn version info:
	
2020-06-03T15:21:54.841+05:30
	
gunicorn (version 19.9.0)
	
2020-06-03T15:21:54.841+05:30
	
INFO:__main__:started gunicorn (pid: 1923)
	
2020-06-03T15:21:54.841+05:30
	
[2020-06-03 09:51:54 +0000] [1923] [INFO] Starting gunicorn 19.9.0
	
2020-06-03T15:21:54.841+05:30
	
[2020-06-03 09:51:54 +0000] [1923] [INFO] Listening at: unix:/tmp/gunicorn.sock (1923)
	
2020-06-03T15:21:54.841+05:30
	
[2020-06-03 09:51:54 +0000] [1923] [INFO] Using worker: gevent
	
2020-06-03T15:21:54.841+05:30
	
[2020-06-03 09:51:54 +0000] [1926] [INFO] Booting worker with pid: 1926
	
2020-06-03T15:21:56.842+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
	
2020-06-03T15:21:56.842+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
	
2020-06-03T15:21:56.842+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
	
2020-06-03T15:21:56.842+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
	
2020-06-03T15:21:56.842+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
	
2020-06-03T15:21:56.843+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
	
2020-06-03T15:21:56.843+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
	
2020-06-03T15:21:56.843+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
	
2020-06-03T15:21:56.843+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
	
2020-06-03T15:21:56.843+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
	
2020-06-03T15:21:56.843+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
	
2020-06-03T15:21:56.843+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
	
2020-06-03T15:21:56.843+05:30
	
[2020-06-03 09:51:56 +0000] [1926] [ERROR] Exception in worker process
	
2020-06-03T15:21:56.844+05:30
	
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 203, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/sagemaker/python_service.py", line 239, in <module>
    resources.add_routes(app)
  File "/sagemaker/python_service.py", line 227, in add_routes
    invocation_resource = InvocationResource()
  File "/sagemaker/python_service.py", line 48, in __init__
    self._handler, self._input_handler, self._output_handler = self._import_handlers()
  File "/sagemaker/python_service.py", line 68, in _import_handlers
    spec.loader.exec_module(inference)
  File "/opt/ml/model/code/inference.py", line 8, in <module>
    import cv2
  File "/usr/local/lib/python3.5/dist-packages/cv2/__init__.py", line 5, in <module>
    from .cv2 import *
	
2020-06-03T15:21:56.844+05:30
	
ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory
	
2020-06-03T15:21:56.844+05:30
	
[2020-06-03 09:51:56 +0000] [1926] [INFO] Worker exiting (pid: 1926)
	
2020-06-03T15:21:56.844+05:30
	
[2020-06-03 09:51:56 +0000] [1923] [INFO] Shutting down: Master
	
2020-06-03T15:21:56.844+05:30
	
[2020-06-03 09:51:56 +0000] [1923] [INFO] Reason: Worker failed to boot.
	
2020-06-03T15:21:56.844+05:30
	
WARNING:__main__:unexpected gunicorn exit (status: 768). restarting.
	
2020-06-03T15:21:56.845+05:30
	
INFO:__main__:gunicorn version info:
	
2020-06-03T15:21:56.845+05:30
	
gunicorn (version 19.9.0)
	
2020-06-03T15:21:56.845+05:30
	
INFO:__main__:started gunicorn (pid: 1932)
	
2020-06-03T15:21:56.845+05:30
	
[2020-06-03 09:51:56 +0000] [1932] [INFO] Starting gunicorn 19.9.0
	
2020-06-03T15:21:56.845+05:30
	
[2020-06-03 09:51:56 +0000] [1932] [INFO] Listening at: unix:/tmp/gunicorn.sock (1932)
	
2020-06-03T15:21:56.845+05:30
	
[2020-06-03 09:51:56 +0000] [1932] [INFO] Using worker: gevent
	
2020-06-03T15:21:56.845+05:30
	
[2020-06-03 09:51:56 +0000] [1935] [INFO] Booting worker with pid: 1935
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
	
2020-06-03T15:21:58.846+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
	
2020-06-03T15:21:58.846+05:30
	
[2020-06-03 09:51:58 +0000] [1935] [ERROR] Exception in worker process
	
2020-06-03T15:21:58.847+05:30
	
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 203, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/sagemaker/python_service.py", line 239, in <module>
    resources.add_routes(app)
  File "/sagemaker/python_service.py", line 227, in add_routes
    invocation_resource = InvocationResource()
  File "/sagemaker/python_service.py", line 48, in __init__
    self._handler, self._input_handler, self._output_handler = self._import_handlers()
  File "/sagemaker/python_service.py", line 68, in _import_handlers
    spec.loader.exec_module(inference)
  File "/opt/ml/model/code/inference.py", line 8, in <module>
    import cv2
  File "/usr/local/lib/python3.5/dist-packages/cv2/__init__.py", line 5, in <module>
    from .cv2 import *
	
2020-06-03T15:21:58.847+05:30
	
ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory
	
2020-06-03T15:21:58.847+05:30
	
[2020-06-03 09:51:58 +0000] [1935] [INFO] Worker exiting (pid: 1935)
	
2020-06-03T15:21:58.847+05:30
	
Exception ignored in: <function _removeHandlerRef at 0x7f40427dc510>
	
2020-06-03T15:21:58.847+05:30
	
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/__init__.py", line 735, in _removeHandlerRef
  File "/usr/lib/python3.5/logging/__init__.py", line 218, in _acquireLock
  File "/usr/lib/python3.5/threading.py", line 141, in acquire
	
2020-06-03T15:21:58.847+05:30
	
TypeError: 'NoneType' object is not callable
	
2020-06-03T15:21:58.847+05:30
	
Exception ignored in: <function _removeHandlerRef at 0x7f40427dc510>
	
2020-06-03T15:21:58.847+05:30
	
Traceback (most recent call last):
  File "/usr/lib/python3.5/logging/__init__.py", line 735, in _removeHandlerRef
  File "/usr/lib/python3.5/logging/__init__.py", line 218, in _acquireLock
  File "/usr/lib/python3.5/threading.py", line 141, in acquire
	
2020-06-03T15:21:58.847+05:30
	
TypeError: 'NoneType' object is not callable
	
2020-06-03T15:21:58.847+05:30
	
2020/06/03 09:51:58 [error] 134#134: *191 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 10.32.0.2, server: , request: "GET /ping HTTP/1.1", upstream: "http://unix:/tmp/gunicorn.sock:/ping", host: "model.aws.local:8080"
	
2020-06-03T15:21:58.847+05:30
	
10.32.0.2 - - [03/Jun/2020:09:51:58 +0000] "GET /ping HTTP/1.1" 502 157 "-" "AHC/2.0"
	
2020-06-03T15:21:58.847+05:30
	
[2020-06-03 09:51:58 +0000] [1932] [INFO] Shutting down: Master
	
2020-06-03T15:21:58.847+05:30
	
[2020-06-03 09:51:58 +0000] [1932] [INFO] Reason: Worker failed to boot.
	
2020-06-03T15:21:58.847+05:30
	
WARNING:__main__:unexpected gunicorn exit (status: 768). restarting.
	
2020-06-03T15:21:59.847+05:30
	
INFO:__main__:gunicorn version info:
	
2020-06-03T15:21:59.847+05:30
	
gunicorn (version 19.9.0)
	
2020-06-03T15:21:59.847+05:30
	
INFO:__main__:started gunicorn (pid: 1941)
	
2020-06-03T15:21:59.848+05:30
	
[2020-06-03 09:51:59 +0000] [1941] [INFO] Starting gunicorn 19.9.0
	
2020-06-03T15:21:59.848+05:30
	
[2020-06-03 09:51:59 +0000] [1941] [INFO] Listening at: unix:/tmp/gunicorn.sock (1941)
	
2020-06-03T15:21:59.848+05:30
	
[2020-06-03 09:51:59 +0000] [1941] [INFO] Using worker: gevent
	
2020-06-03T15:21:59.848+05:30
	
[2020-06-03 09:51:59 +0000] [1944] [INFO] Booting worker with pid: 1944
	
2020-06-03T15:22:00.849+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
	
2020-06-03T15:22:00.849+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
	
2020-06-03T15:22:00.849+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
	
2020-06-03T15:22:00.849+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
	
2020-06-03T15:22:00.849+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
	
2020-06-03T15:22:00.850+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
	
2020-06-03T15:22:00.850+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
	
2020-06-03T15:22:00.850+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
	
2020-06-03T15:22:00.850+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
	
2020-06-03T15:22:00.850+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
	
2020-06-03T15:22:00.850+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
	
2020-06-03T15:22:00.850+05:30
	
/usr/local/lib/python3.5/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
	
2020-06-03T15:22:00.850+05:30
	
[2020-06-03 09:52:00 +0000] [1944] [ERROR] Exception in worker process
	
2020-06-03T15:22:00.852+05:30
	
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/ggevent.py", line 203, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 129, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/workers/base.py", line 138, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.5/dist-packages/gunicorn/util.py", line 350, in import_app
    __import__(module)
  File "/sagemaker/python_service.py", line 239, in <module>
    resources.add_routes(app)
  File "/sagemaker/python_service.py", line 227, in add_routes
    invocation_resource = InvocationResource()
  File "/sagemaker/python_service.py", line 48, in __init__
    self._handler, self._input_handler, self._output_handler = self._import_handlers()
  File "/sagemaker/python_service.py", line 68, in _import_handlers
    spec.loader.exec_module(inference)
  File "/opt/ml/model/code/inference.py", line 8, in <module>
    import cv2
  File "/usr/local/lib/python3.5/dist-packages/cv2/__init__.py", line 5, in <module>
    from .cv2 import *
	
2020-06-03T15:22:00.852+05:30
	
ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory
	
2020-06-03T15:22:00.853+05:30
	
[2020-06-03 09:52:00 +0000] [1944] [INFO] Worker exiting (pid: 1944)
		
No newer events at this moment.

laurenyu · 2020-07-23T17:50:55Z

sorry for the delayed response here.

looks like this is the error:

ImportError: libgthread-2.0.so.0: cannot open shared object file: No such file or directory

according to https://askubuntu.com/questions/1060903/importerror-libgthread-2-0-so-0-cannot-open-shared-object-file-no-such-file-o, there needs to be a Linux library installed in order to use OpenCV.

Unfortunately, there's not a way to do this at runtime, so what you'll have to do is use a Deep Learning Container as a base and install the necessary libraries.

DLC images: https://aws.amazon.com/releasenotes/available-deep-learning-containers-images

nadiaya added the type: question label Jun 9, 2020

laurenyu added the type: unrelated to Python SDK label Jul 23, 2020

bveeramani closed this as completed May 20, 2021

aws locked and limited conversation to collaborators May 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

This issue was moved to a discussion.

by usingin #1546

by usingin #1546

saisubramani commented Jun 4, 2020 •

edited

Loading

nadiaya commented Jun 9, 2020

saisubramani commented Jun 9, 2020 •

edited by laurenyu

Loading

laurenyu commented Jul 23, 2020

This issue was moved to a discussion.

This issue was moved to a discussion.

by usingin #1546

by usingin #1546

Comments

saisubramani commented Jun 4, 2020 • edited Loading

nadiaya commented Jun 9, 2020

saisubramani commented Jun 9, 2020 • edited by laurenyu Loading

laurenyu commented Jul 23, 2020

This issue was moved to a discussion.

saisubramani commented Jun 4, 2020 •

edited

Loading

saisubramani commented Jun 9, 2020 •

edited by laurenyu

Loading