storing ciphertext in variable OSX/Linux results in failed decryption

[mattsb42-aws]

From an initial overview, it appears that this is caused by encoding issues. Apparently PS is not the only one that wraps encoding.

(py36) 186590df9307:aws-encryption-sdk-cli bullocm$ aa=`echo 'asdf' | aws-crypto -e -i - -o - @test/integration/integration_tests.conf`
(py36) 186590df9307:aws-encryption-sdk-cli bullocm$ echo $aa
?x+??Z??6?nypj(?_aws-crypto-public-keyDAnCD9EN1ToOTTS//IzbmxAC8H99wcJzPWELU2WJjXhIOKnG7fbRpugw/IcGMdLWCFA==aws-kmsKarn:aws:kms:us-east-1:248168362296:key/ce78d3b3-f800-4785-a3b9-63e30bb4b183?x???????+Sn?P?n0 `?He.0?H??0| *?H??
        w??z?&???@??;?????dڃ?^??`<i??lZ?Q?5?????=?!??=?6?ow????<???
                                                                   $?m?`lz??=???W??????-?J??+????L)?Z??|g0e0?/#t2g???l7jfm=F??Z)??(?C?va??-??cQw?-,EKI?1?
                                                                                                                                                         =?S??_U?μP3P????ұʹ??0̤??-?????ujbZ?
(py36) 186590df9307:aws-encryption-sdk-cli bullocm$ echo $aa | aws-crypto -d -i - -o - @test/integration/integration_tests.conf
ERROR:aws_encryption_sdk.streaming_client:Error on closing
Traceback (most recent call last):
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/internal/formatting/encryption_context.py", line 132, in read_string
    codecs.decode(source[offset:end], aws_encryption_sdk.internal.defaults.ENCODING),
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/bin/../lib/python3.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa7 in position 172: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk_cli/internal/io_handling.py", line 90, in _single_io_write
    for chunk in handler:
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 262, in next
    return self.readline()
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 240, in readline
    line = self.read(self.line_length)
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 201, in read
    self._prep_message()
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 687, in _prep_message
    self._header, self.header_auth = self._read_header()
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 700, in _read_header
    header, raw_header = aws_encryption_sdk.internal.formatting.deserialize.deserialize_header(self.source_stream)
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/internal/formatting/deserialize.py", line 103, in deserialize_header
    tee_stream.read(ser_encryption_context_length)
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/internal/formatting/encryption_context.py", line 172, in deserialize_encryption_context
    length=key_size
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/internal/formatting/encryption_context.py", line 136, in read_string
    raise SerializationError('Bad format of serialized context.')
aws_encryption_sdk.exceptions.SerializationError: Bad format of serialized context.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 184, in __exit__
    self.close()
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 890, in close
    raise SerializationError('Footer not read')
aws_encryption_sdk.exceptions.SerializationError: Footer not read
Traceback (most recent call last):
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/internal/formatting/encryption_context.py", line 132, in read_string
    codecs.decode(source[offset:end], aws_encryption_sdk.internal.defaults.ENCODING),
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/bin/../lib/python3.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa7 in position 172: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/bin/aws-crypto", line 11, in <module>
    sys.exit(cli())
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk_cli/__init__.py", line 177, in cli
    no_overwrite=args.no_overwrite
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk_cli/__init__.py", line 78, in process_cli_request
    no_overwrite=no_overwrite
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk_cli/internal/io_handling.py", line 118, in process_single_operation
    destination_writer=destination_writer
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk_cli/internal/io_handling.py", line 90, in _single_io_write
    for chunk in handler:
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 262, in next
    return self.readline()
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 240, in readline
    line = self.read(self.line_length)
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 201, in read
    self._prep_message()
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 687, in _prep_message
    self._header, self.header_auth = self._read_header()
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/streaming_client.py", line 700, in _read_header
    header, raw_header = aws_encryption_sdk.internal.formatting.deserialize.deserialize_header(self.source_stream)
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/internal/formatting/deserialize.py", line 103, in deserialize_header
    tee_stream.read(ser_encryption_context_length)
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/internal/formatting/encryption_context.py", line 172, in deserialize_encryption_context
    length=key_size
  File "/Users/bullocm/git/aws-encryption-sdk-cli/.tox/py36/lib/python3.6/site-packages/aws_encryption_sdk/internal/formatting/encryption_context.py", line 136, in read_string
    raise SerializationError('Bad format of serialized context.')
aws_encryption_sdk.exceptions.SerializationError: Bad format of serialized context.

[mattsb42-aws]

For the record, base64-encoding solves this. It looks like this is definitely caused by encoding weirdness.

(py36) 186590df9307:aws-encryption-sdk-cli bullocm$ aa=`echo 'asdf' | aws-crypto -e -i - -o - @test/integration/integration_tests.conf | base64`
(py36) 186590df9307:aws-encryption-sdk-cli bullocm$ echo $aa
AYADeF6YMeeZTCME9fdWnnIwgHEAXwABABVhd3MtY3J5cHRvLXB1YmxpYy1rZXkAREE2OE51dk9vUmNCSSt4bDBZNXI1Ykw5bGNRaktnbldUZmJXRE8xNUc2b1RXZzkrclNtdldWbmNHQm50UWh2bkFnUT09AAEAB2F3cy1rbXMAS2Fybjphd3M6a21zOnVzLWVhc3QtMToyNDgxNjgzNjIyOTY6a2V5L2NlNzhkM2IzLWY4MDAtNDc4NS1hM2I5LTYzZTMwYmI0YjE4MwCnAQEBAHiLxv2Rx9XcK1MVbtlQmW4dst0V6lfDE2sy9gLQD4XsngAAAH4wfAYJKoZIhvcNAQcGoG8wbQIBADBoBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDLNR85u5g+V9NIT9CgIBEIA7PhKTmRu1NwVwj8sZE4ksIppkhbUUQ5xnMsFE3uW+ueZQo1W+yvCkiZ6zcAHy2swp6MTR69VBE1fHeXECAAAAAAwAABAAAAAAAAAAAAAAAAAA3+M3Z5vAWBlbsYtipKKEfv////8AAAABAAAAAAAAAAAAAAABAAAABZ2MWK6ZPi1FwYC8VdFrYDlYEMTy/ABnMGUCMHom6Yb+IGFG8cVa4/LzwBfvDd++eBq5sb6viTIl6klF1FPjpi/hF1heFtzlyxCo9AIxANTO0CkAwqJP6RCd8W6KYsVeIou+7Afo7po8zoleobL4murWx5vgxeBzjgEVftXs1g==
(py36) 186590df9307:aws-encryption-sdk-cli bullocm$ echo $aa | base64 -D | aws-crypto -d -i - -o - @test/integration/integration_tests.conf
asdf

[mattsb42-aws]

This will be addressed by #29.