Skip to content
This repository was archived by the owner on Mar 18, 2022. It is now read-only.

Add the ability to add a docker config image #33

Merged
merged 14 commits into from
Dec 28, 2017
Merged
6 changes: 6 additions & 0 deletions docs/spec.rst
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,12 @@ Following mapping keys are supported (all but the marked once are optional):
- ``pdf``, default: ``false``
- ``epub``, default: ``false``

``build``
Options for setting the docker configuration.

``image``
This sets the docker image to use on the build, as defined `here <https://github.com/rtfd/readthedocs-docker-images/blob/master/CONTRIBUTING.rst#releases>`_.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps "docker image" -> "build image" for consistency on nomenclature. It's not a docker image to end users, nor can arbitrary docker images be used.


``python``
Python specific configuration. All builds are executed inside a
virtualenv. This config can customize the virtualenv before running the
Expand Down
50 changes: 36 additions & 14 deletions readthedocs_build/config/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,8 @@
from .find import find_one
from .parser import ParseError
from .parser import parse
from .validation import validate_bool
from .validation import validate_choice
from .validation import validate_directory
from .validation import validate_file
from .validation import validate_string
from .validation import ValidationError
from .validation import (validate_bool, validate_choice, validate_directory,
validate_file, validate_string, ValidationError)


__all__ = (
Expand All @@ -30,6 +26,18 @@
TYPE_REQUIRED = 'type-required'
PYTHON_INVALID = 'python-invalid'

DOCKER_BUILD_IMAGES = {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make this updatable from a setting as it was, so we can define a custome DOCKER_BUILD_IMAGES in our development instance.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, good call. Hadn't thought of that specific use.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on a setting, we shouldn't change that behavior. This is important for all non-prod instances.

'1.0': {
'python': {'supported_versions': [2, 2.7, 3, 3.4]},
},
'2.0': {
'python': {'supported_versions': [2, 2.7, 3, 3.5]},
},
'latest': {
'python': {'supported_versions': [2, 2.7, 3, 3.3, 3.4, 3.5, 3.6]},
},
}


class ConfigError(Exception):

Expand Down Expand Up @@ -79,6 +87,7 @@ class BuildConfig(dict):
'"python.extra_requirements" section must be a list.')

PYTHON_SUPPORTED_VERSIONS = [2, 2.7, 3, 3.3, 3.4, 3.5, 3.6]
DOCKER_SUPPORTED_VERSIONS = ['1.0', '2.0', 'latest']

def __init__(self, env_config, raw_config, source_file, source_position):
self.env_config = env_config
Expand Down Expand Up @@ -115,13 +124,6 @@ def get_valid_types(self):
'sphinx',
)

def get_valid_python_versions(self):
try:
return self.env_config['python']['supported_versions']
except (KeyError, TypeError):
pass
return self.PYTHON_SUPPORTED_VERSIONS

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason to remove this? Seems we want validation on this still.

def get_valid_formats(self):
return (
'none',
Expand All @@ -145,6 +147,9 @@ def validate(self):
# Validate env_config.
self.validate_output_base()

# Validate the build environment first
self.validate_build() # Must happen before `validate_python`!

# Validate raw_config. Order matters.
self.validate_name()
self.validate_type()
Expand Down Expand Up @@ -205,6 +210,23 @@ def validate_base(self):
base = validate_directory(base, base_path)
self['base'] = base

def validate_build(self):
# Defaults
build = {'image': '2.0'}
if 'build' in self.raw_config:
_build = self.raw_config['build']
if 'image' in _build:
with self.catch_validation_error('build'):
build['image'] = validate_choice(
str(_build['image']),
self.DOCKER_SUPPORTED_VERSIONS,
)
self['build'] = build

# Set the valid python versions for this container
img = build['image']
self.PYTHON_SUPPORTED_VERSIONS = DOCKER_BUILD_IMAGES[img]['python']['supported_versions']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PYTHON_SUPPORTED_VERSIONS is named as a constant and shouldn't be treated as a variable. I think the usage of get_valid_python_versions was more correct. Also, validation steps probably shouldn't have side effects like altering the state here, so that's probably also another good reason to make this a method that merges sources together.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordering of the validation is going to continue to matter (because we need to read the docker image, then determine python version), so it will have to set state somewhere, but agreed on making it a bit nicer.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we are/were doing ordering of validation one level up anyways:
https://github.com/rtfd/readthedocs-build/blob/master/readthedocs_build/config/config.py#L133-L157

So, dependencies would naturally be filled there, as the validation steps are ordered there


def validate_python(self):
python = {
'use_system_site_packages': False,
Expand Down Expand Up @@ -280,7 +302,7 @@ def validate_python(self):
pass
python['version'] = validate_choice(
version,
self.get_valid_python_versions()
self.PYTHON_SUPPORTED_VERSIONS
)

self['python'] = python
Expand Down
81 changes: 64 additions & 17 deletions readthedocs_build/config/test_config.py
Original file line number Diff line number Diff line change
Expand Up @@ -281,23 +281,6 @@ def it_validates_wrong_type_right_value():
build.validate_python()
assert build['python']['version'] == 3

def it_validates_env_supported_versions():
build = get_build_config(
{'python': {'version': 3.6}},
env_config={'python': {'supported_versions': [3.5]}}
)
with raises(InvalidConfig) as excinfo:
build.validate_python()
assert excinfo.value.key == 'python.version'
assert excinfo.value.code == INVALID_CHOICE

build = get_build_config(
{'python': {'version': 3.6}},
env_config={'python': {'supported_versions': [3.5, 3.6]}}
)
build.validate_python()
assert build['python']['version'] == 3.6
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems we should make this error work instead of removing it. We use supported versions for giving intelligent errors to users.



def describe_validate_formats():

Expand Down Expand Up @@ -413,6 +396,70 @@ def it_fails_if_base_does_not_exist(tmpdir):
assert excinfo.value.code == INVALID_PATH


def describe_validate_build():

def it_fails_if_build_is_invalid_option(tmpdir):
apply_fs(tmpdir, minimal_config)
build = BuildConfig(
{},
{'build': {'image': 3.0}},
source_file=str(tmpdir.join('readthedocs.yml')),
source_position=0)
with raises(InvalidConfig) as excinfo:
build.validate_build()
assert excinfo.value.key == 'build'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, why we don't use something like regular unittest.TestCase and assertEqual and those stuffs in this project?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is just how this code was written originally. shrug

assert excinfo.value.code == INVALID_CHOICE

def it_fails_on_python_validation(tmpdir):
apply_fs(tmpdir, minimal_config)
build = BuildConfig(
{},
{
'build': {'image': 1.0},
'python': {'version': '3.3'},
},
source_file=str(tmpdir.join('readthedocs.yml')),
source_position=0)
build.validate_build()
with raises(InvalidConfig) as excinfo:
build.validate_python()
assert excinfo.value.key == 'python.version'
assert excinfo.value.code == INVALID_CHOICE

def it_works_on_python_validation(tmpdir):
apply_fs(tmpdir, minimal_config)
build = BuildConfig(
{},
{
'build': {'image': 'latest'},
'python': {'version': '3.3'},
},
source_file=str(tmpdir.join('readthedocs.yml')),
source_position=0)
build.validate_build()
build.validate_python()

def it_works(tmpdir):
apply_fs(tmpdir, minimal_config)
build = BuildConfig(
{},
{'build': {'image': 'latest'}},
source_file=str(tmpdir.join('readthedocs.yml')),
source_position=0)
build.validate_build()
assert build['build']['image'] == 'latest'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't we check here that 3.6 was the version selected by default?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we're opinionated about this yet. We should be eventually though.

+1 on checks for default version though, that seems to be missing from test cases


def default(tmpdir):
apply_fs(tmpdir, minimal_config)
build = BuildConfig(
{},
{},
source_file=str(tmpdir.join('readthedocs.yml')),
source_position=0)
build.validate_build()
assert build['build']['image'] == '2.0'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here but 2.7, I guess.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we want to specify python magically, based on container image, do we?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already do this to a certain degree, but i think the python version tests are testing some of this. For example:

  • python['version'] = 3 is valid, and exists in the dashboard, we should select the right python 3 version (pretty sure tests exist here)
  • python['version'] = None should return 2.7 for now, until we're opinionated about forcing 3.6 default (not work i think we need to make for ourselves just yet, but i'd like to see this soon)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't want that, what would it happen if the user doesn't specify the python version? should the test fail? do we have a test for that?



def test_build_validate_calls_all_subvalidators(tmpdir):
apply_fs(tmpdir, minimal_config)
build = BuildConfig(
Expand Down
4 changes: 2 additions & 2 deletions requirements/tests.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
mock==1.3.0
pytest>=2.7
pytest-describe>=0.10.1
pytest==3.2.5
pytest-describe==0.11.0
pytest-xdist>=1.12
tox>=2.1