Skip to content

Refactor configuration object to class based #4298

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 64 commits into from
Jul 10, 2018
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
8e6a73c
Create base class
stsewd Jun 25, 2018
10e750c
Use new interface for build.formats
stsewd Jun 25, 2018
0c7fdd8
Implement config.formats
stsewd Jun 25, 2018
3243d12
Linter
stsewd Jun 25, 2018
7799aa3
Test build.python
stsewd Jun 25, 2018
928dd7f
Implement build.python property
stsewd Jun 25, 2018
4c76ce4
Test buil.config property
stsewd Jun 25, 2018
bf055ca
Implement config.build property
stsewd Jun 25, 2018
37c67dd
Test for base and name properties
stsewd Jun 26, 2018
31076bf
Implement missing properties
stsewd Jun 26, 2018
81d0df9
Implement all abstract properties
stsewd Jun 26, 2018
baceda6
Remove setter
stsewd Jun 26, 2018
ea4b0fb
Move some logic from Wrapper object
stsewd Jun 26, 2018
666c952
Thinner Wrapper object
stsewd Jun 26, 2018
a6c18d5
Linter
stsewd Jun 26, 2018
5928a37
Remove output_base from dictionary
stsewd Jun 26, 2018
5d5c44a
Use build_image property
stsewd Jun 26, 2018
dcee59c
Use wrapper attribute
stsewd Jun 26, 2018
3c7bd9f
Use type attribute
stsewd Jun 26, 2018
fefe192
Use base attribute
stsewd Jun 26, 2018
33abd0d
Use python property
stsewd Jun 26, 2018
b49d0a2
Update docs
stsewd Jun 26, 2018
097b8fe
Use python property in wrapper
stsewd Jun 26, 2018
61756c3
Extra guard
stsewd Jun 26, 2018
e377829
Linter
stsewd Jun 26, 2018
cfb0b1d
Make config private
stsewd Jun 26, 2018
4cbc3c3
Use formats attribute
stsewd Jun 26, 2018
985ff7a
Use conda property
stsewd Jun 26, 2018
d0e9f78
Use requirements_file attribute
stsewd Jun 26, 2018
d48996d
Use conf_file attribute
stsewd Jun 26, 2018
2860205
Introducing defaults
stsewd Jun 26, 2018
2509235
Thinner wrapper
stsewd Jun 26, 2018
7f360b0
Don't use ConfigWrapper directly
stsewd Jun 26, 2018
259082f
Use defaults in formats
stsewd Jun 26, 2018
6e8f723
Use load_yaml_config inside PythonEnvironment
stsewd Jun 27, 2018
110ae38
Remove dict inheritance
stsewd Jun 27, 2018
ae20ca8
Test for ConfigNotSupportedError
stsewd Jun 29, 2018
23af026
Raise ConfigNotSupportedError when accessing a invalid attribute
stsewd Jun 29, 2018
668cb27
Test for validate
stsewd Jun 29, 2018
3604d50
Move version
stsewd Jun 29, 2018
c728680
Docs
stsewd Jun 29, 2018
6edf6f0
More tests
stsewd Jun 29, 2018
1b2014a
Remove try except
stsewd Jun 29, 2018
c19f0e0
More tests
stsewd Jun 29, 2018
1964317
Linter
stsewd Jun 29, 2018
fdad93d
Move _config to base class
stsewd Jul 5, 2018
89e0dcf
Implement default for use_system_site_packages
stsewd Jul 5, 2018
215206e
Typo
stsewd Jul 5, 2018
5c66206
Move build_image from ConfigWrapper
stsewd Jul 5, 2018
62a3d98
Move requirements_file from ConfigWrapper
stsewd Jul 5, 2018
711b595
Move python_version from ConfigWrapper
stsewd Jul 5, 2018
b43372b
Move python_full_version and python_interpreter from ConfigWrapper
stsewd Jul 5, 2018
e85e587
Pass build_image as default
stsewd Jul 6, 2018
376962e
Remove ConfigWrapper from tests
stsewd Jul 6, 2018
8d8197a
Goodbye ConfigWrapper
stsewd Jul 6, 2018
811bf72
Fix tests
stsewd Jul 6, 2018
241bbed
More tests
stsewd Jul 6, 2018
cdce197
Rename module
stsewd Jul 6, 2018
6ef45fe
Fix conflict
stsewd Jul 6, 2018
bc67350
Merge branch 'master' into move-configuration-to-class-based
stsewd Jul 6, 2018
8c516ba
Trailing comma
stsewd Jul 6, 2018
84bb4bc
Add TODOs
stsewd Jul 6, 2018
b8bba7e
Rename exception
stsewd Jul 9, 2018
b0bf8c5
Update copy
stsewd Jul 9, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
232 changes: 199 additions & 33 deletions readthedocs/config/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -76,53 +76,39 @@ def __init__(self, key, code, error_message, source_file=None,
super(InvalidConfig, self).__init__(message, code=code)


class BuildConfig(dict):
class BuildConfigBase(object):

"""
Config that handles the build of one particular documentation.

Config keys can be accessed with a dictionary lookup::

>>> build_config['type']
'sphinx'

You need to call ``validate`` before the config is ready to use. Also
setting the ``output_base`` is required before using it for a build.
You need to call ``validate`` before the config is ready to use.
Also setting the ``output_base`` is required before using it for a build.
"""

BASE_INVALID_MESSAGE = 'Invalid value for base: {base}'
BASE_NOT_A_DIR_MESSAGE = '"base" is not a directory: {base}'
NAME_REQUIRED_MESSAGE = 'Missing key "name"'
NAME_INVALID_MESSAGE = (
'Invalid name "{name}". Valid values must match {name_re}')
TYPE_REQUIRED_MESSAGE = 'Missing key "type"'
CONF_FILE_REQUIRED_MESSAGE = 'Missing key "conf_file"'
PYTHON_INVALID_MESSAGE = '"python" section must be a mapping.'
PYTHON_EXTRA_REQUIREMENTS_INVALID_MESSAGE = (
'"python.extra_requirements" section must be a list.')

PYTHON_SUPPORTED_VERSIONS = [2, 2.7, 3, 3.5]
DOCKER_SUPPORTED_VERSIONS = ['1.0', '2.0', 'latest']

def __init__(self, env_config, raw_config, source_file, source_position):
self.env_config = env_config
self.raw_config = raw_config
self.source_file = source_file
self.source_position = source_position
super(BuildConfig, self).__init__()
super(BuildConfigBase, self).__init__()

def error(self, key, message, code):
"""Raise an error related to ``key``."""
source = '{file} [{pos}]'.format(
file=self.source_file,
pos=self.source_position)
pos=self.source_position
)
error_message = '{source}: {message}'.format(
source=source,
message=message
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

File position is generally really useful, we should make sure the file position is raised to users.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is source has the source_position :). The diff is a little messy and gives the impression to be removed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, missed that, yeah this is fine

raise InvalidConfig(
key=key,
code=code,
error_message='{source}: {message}'.format(source=source,
message=message),
error_message=error_message,
source_file=self.source_file,
source_position=self.source_position)
source_position=self.source_position
)

@contextmanager
def catch_validation_error(self, key):
Expand All @@ -135,7 +121,106 @@ def catch_validation_error(self, key):
code=error.code,
error_message=str(error),
source_file=self.source_file,
source_position=self.source_position)
source_position=self.source_position
)

def validate(self):
raise NotImplementedError()
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just leaving this one, as we always need it.


@property
def base(self):
raise NotImplementedError()

def output_base(self):
raise NotImplementedError()

@property
def name(self):
raise NotImplementedError()

@property
def version(self):
raise NotImplementedError()

@property
def type(self):
raise NotImplementedError()

@property
def pip_install(self):
raise NotImplementedError()

@property
def install_project(self):
raise NotImplementedError()

@property
def extra_requirements(self):
raise NotImplementedError()

@property
def python_interpreter(self):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can move this to a function outside the config object

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps it makes somewhere else, yeah. We can probably leave for just now. Feel free to open an issue of you want to track this work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opened #4343

raise NotImplementedError()

@property
def python_version(self):
raise NotImplementedError()

@property
def python_full_version(self):
raise NotImplementedError()

@property
def use_system_site_packages(self):
raise NotImplementedError()

@property
def use_conda(self):
raise NotImplementedError()

@property
def conda_file(self):
raise NotImplementedError()

@property
def requirements_file(self):
raise NotImplementedError()

@property
def formats(self):
"""Documentation formats to be built."""
raise NotImplementedError()

@property
def build(self):
raise NotImplementedError()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An improvement to this pattern would be to remove the need for all of this duplicated code on the base class. I guess the general advice with python is to use duck typing instead of a stricter abstract class pattern.

One option is to use a meta class on the base, which would raise NotImplemented on any missing methods. Another option would be to raise NotImplemented, or a more specific error even, from the v1 or v2 class directly, and avoid requiring properties on the base class for each new field. Do you think it would be better to remove all of these properties on the base class?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is already implemented https://github.com/rtfd/readthedocs.org/pull/4298/files#diff-2b6854693f39acfa183fca7fef27dcfbR148. I think you are seeing an old diff 🤔 There aren't more properties in the base class

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, github sneaked old feedback in. Disregard.



class BuildConfig(BuildConfigBase, dict):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm keeping the dict thing here for a while, so the refactor isn't too big


"""
Version 1 of the configuration file.

Config keys can be accessed with a dictionary lookup::

>>> build_config['type']
'sphinx'
"""

BASE_INVALID_MESSAGE = 'Invalid value for base: {base}'
BASE_NOT_A_DIR_MESSAGE = '"base" is not a directory: {base}'
NAME_REQUIRED_MESSAGE = 'Missing key "name"'
NAME_INVALID_MESSAGE = (
'Invalid name "{name}". Valid values must match {name_re}')
TYPE_REQUIRED_MESSAGE = 'Missing key "type"'
CONF_FILE_REQUIRED_MESSAGE = 'Missing key "conf_file"'
PYTHON_INVALID_MESSAGE = '"python" section must be a mapping.'
PYTHON_EXTRA_REQUIREMENTS_INVALID_MESSAGE = (
'"python.extra_requirements" section must be a list.')

PYTHON_SUPPORTED_VERSIONS = [2, 2.7, 3, 3.5]
DOCKER_SUPPORTED_VERSIONS = ['1.0', '2.0', 'latest']


def get_valid_types(self): # noqa
return (
Expand Down Expand Up @@ -432,6 +517,92 @@ def validate_formats(self):

return True

@property
def version(self): # noqa
return '1'

@property
def name(self): # noqa
return self['name']

@property
def base(self): # noqa
return self['base']

@property
def output_base(self): # noqa
return self['output_base']

@property
def type(self): # noqa
return self['type']

@property
def formats(self): # noqa
return self['formats']

@property
def python(self): # noqa
return self['python']

@property
def build(self): # noqa
return self['build']

@property
def pip_install(self): # noqa
return self['python']['pip_install']

@property
def install_project(self): # noqa
return None

@property
def extra_requirements(self): # noqa
return self['python']['extra_requirements']

@property
def python_interpreter(self): # noqa
ver = self.python_full_version
return 'python{0}'.format(ver)

@property
def python_version(self): # noqa
return self['python']['version']

@property
def python_full_version(self): # noqa
ver = self.python_version
if ver in [2, 3]:
# Get the highest version of the major series version if user only
# gave us a version of '2', or '3'
ver = max(
version
for version in self.get_valid_python_versions()
if version < ver + 1
)
return ver

@property
def use_system_site_packages(self): # noqa
return self['python']['use_system_site_packages']

@property
def use_conda(self): # noqa
return 'conda' in self

@property
def conda_file(self): # noqa
return self['conda']['file']

@property
def requirements_file(self): # noqa
return self['requirements_file']

@property
def build_image(self): # noqa
return self['build']['image']


class ProjectConfig(list):

Expand All @@ -442,11 +613,6 @@ def validate(self):
for build in self:
build.validate()

def set_output_base(self, directory):
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We weren't using this, and it's the only setter. We should keep all the config object as read-only I think.

"""Set a common ``output_base`` for each configuration build."""
for build in self:
build['output_base'] = os.path.abspath(directory)


def load(path, env_config):
"""
Expand Down
Loading