diff --git a/peps/pep-0825.rst b/peps/pep-0825.rst index a89f110b747..3de55594b80 100644 --- a/peps/pep-0825.rst +++ b/peps/pep-0825.rst @@ -12,26 +12,28 @@ Author: Jonathan Dekhtiar , Donald Stufft , Andy R. Terrel PEP-Delegate: Paul Moore -Discussions-To: https://discuss.python.org/t/pep-817-split-wheel-variants-package-format/106196 +Discussions-To: https://discuss.python.org/t/pep-825-wheel-variants-package-format-split-from-pep-817/106196 Status: Draft Type: Standards Track Topic: Packaging Created: 17-Feb-2026 -Post-History: `17-Feb-2026 `__ +Post-History: `17-Feb-2026 `__ Abstract ======== -This PEP proposes variant wheels, an extension to +This PEP provides the data format for variant wheels, an extension to :doc:`packaging:specifications/binary-distribution-format` that permits building multiple variants of the same package while embedding -additional compatibility data. The specific properties are stored inside -the wheel, and expressed via a human-readable variant label in the -filename, which is then mapped to the actual properties via a separately -hosted JSON mapping. This aims to make ``{tool} install {package}`` -capable of selecting the most appropriate variant of packages where -additional compatibility dimensions such as GPU support need to be -accounted for. +additional compatibility data. This data is stored inside the wheel, and +expressed via a human-readable variant label in the filename. + +When wheels are hosted on an index, it is additionally exposed in a +separate JSON file as an optimization. It will be followed by additional +PEPs defining the remaining aspects of variant wheels. The final aim of +the specification is to make ``{tool} install {package}`` capable of +selecting the most appropriate variant of packages where additional +compatibility dimensions such as GPU support need to be accounted for. Motivation @@ -70,6 +72,32 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", document are to be interpreted as described in :rfc:`2119`. +Implementation requirements +--------------------------- + +This specification is written from the perspective of tools producing +the various file formats defined or changed in this specification, such +as variant wheels, variant metadata files or ``pylock.toml`` files. +In the context of format definitions, the word "MUST" specifically +indicates that the requirement must be satisfied for the data to be +considered valid according to this specification. + +Tools that produce the data formats according to this specification MUST +always produce files that meet the requirements of this specification. + +Tools that consume this specification's data formats are not required to +verify that the data meets these requirements. A tool SHOULD NOT rely on +data that the tool has established does not meet the specification +format. The appropriate response depends on the role of the tool, and +follows the same pattern as for invalid Core Metadata in wheel files. +Tools that are in a position to reject invalid data at the point it +enters the ecosystem, such as a package index accepting an upload, +SHOULD do so. Tools that encounter it later, such as an installer +resolving a dependency, SHOULD prefer to degrade gracefully rather than +fail outright, for example by ignoring the variant wheels and selecting +among the remaining ones. + + Variant wheel ------------- @@ -173,26 +201,35 @@ PEP defines the following structure: +- $schema +- default-priorities | +- namespace : list[str] - | +- feature - | +- {namespace} : list[str] = [] - | +- property - | +- {namespace} - | +- {feature} : list[str] = [] +- variants +- {variant_label} +- {namespace} +- {feature} : list[str] = [] -This structure corresponds to the version ``0.1.0`` of the format. The -version number is stored as part of the schema_ URL. Whenever the format -changes, the version number must be incremented. Tools MUST assume that -all variant wheels using an unknown format version are unsupported. +This structure corresponds to the version ``0.1.1`` of the format. The +version number is stored as part of the schema_ URL. The version numbers +follow semantic versioning. + +The numbers starting with zero are reserved for drafts and MUST NOT be +used in production. Tools MUST NOT make any compatibility assumptions +over these versions. Once the proposal is complete, the latest draft +will be promoted to version ``1.0.0``. -The version numbers starting with zero are reserved for drafts and MUST -NOT be used in production. Once the proposal is complete, the latest -draft will be promoted to version ``1.0.0``. +If a backwards incompatible change is done to the specification, +the major version number MUST be incremented, and the remaining version +components MUST be zeroed. Tools MUST reject metadata with a major +version number that they do not support. -The top-level keys are described in the subsequent sections. +If a backwards compatible change is done to the specification, the minor +version number MUST be incremented, and the patch number MUST be zeroed. +Installers and other tools that only consume variant metadata SHOULD +accept metadata with a newer minor version number than the latest +supported, provided that the major version is supported. Tools +outputting variant metadata MUST NOT output a version that they do not +explicitly support. + +The top-level keys, as well as their scope and consistency requirements, +are described in the subsequent sections. Schema @@ -215,40 +252,42 @@ include updated versions of the schema. The schema is available in Default priorities '''''''''''''''''' -The ``default-priorities`` dictionary defines the ordering of variants. -The exact algorithm is described in the `Variant ordering`_ section. +The ``default-priorities`` dictionary defines the ordering of +namespaces which is used in variant ordering. The exact algorithm is +described in the `Variant ordering`_ section. The following key is REQUIRED: - ``namespace: list[str]``: All variant namespaces used in variant wheels for a given package version, ordered in decreasing priority. - This list MUST contain all namespaces used in variant properties. - -It MAY have the following OPTIONAL keys: - -- ``feature: dict[str, list[str]]``: A dictionary with namespaces as - keys, and ordered list of corresponding feature names as values. - The feature names are ordered in decreasing priority. It is used to - override the default feature ordering. + This list MUST contain all namespaces used in variant properties, and + it MUST NOT be empty: a package version providing variant wheels MUST + use at least one variant namespace. -- ``property: dict[str, dict[str, list[str]]]``: A nested dictionary - with namespaces as first-level keys, feature names as second-level - keys and ordered lists of corresponding property values as - second-level values. The feature values are ordered in decreasing - priority. It is used to override the default value ordering. +Default priorities are defined at the project scope. The value in +different wheels SHOULD be identical, or one an extension of the other: +additional namespaces MAY be appended. The metadata is considered +consistent if the longer list starts with the elements of the shorter +list, in the same order. In this case, combining the metadata MUST +result in the longer list being used. Variants '''''''' The ``variants`` dictionary provides a mapping from variant labels -to variant properties. In the variant wheel, it MUST contain the label -present in that wheel's filename. +to variant properties. In an individual variant wheel, it is scoped to +that wheel and it MUST contain exactly one entry, whose key is the +variant label present in that wheel's filename. It has 3 levels. The first level keys are variant labels, the second -level keys are namespaces, the third level are feature names, and the -third level values are sets of feature values, converted to lists, -sorted lexically. +level keys are namespaces, and the third level keys are feature names. +The third level values are sets of feature values, converted to lists +and sorted lexically. + +For the metadata to be consistent, the same keys MUST always correspond +to the same values. When combining metadata, the resulting ``variants`` +dictionary MUST be a union of all the input dictionaries. Example @@ -258,7 +297,7 @@ Example { // The schema URL will be replaced with the final URL on packaging.python.org - "$schema": "https://variants-schema.wheelnext.dev/peps/825/v0.1.0.json", + "$schema": "https://variants-schema.wheelnext.dev/peps/825/v0.1.1.json", "default-priorities": { // REQUIRED: specifies that x86_64 CPU properties are more important than @@ -266,20 +305,6 @@ Example // does not matter), and both are more important than specific BLAS/LAPACK // library: "namespace": ["x86_64", "aarch64", "blas_lapack"], - - // OPTIONAL: makes "library" the most important feature in "blas_lapack" - // namespace - "feature": { - "blas_lapack": ["library"] - }, - - // OPTIONAL: makes ["mkl", "openblas"] the most important values of - // "blas_lapack :: library" feature - "property": { - "blas_lapack": { - "library": ["mkl", "openblas"] - } - } }, "variants": { @@ -303,12 +328,15 @@ Example Index-level metadata -------------------- -For every package version that includes at least one variant wheel, -there MUST exist a corresponding ``{name}-{version}-variants.json`` -file. The ``{name}`` and ``{version}`` placeholders correspond to the -package name and version, normalized according to the same rules as -wheel files, as found in the :ref:`packaging:wheel-file-name-spec` of -the Binary Distribution Format specification. +When a package version that includes at least one variant wheel is +hosted on an index, a corresponding ``{name}-{version}-variants.json`` +file MUST be hosted as well. The purpose of the file is to optimize +variant metadata lookups by removing the need to fetch multiple variant +wheels during dependency resolution. The ``{name}`` and ``{version}`` +placeholders correspond to the package name and version, normalized +according to the same rules as wheel files, as found in the +:ref:`packaging:wheel-file-name-spec` of the Binary Distribution Format +specification. The exact URL where the file is hosted is insignificant, but it MUST be provided in all the responses where the variant wheels are included. @@ -321,14 +349,10 @@ as long as the values do not prevent correct operation. Tools MAY either use or ignore these values. This file uses the same structure as `variant metadata`_, except that -the ``variants`` object MUST list all variants available on the package -index for the package version in question. The tools MUST ensure that -the variant metadata across multiple variant wheels of the same package -version and the index-level metadata file is consistent. They MAY -require that keys other than ``variants`` have exactly the same values, -or they may carefully merge their values, provided that no conflicting -information is introduced, and the result is the same irrespective of -the order in which the wheels are processed. +the ``variants`` object is index-scoped and it MUST list all variants +available on the package index for the package version in question. It +MUST be consistent with the `variant metadata`_ of the individual +wheels, as specified in `metadata consistency`_. This file SHOULD NOT be considered immutable and MAY be updated in a backward compatible way at any point (e.g. when adding a new variant). @@ -345,7 +369,7 @@ like: { // The schema URL will be replaced with the final URL on packaging.python.org - "$schema": "https://variants-schema.wheelnext.dev/peps/825/v0.1.0.json", + "$schema": "https://variants-schema.wheelnext.dev/peps/825/v0.1.1.json", "default-priorities": { // identical to above }, @@ -382,13 +406,62 @@ like: } +Metadata consistency +-------------------- + +The `variant metadata`_ carried by the individual variant wheels of a +package version, and the `index-level metadata`_ file where one is +published, all describe the same release, and are required to agree with +one another. Gathered in one place, and stated in full in the sections +defining the respective keys, the requirements are: + +- ``default-priorities.namespace``: the lists MUST either be identical, + or the longer MUST start with the elements of the shorter one, in the + same order. Combining them MUST yield the longer list. + +- ``variants``: the same variant label MUST always map to the same set + of properties. Combining them MUST yield the union of the + dictionaries. + +Both rules are symmetric, so combining metadata that satisfies them +gives the same result regardless of the order in which the inputs are +processed. + +Meeting these requirements is the responsibility of the publisher of the +package version. Since the data originates from a single source per +project and is copied into the wheels at build time, they are satisfied +by construction unless the wheels of one release are built from +different inputs. + +Tools consuming variant metadata MAY assume that these requirements are +met, and are not required to verify it. Where a tool does establish that +they are not met, the response described in `implementation +requirements`_ applies. + +Where a user draws wheels for the same package from more than one +published source, no publisher is in a position to guarantee consistency +with the others. Ensuring that the sources being combined are consistent +is then the responsibility of the user. Tools are not required to detect +or resolve inconsistencies between sources; `installing wheels from +multiple sources (non-normative)`_ discusses what they can reasonably do +instead. + + Variant ordering ---------------- This specification defines an ordering between different wheels based on the presence of variant metadata. -For the purpose of ordering, variant properties are grouped into +For the purpose of ordering, the combined variant metadata for all +candidate variant wheels MUST be obtained. It can be sourced either from +the `index-level metadata`_ file, or from the individual wheels, +combined as specified in the `variant metadata`_ section. Both sources +yield the same result, since they are required to be consistent, and +tools SHOULD prefer the index-level metadata file where it is available, +as obtaining the data from it is considerably cheaper. + +Variant properties from all the eligible variant wheels are grouped into features, and features into namespaces. For every namespace, the tool MUST obtain a list of compatible features, and for every feature, a list of compatible values. The method of obtaining these lists will be @@ -402,35 +475,14 @@ groups of variant wheels MUST then be ordered according to the following algorithm: 1. Construct the ordered list of namespaces by copying the value of the - ``default-priorities.namespace`` key from `index-level metadata`_. - This is ``namespace_order`` in the example. - -2. For every namespace: - - i. Construct the initial ordered list of feature names by copying the - value of the respective ``default-priorities.feature.{namespace}`` - key. - - ii. Take the ordered list of compatible feature names obtained - previously and iterate over it, in order. For every feature name - that is not present in the constructed list, append it to the - end. - - After this step, a list of ordered feature names is available for - every namespace. This is ``feature_order`` in the example. + ``default-priorities.namespace`` key from the combined variant + metadata. This is ``namespace_order`` in the example. -3. For every feature: +2. For every namespace, take the ordered list of compatible feature + names obtained previously. This is ``feature_order`` in the example. - i. Construct the initial ordered list of values by copying the value - of the respective - ``default-priorities.property.{namespace}.{feature_name}`` key. - - ii. Take the ordered list of compatible feature values obtained - previously and iterate over it, in order. For every value that is - not present in the constructed list, append it to the end. - - After this step, a list of ordered property values is available for - every feature. This is ``value_order`` in the example. +3. For every feature, take the ordered list of compatible values + obtained previously. This is ``value_order`` in the example. 4. For every group, determine the most preferred value corresponding to every variant feature present in the variant properties corresponding @@ -472,7 +524,13 @@ sorted from the most preferred to the least preferred. The tools MAY provide options to override the default ordering, for example by specifying a preference for specific namespaces, features or properties. The tools MAY also provide options to exclude specific -variants, or to select a particular variant. +variants, or to select a particular variant. These options operate on +the wheels that were found compatible, so they MAY reorder or narrow +that set, but MUST NOT cause selection of a wheel with properties the +target system does not support. Installing a variant wheel for a system +other than the one being installed to is instead a matter of overriding +which properties are considered supported, which is `out of scope`_ for +this PEP. Alternatively, the sort algorithm for variant wheels could be described using the following pseudocode. For simplicity, this code does not @@ -494,34 +552,25 @@ compatibility tags. ... - # default-priorities dict from index-level metadata + # default-priorities dict from combined variant metadata default_priorities = { "namespace": [...], # : list[str] - "feature": {...}, # : dict[str, list[str]] - "property": {...}, # : dict[str, dict[str, list[str]]] } - # 1. Construct the ordered list of namespaces. + # 1. Obtain the ordered list of namespaces from the variant metadata. namespace_order = default_priorities["namespace"] - feature_order = {} - value_order = {} - - for namespace in namespace_order: - # 2. Construct the ordered lists of feature names. - feature_order[namespace] = default_priorities["feature"].get(namespace, []) - for feature_name in get_compatible_feature_names(namespace): - if feature_name not in feature_order[namespace]: - feature_order[namespace].append(feature_name) - - value_order[namespace] = {} - for feature_name in feature_order[namespace]: - # 3. Construct the ordered lists of feature values. - value_order[namespace][feature_name] = ( - default_priorities["property"].get(namespace, {}).get(feature_name, []) - ) - for feature_value in get_compatible_feature_values(namespace, feature_name): - if feature_value not in value_order[namespace][feature_name]: - value_order[namespace][feature_name].append(feature_value) + # 2. Obtain the ordered lists of features. + feature_order = { + namespace: get_compatible_feature_names(namespace) + for namespace in namespace_order + } + # 3. Obtain the ordered lists of feature values. + value_order = { + namespace: { + feature_name: get_compatible_feature_values(namespace, feature_name) + for feature_name in feature_order[namespace] + } for namespace in namespace_order + } def best_value_property(namespace: str, feature_name: str, feature_values: str) -> str: @@ -533,7 +582,7 @@ compatibility tags. def property_key(prop: tuple[str, str, str]) -> tuple[int, int, int]: - """Construct a sort key for variant property (akin to step 5.)""" + """Construct a sort key for variant property (akin to step 5)""" namespace, feature_name, feature_value = prop return ( namespace_order.index(namespace), @@ -550,7 +599,7 @@ compatibility tags. properties: dict[str, dict[str, list[str]]] def best_value_properties(self: Self) -> list[tuple[str, str, str]]: - """Determine the most preferred values for every feature, step 4.""" + """Determine the most preferred values for every feature, step 4""" return [ ( namespace, @@ -562,11 +611,11 @@ compatibility tags. ] def sorted_properties(self: Self) -> list[tuple[str, str, str]]: - """Sort the list of features with their best values (step 6.)""" + """Sort the list of features with their best values (step 6)""" return sorted(self.best_value_properties(), key=property_key) def __lt__(self: Self, other: Self) -> bool: - """Variant comparison function for sorting (part of step 7.)""" + """Variant comparison function for sorting (part of step 7)""" self_properties = self.sorted_properties() other_properties = other.sorted_properties() # Proceed from the first to the last common sort best-value property. @@ -596,50 +645,110 @@ Environment markers ------------------- Four new :ref:`environment markers -` are introduced in -dependency specifications: - -1. ``variant_namespaces`` corresponding to the set of namespaces of all - the variant properties that the wheel variant was built for. -2. ``variant_features`` corresponding to the set of - ``namespace :: feature`` pairs of all the variant properties that the - wheel variant was built for. -3. ``variant_properties`` corresponding to the set of - ``namespace :: feature :: value`` tuples of all the variant - properties that the wheel variant was built for. -4. ``variant_label`` corresponding to the exact variant label that the - wheel was built with. For the non-variant wheel, it is an empty +` are introduced in +dependency specifications. Unlike the markers defined by the +:ref:`packaging:dependency-specifiers` specification, their values are +not the same for every wheel: they are scoped to the variant properties +that the wheel being processed was built for. They MUST be obtained as +described in `evaluating variant markers`_. These are: + +1. ``variant_label``: a string, expressing the exact variant label of + the wheel being processed. For non-variant wheels, it is an empty string. +2. ``variant_properties``: a set of all + ``namespace :: feature :: value`` tuples of the properties + corresponding to the variant label that are compatible with the + target system. For non-variant wheels, it is an empty set. +3. ``variant_features``: a set of all ``namespace :: feature`` pairs + corresponding to the properties in ``variant_properties``. +4. ``variant_namespaces``: a set of all namespaces of all the properties + in ``variant_properties``. -The markers evaluating to sets of strings MUST be matched via the ``in`` -or ``not in`` operator, e.g.: +``variant_label`` is a ``String`` field, while ``variant_properties``, +``variant_features`` and ``variant_namespaces`` are ``Set of String`` +fields. The operators available for each, and their semantics, are those +defined for the respective field types in +:ref:`packaging:dependency-specifiers`. -.. code:: +Implementations MUST ignore differences in whitespace around the ``::`` +separators when matching features and properties. - # satisfied by any "foo :: * :: *" property - dep1; "foo" in variant_namespaces - # satisfied by any "foo :: bar :: *" property - dep2; "foo :: bar" in variant_features - # satisfied only by "foo :: bar :: baz" property - dep3; "foo :: bar :: baz" in variant_properties -The ``variant_label`` marker is a plain string: +Evaluating variant markers +'''''''''''''''''''''''''' + +The variant markers MUST only be used in dependency specifiers and MUST +NOT take part in selecting a wheel. These markers gate the individual +dependency specifiers of a wheel that has already been selected. They +MUST be evaluated only once variant wheel selection, as described in +`variant ordering`_, has taken place. + +Their values MUST be determined as follows: + +1. ``variant_label`` is the variant label of the selected wheel, as + found in its filename. + +2. The properties that the ``variants`` dictionary of the `variant + metadata`_ maps to that label are taken, and expanded into + ``namespace :: feature :: value`` triples for every listed feature + value. Either the `variant metadata`_ contained in the wheel itself + or the `index-level metadata`_ MAY be used for this, as the + consistency requirements guarantee that the two agree. + +3. The set of expanded triples is filtered to the variant properties + that the target system supports, as already determined during variant + wheel selection. ``variant_properties`` is the result. Since a wheel + can only be selected if the target system supports it, and that + requires at least one supported value for every feature the wheel + declares, this step never removes a feature entirely. + +4. ``variant_features`` and ``variant_namespaces`` are derived from + ``variant_properties``. + +For non-variant wheels, ``variant_label`` is an empty string and the +three set-valued markers are empty sets. No variant metadata is needed +in order to evaluate variant markers for such wheels. + + +Example +''''''' + +The following dependency specifiers illustrate the four markers and the +operators permitted with each of them: .. code:: # satisfied by the variant "foobar" - dep4; variant_label == "foobar" - # satisfied by any wheel other other than the null variant + dep1; variant_label == "foobar" + # satisfied by any wheel other than the null variant # (including the non-variant wheel) - dep5; variant_label != "null" + dep2; variant_label != "null" # satisfied by the non-variant wheel - dep6; variant_label == "" + dep3; variant_label == "" + # satisfied by any "foo :: * :: *" property + dep4; "foo" in variant_namespaces + # satisfied by any "foo :: bar :: *" property + dep5; "foo :: bar" in variant_features + # satisfied only by "foo :: bar :: baz" property + dep6; "foo :: bar :: baz" in variant_properties + # equivalent + dep7; "foo::bar::baz" in variant_properties -Implementations MUST ignore differences in whitespace while matching the -features and properties. +The filtering in step 3 is only observable where a variant feature lists +multiple values, of which the target system needs to support just one +(see `variant properties`_). Consider a wheel whose properties include +both of: -Variant marker expressions MUST be evaluated against the variant -properties stored in the wheel being installed. +.. code:: text + + nvidia :: sm_arch :: 120_real + nvidia :: sm_arch :: 110_real + +On a system that provides both architectures, ``variant_properties`` +contains both properties. On a system that provides only the former, it +contains ``nvidia :: sm_arch :: 120_real`` alone, and a dependency +specifier testing for ``nvidia :: sm_arch :: 110_real`` is therefore not +satisfied. Integration with pylock.toml @@ -650,14 +759,29 @@ as wheels with different Platform compatibility tags: either all variant (and non-variant) wheels can be listed, or a subset of them. A new ``[packages.variants-json]`` subtable is added to the file. It -MUST inline the contents of the `index-level metadata`_ file, converting -the JSON structure into the respective TOML types. The ``$schema`` key -MUST be preserved to facilitate versioning. The tools MAY remove keys -that are not relevant to variant wheels present in ``pylock.toml``. +MUST inline combined variant metadata, following the same format as the +`index-level metadata`_ file, converting the JSON structure into the +respective TOML types. The ``$schema`` key MUST be preserved to +facilitate versioning. The tools MAY remove the entries of the +``variants`` dictionary whose labels do not occur among the wheels +listed for the package, along with any namespaces that are thereby left +unused in ``default-priorities.namespace``. The entry for every label +that does occur MUST be retained in full, as `evaluating variant +markers`_ requires the complete set of properties corresponding to the +selected label. If variant wheels are listed, the tool SHOULD resolve variants to select the best wheel file. +Variant `environment markers`_ occurring in the dependency specifiers of +a locked package are evaluated in the context of the wheel selected for +that package's entry, as described in `evaluating variant markers`_. No +further context is needed in the lock file: the markers are evaluated +only once that wheel has been selected, and every package entry resolves +its own wheel. Since variant markers may only be used in dependency +specifiers, they cannot occur in ``packages.marker`` or in the top-level +``environments`` key, neither of which is scoped to a selected wheel. + Proposed specification update ''''''''''''''''''''''''''''' @@ -665,8 +789,8 @@ Proposed specification update The proposed text for :doc:`packaging:specifications/pylock-toml` follows: -``[packages.variant_json]`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +``[packages.variants-json]`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - **Type**: table - **Required?**: no @@ -676,8 +800,8 @@ follows: - The structure of this table MUST conform to the JSON Schema. - Tools that support variant-aware resolution MUST validate this table against the referenced schema. - - Tools that do not support variant-aware resolution MAY ignore this table - but SHOULD preserve it when rewriting the lock file. + - Tools that do not support variant-aware resolution MAY ignore this + table but SHOULD preserve it when rewriting the lock file. Example @@ -700,13 +824,13 @@ Example { url = "https://pypi.anaconda.org/mgorny/simple/numpy/2.3.4/numpy-2.3.4-cp314-cp314-macosx_13_0_x86_64-openblas.whl", hashes = {} }, ] - [packages.variant_json] - "$schema" = "https://variants-schema.wheelnext.dev/peps/825/v0.1.0.json" + [packages.variants-json] + "$schema" = "https://variants-schema.wheelnext.dev/peps/825/v0.1.1.json" - [packages.variant_json.default-priorities] + [packages.variants-json.default-priorities] namespace = [ "x86_64", "aarch64", "blas_lapack" ] - [packages.variant_json.variants] + [packages.variants-json.variants] null = { } x86_64_v3_openblas = { "blas_lapack" = { "library" = ["openblas"]}, "x86_64" = { "level" = ["v3"]} } x86_64_v4_mkl = { "blas_lapack" = { "library" = ["mkl"]}, "x86_64" = { "level" = ["v4"]} } @@ -727,14 +851,19 @@ behavior would be to: 3. Filter available wheels based on Platform Compatibility Tags. 4. Determine if any of the remaining wheels are variant wheels. If not, proceed as with non-variant wheels. -5. If any wheels feature a `variant label`_, download the `index-level - metadata`_ file, ``{name}-{version}-variants.json``. If this - file is missing, assume all variant wheels are incompatible and - proceed as with non-variant wheels. +5. If any wheels feature a `variant label`_, obtain the combined variant + metadata. Normally this means downloading the `index-level + metadata`_ file, ``{name}-{version}-variants.json``. If the source + does not provide that file, the combined metadata can instead be read + from the candidate wheels. Only one wheel per distinct variant label + needs to be inspected, and only the `variant metadata`_ file within + it, rather than the whole wheel. Where this is still too expensive, a + tool may treat the variant wheels as incompatible and proceed with + the non-variant wheels instead. 6. Map the variant labels into sets of variant properties using the - index-level variant metadata file. If any of the labels present in - wheel filenames are missing in the file, assume that the respective - wheels are incompatible. + combined variant metadata. If any of the labels present in wheel + filenames are missing from it, assume that the respective wheels are + incompatible. 7. Obtain the ordered lists of compatible variant properties. The mechanism for this will be specified in a subsequent PEP. 8. Filter and order variants based on the lists of compatible @@ -744,18 +873,22 @@ behavior would be to: 9. If multiple wheels for a given version share the same variant label, order them by Platform compatibility tags and build number, and select the best wheel. +10. Read the dependencies of the selected wheel, and evaluate the + `environment markers`_ occurring in them, using the label of the + selected wheel and those of its variant properties that the target + system supports, as described in `evaluating variant markers`_. -Note that steps 4. through 8. are introduced specifically for variant +Note that steps 4 through 8 are introduced specifically for variant wheels. The remaining steps correspond to the current installer -behavior. +behavior. Step 10 is modified through the presence of new environment +markers. -When installing from a source that does not provide an `index-level -metadata`_, the same algorithm can be used, except that the variant -metadata needs to be read directly from the wheels. +The same algorithm applies to sources other than an index, such as a +local directory of wheels. -Installing a local wheel -'''''''''''''''''''''''' +Installing a specific local wheel +''''''''''''''''''''''''''''''''' When asked to install a local wheel file, the proposed behavior would be to: @@ -768,6 +901,10 @@ to: 4. Obtain the ordered lists of compatible variant properties. The mechanism for this will be specified in a subsequent PEP. 5. Verify the wheel compatibility via compatible properties. +6. Read the dependencies of the wheel, and evaluate the `environment + markers`_ occurring in them, using the label of the wheel and those + of its variant properties that the target system supports, as + described in `evaluating variant markers`_. Publishing variant wheels on an index @@ -783,20 +920,202 @@ If the index is responsible for generating the file, it should use some mechanism to defer publishing it until the release is fully uploaded (for example, :pep:`694`). -To generate the ``{name}-{version}-variants.json`` file: +To generate the ``{name}-{version}-variants.json`` file, take the +``*.dist-info/variant.json`` files of all the variant wheels for a given +package version and combine them, as specified for the individual keys +in the `default priorities`_ and `variants`_ sections. The result does +not depend on the order in which the wheels are processed. + + +Installation example (non-normative) +------------------------------------ -1. For the first variant wheel for a given package version, copy the - data from its ``*.dist-info/variant.json`` file. -2. For subsequent wheels, merge the data from their - ``*.dist-info/variant.json`` files into the existing data: +Let's say that PyTorch publishes a number of variant wheels: + +.. code-block:: text + + torch-2.13.0-{py}-{abi}-{platform}-cuda12.6.whl + ^^^^^^^^ + torch-2.13.0-{py}-{abi}-{platform}-cuda13.0.whl + ^^^^^^^^ + torch-2.13.0-{py}-{abi}-{platform}-cuda13.2.whl + ^^^^^^^^ + torch-2.13.0-{py}-{abi}-{platform}-rocm7.2.whl + ^^^^^^^ + torch-2.13.0-{py}-{abi}-{platform}-null.whl # CPU-only + ^^^^ + +The highlighted filename parts are the variant label. + +Each of these wheels carries a variant metadata file that contains: + +.. code:: json5 + + { + "$schema": "https://variants-schema.wheelnext.dev/peps/825/v0.1.1.json", + "default-priorities": { + "namespace": ["nvidia", "amd"] + }, + "variants": { + // ... + } + } - - disjoint keys of ``variants`` dictionary are merged together - - common keys of these dictionaries must have exactly the same value - - ``default-priorities.namespace`` list can be replaced if the new - value starts with the old value - - ``default-priorities.feature`` and ``default-priorities.value`` - keys can be added if they were not present in the previous - ``default-priorities.namespace`` value +In every wheel, the ``variants`` dictionary contains a single key that +is the variant label and whose value lists all properties corresponding +to that label. For example, the ``cuda*`` variant wheels contain +properties expressing the compatibility with NVIDIA GPUs, whereas +``rocm*`` variant wheels the compatibility with AMD GPUs. The ``null`` +variant has no properties. + +In addition to these variant wheels, a ``torch-2.13.0-variants.json`` +file is published with the variant metadata merged from individual +wheels. It has the same contents as the example, except that the +``variants`` dictionary includes all variant labels and their +properties. + +When a package manager is requested to install ``torch``, in order: + +1. The index is queried to determine available wheels. It is + established that 2.13.0 is the newest version and it is selected. + +2. The available 2.13.0 wheels are filtered by Platform compatibility + tags. Only wheels that are compatible with the current system remain. + +3. The ``torch-2.13.0-variants.json`` file is found in the index + response and it is downloaded. Its contents are read to determine the + mapping from variant labels to sets of variant properties, as well as + the namespace preference order. + +4. The lists of compatible features and feature values are obtained for + all namespaces that are used in the ``variants`` dictionary. Wheels + with variant labels corresponding to properties that aren't on these + lists are incompatible and are filtered out. + + For example: + + - The ``cuda*`` labels map to a + ``nvidia :: cuda_version_lower_bound`` property whose value + specifies the minimum CUDA driver version, and + ``nvidia :: sm_arch`` properties whose values list supported GPUs. + The installer queries the driver (if available) to determine + whether a compatible runtime and one of the compatible GPUs are + available. If it cannot find the driver, a compatible runtime + version or a compatible GPU, the wheels are removed from the list. + + - Similarly, the ``rocm*`` labels map to a ``amd :: rocm_version`` + property that specifies the supported ROCm version and + ``amd :: gfx_arch`` properties that list supported GPUs. The + installer queries the appropriate driver in a similar manner as + before. If it cannot find the driver, a compatible runtime version + or a compatible GPU, the wheels are removed from the list. + + - The ``null`` label always corresponds to an empty property set, + and it is therefore always compatible. + + On a system with a compatible NVIDIA GPU and a CUDA runtime, at least + one of the ``cuda*`` wheels and the ``null`` wheels will remain on + the list. + +5. If variant wheels with multiple different labels remain on the list, + the results are sorted per the algorithm in `variant ordering`_. + The most preferred label is selected. + + For example, in this case the ``cuda*`` wheels are ordered by their + properties, using the lists obtained in step 4. The wheels for CUDA + 13 will sort before the ones for CUDA 12, as newer CUDA versions are + preferred. The ``null`` wheel always sorts last, so it would be + selected only if none of the GPU wheels were compatible. + +6. If at this point multiple wheels with the same label remain on the + list, the final selection is performed based on Platform + compatibility tags. This is a rare occurrence and it does not apply + here. + + It could happen, for example, if the same wheel variant was + provided both with ``abi3`` and ``cp315`` tags. When installing for + Python 3.15, the installer would choose between these two based on + the tag. + +7. The metadata for the selected wheel is processed. The variant + properties corresponding to the wheel, narrowed to those the system + was found to support in step 4, are used to process `environment + markers`_. This results in additional CUDA-related dependencies being + selected. + + Suppose the wheel also depends on a library that supports only one + of the architectures the wheel supports: + + .. code:: text + + fast-gemm; "nvidia :: sm_arch :: 120_real" in variant_properties + + On a system with an older GPU, ``120_real`` is not among the + supported properties found in step 4, so the narrowing removes it and + this dependency is not selected. + +8. The wheel is downloaded and installed. + + +Installing wheels from multiple sources (non-normative) +------------------------------------------------------- + +As of the time of writing, there are no accepted standards addressing +the support for installing packages from multiple sources, and the +existing tools differ on the exact behavior. This problem is described +in more detail in the informational :pep:`766`. Variant wheels extend +these differences. The consistency requirements that make variant +metadata combinable hold within a single source tree, and nothing +obliges independent publishers to meet them with respect to one another: +the same variant label may map to different properties, and namespace +orderings chosen independently need not be extensions of one another. +Establishing whether the metadata from two sources happens to be +combinable is possible, but the cost of doing so in the general case is +prohibitive, and there is no correct answer when it is not. For these +reasons, the specification does not attempt to standardize a behavior, +but instead considers it implementation-defined and provides a few +non-normative, suggested solutions, including using non-variant wheel. + +It is entirely valid for tools not to support this behavior, either by +not providing support for using multiple sources at all, or by rejecting +to proceed if more than one of the sources includes variant wheels. + +When processing variant wheels from different sources, it is recommended +to consider their variant metadata in isolation. Matching variant labels +or namespaces do not establish that they have the same meaning across +sources. + +A tool that searches sources in priority order (for example, the "index +priority" in :pep:`766`) can order variants from one source at a time +using the `variant ordering`_ algorithm, and proceed to the next source +only if the current one has no viable candidates. No cross-source +metadata merge is necessary. + +A tool that collates candidates from all sources before selecting among +them (for example, "version priority" in :pep:`766`) must compare wheels +from different sources directly. For a package version, if exactly one +source provides compatible variant wheels, those variants can be +selected ahead of non-variant wheels, as generally variant wheels are +preferred over non-variant wheels. If multiple sources provide +compatible variants, their metadata can be combined unambiguously only +when the metadata is consistent as specified in the sections +corresponding to the individual metadata keys. + +When metadata cannot be combined unambiguously, there is no uniquely +correct global ordering. Valid tool-specific choices include: + +- refusing to install with an error +- printing a warning and ignoring variant wheels, falling back to + selection among non-variant wheels +- using a deterministic criterion for selecting among multiple + compatible variant wheels, for example by preferring the wheel from + the index that came earlier in the option arguments +- requesting additional input from the user + +These choices can produce a valid installation without guaranteeing a +globally optimal selection. Silently fabricating a combined order or +resolving conflicting label mappings is technically possible, but +discouraged: it fabricates semantics that no source declared. Rationale @@ -831,6 +1150,15 @@ issues because of their length, the property lists are stored inside the wheel and mapped to a short label that is chosen by the package maintainer and intended to be human-readable. +Variant metadata is stored in an additional JSON format file rather than +being added to the :doc:`packaging:specifications/core-metadata`. It is +versioned independently, and tools that are not specifically concerned +about variant metadata can ignore its compatibility rules. Its +versioning is similar in spirit to that of Core Metadata. JSON provides +a more convenient format for structured data, as well as more natural +conversion from TOML (so that the data can be sourced from +``pyproject.toml``). + Wheel filenames alone do not provide sufficient metadata to drive variant wheel selection. To avoid tools having to fetch the variant metadata straight from multiple wheel files, the metadata from wheels @@ -884,9 +1212,7 @@ provided. However, namespaces are governed independently and considered on equal footing, and therefore there will be no standard ordering for them. Instead, the ordering of namespaces will be explicitly stated in the variants metadata, which in turn will be provided by the package -maintainer as part of the build process. For completeness, it will also -be possible to provide overrides for the ordering of features and values -via the same mechanism. +maintainer as part of the build process. In the vast majority of real use cases, ordering based on properties will suffice. However, in a pathological case two different variant @@ -910,6 +1236,73 @@ due to the variant metadata being updated or being generated in a way that does not guarantee stable bytewise output. +Variant environment markers +--------------------------- + +Variant properties take part in installing a variant wheel at two +distinct points, and only the second of them concerns markers. An +installer first selects a package version, filters the wheels for that +version by Platform compatibility tags, and then filters and orders the +remaining variant wheels using their variant properties, checked against +the properties that the target system supports; tools may override the +resulting choice. Only afterwards are the dependencies of the selected +wheel read, and the variant markers occurring in them evaluated. + +Filtering and ordering wheels is therefore driven by variant properties, +not by markers. Markers never gate the selection of a wheel; they only +gate individual dependency specifiers, and they are evaluated only once +a wheel has been selected. Were it otherwise, a marker would have to be +evaluated before the wheel whose properties it refers to was known. + +The values of the three set-valued markers are filtered to the variant +properties that the target system supports. A wheel lists every value it +runs on, so without that step a dependency gated on one of them would be +installed wherever the wheel runs. A wheel built for GPU architectures +from ``80_real`` to ``120_real`` may depend on a library that supports +only ``120_real``, much as a dependency that supports a single CPU +architecture is gated on ``platform_machine``. Where a dependency does +exist for every value of a feature, it can instead be published as a +variant package and depended upon unconditionally, leaving the choice to +`variant ordering`_. + +Because of this filtering, the variant markers do not depart from the +established meaning of an environment marker. Every property in +``variant_properties`` is by construction supported by the target +system, so the three set-valued markers describe the environment, just +as the markers defined in :ref:`packaging:dependency-specifiers` do. +What is specific to variant wheels is the vocabulary rather than the +semantics. The existing markers expose a fixed set of environment +attributes to every wheel alike, whereas a variant property has to be +declared by the wheel before it can be observed at all: the wheel's +`variant metadata`_ determines which part of the environment its +dependency specifiers are able to see. ``variant_label`` is the +exception, as it names the selected variant and says nothing about the +environment beyond the fact that this variant was deemed compatible. + +Three consequences of this design are worth noting: + +- Filtering never removes an entire feature or namespace, since a wheel + can only be selected if the target system supports at least one value + for every feature it declares (see `variant properties`_). + ``variant_features`` and ``variant_namespaces`` therefore always list + every feature and namespace the wheel was built for. This is why the + overrides permitted in `variant ordering`_ may not reach past the + compatibility filter: a wheel selected in spite of being unsupported + could have its properties filtered away, and the dependencies gated on + them would silently disappear. + +- For the null variant, ``variant_label`` is ``"null"`` and the three + set-valued markers are empty sets, as the null variant has zero + properties. ``variant_label`` is consequently the only marker that + distinguishes a null variant from a non-variant wheel. + +- A marker referring to a property, feature or namespace that the wheel + does not declare cannot be satisfied in any environment, since + filtering only ever removes properties. Such markers can therefore be + resolved at build time, which is what makes the partial evaluation + described in `Backwards Compatibility`_ possible. + + Backwards Compatibility ======================= @@ -1061,6 +1454,52 @@ justifies the benefit of more compact, and slightly more consistent naming. +Removing ordering information from wheel files +---------------------------------------------- + +The specification proposes that all the data needed to order wheel +variants is stored within the variant metadata (the `default +priorities`_ dictionary). This data is expected to originate from a +common source per project (a subsequent PEP will propose an integration +within ``pyproject.toml`` file) and to be copied into every variant +wheel at build time, and afterwards into the `index-level metadata`_ +file. + +It has been argued that this makes the ordering data a wheel-level +property which, once inserted into a particular wheel, says something +about other wheels. That is not the case. The data is project-level +metadata that is copied into the wheel, and like the other project +metadata carried there, it does not reference other wheels. The ordering +data can specify namespaces for which no variant wheels exist in a +particular release, and nothing in the format depends on the presence of +any specific other wheel. + +Furthermore, this design specifically ensures that the index-level +metadata file is a cache rather than a first-order data source. Since +all the data is stored in the wheels: + +1. It is possible to generate the index-level metadata file based on + available wheels alone, without any additional input data. Notably, + this empowers indexes to generate it automatically without requiring + any changes to the tools or workflows used to upload wheels. + +2. It is possible to select across multiple wheels even in absence of + index-level metadata file, for example when installing from a local + directory. + +There is indeed a real risk that two wheels built at different times +and with different tooling may end up having inconsistent metadata. +However, the specification requires consistency and makes the publisher +of the package version responsible for it, as described in `metadata +consistency`_. + +It has also been suggested that detaching the ordering data from variant +metadata would make it possible for tools to accept an override of that +data in a standard format. However, there is no relation between the +two; such a format can be introduced either way, for example using a +subset of variant metadata. + + Out of scope ------------ @@ -1071,12 +1510,22 @@ The following problems are deferred to subsequent PEPs in the series: - overriding the compatibility detection using static data - building variant wheels -In addition to that, the following matters are left -implementation-defined: -- Selecting variant wheels from multiple sources. Currently, there is no - standard defined behavior for regular wheels, nor consensus across - different packaging tools on how to handle that. +Open Issues +=========== + +These questions must be resolved before this PEP can be accepted. + +Use of variant environment markers +---------------------------------- + +The design of the variant `environment markers`_ is not yet settled. The +same effect can be achieved through Dynamic dependencies; the open +question is whether markers are the right mechanism for obtaining it +while keeping dependency metadata static across a release. This is under +discussion in `this thread +`__. +The specification reflects the current design. Acknowledgements @@ -1097,6 +1546,34 @@ and Zanie Blue. Change History ============== +- 10-Aug-2026 + + - Decoupled most of the specification from `index-level metadata`_, + clarifying that it is only an optimization for scenarios where + wheels are published on an index. + - Added non-normative guidance for installing variant wheels from + multiple sources. + - Added an explicit "Implementation requirements" section. + - Clarified the scope of the individual variant metadata keys, and + stated the consistency requirements for each of them alongside. + - Added "Removing ordering information from wheel files" to rejected + ideas. + - Removed ``default-priorities.feature`` + and ``default-priorities.property``. + - Made schema versioning use semantic versioning, with its backwards + compatibility implications. + - Improve environment marker content. Make ``variant_properties`` + marker use variant properties compatible with the system rather + than all the properties specified in the metadata. + - Update pylock.toml section to explain environment marker usage. + - Various smaller fixes for language and design consistency. + +- 11-May-2026 + + - Added replacing platform compatibility tags entirely to rejected + ideas. + - Clarified interpretation of sorting algorithm and index support. + - 06-Apr-2026 - Added a formal requirement that Python tags must not start with @@ -1123,12 +1600,6 @@ Change History - Changed ``pylock.toml`` integration to inline variant metadata rather than storing a URL and a hash. -- 11-May-2026 - - - Added replacing platform compatibility tags entirely to rejected - ideas. - - Clarified interpretation of sorting algorithm and index support. - Appendices ========== diff --git a/peps/pep-0825/appendix-variant-json-schema.rst b/peps/pep-0825/appendix-variant-json-schema.rst index 6505ba74357..082374e286b 100644 --- a/peps/pep-0825/appendix-variant-json-schema.rst +++ b/peps/pep-0825/appendix-variant-json-schema.rst @@ -5,7 +5,7 @@ Appendix: JSON Schema for Variant Metadata ========================================== -.. literalinclude:: variant-schema-0.1.0.json +.. literalinclude:: variant-schema-0.1.1.json :language: json :linenos: :name: variant-json-schema diff --git a/peps/pep-0825/variant-schema-0.1.0.json b/peps/pep-0825/variant-schema-0.1.1.json similarity index 51% rename from peps/pep-0825/variant-schema-0.1.0.json rename to peps/pep-0825/variant-schema-0.1.1.json index 2b792fd1a8f..82288a313df 100644 --- a/peps/pep-0825/variant-schema-0.1.0.json +++ b/peps/pep-0825/variant-schema-0.1.1.json @@ -1,7 +1,7 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://variants-schema.wheelnext.dev/peps/825/v0.1.0.json", - "title": "Variant metadata, v0.1.0", + "$id": "https://variants-schema.wheelnext.dev/peps/825/v0.1.1.json", + "title": "Variant metadata, v0.1.1", "description": "The format for variant metadata (variant.json) and index-level metadata ({name}-{version}-variants.json)", "type": "object", "properties": { @@ -22,54 +22,9 @@ }, "minItems": 1, "uniqueItems": true - }, - "feature": { - "description": "Default feature priorities (by namespace)", - "type": "object", - "patternProperties": { - "^[a-z0-9_]+$": { - "description": "The most preferred features", - "type": "array", - "items": { - "type": "string", - "pattern": "^[a-z0-9_]+$" - }, - "minItems": 0, - "uniqueItems": true - } - }, - "additionalProperties": false, - "uniqueItems": true - }, - "property": { - "description": "Default property priorities (by namespace)", - "type": "object", - "patternProperties": { - "^[a-z0-9_]+$": { - "description": "Default property priorities (by feature name)", - "type": "object", - "patternProperties": { - "^[a-z0-9_]+$": { - "description": "The most preferred feature values", - "type": "array", - "items": { - "type": "string", - "pattern": "^[a-z0-9_.]+$" - }, - "minItems": 0, - "uniqueItems": true - } - }, - "additionalProperties": false, - "uniqueItems": true - } - }, - "additionalProperties": false, - "uniqueItems": true } }, "additionalProperties": false, - "uniqueItems": true, "required": [ "namespace" ] @@ -78,14 +33,15 @@ "description": "Mapping of variant labels to properties", "type": "object", "patternProperties": { - "^[a-z0-9_.]{1,16}$": { + "^[a-z0-9_.]+$": { "type": "object", "description": "Mapping of namespaces in a variant", "patternProperties": { - "^[a-z0-9_.]+$": { + "^[a-z0-9_]+$": { "description": "Mapping of feature names in a namespace", + "type": "object", "patternProperties": { - "^[a-z0-9_.]+$": { + "^[a-z0-9_]+$": { "description": "List of values for this variant feature", "type": "array", "items": { @@ -96,16 +52,13 @@ "uniqueItems": true } }, - "uniqueItems": true, "additionalProperties": false } }, - "uniqueItems": true, "additionalProperties": false } }, - "additionalProperties": false, - "uniqueItems": true + "additionalProperties": false } }, "required": [ @@ -113,6 +66,5 @@ "default-priorities", "variants" ], - "additionalProperties": false, - "uniqueItems": true + "additionalProperties": false }