Renaming identity column with declarative schema leads to SQL error

Question

I have the following db_schema.xml:

<?xml version="1.0"?>
<schema xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Setup/Declaration/Schema/etc/schema.xsd">
    <table name="vendor_document" comment="VENDOR documents available to download by customers" charset="utf8"
           collation="utf8_general_ci">
        <column name="id" xsi:type="int" unsigned="true" identity="true" nullable="false" comment="Document ID"/>
        <column name="filename" xsi:type="varchar" nullable="false" length="255" comment="Filename of the document"/>
        <column name="category" xsi:type="varchar" nullable="false" length="255"
                comment="Document category the document is assigned to"/>
        <column name="created_at" xsi:type="timestamp" default="CURRENT_TIMESTAMP" nullable="false"
                comment="Document created at"/>
        <column name="updated_at" xsi:type="timestamp" default="CURRENT_TIMESTAMP" nullable="false" on_update="true"
                comment="Document updated at"/>

        <constraint xsi:type="primary" referenceId="PRIMARY">
            <column name="id"/>
        </constraint>
    </table>

    <table name="vendor_document_store" comment="VENDOR Document To Store Linkage Table">
        <column name="document_id" xsi:type="int" unsigned="true" nullable="false" comment="Document ID"/>
        <column xsi:type="smallint" name="store_id" padding="5" unsigned="true" nullable="false" identity="false"
                comment="Store Id"/>

        <constraint xsi:type="primary" referenceId="PRIMARY">
            <column name="document_id"/>
            <column name="store_id"/>
        </constraint>
        <constraint xsi:type="foreign" referenceId="VENDOR_DOCUMENT_STORE_DOCUMENT_ID_VENDOR_DOCUMENT_ID"
                    column="document_id" referenceTable="vendor_document" referenceColumn="id" onDelete="CASCADE"/>
        <constraint xsi:type="foreign" referenceId="VENDOR_DOCUMENT_STORE_STORE_ID_STORE_STORE_ID" column="store_id"
                    referenceTable="store" referenceColumn="store_id" onDelete="CASCADE"/>
    </table>
</schema>

I am implementing the relation between the model like explained here. To get the store ids associated with the document I use \Magento\Framework\EntityManager\EntityMetadataInterface::getLinkField to get the the link field but because my primary key column is called "id" Magento generates a wrong query.

I now want to change the identity column name to "document_id" so Magento generates the correct query. I changed the db_schema.xml resulting like this:

<?xml version="1.0"?>
<schema xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:Setup/Declaration/Schema/etc/schema.xsd">
    <table name="vendor_document" comment="VENDOR documents available to download by customers" charset="utf8"
           collation="utf8_general_ci">
        <column name="document_id" xsi:type="int" unsigned="true" identity="true" nullable="false" comment="Document ID"/>
        <column name="filename" xsi:type="varchar" nullable="false" length="255" comment="Filename of the document"/>
        <column name="category" xsi:type="varchar" nullable="false" length="255"
                comment="Document category the document is assigned to"/>
        <column name="created_at" xsi:type="timestamp" default="CURRENT_TIMESTAMP" nullable="false"
                comment="Document created at"/>
        <column name="updated_at" xsi:type="timestamp" default="CURRENT_TIMESTAMP" nullable="false" on_update="true"
                comment="Document updated at"/>

        <constraint xsi:type="primary" referenceId="PRIMARY">
            <column name="document_id"/>
        </constraint>
    </table>

    <table name="vendor_document_store" comment="VENDOR Document To Store Linkage Table">
        <column name="document_id" xsi:type="int" unsigned="true" nullable="false" comment="Document ID"/>
        <column xsi:type="smallint" name="store_id" padding="5" unsigned="true" nullable="false" identity="false"
                comment="Store Id"/>

        <constraint xsi:type="primary" referenceId="PRIMARY">
            <column name="document_id"/>
            <column name="store_id"/>
        </constraint>
        <constraint xsi:type="foreign" referenceId="VENDOR_DOCUMENT_STORE_DOCUMENT_ID_VENDOR_DOCUMENT_DOCUMENT_ID"
                    column="document_id" referenceTable="vendor_document" referenceColumn="document_id" onDelete="CASCADE"/>
        <constraint xsi:type="foreign" referenceId="VENDOR_DOCUMENT_STORE_STORE_ID_STORE_STORE_ID" column="store_id"
                    referenceTable="store" referenceColumn="store_id" onDelete="CASCADE"/>
    </table>
</schema>

I pretty much just renamed "id" to "document_id" and updated the referenceIds on the foreign constraints. But when running setup:upgrade it produces the following error:

SQLSTATE[42000]: Syntax error or access violation: 1075 Incorrect table definition; there can be only one auto column and it must be defined as a key, query was:
ALTER TABLE `vendor_document` MODIFY COLUMN `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP  COMMENT "Document created at",
MODIFY COLUMN `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT "Document updated at",
ADD COLUMN `document_id` int(11) UNSIGNED NOT NULL  AUTO_INCREMENT COMMENT "Document ID", AUTO_INCREMENT = 1,
DROP PRIMARY KEY,
ADD CONSTRAINT  PRIMARY KEY (`document_id`)

Am I doing something wrong or is it a bug of Magento not generating the SQL properly?

Update:
The error above was because I regenerated the db_schema_whitelist.json so the "id" entry was missing and Magento wasn't allowed to drop the table. I added the entry again and Magento tries to drop the column now but now I get the following error:

SQLSTATE[HY000]: General error: 1829 Cannot drop column 'id': needed in a foreign key constraint 'vendor_shop/VENDOR_DOCUMENT_STORE_DOCUMENT_ID_VENDOR_DOCUMENT_ID' of table `vendor_shop`.`vendor_document_store`, query was:
ALTER TABLE `vendor_document` MODIFY COLUMN `created_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP  COMMENT "Document created at",
MODIFY COLUMN `updated_at` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP COMMENT "Document updated at",
ADD COLUMN `document_id` int(11) UNSIGNED NOT NULL  AUTO_INCREMENT COMMENT "Document ID",
AUTO_INCREMENT = 1,
DROP COLUMN `id`,
DROP PRIMARY KEY,
ADD CONSTRAINT  PRIMARY KEY (`document_id`)

Answer 1

The problem was in handling the db_schema_whitelist.json.

The DevDocs state:

As a best practice, you should generate a new whitelist file for each release. You must generate the whitelist in any release that contains changes in the db_schema.xml file.

This statement let to confusion and I thought it should be new generated. But reading a bit above:

The module_vendor/module_name/etc/db_schema_whitelist.json file provides a history of all tables, columns, and keys added with declarative schema. It is required to allow drop operations.

When renaming columns it is important that the db_schema_whitelist.json is not completely regenerated. It needs to contain the current column name and the current constrain references otherwise Magento is not able to drop them.

Instead of deleting the whitelist and generating it again it should be just updated with:

bin/magento setup:db-declaration:generate-whitelist --module-name=Vendor_Module

If there is already a whitelist Magento will just update it accordingly by keeping the old values and adding the new ones.