Custom Drupal Commerce 2.x Migrations

We recently were tasked with migrating a Drupal Commerce 1.x store into Drupal Commerce 2.x. The migration requirements were pretty simple, but did require that we migrate all product, customer, order, and payment data into the new site. This blog contains an overview of all of the steps, hiccups and and resources referenced along the way. The following example repo can be used for as a reference when writing your own migrations. We do not recommend copying this directly as it is not meant to be a drop-in migration module for Drupal Commerce. Feel free to provide merge requests if you find bugs or items that could be useful to others. See https://gitlab.com/blueoakinteractive/boi-migrate for Drupal migration examples referenced in this article. ## Thank you Commerce Guys & Acro Media [The Commerce Guys](https://commerceguys.com/) and [Acro Media](https://www.acromedia.com) have done a great job building [commerce_migrate](https://www.drupal.org/project/commerce_migrate) for Drupal Commerce 2.x to be a base for migrating other e-commerce platforms into [Drupal Commerce](https://www.drupal.org/project/commerce). There are three sub-modules that come with the base module for migrating from Ubercart, Drupal Commerce 1.x (Commerce Kickstart) and WooCommerce. ## Commerce Migrate, Out of the Box The Drupal Commerce migration in commerce_migrate is based on [migrate_upgrade](https://www.drupal.org/project/migrate_upgrade) and can run an entire 1.x to 2.x migration from Commerce Kickstart with just a few Drush commands. ``` // Example provide by Matt Glaman. drush en -y commerce_migrate_commerce # Drush does not rebuild caches when modules installed. Do it for safe measure. drush cr drush migrate-upgrade \ --legacy-db-url=mysql://user:pass@localhost/db_to_migrate \ --legacy-root=http://mystore.com ``` ## Customizing Migrations For our purposes, we needed more control to run migrations individually as well as the ability to only migrate the specific entities that we needed. For example, we did not need old shopping carts, canceled orders or any of the related entities that go along with them. To provide this flexibility, we also looked to [migrate_plus](https://www.drupal.org/project/migrate_plus) and [migrate_tools](https://www.drupal.org/project/migrate_tools), which provide extensions and examples for migrating data into Drupal 8. I highly recommend reviewing the included example modules that come with migrate_plus. ## Getting started The first step in building out your custom migrations is to create a new custom Drupal 8 module (https://www.drupal.org/docs/8/creating-custom-modules). At a minimum, you'll need a [module.info.yml](https://gitlab.com/blueoakinteractive/boi-migrate/blob/master/boi_migra…) file and a config/install directory to contain your migration definitions. In Drupal 8, migrations are based on config entities defined in .yml files. A lot of the heavy lifting required to write Drupal Commerce migrations is provided to you in the migration plugins that come with Drupal core, commerce_migrate and migrate_plus. ## Database Connection It's assumed that you already have a fully installed version of Drupal 8 with the necessary Drupal Commerce modules installed. Next, you'll need to update your settings.php file to contain the database connection details for your legacy (Drupal 7) database. The format for this array is the same as the default settings array, but is keyed with "legacy". You can name this whatever you want, but the name has to be the same as what's defined in your migration group (below). ``` $databases['legacy']['default'] = [ 'database' => 'mysql', 'username' => 'mysql', 'password' => 'mysql', 'prefix' => '', 'host' => 'localhost', 'port' => 3306, 'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql', 'driver' => 'mysql', ]; ``` ## Migration groups The first config file you should create, defines a migration group for all of your commerce migrations. In the [example repo](https://gitlab.com/blueoakinteractive/boi-migrate/blob/master/config/in…), this file is `config/install/migrate_plus.migration_group.boi_commerce.yml`. The filename structure of this and the other migration configurations is important so that migrate_plus and migrate_tools can locate them automatically. #### Migration config filename requirements: * Must be in config/install folder of your custom module. * Must start with migrate_plus. * Allows migrate plus to auto-locate the file. * Must contain either "migration_group" or "migration". * Defines if the file is a group or migration. * The remainder of the filename should match your migration or group id (ie: boi_commerce). * Must end with .yml. In this file, you define the human and machine readable names for your migration group as well as the database configuration to use (from your settings.php). The source.key value (ie: legacy) needs to match what you used in your settings.php. ## Your first migration configuration A new .yml file is required for each entity type that you're migrating into Drupal 8. You'll need to understand the relationship that entities have to one another in your source database. For example, orders belong to users, so you should migrate your user data first. This dependency chain can also be defined in your `migration_dependencies` section of your migration config. #### The configuration file Create a new file called `migrate_plus.migration.user.yml` ([example repo](https://gitlab.com/blueoakinteractive/boi-migrate/blob/master/config/in…)). Again, the structure of your filename is important. It should contain migrate_plus.migration (instead of migration_group) and the id or your migration. One of the awesome things about Drupal 8 migrations is that you can find migration templates for just about any type of entity in the source module's migrate_templates or migrations sub-directory. The example below is copied from Drupal core's `core/user/migration_templates/d7_user.yml` and has been edited, for our purposes, to remove unnecessary definitions. Since migrate is in Drupal 8 core, other core and contributed modules provide their own migration templates and plugins for migrations. We'll get into plugins later, for now, let's review the basic template file for our user migration. ``` // The machine name of your migration. id: user // The human readable name of your migration. label: User migration // The migration group this migration belongs to (important!). migration_group: boi_commerce // Optional tags for organization. migration_tags: - Drupal 7 // Migration plugin for reading from source data. source: plugin: d7_user // Migration plugin for writing to destination data. destination: plugin: entity:user // Field and property value maps (destination: source). process: uid: uid name: name pass: pass mail: mail access: access status: status created: created ``` It it important that the `migration_group` value match the migration group you defined above, so that Drupal migrate knows how to bootstrap your source database. This can also be a useful way to keep your migrations organized. #### Defining the source plugin The `source` property defines which migration source plugin to use to fetch data from the source database. In this example, the `d7_user` plugin (see `core/modules/user/src/Plugin/migrate/source/d7/User.php`) is defined to fetch Drupal 7 user data from the source database. All of the magic is provided for you in the plugin's PHP class and is invoked simply by defining `source: d7_user` in your migration.yml. See https://www.drupal.org/docs/8/api/migrate-api/migrate-source for more information on migration sources. #### Defining the destination plugin Similar to the source plugin, the destination plugins can be invoked the same way. In this example we use the `destination: 'entity:user'`, which tells the migration module to use it's plugins for entity migrations. See https://www.drupal.org/docs/8/api/migrate-api/migrate-destination for more information on migration destinations. #### Processing the data along the way In the process section, we define field and property mappings that will be used to read from the source and write to the destination. It's important to understand that these are defined in reverse order than what may be expected. The destination comes first and the source comes second (destination: source). The examples in this user migration are very simple 1 to 1 mappings that the migration module knows how to handle. For example, the user id of the Drupal user in Drupal 6 will now be the same as it is in the Drupal 8 site (uid: uid). If that is not the desired result, you can omit the uid mapping or define your own plugin for setting the Drupal 8 user id. See https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins for more information on migration processes. ## Migrating other dependent entities The next migration we needed to run was to bring in file entities from Drupal 7. For this, we followed https://drupal.stackexchange.com/a/232462/53118 as an example to do so. This method uses a 1 to 1 mapping for fid (file id), so that, like uid above, the file id in Drupal 8 will be the same as Drupal 7. This allows our other migration to use the same file id and not worry about having to invoke the file migration to determine the new id. If using the example repo, you'll want to update `source_base_path` in the `migrate_plus.migration.file.yml` file to match your Drupal 7 root directory. ## More advanced migration templates You'll probably realize that the templates and classes that come with core and contributed modules get you 90% of the way there. The amount of custom coding required will depend on the complexity of your Drupal 7 source and your Drupal 8 destination, as well as the requirements of your particular project. #### Product attribute entities The new product architecture in Drupal Commerce 2.x abstracts product attributes into their own entity type. For Drupal Commerce 1.x you added fields to your product type to define attributes, typically via taxonomy references. In Drupal Commerce 2.x, you create a new product variation entity which contains all of the options available to that attribute. You then associate a product variation bundle with that attribute type. Our migration contained a physical size attribute called "container". It's possible to migrate the product attribute entity destination in it's own migration (see commerce_migrate for examples), but for simplicity, we created this manually in our Drupal 8 installation. To do the same, visit /admin/commerce/product-attributes in your Drupal Commerce 2.x installation and click "Add product attribute". Take note of the machine name that you pick. In our instance, we named it the same as the taxonomy vocabulary in Drupal 7 to keep the mapping simple. ### Custom source plugins We then needed to migrate the product variation values from a Drupal 7 taxonomy vocabulary into a Drupal Commerce 2.x product attribute. To do this we needed to create a custom source plugin. To do this, we created `boi_migrate/src/Plugin/migrate/source/boi/ProductAttributeValue.php`([example repo](https://gitlab.com/blueoakinteractive/boi-migrate/blob/master/src/Plugi…)), which is new class to extend the `FieldableEntity` class provided by migrate. `FieldableEntity` is an awesome base class that understands how to initialize Drupal 7 entities and all of their field values, which saves a lot of time writing custom queries. Take note of the [annotation](https://www.drupal.org/docs/8/api/plugin-api/annotations-based-plugins) in the comments of the class that defines "@MigrateSource", this tells the migration module that' we're defining a migration source plugin called "boi_product_attribute_value". It also tells the base `FieldableEntity` class that our source object is a taxonomy_term. The important part our our custom source definition is the overridden `query()` method. This is a standard Drupal query object with a query condition based on the configuration defined in our migration yml. `$this->configuration['vocabulary']` references the `source.vocabulary` value defined in the migration configuration (See `migrate_plus.migration.product_attribute_value.yml`). If we had more product attributes like color, we could append those to 'source.vocabulary' in the migration config .yml and they'd automatically be added to the source query. Other custom query plugin examples can be found in the `boi_migrate/src/Plugin/migrate/source/boi` directory including Order.php, Payment.php, and LineItem.php. In those examples, we're extending the migration source plugins that commerce_migrate provides and limiting the results returned by the query. We only wanted to migrate commerce entities related to pending or completed orders. ### Custom process plugins If you realize that the data from your source migration isn't being formatted to your desired output, you can create a custom process plugin. However, I first encourage you to make sure there isn't already provided by the migration module or modules that extend it. See https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins for documentation on process plugins. The [example repo](https://gitlab.com/blueoakinteractive/boi-migrate/blob/master/src/Plugi…) includes a simple process plugin to transform the integer amount provided into the float amount stored by Drupal Commerce 2.x. The code for this transformation lives in `boi_migrate/src/Plugin/migrate/process/boi/Amount.php` which extends the migration module's `ProcessPluginBase` class. Again, the annotation is used to define the name of this plugin as "boi_amount". We override the base classes `transform()` method and convert the amount passed from source into the amount expected by the destination. Without this, payment transactions would come through as 100x their actual amount. Once the plugin is created, rebuilding the cache allows it to be located by the migration plugin helper and we can use it in our migration config. Note that process plugins are chainable, so you can append as many as you need to your process key to get the desired output. ## Updating your migration configuration The migration configurations that have been defined in `config/install/*.yml` are imported into Drupal's configuration system when the module is enabled. When you make changes to these files, they are not updated in Drupal automatically. Since you'll likely be running and re-running your migrations until you get things right, you should use [config_devel](https://www.drupal.org/project/config_devel) (in development only) to re-import your configurations after making changes. Download and enable config_devel. ``` composer require drupal/config_devel --dev drush en config_devel -y``` Re-import the user migration after making changes. The drush command `cdi1` is an alias of `config-devel-import-one` which allows you to re-import one config file from it's filesystem location. ```drush cdi1 modules/boi_migrate/config/install/migrate_plus.migration.user.yml``` ## Summary The level of effort for us to write this migration was, initially, pretty high. We have a lot of experience with Drupal 7 and Commerce 1.x migrations, which in some cases may have made things more difficult. We were used to having the migrate_tools utilities available out of the box and were expecting to have to register the migrations manually. Instead we had to grab the extra module(s) and use the correct naming conventions for the config files and migrate_plus. It was also a challenge initially to determine where to configure the database connect. I expected to put it in settings.php in the standard array format, but it wasn't clear how the migrations would know which database to use. Once we got over the initial hurdles, things fell into place pretty quickly. With the migration module in core, modules providing their own plugins and templates, and commerce_migrate providing a ton of examples, we now have a solid platform to migrate data into Drupal easily. The next migration should be very straight forward for us to execute and should only take a fraction of the time. Hopefully this blog will put you on the fast track to success with your migration! ### Additional Resources * https://www.drupaleasy.com/blogs/ultimike/2016/04/drupal-6-drupal-81x-c… * https://github.com/mglaman/commerce-migrate-environment ### Need Help? Blue Oak Interactive offers Drupal and Drupal Commerce migration consulting and development. [Contact us](/contact) for more information.