Custom Drupal Commerce 2.x Migrations

Custom migrations for Drupal Commerce 2.x using Migrate Plus and Migrate Tools
Mon, August 14th, 2017
andyg5000
Migrate Drupal Commerce

We recently were tasked with migrating a Drupal Commerce 1.x store into Drupal Commerce 2.x. The migration requirements were pretty simple, but did require that we migrate all product, customer, order, and payment data into the new site. This blog contains an overview of all of the steps, hiccups and and resources referenced along the way.

The following example repo can be used for as a reference when writing your own migrations. We do not recommend copying this directly as it is not meant to be a drop-in migration module for Drupal Commerce. Feel free to provide merge requests if you find bugs or items that could be useful to others.

See https://gitlab.com/blueoakinteractive/boi-migrate for Drupal migration examples referenced in this article.

Thank you Commerce Guys & Acro Media

The Commerce Guys and Acro Media have done a great job building commerce_migrate for Drupal Commerce 2.x to be a base for migrating other e-commerce platforms into Drupal Commerce. There are three sub-modules that come with the base module for migrating from Ubercart, Drupal Commerce 1.x (Commerce Kickstart) and WooCommerce.

Commerce Migrate, Out of the Box

The Drupal Commerce migration in commerce_migrate is based on migrate_upgrade and can run an entire 1.x to 2.x migration from Commerce Kickstart with just a few Drush commands.

// Example provide by Matt Glaman.
drush en -y commerce_migrate_commerce
# Drush does not rebuild caches when modules installed. Do it for safe measure.
drush cr
drush migrate-upgrade \
   --legacy-db-url=mysql://user:[email protected]/db_to_migrate \
   --legacy-root=http://mystore.com

Customizing Migrations

For our purposes, we needed more control to run migrations individually as well as the ability to only migrate the specific entities that we needed. For example, we did not need old shopping carts, canceled orders or any of the related entities that go along with them.

To provide this flexibility, we also looked to migrate_plus and migrate_tools, which provide extensions and examples for migrating data into Drupal 8. I highly recommend reviewing the included example modules that come with migrate_plus.

Getting started

The first step in building out your custom migrations is to create a new custom Drupal 8 module (https://www.drupal.org/docs/8/creating-custom-modules).

At a minimum, you'll need a module.info.yml file and a config/install directory to contain your migration definitions. In Drupal 8, migrations are based on config entities defined in .yml files. A lot of the heavy lifting required to write Drupal Commerce migrations is provided to you in the migration plugins that come with Drupal core, commerce_migrate and migrate_plus.

Database Connection

It's assumed that you already have a fully installed version of Drupal 8 with the necessary Drupal Commerce modules installed. Next, you'll need to update your settings.php file to contain the database connection details for your legacy (Drupal 7) database. The format for this array is the same as the default settings array, but is keyed with "legacy". You can name this whatever you want, but the name has to be the same as what's defined in your migration group (below).

$databases['legacy']['default'] = [
  'database' => 'mysql',
  'username' => 'mysql',
  'password' => 'mysql',
  'prefix' => '',
  'host' => 'localhost',
  'port' => 3306,
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
];

Migration groups

The first config file you should create, defines a migration group for all of your commerce migrations. In the example repo, this file is config/install/migrate_plus.migration_group.boi_commerce.yml. The filename structure of this and the other migration configurations is important so that migrate_plus and migrate_tools can locate them automatically.

Migration config filename requirements:

  • Must be in config/install folder of your custom module.
  • Must start with migrate_plus.
    • Allows migrate plus to auto-locate the file.
  • Must contain either "migration_group" or "migration".
    • Defines if the file is a group or migration.
  • The remainder of the filename should match your migration or group id (ie: boi_commerce).
  • Must end with .yml.

In this file, you define the human and machine readable names for your migration group as well as the database configuration to use (from your settings.php). The source.key value (ie: legacy) needs to match what you used in your settings.php.

Your first migration configuration

A new .yml file is required for each entity type that you're migrating into Drupal 8. You'll need to understand the relationship that entities have to one another in your source database. For example, orders belong to users, so you should migrate your user data first. This dependency chain can also be defined in your migration_dependencies section of your migration config.

The configuration file

Create a new file called migrate_plus.migration.user.yml (example repo). Again, the structure of your filename is important. It should contain migrate_plus.migration (instead of migration_group) and the id or your migration.

One of the awesome things about Drupal 8 migrations is that you can find migration templates for just about any type of entity in the source module's migrate_templates or migrations sub-directory. The example below is copied from Drupal core's core/user/migration_templates/d7_user.yml and has been edited, for our purposes, to remove unnecessary definitions. Since migrate is in Drupal 8 core, other core and contributed modules provide their own migration templates and plugins for migrations. We'll get into plugins later, for now, let's review the basic template file for our user migration.

// The machine name of your migration.
id: user

// The human readable name of your migration.
label: User migration

// The migration group this migration belongs to (important!).
migration_group: boi_commerce

// Optional tags for organization.
migration_tags:
  - Drupal 7

// Migration plugin for reading from source data.
source:
  plugin: d7_user

// Migration plugin for writing to destination data.
destination:
  plugin: entity:user

// Field and property value maps (destination: source).
process:
  uid: uid
  name: name
  pass: pass
  mail: mail
  access: access
  status: status
  created: created

It it important that the migration_group value match the migration group you defined above, so that Drupal migrate knows how to bootstrap your source database. This can also be a useful way to keep your migrations organized.

Defining the source plugin

The source property defines which migration source plugin to use to fetch data from the source database. In this example, the d7_user plugin (see core/modules/user/src/Plugin/migrate/source/d7/User.php) is defined to fetch Drupal 7 user data from the source database. All of the magic is provided for you in the plugin's PHP class and is invoked simply by defining source: d7_user in your migration.yml.

See https://www.drupal.org/docs/8/api/migrate-api/migrate-source for more information on migration sources.

Defining the destination plugin

Similar to the source plugin, the destination plugins can be invoked the same way. In this example we use the destination: 'entity:user', which tells the migration module to use it's plugins for entity migrations.

See https://www.drupal.org/docs/8/api/migrate-api/migrate-destination for more information on migration destinations.

Processing the data along the way

In the process section, we define field and property mappings that will be used to read from the source and write to the destination. It's important to understand that these are defined in reverse order than what may be expected. The destination comes first and the source comes second (destination: source). The examples in this user migration are very simple 1 to 1 mappings that the migration module knows how to handle. For example, the user id of the Drupal user in Drupal 6 will now be the same as it is in the Drupal 8 site (uid: uid). If that is not the desired result, you can omit the uid mapping or define your own plugin for setting the Drupal 8 user id.

See https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins for more information on migration processes.

Migrating other dependent entities

The next migration we needed to run was to bring in file entities from Drupal 7. For this, we followed https://drupal.stackexchange.com/a/232462/53118 as an example to do so. This method uses a 1 to 1 mapping for fid (file id), so that, like uid above, the file id in Drupal 8 will be the same as Drupal 7. This allows our other migration to use the same file id and not worry about having to invoke the file migration to determine the new id.

If using the example repo, you'll want to update source_base_path in the migrate_plus.migration.file.yml file to match your Drupal 7 root directory.

More advanced migration templates

You'll probably realize that the templates and classes that come with core and contributed modules get you 90% of the way there. The amount of custom coding required will depend on the complexity of your Drupal 7 source and your Drupal 8 destination, as well as the requirements of your particular project.

Product attribute entities

The new product architecture in Drupal Commerce 2.x abstracts product attributes into their own entity type. For Drupal Commerce 1.x you added fields to your product type to define attributes, typically via taxonomy references. In Drupal Commerce 2.x, you create a new product variation entity which contains all of the options available to that attribute. You then associate a product variation bundle with that attribute type.

Our migration contained a physical size attribute called "container". It's possible to migrate the product attribute entity destination in it's own migration (see commerce_migrate for examples), but for simplicity, we created this manually in our Drupal 8 installation. To do the same, visit /admin/commerce/product-attributes in your Drupal Commerce 2.x installation and click "Add product attribute". Take note of the machine name that you pick. In our instance, we named it the same as the taxonomy vocabulary in Drupal 7 to keep the mapping simple.

Custom source plugins

We then needed to migrate the product variation values from a Drupal 7 taxonomy vocabulary into a Drupal Commerce 2.x product attribute. To do this we needed to create a custom source plugin.

To do this, we created boi_migrate/src/Plugin/migrate/source/boi/ProductAttributeValue.php(example repo), which is new class to extend the FieldableEntity class provided by migrate. FieldableEntity is an awesome base class that understands how to initialize Drupal 7 entities and all of their field values, which saves a lot of time writing custom queries. Take note of the annotation in the comments of the class that defines "@MigrateSource", this tells the migration module that' we're defining a migration source plugin called "boi_product_attribute_value". It also tells the base FieldableEntity class that our source object is a taxonomy_term.

The important part our our custom source definition is the overridden query() method. This is a standard Drupal query object with a query condition based on the configuration defined in our migration yml. $this->configuration['vocabulary'] references the source.vocabulary value defined in the migration configuration (See migrate_plus.migration.product_attribute_value.yml). If we had more product attributes like color, we could append those to 'source.vocabulary' in the migration config .yml and they'd automatically be added to the source query.

Other custom query plugin examples can be found in the boi_migrate/src/Plugin/migrate/source/boi directory including Order.php, Payment.php, and LineItem.php. In those examples, we're extending the migration source plugins that commerce_migrate provides and limiting the results returned by the query. We only wanted to migrate commerce entities related to pending or completed orders.

Custom process plugins

If you realize that the data from your source migration isn't being formatted to your desired output, you can create a custom process plugin. However, I first encourage you to make sure there isn't already provided by the migration module or modules that extend it. See https://www.drupal.org/docs/8/api/migrate-api/migrate-process-plugins for documentation on process plugins.

The example repo includes a simple process plugin to transform the integer amount provided into the float amount stored by Drupal Commerce 2.x. The code for this transformation lives in boi_migrate/src/Plugin/migrate/process/boi/Amount.php which extends the migration module's ProcessPluginBase class. Again, the annotation is used to define the name of this plugin as "boi_amount". We override the base classes transform() method and convert the amount passed from source into the amount expected by the destination. Without this, payment transactions would come through as 100x their actual amount. Once the plugin is created, rebuilding the cache allows it to be located by the migration plugin helper and we can use it in our migration config. Note that process plugins are chainable, so you can append as many as you need to your process key to get the desired output.

Updating your migration configuration

The migration configurations that have been defined in config/install/*.yml are imported into Drupal's configuration system when the module is enabled. When you make changes to these files, they are not updated in Drupal automatically. Since you'll likely be running and re-running your migrations until you get things right, you should use config_devel (in development only) to re-import your configurations after making changes.

Download and enable config_devel. composer require drupal/config_devel --dev drush en config_devel -y

Re-import the user migration after making changes. The drush command cdi1 is an alias of config-devel-import-one which allows you to re-import one config file from it's filesystem location.

drush cdi1 modules/boi_migrate/config/install/migrate_plus.migration.user.yml

Summary

The level of effort for us to write this migration was, initially, pretty high. We have a lot of experience with Drupal 7 and Commerce 1.x migrations, which in some cases may have made things more difficult. We were used to having the migrate_tools utilities available out of the box and were expecting to have to register the migrations manually. Instead we had to grab the extra module(s) and use the correct naming conventions for the config files and migrate_plus.

It was also a challenge initially to determine where to configure the database connect. I expected to put it in settings.php in the standard array format, but it wasn't clear how the migrations would know which database to use.

Once we got over the initial hurdles, things fell into place pretty quickly. With the migration module in core, modules providing their own plugins and templates, and commerce_migrate providing a ton of examples, we now have a solid platform to migrate data into Drupal easily.

The next migration should be very straight forward for us to execute and should only take a fraction of the time. Hopefully this blog will put you on the fast track to success with your migration!

Additional Resources

Need Help?

Blue Oak Interactive offers Drupal and Drupal Commerce migration consulting and development. Contact us for more information.