For the past two years I’ve been working on something less visible but no less important.
Since DrupalCon Amsterdam 2019 (an actual in-person conference — sounds surreal in 2021, doesn’t it?!) I’ve been working on Acquia Migrate Accelerate, or “AMA” for short. In a few days, another DrupalCon Europe is starting … so perfect timing for a recap! 😀
Why?
Drupal 8 comes with an awesome migration system built in, originating in the Migrate Drupal 7 module. It standardized many migration best practices. But it still required a huge time investment to learn it.
Of course, there’s the “Migrate Drupal UI” (migrate_drupal_ui) module in Drupal core. But that does not allow for granular migrations. It allows for a one-shot migration: you see which things will be migrated and which won’t. You can click a button and hope for the best. It only works for the very simplest of sites. It is impressively minimal in terms of the code it needs, but unfortunately it also means one pretty much needs to be a expert in migrations to use it successfully.
It will be of little help as soon as you run into important data you want to migrate for which no migration path exists.
See Mauricio Dinarte’s excellent “31 days of Drupal migrations”. In those 31 blog posts, you’ll learn to know and appreciate the migration system (I sure did!). Unfortunately, that still won’t fully prepare you: you’ll need to decyher/reverse engineer the intricacies of how the data gets stored in Drupal 7 with its entities, revisions and fields — and with each field type having its own intricacies — and map that to Drupal 9 equivalents.
And how does one migrate straight from Drupal 7 with its more fragmented ecosystem? 1
For example: media handling. There are easily a dozen approaches possible in Drupal 7. Each in use on tens of thousands of sites. In Drupal 8 & 9, everything has standardized on Media and Media Library. But how do you get your Drupal 7 site’s content in there?
Another example: location data. location was very popular, but is now dead. geofield was equally popular but still alive. geolocation was less popular but now more. addressfield was popular, address’ is the successor. None of the Drupal 9 modules offer Location’s feature set. How do you migrate this data?
Goal
The goal for AMA (the vision of https://www.drupal.org/u/grasmash and especially https://www.drupal.org/u/webchick!) is to empower the non-technical user to be able to perform migrations. A UI that is aimed at the site builder POV: one should be able to select which content types (also vocabularies, menus, et cetera) make sense to migrate, and then not have to bother with technical details such as “migration plugins” or YAML files.
For example, AMA shows just “Page” in the UI. Under the hood (and you can see this in the UI too, but it’s just not prominent), that corresponds to the following migration plugin definitions:
d7_node_type:paged7_field_instance:node:paged7_field_formatter_settings:node:paged7_field_instance_widget_settings:node:paged7_node_complete:paged7_url_alias:node:pagenode_translation_menu_links:node:page
In other words: the support configuration for nodes of the page bundle (the first 4), then all actual entity/field data (d7_node_complete:page), followed by URL aliases and menu links referencing pages.
However, to be able to do this, we need many more migrations in Drupal core to be derived: view modes, fields, formatters and widget should all have an entity type+bundle-specific derivative. That’d allow each bundle to be migrated individually. Which enables the site builder to check that their pages and everything related to it has been correctly migrated before moving on to the next data concept to migrate. So far we’ve not yet been able to convince the migration system maintainers of the value of this. 2
(AMA does many more things, but that’s not in scope of this blog post.)
Closed & Open
Acquia understandably wants its customers to be able to use AMA, and not its competitors’ users. The module is GPL-2+
Like all Drupal modules, the AMA module is GPL 2+ licensed. Only the React UI is closed source. The automated recommendations engine is closed source. Obviously the code to spin up AMA environments in Acquia Cloud is closed source 3.
But … all of the work that goes into making migrations reliable is open source. At the time of writing, we have ~100 unique patches that are being applied, 39 of which to Drupal core! While I was writing this, https://www.drupal.org/project/drupal/issues/3190818 got committed, plus a few were committed recently but did not yet ship in a 9.2.x point release, so soon that number will be lower 🙂
An overview
In the past 20 months we’ve hardened many migrations, and created new ones from scratch. Thousands of Drupal sites have already benefited — many more than there are Acquia customers.
The highlights:
- New migration:
field_collection→paragraphs+paragraphs— this is the first major migration we helped get to a point where it works! - New migration: https://www.drupal.org/project/location_migration, to migrate from the
locationpseudo-field data intoaddressfields andgeolocationfields. - New migration: https://www.drupal.org/project/bean_migrate, to migrate from the
beanmodule into Drupal 9 coreblock_customentities. - Vastly expanded migration: https://www.drupal.org/project/media_migration, we became co-maintainers (and effectively lead maintainers) for this module that Lullabot’s https://www.drupal.org/u/juampynr started, expanding it to migrate data from
file_entity,media_wysiwyg,ckeditor_link_fileand many more, but especially to migrate these to equivalent sensible “media types” in Drupal 9, thus making the Media Library in Drupal 9 an instantly highly valuable UX after migrating from Drupal 7! - Stabilize many contrib migrations, most notably as
metatag(shipping in stable releases) andparagraphs(partially committed). - And developer-focused modules such as https://www.drupal.org/project/migrate_devel_file_copy and https://www.drupal.org/project/migmag (which contains more advanced alternatives to process & destination plugins in Drupal core).
Overall, 29 Drupal core patches 4 and 18 Drupal contrib patches have been committed! Plus another 36 core patches 5 and 32 contrib patches are successfully being used, and will hopefully land in the near future. (Not counting commits to the migration modules we now (co-)maintain.) Many dozens of migration paths from Drupal 7 have been stabilized, especially by https://www.drupal.org/project/media_migration.
A comprehensive overview (all patches are uncommitted unless stated otherwise):
acl: #3231034 — committed, not yet in releaseagreement: #3232859 — committed and released!block_class: #3225653blockreference→block_field: #3219319color_field: #3224459 — committed, not yet in releasefield_group: #3177720 + #3177724 are committed and shipped in the 3.2 release, #3177722 is not yet committedfield_formatter_class: #3208715 + #3217829google_analytics: #3170816htmlpurifier: #3167171mediaelement: #3223959metatag: #3158613 + #3165112 + #3169952 (all committed and released, thanks https://www.drupal.org/u/damienmckenna!) + #3190804 (uncommitted) + #3228362 (committed, in upcoming release)multifield→paragraphs: #2977853multiple_selects: #3188284node_queue→entity_queue: #3206490pathologic: #2693695paragraphs: #2911244 (committed) + #3145755 + #3203739 (committed) + #3226658 + #3221275 + #3228554 — all uncommitted except two, neither is releasedpathauto: #3179835 + #3179865 + #3079275 + #3190980 + #3182708 — all uncommittedpathologic: #2693695printandprint_*→entity_print: #3238952redirect: #3082364 + #3162128scheduler: #3238953shortcode: #3229599user_hash: #3232037 — committed and releasedwebform_migrate: #3172139 + #3164697 (committed) + #3170195 (committed) + #3172154 + #3172161 (committed) + #3172162 (committed) + #3172176 (committed) + #3172417 + #3172473 + #3182468 (committed)— but none of these shipped in a release
D7 → D9 for all
We aim to continue to do to the work to get patches committed: address feedback, add test coverage, and so on. We want to help everyone migrate from Drupal 7 to 9!
Teamwork
These many hardened migrations are thanks to the titanic work of:
- Angie aka https://www.drupal.org/u/webchick
- Zoltan aka https://www.drupal.org/u/huzooka
- Gabe aka https://www.drupal.org/u/gabesullice
- Peter aka https://www.drupal.org/u/zrpnr
- Matthew aka https://www.drupal.org/u/grasmash
- Naren aka https://www.drupal.org/u/narendrar
- Srishti aka https://www.drupal.org/u/srishtibankar
- Omkar aka https://www.drupal.org/u/omkarpodey
- … and of course many in the Drupal community with whom we’ve collaborated along the way, especially https://www.drupal.org/u/quietone!
If you found this interesting, check out Gabe’s write-up of the application architecture that powers the awesome React-based UI that Peter built.
-
Some would say richer. ↩︎
-
It also implicitly reveals one of the most fundamental assumptions in the migration system: that only the final state after running all migrations matters. For developers who know both Drupal 7 and 9’s data models really well, this may be fine. But for a non-expert, it’d be simpler if they were able to migrate each the entities of each entity type+bundle and then inspect the results, not to mention that it’d take less time to get some confidence in the migration! For example, first the “tags” taxonomy terms, then the “image” media items, then the “blog post” content items. Verifying the correct validation of each of those clusters of data is simpler conceptually. Site builders, if you want this, please leave a comment in https://www.drupal.org/node/3097336! ↩︎
-
Acquia Cloud handles the creation of an ephemeral Drupal 9 migration environment, with a Drupal 9 site automatically generated, with all equivalent D9 modules pre-
composer required, and all modules with a vetted migration path pre-installed. For Acquia the value is obvious: its customers are more likely to succesfully mgirate to Drupal 9 sooner, the customer is more likely to stay a customer. We’ve currently got over 100 customers using AMA.) ↩︎ -
Committed core patches: #3096676, #2814953 (this one has the majority of the work done by the community!), #3122056, #3126063, #3126063, #2834958 (from those first 14), #3152789, #3151980, #3151993, #3153791, #2925899, #3165944, #3176394, #3178966, #3187320, #3187415, #3187418, #3187463, #3189463, #3187263 (from 2020), #3190815, #3190818, #3191490, #3097312, #3212539, #3213616, #3224620, #3227549, #3085192 (from 2021). ↩︎
-
We started out with 14 core patches. Of those, #3115073, #3122649, #3096972, #3108302, #3097336, #3115938, #3123775 still remain. Other core patches we’ve contributed in 2020 that are not yet committed: #2845340, #3151979, #3051251, #3154156, #3156083, #3156730, #3156733, #3165813, #3166930, #3167267, #3186449, #3187334, #3187419, #3187474, #3187616. And those in 2021: #2859314, #3200949, #3204343, #3198732, #3204212, #3202462, #3118262, #3213636, #3218294, #3219078, #3219140, #3226744, #3227361, #3227660 ↩︎
