Drupal comes with its own built in cron. This means that you can add your own job to the list of jobs that are executed when the Drupal cron runs.
There have been a lot of people that are very much interested in the “DevOps” concept and when I sat down with some of these, the direction of the conversation went down to many interesting paths.
They started talking about deployment best practices, rollbacks, hot deployment etc.
But, when there were some mentions about “Blue-Green Deployment” - complete silence.
Therefore, this gave me an idea to tell the rest of the world that with all the microservices, native cloud and what not technology, blue-green deployment is not a silver bullet, but it is an element to usefulness.
Well, you got to read ahead.What do we understand by blue-green deployment?
A blue-green deployment is a management approach for releasing software code.Two identical hardware environments are configured in the exact same way in Blue-green deployments, which is also known as A/B deployments
Only one of the environments is live at a single time, where the live environment serves all the production traffic. For example, if blue is currently live then green would be idle and vice-versa.
Blue-green deployments are usually utilized for consumer-facing applications and the applications which have critical uptime requirements. The new code is delivered to the inactive environment, where it is completely tested.
How it reduces the risk?
Achieving automation and continuous delivery at any level of production is a holy grail, and avoiding downtimes and risks are high up on the list of priorities. Blue-green deployment provides you with simple ways of achieving these goals by eliminating risks that are witnessed in the deployment.
- You will never encounter surprise errors
When you fill a particular form online, what all credentials do you fill? Your name, phone number, address, street and probably your bank details if you are making an online purchase. Right?
You press the “pay now” button and check on the “receive spam emails” but unfortunately, your order wasn’t able to get processed as you desired. If you are lucky enough you get an error message equivalent to “application is offline for maintenance” all your efforts and time goes in vain. But with blue-green deployment, you never have to worry about this maintenance screen.
There is a list of item’s upon one click and upon next click, you are eligible to see the new menu that you add. This would keep furious emails about error screen from flooding your inbox.
- Testing the production environment
Ensuring that your pre-production environment is as close to your production environment as possible is not only important but essential too. With the help of blue-green deployment, this task is easily achievable. The user can test any application while it is disconnected from the main traffic. The team has the eligibility to even load the test if they desire too.
- Makes sure that the traffic is seamless
Customer needs and desires are more global than ever and there is no longer an essential good time to do deployment, especially if you work in an enterprise where the business needs to be running around the clock. If you have a customer facing application then there are chances that they might switch their platform to some other website, if they don’t find what they desire. This means a decrease in sale and business.
Blue-green deployment assures that your traffic never stops. That customer can place their order just fine without disruption. Which means that the employees overseas continue to do their job without any interruption, saving companies money.
- Easy Recovery
You might witness times where you would get introduced to bugs and viruses. We can either spend a lot of money on its fix or we can inevitably find them and recover them. With the help of blue-green deployment, we have our older and more stable version of our applications to come back online at a moment’s notice by evading the pain to roll back a deployment.Source: Martin FowlerHow does this process work?
As we know that blue-green deployment technique involves running two identical production environments where they are configured in the same way, therefore, let us assume that the current deployment is in the green environment in 2.3 release. The next deployment which would take place would be in a blue environment that would be in 2.4 release.
The environment would then be tested and evaluated until it is confirmed to be stable and responding. Once it is in production the server would be redirected, thus becoming the new production environment that the users are routed to.
The entire design is used to provide fast rollbacks in a case a deployment fails or does not pass a QA. When deployment fails or critical bugs are identified, a rollback to the green environment will be initiated. Once the bugs are fixed the version is re-deployed to the blue environment and the traffic is rerouted back the moment it is stable.
While deploying the preceding version i.e version 2.5, the deployment would switch to the green environment and would be extensively be tested and evaluated. Traffic would be rerouted to the green zone once it passes the quality assessment.
This way both green and blue environment are regularly cycled between live versions and staging to the next version.
Let us imagine that you constructed a website with the help of Drupal, now you are getting high traffic in it. Normally for developing, updating and testing a website (without risking the live integrity), you follow these steps:
Development: The development process starts with developers working on new features, bug fixes, theming and configuration in the local environment. It makes it possible to easily roll back to the previous stage of development.
Testing: Typically this environment is not available for client viewing and it is intended for testing developmental work against a lateral host.
Staging: This stage is used for presenting the changes to the client for approval. QA (quality assurance) and UAT (user acceptance testing) are most often carried out on the staging stage.
Production: This is the live site on the web available visitors. It contains new features that have been proven safe to go live.
As you can see that this process can be long and time-consuming, maintaining and constructing site can be irritating therefore blue-green deployment rescues you at times like these.
It would provide near to zero downtime and would present easy rollbacks capabilities. The fundamental idea behind blue/green deployment is to shift traffic between two identical environments that running differently in different applications.
Blue-Green Deployment for Drupal websites with Docker
Drupal Deployments are hard. The user has to make sure that that the code is deployed, composer dependencies are pulled, schema updates are pulled, scheme updates are performed and all the caches are cleared.
All with keeping the website up and responsive to the users. But if anything goes wrong and you wish to rollback? Do you stop the deployment? Well, no blue-green deployment is the answer to it.
Docker makes it easy to build, shift and run applications. On the EC2 instance, there are always two raised docker containers of “blue” and “green”, and ngnix works as a reverse proxy on the same instance. The user can build a Drupal site that is running parallelly in the “blue” and “green” environment and serve both from MySQL database. we install Apache, PHP, and Drupal in baseimage-docker.
Drupal with Blue-Green Deployment in AWS Beanstalk
Within the help of ECS, the user can create task definitions, which are very similar to a docker-compose.yml file.
A task definition is a collection of the container, each of which has a name, the Docker image runs, and have the option to override the image’s entry point and command. The container definition is also where the user can define environment variables, port mappings, volumes to mount, memory and CPU allocation, and whether or not the specific container should be considered essential, which is how ECS knows whether the task is healthy or needs to be restarted.
The Amazon web service solution allows the user to quickly and easily manage the deployment and scalability of web platforms. The deployment helps in configuring a high-availability environment that seamlessly runs a Drupal website. Running a DB instance that is external to Elastic beanstalk decouples the database from the lifecycle of the environment, and lets the user connect to the same database from multiple environments, swap out one database from another and perform a blue-green deployment without affecting the database.
The below image shows how green-blue deployment work in AWS environment.
Now that we understand how blue-green deployment works, let’s cover some of the best practices that are related to it:
Load balancing helps you to automatically set a new server without depending on any other mechanism, without depending on the DNS mechanism. The DNS record will always point to the Load Balancer and the user would only modify the servers behind it. This way they can be absolutely sure that all traffic comes to the new production environment instead of the old one.
To avoid downtime the user can execute rolling update which means instead of switching from all blue server to all green server in a single cut-off you are eligible to work with an integrated environment. This indicates that rather than switching from all blue servers to all green servers in a single cut-off, the user can control with an integrated environment
Monitoring the environment
Monitoring the productive as well as the non-productive environment is important. Since the same environment can play both as production and as non-production, all you would need is to toggle the alerting between the two states.
The user can script as many actions as possible in the witch process, instead of doing a manual set of actions. This brings huge benefits. The process becomes quicker, easier, safer and enables self-service.
Deployment in cloud
If your servers run in the cloud, there is an interesting variation of the Blue-Green method in which instead of going back and forth between two static environments, you can just create the next environment from scratch.
This process is also valuable for avoiding the danger of servers becoming snowflakes, which are servers that have a unique configuration set that isn’t documented anywhere. Once these snowflakes get erased for some reason, you have no easy way to properly recreate them. Whatever may be the choice it is important to keep the newest test and release technology to ensure that the release is smooth.
Deployments are one of the most important parts of the software development lifecycle, therefore all the activities involved should thoroughly be researched and tested to ensure that they are a perfect fit for your system architecture and business.
At OpenSense Labs, we have a pool of Drupal developers and experts that work on technologies that use these tools and services. Contact us now at email@example.com, our experts would guide you with the queries and questions that are related to this topic.
About a year ago, I only just learned about the principles of IndieWeb, which in a way is a bit of a shame. Fast forward to now, and I'm proud to announce the first stable release for Drupal 8. Together with this milestone, I also pushed a new version of Indigenous so that both are aligned feature wise.
It's been a great journey so far, and while there's still a lot to do for both projects, the stability and feature set warrants a stable tag. It has changed the way I interact with (social) media day to day now since the last half year, both in reading and posting, being in full control of every aspect. It's great, everyone should try it!What's next?
I've been thinking the last few weeks to raise funding, but after much consideration, I'm not going forward on that path. Even though my public GitHub profile lists over 1300 contributions the last year (about 3.5 per day), which somehow is simply crazy, I still have more than enough spirit and motivation to keep on going. Just a little slower from now on, since many features for both projects are not mission critical - even though they are awesome. Of course, I won't mind if someone would suddenly feel the urge to sponsor me.
Slowing down now you think, that can't be true ? Right. As already announced a few weeks ago, the next focus will be writing an Activitypub module for Drupal so you can communicate with your site on the Fediverse. I'm currently using Bridgy Fed for this, but, in the IndieWeb spirit, it's time to bring this home!
But first, time to make sure I don't mess up my tryouts of the Moonlight sonata. No commits until after March 31st - I promise :)
We’re back with an overview of the top Drupal blog posts from last month. Have a read and get yourself up to speed on the most recent goings-on within the Drupal community!READ MORE
This blog has been re-posted and edited with permission from Dries Buytaert's blog.
Three stars will align and the Open Web will win.
Today, the world wide web celebrates its 30th birthday. In 1989, Sir Tim Berners-Lee invented the world wide web and changed the lives of millions of people around the globe, including mine.
Tim Berners-Lee, inventor of the World Wide Web, in front of the early web.
Milestones like this get me thinking about the positive impact a free and Open Web has had on society. Without the web, billions of people would not have been able to connect with one another, be entertained, start businesses, exchange ideas, or even save lives. Open source communities like Drupal would not exist.
As optimistic as I am about the web's impact on society, there have been many recent events that have caused me to question the Open Web's future. Too much power has fallen into the hands of relatively few platform companies, resulting in widespread misinformation, privacy beaches, bullying, and more.
However, I'm optimistic that the Open Web has a chance to win in the future. I believe we'll see three important events happen in the next five years.
First, the day will come when regulators will implement a set of laws that govern the ownership and exchange of data online. It's already starting to happen with GDPR in the EU and various state data privacy laws taking shape in the US. These regulations will require platforms like Facebook to give users more control over their data, and when that finally happens, it will be a lot easier for users to move their data between services and for the Open Web to innovate on top of these data platforms.
Second, at some point, governments globally will disempower large platform companies. We can't leave it up to a handful of companies to judge what is false and true, or have them act as our censors. While I'm not recommending governments split up these companies, my hope is that they will institute some level of algorithmic oversight. This will offer an advantage to the Open Web and Open Source.
Third, I think we're on the verge of having a new set of building blocks that enable us to build a better, next-generation web. Thirty years into the web, our data architectures still use a client-server model; data is stored centrally on one computer, so to speak. The blockchain is turning that into a more decentralized web that operates on top of a distributed data layer and offers users control of their own data. Similar to building a traditional website, distributed applications (dApps) require file storage, payment systems, user data stores, etc. All of these components are being rebuilt on top of the blockchain. While we have a long way to go, it is only a matter of time before a tipping point is reached.
In the past, I've publicly asked the question: Can we save the Open Web? I believe we can. We can't win today, but we can keep innovating and get ready for these three events to unfold. The day will come!
With that motivation in mind, I want to wish a special happy birthday to the world wide web!
To help the initiative to update all deprecated code for Drupal 9 we need a standardized format for deprecation messages.New issue for discussion:
Issue #3024461: Adopt consistent format for deprecation messages.
Having a machine readable format for deprecation messages will allow us to develop tools on api.drupal.org to keep track of the current status of deprecated code in Drupal core and contributed modules. This will help drive the initiative to update all deprecated code before the release of Drupal 9.
You can get started quickly by helping us to update an issue summary or two or dive in and check out the full list of open proposals and see if there's anything you'd like to champion!
Layout Builder is here.
Chances are you’ve heard about Drupal 8’s Layout Builder, and maybe even seen one of those fancy demos. If 2018 was the year of promises for Layout Builder and other exciting Drupal 8 improvements, then 2019 is the year those improvements get into the hands of Drupal developers and content editors. 2019 is the year these features are live.
Every year community members from across the globe meet in Orlando for Florida Drupal Camp. This year Adam, Ryan, and Jonathan from Hook 42 attended. It was a fantastic time to connect with people, to learn, and enjoy some warmer weather. Plus, alligators!
Jacob Rockowitz: A request for a Webform logo and possibly a logo kit for Drupal contributed projects
Thank you for backing the Webform module's Open Collective
First off, I want to thank the backers of the Webform module's Open Collective. After my last blog post, Open email asking organizations to back the Webform module and Drupal-related Open Collectives, we have 14 backers and a current balance of $908.44 that needs to be spent.
I also received a comment and an email about the need for some process for paid support. It’s worth noting that the Drupal Association is exploring a paid support model for assisting with security releases. We should recognize that Drupal 8 was a major software change and it is one that is still changing the Drupal community. And while I am thinking about how the Drupal community is changing and how we can develop better support around the Webform module, one of my more immediate concerns is improving the Webform module's Open Collective, and brand is the first thing I want to address.
Improving the Webform module's Open Collective
There are some useful tips and guides for building and maintaining an Open Collective. I appreciate Pia Mancini’s statement that "A collective is not a sprint," which makes me feel comfortable taking time to build out the Webform module's Open Collective.
Defining and strengthening the Webform module's mission will help clarify to backers what they are supporting and getting from the Webform module. The product summary for the...Read More
DrupalCon Seattle is about a month away, and we're putting the finishing touches on this year's plans. Drupal's biggest annual conference affords us the opportunity to support the project, share our expertise, and connect with our colleagues from far and wide. We love DrupalCon. Here's what we've got in store this year.
Here's a quick recap of our blog posts from February - we hope you enjoy them!READ MORE
Few things are as good for a business as a website that looks great and runs well. When you’ve established a strong digital presence and rank well on Google, then you’re set to chase and convert leads to your heart’s delight.
Before any of those gains materialize, however, there’s the tricky task of building that website.
If you’ve chosen to use the Drupal CMS, then congratulations: you’ve made the right choice. Drupal is agile, powerful, and home to a wide community of developers and entrepreneurs.
Moreover, Drupal makes it possible to choose themes: templates that do most of the legwork of designing a site so you need only worry about the parts that matter (ex. your lead magnets, SEO, and copy).
Since not all themes are created equal... we’ve compiled a rundown of the 7 best Drupal themes to use in 2019:
1. Progressive by NikaDevs
Progressive is a theme that offers great value for money.
Its creators have packaged it with over 200 interactive elements, meaning you’ll be sure to find a function that lets your site move and behave just the way you envisioned it. It comes with video hosting, unique slider effects, and visual features that are guaranteed to capture an audience’s attention.
Our favorite thing about the theme is the set of four homepages offered at entry. While other themes force you into a single aesthetic mode, Progressive offers its solutions without placing implicit restrictions on your design options.
Compatibility: Drupal 7 & 8
Best used for: Businesses of all shapes and sizes.
Case Study: Time Center
2. TheMAG by PinkDexo
Today’s marketing is all about content that informs, entertains, or intrigues. It’s for this reason that TheMAG by PinkDexo ranks on our list of the best Drupal themes for 2019: it’s built to house content and house it well.
TheMAG is ideal for sites that want to rake in ad revenue, profit from content, or showcase products. It comes with a wide range of layout options and interactive elements, but it shines the most when used to present content in the style of a --you guessed it-- magazine.
We recommend taking a look at the theme before getting too excited. It’s a perfect fit for certain niches (blogging, journalism, entertainment) but other types of businesses would do best to continue reading through our list.
Compatibility: Drupal 7, Drupal 8, Drupal Thunder, & Drupal Commerce
Best used for: Media and Entertainment. Businesses that lean on blogging activity.
Case Study: Afaq Online Magazine
3. Winnex by gavias
Winnex perfects soft and professional design through its clean look and implicit geometry. It took the best lessons out of the adage, “less is more,” and used them to create an intuitive theme for professionals looking to market their services.
The theme gives users an easy time of site development thanks to its block-based, drag-and-drop interface, its support staff, and the numerous video tutorials they’ve produced.
Winnex gets all the basics right, but our favorite feature is how effortless it is to upload and display videos. After all, when it comes to pitching a service, it pays to set your site up with content that builds user trust.
Compatibility: Drupal 8
Best used for: Consultancy and other B2B services.
Case Study: Amman Stock Exchange (ASE)
4. Porto by refaktor
Part of a CMS’ job is to simplify the job of web design. In this regard, Porto is a theme that surpasses expectations. We’re hard-pressed to think of any other theme that can get so much done in so simple a way.
Porto features an unparalleled level of customization, with nearly endless options for page layout, header design, color mixing, and media hosting. This means that any business -- from consultants to home repair services, to e-commerce businesses -- can build their dream site using the theme.
Compatibility: Drupal 7, Drupal 8, Drupal Commerce, & Bootstrap 3.x
Best used for: E-Commerce. Businesses of all shapes and sizes.
Case Study: ProEquest - The Premier Equestrian E-Commerce Platform
5. Jango by NikaDevs
NikaDevs is back on the list with another highly versatile theme. Like Progressive, Jango offers endless potential for web design. Unlike Progressive, however, Jango presents itself as a theme for more timeless web design.
You’ll see fewer colors and gimmicks in their marketing, which can work well to favor businesses that need a site that keeps things simple. As with all themes on this list, Jango is structured, responsive, and offers support for its users.
Jango is a straightforward Drupal theme in the sense that it performs as needed, with minimal flash and maximum impact.
Compatibility: Drupal 7, Drupal 8, Drupal Commerce, & Bootstrap 3.x
Best used for: Any business that wants a simple and effective website.
Case Study: United Nations Relief and Works Agency (UNRWA)
6. OWL by gavias
Cafe and restaurant owners can breathe easy knowing there’s a Drupal theme specially designed for them. OWL is a quick and easy to use theme that knows the needs and demands of the food industry.
Our favorite feature of OWL is its capacity to host high resolution, Retina-ready images that food photographers are sure to appreciate. Coupled this with a simple interface and support for all kinds of modern programming tools (ex. Bootstrap, Custom CSS, SASS, etc.) and you have the perfect theme for restaurateurs.
Compatibility: Drupal 8, Bootstrap 3.x
Best used for: Hospitality. Restaurants and Cafes
Case Study: Makarem Hotels
7. Edmix by gavias
Our final entry, and our third from developer gavias, is a theme that caters to educational sites and businesses that offer online courses or tutorials. Edmix appears to take the best design features from of popular sites like Coursera and Udemy, giving its users the ability to shine a spotlight on their courses and videos.
As is the standard for gavias themes, the interface is easy to use and comes with assistance in the form of their support team, and library of tutorial videos.
Compatibility: Drupal 8, Drupal Commerce, & Bootstrap 3.x
Best used for: Online universities, masterclass services, etc.
Case Study: Georgetown University
This list covers what we think to be the best Drupal themes for a business operating in 2019. As you’ll notice, the best themes are versatile, and they allow for a high degree of customization to let businesses present the best sides of themselves without sacrificing style.
If you’ve had luck with any of the themes above, or if you know of a theme that you feel deserves a place on the list, feel free to write to us or leave a comment below.
Likewise, if you need help using Drupal or deploying any of the themes we’ve listed, Varbase is a powerful website builder platform that empowers businesses to build fast, reliable, and search optimized websites in record time.
From Drupal experimentalist, to migrating the Oak Ridge National Laboratory's website, to thru-hiking the Appalachian Trail, Jim Smith is bridging communities and sharing his story.
Search is a key feature in web experience, and for a lot of people, it's the go-to method to find content. We use search countless times a day on our smartphones in various contexts. And yet, when we're building out websites, search is often an afterthought that we don't spend much time on. Search gets added to the laundry list of site features, like meta tags and social media links.Drupal Core Search
Drupal is fantastic at managing content. It gives you loads of flexibility when it comes to building out your information architecture and categorizing content. But we often don't set aside a lot of time to build out a customized search UI to find that content. At the end of a project, you might just turn on the core search module and call it done. And then we find out that people use Google to search our website.
Drupal's core search functionality hasn't changed much in the last 10 years, and is lacking features that users expect. It can be slow, and it doesn't correct for misspellings or allow you to prioritize results. Search should make your content easy to find, and make your site more engaging for users. Over the years, we've worked on lots of websites that integrate with Solr, to provide an enterprise-level search engine on top of Drupal. But setting up Solr takes time, and can be tricky if you don't have a lot of time to set it up, or the know-how to configure your server.Why Cludo?
We recently decided to add search to evolvingweb.ca, and decided to try out Cludo. It's a "search as a service" tool that allows you to add on a search interface to an existing site. Kind of like you'd add Disqus, or Google Analytics. It was pretty easy for our developers to set up Cludo. Besides some challenges setting up search of the French language side of our site, particularly searches with UTF8 characters, the setup was straight-forward and only took a few hours to add.
The immediate advantage is that you don't have a lot of setup time for a search that just works how users expect. But after it was all set up, I realized that there are a lot of extra features that you get that we wouldn't normally take the time to configure for a basic search:
- Autocomplete - start typing the title of a node and it'll autocomplete
- Customize the index - you can pick and choose what's searchable and what's not
- Analytics - you can see who searches for what, giving non-technical users visibility on how users search for content
- Boosting - you get nice defaults for results ordering, but you can also customize the criteria to prioritize certain types of pages or certain criteria
- Machine learning - an add-on feature that does the boosting and changes the autocomplete ordering for you based on user behaviour
- Easy-to-use interface - non-technical users can update all the settings through Cludo's UI
Before you ask, yes there's a module for that! The Cludo module was released a couple weeks ago. It's still in development, but you can try it out. You just have to add your Cludo account number and key, and it provides a search form block that you can place on the page.
Here are some examples of websites using Drupal:
- This website (just click on that search icon in the header)
- Larimer County
- Parkinson's UK
- Blank Rome
So what's the catch? Cludo isn't a free service, it comes with a $200/month price tag for most websites. And it will cost more than that if your site has more than 20,000 pages or you want bells and whistles like document search, machine learning or searching private content. There are discounts for non-profits and educational organizations.
There's a trend towards using third-party services for everything from marketing automation tools to comments and now search. I know a lot of Drupal developers prefer to use open source tools as much as possible. I think the great thing about third-party tools is that it gives us another option. We can offer our clients a way to get a search interface up-and-running quickly, without a lot of up-front development time. It gives the end-user something that's easier to configure.
On the other hand, for a large website with a lot of content, we might want more control over the functionality and costs. And for an intranet, we might want more control over where the data is stored. If we have a lot of site installs, Cludo could start to become very pricey. In these cases, using Search API would be a better option. But for lots of use cases, when that "instant" quality is the priority, Cludo is a great option, to make sure your content is discoverable and that your users can find it.+ more awesome articles by Evolving Web
I often work on migrations that involved dates that need to be migrated into Drupal 8. While many times the dates are for entity created and updated dates, and therefore in Unix timestamp format, sometimes (when migrating events, for example), I'll need to migrate a date in some other format into Drupal's date field.
In the past, I've ended up writing a custom process plugin or callback to convert the date into the proper format for Drupal core's date field, but I recently discovered the format_date process plugin (I'm pretty sure I'm late to the party on this) and realized I was doing more work than I had to.
The short version is this - the format_date plugin takes a date in any format (as long as it can be described using the standard PHP Date patterns) and converts it to the format Drupal requires. Oh, and it also has timezone support!
Here's an example of taking a datetime in the format of "11/04/18 08:30:00" (EST) and converting it to the format that Drupal's core date field requires, "2019-11-04T08:30:00" (UTC).field_eventdate/value: plugin: format_date from_format: 'm/d/y H:i:s' to_format: 'Y-m-d\TH:i:s' from_timezone: 'America/New_York' to_timezone: 'UTC'
It's pretty simple! Less custom code usually means less opportunities for mistakes and less code to maintain in the future!
One of the pillars of the consulting side of the work we do here at DrupalEasy is data migration into Drupal sites. Over the past few years, we've been focused on migrating data into Drupal 8 using the most excellent core migrate modules along with contrib modules like Migrate Tools, Migrate Plus, and Migrate Source CSV.
If there's one thing I've learned over the years is that data to be migrated is never, ever, ever, ever, never as clean as it is claimed to be. There's always something that has to be massaged during the migration process in order to get things working.
One of the most common things I see when migrating data from a .csv is strings with trailing spaces. If you take a cursory look at the data in a spreadsheet, you might see something like "Bread", but if you look at the same data in a text .csv file, you'll see that the string is actually "Bread " (trailing space). If you're migrating this field to a vocabulary using the entity_lookup process plugin, that trailing space will cause the term to not be found, and therefore not migrated.
The solution? Well, you could clean up the data, but there's actually a much easier solution that I use by default on almost all string data being migrated - I use the "callback" plugin to in-turn call the PHP trim() function on incoming strings in the "process" section of the migration configuration. Here's an example:field_topics: - plugin: callback callable: trim source: Topic - plugin: entity_lookup entity_type: taxonomy_term bundle: topics bundle_key: vid value_key: name ignore_case: true
Using this method allows for the incoming data to be a little dirty without affecting the migration.
Code flows up, data flows down.
I repeat this phrase in just about every workshop I teach - it is one of the basic principles of being a professional web developer. The idea is that we should be working locally, then pushing our changes (using Git) up to the project's dev, then QA, the live environments. As for the project's data (database and files directory for Drupal sites), the direction is opposite, we should only be moving data "down" - from live to QA, or live to dev, or live to local.
There are, of course, exceptions to every rule, and certainly in this case as well.
One exception is when the project is just getting started. Consider the example where you've started a new project on your local, you've reached the first milestone of development and are ready to move everything to a shared development environment where the client can catch their first glimpse of the project. In this case, you'll likely be moving everything "up" - code, database, and files.
I had this exact scenario recently, I was migrating a rather large site to Drupal and had the initial migration looking good, and was in the process of getting it up-and-running on Pantheon. I successfully pushed the code as well as SFTPd the 1.6GB files directory to the Pantheon dev environment. The database was a bit larger than the 100MB maximum Pantheon allows to be uploaded through the browser, so I was using their "URL" method.
My plan was to put the database dump in a public Dropbox folder, then copy/paste the URL of the file in the Pantheon Dashboard interface. Unfortunately, it didn't work. I tried both .sql and .sql.gz formats, I tried doing the database import using Terminus (Pantheon's command-line interface) - each time I was provided with either no error message, or one that wasn't very helpful.
The solution? Turns out it is a bit of a DropBox issue, albeit one that is pretty easy to fix.
When copying/pasting the URL for a public file on DropBox, the URL ends in dl=0 - turns out that this prevents Pantheon from being able to import the file. Simply change it to dl=1 and the problem is solved (this works in both the Dashboard and Terminus)!