Feeds
Mario Hernandez: Drupal base path
Recently I was building a component that required a static image which was not stored in the database but instead needed to be stored somewhere in the file system of the site. There are several ways for serving a static image for example we could have stored the image in the sites/default/files/images/ directory. A very common approach which in many cases would work just fine, however, in my case I was building a component and I wanted for the component image to be located within the same component's directory. This makes sense because if I wanted to reuse or share this component, all component assets would be included in a single directory.
RequirementsMy goal with this task was to dynamically point to the image regardless the site this component was running on. Oh yeah, we are running a multi-site architecture with hundreds of sites and a single code base. So this component needed to work in all of these hundreds of sites. Now the challenge seems a little more... "challenging".
¡Manos a la hobra'!'I started by doing research to determine the best way possible to achieve this. I read about using a pre-process function that would generate a dynamic base path of the site but I was hoping I could keep things simple and do everything on the front-end with only Twig. This would make it a more appealing approach for front-end developers.
After some research, I came across two little gems that became game-chargers for my project. One of these gems is the {{ url('<front>') }} Twig function. This will provide the current site's homepage/base path. The other very handy Twig function is {{ active_theme_path() }} which prints the path of the current active theme (themes/custom/my_theme). While researching for this task, I also found you can use the {{ directory }} Twig variable in your theme's templates to print the active theme's path. A word of coution when using either the {{ active_theme_path() }} function of the {{ directory }} variable as these could have different results depending on your whether you are using them in a base or sub theme. Here's a drupal.org issue that discusses this in more detail.
Armed with these two little functions, and one Twig variable, we can now work in generating a dynamic path to our theme's directory where the static image for our component is located. So this may seem like a simple thing but remember, our component's image should work regardless of the site the component is used on within our multi-site architecture. Some sites even use a different sub-theme but the parent theme is always the same which is where our image is stored.
Before we can use the first function we need to run it through the |render Twig filter. Since Twig will return an array from the {{ url() }} function, we need to convert it to a string because we need the value of the function. Let's take a look:
{{ url('<front>')|render }} # This will give us http://my-site.com/Next let's work with the theme path function. Similarly to the function above, we will use the |render Twig filter to convert it from an array to a string.
{{ active_theme_path()|render }} # This will give us themes/custom/my-themeNow that we have two strings we can joint them together to compose the full path to our image:
<img src="{{ url('<front>')|render }}{{ active_theme_path()|render }}/images/image.jpg" alt="alt text" />If we want to get fancy we could actually set a variable to shorten things a bit:
{% set theme_url = url('<front>')|render ~ active_theme_path()|render %} <img src="{{ theme_url ~ '/images/image.jpg' }}" alt="alt text" />And there you have it. A dynamic path that will work on any of our sites.
I realized some people reading this already knew all of this but I didn't. So I figured I would share it because I bet there are others out there that also do not know about the {{ url('<front>') }} or {{ active_theme_path() }} Twig functions as well as the {{ directory }} variable. As I said before, there are many ways to handle this challenge, but in my case this is exactly how I wanted to approachh it. I hope this was helpful. Cheers!
Mario Hernandez: Five principles for building better components
When working on a component-based project, building components or patterns, can be a liberating experience because we are able to draft the best markup possible to build the most flexible, scalable and reusable components. If you work with third party platforms such as SharePoint, WordPress, Drupal, or others, this becomes even more rewarding as in most cases content management systems are not known for producing the best markup possible. However, this liberating experience can come a a price if you are not taking the appropriate measures and checks when building your library of patterns.
In this post I am going to discuss five critical principles I follow when building components. My goal is always to achieve first and foremost, semantic markup that works in all devices including assistive technologies, reusability, scalability, and finally, ease of use. In some cases it is hard to achieve all of these goals so it is important to understand where you draw the line and what is more important in the long term.
Disclaimer: Components and Paterns are used interchangeably.
Find patterns in your patternsIdeally, before you begin building patterns, or components, you want to fully understand how and when these patterns will be used. Certainly there are times when we don’t have all this information when we first start working on a project, but if you do, take a detailed inventory of your patterns. Doing this will help you build better and more reusable patterns.
Failing to identify the patterns in your patterns can lead to bad decisions when building them. For example, let’s say we have a card component for displaying latest news articles, if we don’t know that there are cards for events that look similar to news articles, we may miss an opportunity to turn the card component into one we can use for multiple purposes in our website. But don’t let this get you down, building a website is an iteration process. If you later discover that the news card component can be leveraged for events, it’s perfectly okay to update the card component so it can be used with events as well. My point is that it is preferred if you can catch these patterns early on, but we all know that sometimes designs are not always available in full when a project is started and when they become available we may need to re-think our process of component-building.
Reusability when working with components is important and should be at the forefront when building components, however, don’t paint yourself in a corner by over-engineering your components because you found a way to turn a component into a multi-level reusable machine. Trying to make your components do too much can lead to over complex development and in the long run, very hard to maintain. It is better to have components that are simple and easy to maintain even if it means you may have some components that are similar. It is hard to find the balance between simplicity and reusability and this discovery will only come as you work on your components. If you have to choose between simple and reusable, simple may be the way to go.
One set of markup for all devicesEarlier I mentioned that before you start building components you should have a full picture of how and when those components will be used by looking at the design comps. Of course this is not always possible, but in an ideally designs comps give us opportunities to identify areas of improvement or areas that could lead to potential problems. An example of a problem would be if you are provided the designs for mobile, tablet, and desktop. This is great, but let’s say the information displayed at each device type changes in such a way that you don’t know how to best write the markup so it works on all devices. In a perfect world, you should not have to alter the markup of your components per breakpoint. Doing so could lead to confusion on how content is rendered, not to mention problems with accessibility or even SEO. If you find that markup may be a challenge from breakpoint to breakpoint, it is perfectly okay to go back to the designer and propose updates to the design so a unified markup structure could serve all devices. In my experience, the designer would be okay with your recommendations as long as you are not completely changing the layout of the website but instead are simply proposing minor adjustments that don’t change the overall look and feel of the website.
In some cases a designer may not be fully aware that the designs they are proposing could lead to issue when the site is rendered in different device sizes, so it’s okay to bring these issues to their attention and you may find that they are onboard with your suggestions.
Another way to paint yourself in a corner is by the name you choose for your components. Component names should be generic and yet descriptive so they can be used on different scenarios. Conversely, names that are too specific will limit when and how you can use those components. Let’s take a look at some bad component names and propose a better name:
-
News Card: If we think back of how we can reuse components, naming a component News Card will limit us on what kind of content we can use this component on. What if Events use a similar card with minor differences? A better name for this component may be Card or Content Card. This allows us to use on any type of content.
-
FAQ Accordion: We have seen this before, we want to build a collapsible list of Frequently Asked Questions so the name makes sense. However, what if we want the same functionality for other type of content such as Forums or Knowledge base content? A better name would simply be Accordion which describes the functionality of the component, not the content that it can hold.
-
Latest News List: Again, we are limiting ourselves to only showing content of type news. A list of content should be generic so we can use them for any kind of content (i.e. Blog posts, events, popular content etc.). A more appropriate name would be Content List.
-
Finally, Homepage Hero: It is not uncommon for Hero sections to be in multiple pages other than the homepage. By naming this component simply Hero, we can leverage it and use it in more than just the homepage. You may wonder, what if I have other types of heroes in non-homepage pages? Well, perhaps we can find alternative names for those, things like Hero Short, or Page Breaker, something that is descriptive and yet provides the ability to be used in multiple pages.
The above are only some examples of how a bad name can not only limit where or what type of content you can use your components with, but it can also lead to building unnecessary components.
Document your componentsOne of the great things about design systems like Pattern Lab, Storybook and others, is that they provide a way to document the details of your components. Documenting components can go a long way to ensure your entire team is fully aware of the why and how of your components.
I practice that I follow when building components is documenting the technical aspects of the component such as its name, where it is used, and details about any variants of the components. Just as important, I like to define the components fields, their data type and values they accept. This architectural information ensures gives your team and your stakeholders a detailed view of a components role in your project.
In closing, before you begin building components, take the time to study your design comps and take notes of how you feel you can make the most out of your components library. You will not get it right the first time, that’s okay. As long as you are giving yourself the room to iterate through the process, you can always go back and update a component if it will give you more flexibility and will turn your project into a solid component-based system.
In closingYou may not always get it righ the first time, but keeping these principles in mind will help you achieve better implementation of your website's patterns.
Mario Hernandez: Demystifying components integration with Drupal
Component-based development is something I have been doing for at least five years and it is incredible that I still find myself learning something new every time I work on a new project. In addition to development, I write training curriculums on the topic and this gives me more exposure to this topic than most people. You’d think by now I should have learned all there is to learn about components and integrating them with Drupal, but that’s not the case, and I love it because I alway look forward to learning something new.
The inspiration for this postI came across an outstanding blog series about Building with Emulsify Part 3: Component Complexity by Evan Willhite of Four Kitchens.
It's not you, it's meI’d say I’m about 90% onboard with how Evan approaches component integration, but the part I am not a fan of is writing preprocess functions to achieve full integration, mainly because I am not well-versed in preprocess functions or PHP in general. There is absolutely nothing wrong with this, but what if you are someone like me who is not comfortable with preprocess functions or custom modules? So I decided to use most of the work Evan put together, but will perform the integration slightly differently so we only work with Twig and not PHP.
Both instances for which PHP code was required in Evan’s post were the result of using a Paragraph type to reference nodes. My approach will be to eliminate the paragraph type and work directly with the node entity.
Let’s start. I will pick up from Architecture since everything prior to this is typically the approach I take as well.
ArchitectureWe want administrators to create Nodes of any type and using a Display View Mode we can automatically display them using the card component. The one difference for me is that I don’t have a card variation.
Passing data between entitiesFollowing Evan’s instructions, create a new Display view mode called Card. Here's another minor difference in that my Card uses an image field and date field in addition to title, body, and link fields. Here’s what the card looks like.
In the Card display view mode of an Article content type, ensure Body and Image are available. All other fields should be placed under Disabled. The Title and Date fields don’t display in the Manage Display screens, but they are always available in all Entity templates.
Integrating the Card component within a Node templateJust like in Evan’s post, we still need to associate a Node template with our Card. For this we will use the same template suggestion as Evan, node--card.html.twig.
{% include '@molecules/card/card.twig' with { 'image': content.field_image, 'date': date, 'title': label, 'body': content.body, 'link_url': url, 'link_text': "Read the article", } only %} Displaying a list of cards with ViewsNow that the Card component has been integrated with Drupal, let’s create a simple view to display a collection of nodes displayed as Cards.
The views' settings can be seen below with description of the important configurations to note.
A few things about the view:
- It’s called Latest Articles
- The page built by the view can be viewed at /blog
- Rather than using fields it uses the Card view mode we created earlier. This is the biggest advantage of integrating the Card component with the Entity, any time a node is displayed using the Card Display View Mode, the node will be displayed automatically as a Card.
- It only pulls nodes of type Article
With some minor CSS the articles would look like this:
In closingThere will be times when writing a preprocess will be required based on the requirements, but whenever possible I’d like to avoid it.
Mario Hernandez: Styling Forms Elements
Whether you are a veteran or just getting started with web development, working with web forms is not the sexiest thing but there is no avoiding them. Forms on a website are critical to the website's mission and the visitors using them. In this post I am going to show you how to properly style webforms and form elements so you can ensure they are consistent with your brand and provide a pleasant user experience.
Resources:I have recording video tutorials on some of the concepts covered in this post. Scroll to the bottom if you are the kind of person who learns better watching tutorials.
Where to startOne thing about forms you can count on is that they are not consistent. In today's API era forms can be generated from any source and you don't always have control of their HTML. I personally work with Drupal a lot and even within it forms can originate from Views, Blocks, Nodes, Paragraphs, Modules, and more. I imagine this is the case with most Content Management Systems. So how do you address something where you can't always predict source or markup of forms?
If you can't beat them, join themWhether is Drupal or other system, it is best to let that system dictate the markup of forms. Try to bend a system to your needs can prove challenging and not worth the trouble. This is the approach I take and so far this has worked very well for me.
Attributes are your friendsAs a Front-End developer for many years, I love a good css class. They make my job a lot easier than not having a class. When it comes to form elements however, it is best to ignore classes and rely solely in attributes. What attributes you may be asking yourself? Let's take a look
From element typesThe type attribute in form elements make it easy to style them in a global manner. Rather than styling form elements based on their css class or ID, the type attribute is the best approach. Each form field has a type attribute which makes it possible for browsers to render the element as expected. The type attribute can also play a factor in Javascript and form handling, but for now we are going to focus on styling.
-
Input fields: Input fields such as text fields, are probably the most common form elements in a form. But don't let its simplicity fool you because text fields come in many types; the most common being text. this is the type for a typical text box where you can type just about any kind of information. This sometimes leads to developers using it for anything including for data that should be in a different form type. Let's take a look at the various types a text field can be:
- text: Used for most text fields and accept any string of text
- email: Used to capture email address. Using this type is recommended for email addresses as it provides regular expressions to validate the data entered in the field meets the patterns of an email addresses. In addition, using this type of field makes it possible for your mobile devices to show a custom keyboard where the "@" sign is available without having to toggle your keyborad characters to find it.
- password: As you can imagine, this makes it possible to hide passwords from view as you type them.
- search: Elements of type search are text fields designed for the user to enter search queries into. These are functionally identical to text inputs, but may be styled differently by the user agent.
- tel: Similarly to the email address, using this type allows for the keyboard on mobile devices to display numbers by default rather than alpha characters.
- url: Similarly to email, this provides you with a custom keyboard on mobile devices that make it easier to type a common url.
As you can see from the examples above, text fields may seem simple but they come in all flavors. In most cases, the goal of form elements is to be styled the same across an entire site. So it is recommended you style these elements globally so no matter where they are used they automatically inherit the intended styles. So how do you style them globally? Let's take a look:
Here are a couple of examples of differnt input fields which we will style to ensure they all look similarly everywhere.
<input type="text" name="firstname"> <input type="password" name="password"> <input type="email" name="email"> Styling input[type='text'], input[type='password'], input[type='email'], input[type='tel'], input[type='search'], textarea { background-color: #ffffff; border: 1px solid #000000; height: 40px; padding: 10px; width: 100%; } textarea { height: initial; }By using the element's type attribute we can collectively style them with a single set of rules. As you may have noticed I included textarea as on of the elements because I want textareas boxes (multi-row textbox), to inherit all of the styles from text boxes with the exception of the height property; thus we override the height property while keeping all other styles the same as text boxes.
Button elementsButtons are elements you typically want to display with consistent styles across your entire website. This includes buttons you may create yourself and those provided by the content management system you may be working with. So similarly to input fields above, we can't always rely on IDs or classes to style them. However, similarly to the input fields above, we can rely on the type attribute to style the different type of buttons in our site. Different type of buttons? yes, there are different types. Let's tae a look.
Markup <a class='button'>I want to look like a button</a> <button>I'm an actual button</button> <input type='submit' value='Yo tambien'> <input type='reset' value='Me too'>- All elements above will have the same look and feel.
- The first 3 elements should look identical, but the last one (type=['reset']) should look slightly different because it's not a primary button, it's more of a secondary button.
- Sometimes this secondary button will be used to cancel or to reset someting on a page. While the top 3 will be used for important actions such as submit, save, etc.
- Let's style them now
- We are including all the button elements in our initial set of styles. The reason for this is that we want all the buttons (regardless of their type of function), to share the same shape, font size, font-weight, border width, hover behavior, and other properties.
- Then for the reset type only, we are styling this button with white background and purple text. This will ensure this button will not stand out as much as the other buttons.
Probably the hardest elements to style due to their dynamic nature, they have a hard time fitting in with the rest of HTML elements. Rather than link you to another blog post where you can copy and paste the necessary CSS to style these elements, I am walk you through the process of styling them.
Checkout the Codepen for these elements.
MarkupThe typical markup for a checkbox or radio button looks like this:
<div class="checkbox__item"> <input type="checkbox" id="checkbox1" name="checkboxname" value="Item 2"> <label for="checkbox1">Item 2</label> </div> <div class="checkbox__item"> <input type="checkbox" id="checkbox2" name="checkboxname" value="Item 2"> <label for="checkbox2">Item 2</label> </div> <div class="radio__item"> <input type="radio" id="radio1" name="radioname" value="Item 2"> <label for="radio1">Item 1</label> </div> <div class="radio__item"> <input type="radio" id="radio2" name="radioname" value="Item 2"> <label for="radio2">Item 2</label> </div> StylesOut of the box, you can not style the native checkbox nor radio button elements. To be able to apply custom styles we need to do it in a hacky way. This is my favorite way to style these elements.
Drawing the checkbox /* visually hide the native checkbox and radio buttons. */ input[type='checkbox'], input[type='radio'] { position: absolute; opacity: 0; }- By setting absolute position and opacity to zero, we are hiding the native radio and checkbox elements from view. However, we need the elements to still function as expected so using visibility-hidden or display: none are not viable options.
- Although we are visually hiding the elements, they are still available in the DOM so we can interact with them and also so they can be available to assistive technologies such as screen readers.
- We set relative position on the label as we prepare to add pseudo elements in the next steps.
- The left padding here will allow us to draw a box or a circle for the checkbox/radio button. More on this later.
- First we define a sibling element of the checkbox with the + sign. The sibling element being the label element. If you look at the markup we wrote above, you will notice that the checkbox and the label are siblings. This is also true for the radio button and its label.
- Using a ::before pseudo element on the label, we are drawing a 24px square box.
- For any pesudo element to be visible on the page, we need to assign the property of content:, even if it's value is empty as shown above.
- We then add a 2px border on the box. This box is what will be presented as the actual checkbox for the user to interact with (check/uncheck).
- By setting the box with absolute position, we are positioning just to the left of the label. The position of the box is in relation to the label's position (hence relative position on the label).
- The same way we drew the box previously, now we draw a checkmark symbol using a ::after pseudo element on the label. This means we are drawing the checkmark symbol after the label element.
- In reality, we are drawing a rectangular box with left and bottom borders of 3px. By rotating the box negative 45 degrees, this makes it look like a checkmark.
- We then use absolute position on the checkmark so we can position it right in the middle of the box we drew before.
Before proceeding with the styles, let's repeat the steps above but this time for the radio buttons. These styles are extremely similar with the exception of in stead of drawing a square box, we will be drawing a circle with an outline, and instead of a checkmark, we will be drawing a filled circle in the middle of the outlined circle.
/* Draw a circle and add borders to it */ input[type='radio'] + label::before { border-radius: 50%; border: 2px solid #e15b00; content: ''; display: inline-block; height: 24px; left: 2px; position: absolute; top: -4px; width: 24px; } /* Draw an inner circle */ input[type='radio'] + label::after { background-color: #e15b00; border-radius: 50%; content: ''; display: inline-block; height: 20px; left: 6px; position: absolute; top: 0; width: 20px; }- This time we are targeting the input type of radio and we are using border-radius to draw a perfect circle.
- By default, we don't want either the checkboxes or radio buttons to be checked. So we set the value of content: to none. This will present the checkboxes and radio buttons unchecked.
- Using the :checked pseudo class (input[type='radio']:checked, input[type='checkbox']:checked), we then show the checkmark or inner circle by setting the content: property's value to empty ('' i.e. removing none). The :checked pseudo class becomes available when the checkbox or radio buttons are click-checked and when that happens, then we show the checkmark or inner circle.
- Since the checkmark we drew is white, we are adding a background color inside the checkbox to display the checkmark. We are doing this only when the :checked pseudo class is present.
- Finally, using the :focus pseudo class, we are adding basic outline styles to show the "on focus" state of the inputs. We need to do this since we are not using the native input elements. If we were, the browser would automatically add these styles to our inputs.
I have recorded tutorials on severalof these things. Check them out if you are the kind of person who learns better through video tutorials.
Mario Hernandez: Building an automated DDEV-based Drupal environment
A successful training experience begins before we step foot in the training room. Or, in these days of distance learning, it begins before students login to your training platform. The challenge is having an environment that is easy for students to setup and provides all the tooling required for the training. Configuring a native development environment is no easy task. Web development tools have gotten more complex with the years and it's a huge barrier for even experienced developers. My goal in setting up this new environment was to have everything completely configured and automated so students only needed to run one command. Very ambitious.
WARNING: The codebase shared in this post is only intended for local development. DO NOT use this project in a production website.
About DDEV: The official name is DDEV-Local. For simplicity I use DDEV in this tutorial.
Here are the tools and configuration requirements for this environment:
- Docker
- DDEV
- Composer
- Drush
- Drupal 8 and contrib modules
- Pre-built Drupal entities (content types, paragraph types, views, view modes, taxonomy, image styles, and more)
- Custom Drupal 8 theme
- Twig debugging enabled by default and Drupal cache disabled by default
- NodeJS, NPM, and NVM
- Pattern Lab
- Gulp, ESLint, Sass Lint, BrowserSync, Autoprefixer, and many many more node dependencies
My main objective was to simplify the building and interaction with the environment, by:
- Only require Docker and DDEV to be installed on host computer
- Reducing the number of steps for building the environment to one command, ddev start
- Execute most if not all commands in the containers, not the host machine (Drush, Composer, Pattern Lab, etc.)
- Access Pattern Lab running in the web container, from the host machine
Yes, it is crazy, but let's see how this turned out
First things first. As I said before, installing Docker and DDEV are the only two things students will need to install. There is no way around this. Luckily there are a lot of resources to help you with this. The DDEV docs is a great place to start.
Setting up a Drupal siteBefore we start, I'd like to clarify that this post focuses on outlining the process and steps for automating a local environment. However, before we can automate, we need to build all the pieces. If you just want to grab the final product here's the repo, otherwise, read on.
As of Drupal 8.8.0, Composer project templates are now available as part of Drupal core. These project templates are recommended for building new Drupal sites as they serve as a starting point for creating a Composer-managed Drupal site.
Two things we will be doing with our Drupal project:
- Drupal dependencies and modules will not be commited to the git repo. They will be downloaded with Composer when the project is being built.
- Composer will not be installed in the host system, instead, we will use the composer version that comes with DDEV's web container.
-
Create a new directory for your project. The directory name should be lowercase and alpha-numeric characters only. For this example I will name it drupaltraining and will create it in my /Sites directory. Using your command line tool create the new directory
mkdir drupaltraining -
Now navigate into the newly created directory
cd drupaltraining -
Run the DDEV command below to setup a new Drupal project
ddev config --project-type=drupal8 --docroot=web --create-docroot- This will create a .ddev directory with basic Drupal configuration.
- It will also create settings.php and settings.ddev.php` files inside web/sites/default.
- Finally, it creates a Drush directory.
- Keep in mind, Drupal is not in place yet, this simply sets the environment for it.
-
Now start DDEV to create the containers.
ddev start- After the project has been built, you will see a link to open Drupal in the browser. For this project the link should be https://drupaltraining.ddev.site (ignore for now). If you have not installed mkcert (mkcert -install), the link you see may use http instead of https. That's fine, but I'd recommend always using https by installing mkcert.
-
Now we are going to create the codebase for Drupal by using the official Drupal community
ddev composer create "drupal/recommended-project:^8"
Composer template.Respond Yes when asked if it's okay to override everything in the existing directory.
- This will setup the codebase for Drupal. This could take a while depending on your connection.
- composer.json will be updated so Drupal is added as a project's dependency.
- New settings.php and settings.ddev.php files are created.
- Using ddev composer create uses the version of composer that comes as part of the DDEV's web container. This means composer is not required to be installed in the host's computer.
- One thing I love about using the new composer template is the nice list of next steps you get after downloading Drupal's code base. So useful!
-
Now let's grab some modules
ddev composer require drupal/devel drupal/admin_toolbar drupal/paragraphs drupal/components drupal/viewsreference drupal/entity_reference_revisions drupal/twig_field_value- This will download the modules and update composer.json to set them as dependencies along with Drupal. The modules and Drupal's codebase will not be commited to our repo.
-
Let's add Drush to the project
ddev composer require drush/drush- Installing Drush as part of the project which will run in the web container will make it possible to run drush commands (ddev drush <command>), even if the host does not have drush installed.
-
Now launch Drupal in the browser to complete the installation
ddev launch- This will launch Drupal's intall page. Drupal's url will be https://drupaltraining.ddev.site. Complete the installation by using the Standard profile. Since the settings.ddev.php file already exists, the database configuration screen will be skipped from the installation.
-
Make a copy of example.settings.local.php into web/sites/default
cp web/sites/example.settings.local.php web/sites/default/settings.local.php- This is recommended to override or add new configuration to your Drupal site.
-
Update settings.php to include settings.local.php.
if (file_exists($app_root . '/' . $site_path . '/settings.local.php')) { include $app_root . '/' . $site_path . '/settings.local.php'; }- I'd suggest adding the above code right after the settings.ddev.php include already in settings.php. This will allow us to override configuration found in settings.ddev.php.
-
Let's change Drupal's default config directory. Doing this will export any configuration outside the web directory in Drupal. This is a good security measure. Let's also ignore a couple of folders to avoid drupal errors.
if (empty($settings['config_sync_directory'])) { $settings['config_sync_directory'] = '../config/sync'; } $settings['file_scan_ignore_directories'] = [ 'node_modules', 'bower_components', ];- First block changes the default sync directory to drupaltraining/config/sync. If you look inside settings.ddev.php you will see that this directory is inside web. We want to store any configuration changes outside the web directory.
- Second block sets up Drupal to ignore node_modules and bower_components. Drupal may look for twig templates inside these directories and could cause Drupal to crash. Ignoring these directories solves these issues.
-
Since we've made changes to DDEV's configuration, restart DDEV
ddev restart
That's quite the process, isn't it? The good news is this entire process will be eliminated when we finish automating the environment.
Drupal 8 custom themeThe project's theme is called training_theme. This is a node-based theme, and will be built with Mediacurrent's theme generator, which will provide:
- A best-practices Drupal 8 theme
- Pattern Lab integration
- Automated Front-End workflow
- Component-based-ready environment
- Production-ready theme
The final DDEV project will include the new Drupal 8 theme so there is no need to create it now, but if you want to see how the Theme Generator works, Watch the video tutorial I recorded.
Automating our environmentNow that Drupal has been setup let's begin the automation process.
DockerfileA web Docker container comes with Node and NPM installed. This will work in most cases, but the Drupal theme may use a version of node not currently available in the container. In addition, the web container does not include Node Version Manager (NVM), to manage multiple node versions. If the tools we need are not available in the web or db images/containers, there are ways to modify them to include the required tools. One of those ways is an add-on Dockerfile in your project's .ddev/web-build or .ddev/db-build, depending which container you are trying to modify.
Inside .ddev/web-build create a file called Dockerfile (case sensitive), and in it, add the following code:
ARG BASE_IMAGE FROM $BASE_IMAGE ENV NVM_DIR=/usr/local/nvm ENV NODE_DEFAULT_VERSION=v14.2.0 RUN curl -sL https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh -o install_nvm.sh RUN mkdir -p $NVM_DIR && bash install_nvm.sh RUN echo "source $NVM_DIR/nvm.sh" >>/etc/profile RUN bash -ic "nvm install $NODE_DEFAULT_VERSION && nvm use $NODE_DEFAULT_VERSION" RUN chmod -R ugo+w $NVM_DIRThe code above sets an environment variable for default node version (v14.2.0), which is the node version the theme uses at the time of this setup. It installs and configures NVM, and makes NVM executable by updating the container's bash profile. A Dockerfile runs while the image or container is being built to alter any default configuration with the code found in the Dockerfile.
Custom DDEV commandsThe Drupal 8 theme for this project uses Node for most of its tasks. To compile CSS, JavaScript, and Twig we need to run commands such as npm install, npm run build, npm run watch, and others. Our goal is to be able to run these commands in the web container, not the host computer. Doing this eliminates the need for students to install any node-related tools which can get really complicated. While we could achieve this by asking students to first SSH into the web container (ddev ssh), then navigate into /web/themes/custom/training_theme/, then run the commands, I want to make it even easier for them. I want them to be able to run the commands from any directory within the project and to not have to ssh into the web container. To achieve this we need to create a couple of custom commands.
Custom commands can be created to run in containers as well as the host machine. Since we want to run these commands in the web container, we are going to create the commands inside .ddev/commands/web/. Custom commands are bash script files.
NVM custom command- Inside .ddev/commands/web/ create a new file called nvm
- Add the following code in the file:
- Since this is a bash script, #!/bin/bash is required as the first line in the file.
- A description of the script is a good practice to explain what the script does.
- Pay close attention to ## Usage: nvm [flgas] [args]. This is what makes the commands work. The [flags] and [args] are ways to pass arguments to the command. For example, nvm on its own won't do much, but use or install can be passed as parameters to complete the commands nvm use or nvm install. Being able to run these commands will allow us to install new versions of Node later on if needed.
- Next we are adding examples of potential commands that can be run.
- Finally, you see the code or actual commands. source /etc/profile is basically resetting the container's bash profile so NVM can run. Then we navigate into the training_theme directory within the container where the nvm commands will be executed. So technically in the script above we are running 3 commands in one. Using && in between each command lets us combine or concatenate them. The $@ after nvm represent the flags or arguments we can pass (i.e. use or install).
Running the new commands: Every custom command needs to be executed by adding ddev before the command. For example: ddev nvm use. Using ddev infront of the command instructs the system to run the commands in the containers, rather than the host computer.
NPM custom commandsWe will also create a custom script to run NPM commands. This will be similar to the NVM script. This new script will be executed as ddev npm install or ddev npm run build, etc. The npm commands will allow us to install node dependencies by the theme as well as execute tasks like compiling code, linting code, compressing assets, and more.
- Create a new file inside .ddev/commands/web and call it npm
- Add the following code in the file:
- Most of the code here is similar to the previous script, except instead of nvm we will run npm.
- Notice we are again navigating into the theme directory before running the command. This make is possible for the custom commands we are creating to be ran from any directory within our project while still being executed inside the training_theme directory in the container.
Let's create one last custom command to run drush commands within the DDEV containers
- Create a new file inside .ddev/commands/web and call it drush
- Add the following code in the file:
- This will allow us to run drush commands in the container but using ddev drush <command> (i.e. ddev drush cr, ddev drush updb -y, etc.).
So that's it for custom commands. By having custom commands for nvm and npm, we can now successfully run any theming related tasks.
Automating Drupal's setupWe want to streamline the drupal installation process. In addition, we want to be able to import a custom database file to have access to all the infrastructure needed during training. This includes content types, paragraph types, views, view modes, image styles, and more. Enter DDEV hooks.
DDEV HooksHooks are a great way to perform tasks before or after DDEV starts. There are tasks that need to happen in specific sequence and hooks allow us to do just that. So what's the differnce between custom commands and hooks? Technically hooks can be considered custom commands, but the difference is that they are executed automatically before or after DDEV starts, whereas custom commands are ran on demand at any time. DDEV needs to be running if custom commands are intended to run in containers. Back to hooks, Let's build a post-start hook.
Hooks can be ran as pre-start, post-start, and after-db-import. Also, hooks can be executed inside containers and/or the host machine. In our case all the tasks we outlined above will be ran after DDEV starts and most of them in Docker's containers.
- Open .ddev/config.yaml and add the following code at the bottom of the file. There may already be a hooks section in your file. Be sure indentation in the file is correct.
- post-start: indicates tasks declared will run after DDEV container's have started. We want the web and db containers available before running any of the tasks.
- composer: install will download all dependencies found in composer.json (Drupal core, modules, drush and others).
- exec: /var/www/html/db/import-db.sh is a custom script which imports a custom database file (provided in this project). We will go over this script shortly. Importing a custom database builds the Drupal site without having to install Drupal as long as Drupal's codebase exists.
- exec: drush cim -y, exec: drush updb -y, and exec: drush cr, are basic drush commands.
- exec-host: cp -rf web/sites/example.development.services.yml web/sites/development.services.yml will create a new development.services.yml. This is needed because when Drupal is setup, development.services.yml is overridden and since we are using custom configuration in that file to enable twig debugging, we need to restore the configuration by replacing the file with a copy of our own.
- exec-host: cp -rf web/assets/images/* web/sites/default/files/images/ copies a collection of images used in the demo content added to the site.
- Finally, exec-host: ddev launch /user will open a fully configured Drupal website in the browser.
Login to Drupal: Username: admin, password: admin
Import database scriptNow let's write the script to import the database we are using in the hook above.
- In your project's root, create a new directory called db
- In the new directory create a new file called import-db.sh and add the following code:
-
This is again another bash script which performs a database import but only if the database is empty or clean. We do this by checking if one of the tables we expect in the database exists (node__field_hero). If it does, the database is not imported, but if it doesn't it will import the database. This table can be any table you know it should exist.
-
Notice the script calls for a database file named drupaltraining.sql.gz. This means the database file shold exist inside the db directory alongside the import-db.sh script. This database file was created/exported after Drupal was configured with all training required infrastructure and settings.
-
Make the script executable
- This will ensure the script can be executed by DDEV, otherwise we would get a permission denied error.
This does it for automation. The next few tasks are things that improve the development environment. Some of these tasks are optional.
Exposing Pattern Lab's port in host computerSince our goal is to not have to install any Front-End tools in the host computer, running Pattern Lab has to be done in the web container. The problem is we can't open Pattern Lab in the host's browser if Pattern Lab is running in the container. For this to work we need to expose the port in which Pattern Lab runs to the host machine. In this environment, that port is 3000 (this port number may vary). We identify this port by running npm run watch inside the training_theme directory. This will provide a series of links to access Pattern Lab in the browser.
Under the hood, DDEV uses docker-compose to define and run the multiple containers that make up the local environment for a project. docker-compose supports defining multiple compose files to facilitate sharing Compose configurations between files and projects, and DDEV is designed to leverage this ability.
Creating a new docker-compose.*.yamlA docker-compose file allows to do many things including exposing ports from the containers to the host computer.
On a typical Pattern lab project if you run npm start you will see Pattern Lab running on http://localhost:3000. In this environment the equivalent command is ddev npm run watch. Since this environment is running Pattern Lab in the web container, the only way to access Pattern Lab in the browser is by having access to the container's port 3000. Exposing the port via the http or https protocols makes it possible to access Pattern Lab's UI page from the host machine.
-
Inside .ddev/, create a new file called docker-compose.patternlab.yaml
-
In the new file add the following code:
# Override the web container's standard HTTP_EXPOSE and HTTPS_EXPOSE # to access patternlab on https://drupaltraining.ddev.site:3000, or http://drupaltraining.ddev.site:3001. version: '3.6' services: web: # ports are a list of exposed *container* ports ports: - "3000" environment: - HTTP_EXPOSE=${DDEV_ROUTER_HTTP_PORT}:80,${DDEV_MAILHOG_PORT}:8025,3001:3000 - HTTPS_EXPOSE=${DDEV_ROUTER_HTTPS_PORT}:80,${DDEV_MAILHOG_HTTPS_PORT}:8025,3000:3000- The name of the file is completely optional. It makes sense to use a name that is related to the action, app, or service you are trying to implement. In this example the name docker-compose.patternlab.yaml made sense.
- How did we arrive at the content above for this file? DDEV comes with two files that can be used as templates for new configuration, one of those files is called .ddev-docker-compose-base.yaml. You can find all the code we added above in this file.
-
Restart DDEV to allow for the new changes to take effect:
ddev restart- The basics of the code above is modifying the web container's port 3000. We are exposing this port through the http and https protocols on the host machine.
-
For the above to work Pattern Lab needs to be running in the container:
ddev npm run watch
Viewing Pattern Lab in the host machine
Using NFS (optional)Last but not least, enabling NFS in DDEV can help with performance of your application. NFS (Network File System) is a classic, mature Unix technique to mount a filesystem from one device to another. It provides significantly improved webserver performance on macOS and Windows. This is completely optional and in most cases you may not even need to do this. The steps below are for macOS only. Learn more about NFS and how to enable it in other Operating Systems like Windows.
- In your command line run
- Make note of uid (user id) and gid (group id).
- Open etc/exports in your code editor, and add the following code, preferably at the top of the file:
-
Replace xxxx with your username. The full path shown above is required if you are using macOS Catalina.
-
Replace Sites/Docker (This is my personal project's directory), with the directory name where your DDEV projects are created. Most people would mount the entire user's home directory, but I think only mounting the directory where your projects live is good enough (/Sites/Docker/).
-
Replace 502 and 20 with the values you got when you ran the id command above.
-
Update DDEV's config.yml to enable NFS
- Restart DDEV
You should notice an improvement in performance in your Drupal website.
In closingI realize there is a lot here, but I am pretty happy with how this turned out. Thanks to all the work in this article, when one of the students wants to setup their training environment, all the have to do is run ddev start. I think that's pretty sweet! 🙌 Happy DDEVing!
Giving creditBefore I started on this journey, I knew very little about DDEV. Thanks to the amazing help from Randy Fray from Drud.com and Michael Anello from DrupalEasy.com, I have learned a lot in the past weeks and wanted to share with other community members. You can find both, Randy and Michael on Drupal's DDEV Slack Channel. They are extremely helpful and responsive.
ResourcesMario Hernandez: Building a Drupal Theme with the Theme Generator
On one of my last training workshops I took a chance and decided to let students pick their environment of choice to use during training. As always I hosted an online call prior to training to assist anyone who needed help setting up their environment. About five of the students showed up to the call. This is a first. In the past when I've used a preconfigured training environment typically no one shows up to this prep call because the environment I've put together for them has been fully tested and any potential issues have been addressed.
Although I was able to help everyone get ready for training, and no big issues were encountered during training, I learned that perhpas having a preconfigured training environment is the best way to go. Having done this in the past I found that a preconfigured environment not only provides a consistent experience for everyone but it makes things more predictable for everyone.
I'm going to show you the latest setup I am using when training people. This is a new setup I put together using DDev with a host of other tools including Drupal
See the Theme Generator's project page on Github.
Watch the full tutorial below:
Mario Hernandez: Our best training yet
For the past few years we've been working on a training curriculum around Component Based Development and every time we finish updating it for an upcoming event we feel really good about it and think we finally got it. In reality, a training curriculum for a topic so complex as Component Development is probably never finished.
After BADCamp's training workshop last year we thought we were done updating the training material and we felt it was so good that it would carry us all the way to DrupalCon Seattle 2019. We could not have been more wrong. Soon after BADCamp we made the decision to move from KSS Node to Pattern Lab for handling our pattern's workflow and living styleguide. This meant our entire training material needed to be updated in time for DrupalCon. Needless to say we spent months ensuring the material was current, relevant and effective.
DrupalCon Seattle was a total success. Our training sold out in about two weeks and end up with almost 50 people in attendance which is probably a record for us. But with more students the chances of more issues or something going wrong increase. We make a big effort to ensure we provide an automated environment for student to use. This allows us provide all tools needed during training pre-configued and tested ensuring a more predictable and consistent behavior by students' systems. However, there are always cases in which a student decides not to use our environment and either due to technical skills or strict restrictions on their systems they need to use a workflow we have not tested.
This was certainly the case at DrupalCon and we had to find ways to help those students to be able to follow along during training.
One way to help people with their local setup is by conducting what we call "Office Hours". We reserve an hour or two weeks before the training where anyone who needs assistance with their local environment can join us online and we attempt to get them up and running. We helped several individuals this way. Although it is more work for us it allows us to address any issues off hours rather than spending valuable training time fixing computer problems.
The training was engaging and we received great feedback. Our curriculum, although not perfect, is relevant and well received. Having done this a few times at small and large events gives us confidence and makes it easier next time we do it.
Our team is dedicated and passionate about training and this would not be possible without them. Especial thanks go to Eric Huffman who has done most of the heavy lifting with the local environment automation. Kelly Dassing, who as a Director of Project Management provides a unique perspective to those in attendance who may not be developers. Tobias Williams who is an all around great FE develper and Engineer, and Mark Casias, who is able to answer the hard Back-end question that the rest of the team may not be as proficient as he is.
Should you have training needs please reach out and we would be happy to put together a curriculum that fits your specific needs or that of your team.
For a more in-depth look at how we prepare for a training workshop ready my blog post on Planning for an Effective Training Workshop.
Until next time ... Cheers!.
Mario Hernandez: Handling Drupal attributes in components
In Drupal's twig templates you'll often see an attributes variable being output within the template. This variable is how core and contrib modules inject their CSS classes, an ID, or data attributes onto template markup. You'll also find title_prefix and title_suffix variables. These are used by core and contrib modules to inject markup into twig templates. A good example of this is the core Contextual Links module. If you were to remove the attributes , title_prefix , and title_suffix variables from a node template, for example, then the Contextual Links module would no longer have a way to add its drop-down to the display of nodes.
In some cases this may not be an issue for you, but in general it's best to plan to accommodate those Drupal-specific variables in your component markup so that when you integrate Drupal content into your components, other features can be available too.
Since the attributes variable can include class, id, and data attributes in one variable, we need to make sure we only combine Drupal’s classes with ours, and let the other attributes render without Drupal classes. This can be accomplished on the main wrapper of the component template.
<article class="card{{ attributes ? ' ' ~ attributes.class }}" {{ attributes ? attributes|without(class) }}> {{ title_prefix }} {{ title_suffix }} {% if image %} <div class="card__image"> {{ image }} </div> {% endif %} <div class="card__content"> {% if heading %} {% include '@components/heading/heading.twig' with { "heading": { "title": heading.title, "url": heading.url, "heading_level": heading.heading_level, "classes": 'card__heading' } } only %} {% endif %} </div> </article>Note that the without twig filter in this example is a Drupal-specific filter, so for the component we'll want to make sure we’re using one that supports Drupal’s custom filters (most design systems such as KSS node, and Pattern Lab have configuration options that support Drupal twig filters).
Now if we integrate our card component with Drupal (i.e. node--card.html.twig), we can ensure Drupal's attributes and contextual links will be available when the component is rendered. The node template, also known as presenter template, would look something like this:
{% set rendered_content = content|render %} {% set heading = { title: label, url: url, heading_level: '4', attributes: title_attributes } %} {% embed '@components/card/card.twig' with { attributes: attributes, title_prefix: title_prefix, title_suffix: title_suffix, heading: heading, image: content.field_image is not empty ? content.field_image, } only %}- First we're triggering a full render of the content variable.
- Then we set up a variable for the Heading field,
- Finally we are using an embed twig statement to integrate the Card component. In the embed we are mapping all the Card fields with Drupal's data. We also pass in Drupal-specific items such as title_prefix, title_suffix, attributes, etc.
Our card component will be rendered with all Drupal's attributes and the ability to be edited inline thanks to the contextual links.
Hope this helps you in your component development journey.
Mario Hernandez: Surviving the fire
We are used to watching the news and somewhere in California there is a wildfire destroying nature and people's homes. In 30 years I have lived in the Los Angeles area I have never experienced this first hand. I've have been a victim of earthquakes but this was my first as a fire victim.
We first heard about the fires earlier in the week and the night of November 8th we knew the fires were escalating, but they were nowhere near our neighborhood. All this changed when we went to bed Thursday night and we noticed winds were pretty strong to the point that they would make loud and weird noises as they blew through trees and roof of our place.
We went to bed without a single concern. At 3:30 am. the morning of Friday November 9th, we woke up to loud screams and knocks on our door by the Fire Department. As I opened the door still half asleep, they told me we had 15 min minutes to get out and to head to the nearest shelter in place. They had to repeat this information about 3 times as I was still in disbelief and completely confused. As I looked out our door, this was the image I saw:
You probably can't tell from the picture but that fire is only feets away from our house. That is the same hills I hike on and now they are completely burned.
It gets worseMy wife and I got the kids ready and were only able to pick up passports and personal documents. We didn't have enough time to grab anything else except change cloths to something more practical for the weather.
We arrived at the shelter in place at around 4:00 am. and it was packed with people and pets. We also had a pet of our own. After registering we sit there for a while waiting to hear news but nothing was coming in. At around 5:00 am. we decided it would be best to leave to my parent's house, about 28 miles away. As we are ready to leave we see one of our neighbor who proceeds to give us the worst news of the morning. He informed us our building complex has been completely burned and our house is gone. We didn't know how to respond or react to these news. The thought that we no longer had a house was unreal. I recall telling my wife I didn't believe it but I had no way to prove it. I was probably in denial or perhaps hoping it wasn't true.
As we leave the shelter in place at around 5:30 am., my wife asked me to drive by our house to see for ourselves what it looked like. To our surprise, there was zero damage to our building or buildings in our neighborhood. I was both relieved and extremely upset at my neighbor for giving us such as bad and inaccurate information. I hope I can put this behind us, but for now I am still bothered by this.
The fire stopped short of our place and we could not be more fortunate. I was relieved and felt like we had made it through the worst when we saw our place still standing.
Then it gets betterWhile at the shelter in place, I notified several people at my job, Mediacurrent.com, that I was being evacuated and would not be able to work that day. The response from everyone was unbelievable. Mediacurrent's partner Paul Chason responded within seconds of my email to let me know that they were behind us and to let him know of anything we needed. This was the same message I received from many colleagues and friends. I have always felt fortunate to be part of a great team like Mediacurrent's, but they go above and beyond to support their employees. This is not the first time they have shown me this kind of support. The first time was when my wife was diagnosed with Chronic Kidney disease and Mediacurrent's response was unbelievable.
As other friends in social media learned about our situation, the flood of support was so great and it gave us strength and hope that we were going to be okay.
As we started communicating with other neighbors we began to feel more confident that we would be able to return home soon.
We came back home on Friday evening to pick up some stuff, including my wife's dialysis machine, which she uses every night as treatment for her chronic kidney disease, and also to clean up and change cloths. Things by then looked pretty normal in our neighborhood but the air quality was still pretty bad so we decided to go back to my parent one more night. On Saturday evening we returned home for good and began the cleaning process which was not as bad as it could had been.
In closingI hope we never have to go through something like this again, and really hope our friends and family never have to go through this, but having the support of our family, friends and coworkers made this horrible experience more manageable. We want to thank our family for always being there, and for providing a place to stay during this emergency. Our Mediacurrent family for showing us that we are more than coworkers, we care for each other. Thank you to our friends who would continuously check on us and offer help. Finally, our deepest and most special thanks to the everyday heroes who risk their lives to help others; first responders, fire fighters and law enforcements who worked and continue working tirelessly to protect us.
Thank you - The Hernandezes. 🙏 ❤️
Mario Hernandez: Adding Social Share Links to Gatsby
I've been working on my personal blog (this site), for a while. I built it with Gatsby and little by little I have been adding extra functionality. Today I'm going to show you how I added social sharing links to allow visitors to share my posts with others using Twitter, Facebook, LinkedIn, and other channels.
For an example of the Sharing links, look at the icons above the hero image on this and every post on this site.
There are many ways to accomplish this but from the begining I wanted to use something that was simple and did not require too much overhead to run. There are solutions out there that require third libraries and scripts and I wanted to avoid that. A while back I was introduced to Responsible Social Share Links. The beauty of Responsible Social Links is that they do not need any Javascript to work. They use the sharing links available for most social media channels.
Let's take a look at some examples of what these links look like:
Most of these links accepts several parameters. You can see these parameters in more details at the Responsible Social Share Links page for additional information. In addtion, some systems may require you to encode the links but luckily for us Reacts does this for us automatically.
Using the links in a Gatsby site (or React for that matter)You may think, that's so easy, just modify each of the links with my personal information and done. That's true to an extend. However, the tricky part is dynamically passing the current page's URL and post title to your sharing link. So here's how I did it:
-
Edit your blog post template. In my case my blog post template is /src/templates/blog-post.js This is based on the Gatsby starter I used. Your mileage may vary.
-
Add the following code where you wish to display the sharing links to generate a twitter share link:
The example above creates a twitter share link and uses the data variables I am already using to print the blog post content. As you know, Gatsby uses GraphQL to query the posts and by doing this you have access to each of the fields in your post (i.e. title, path, tags, date, etc.).
In the example above, I am passing ${post.frontmatter.title} so when the post is shared the title of the post is included as your tweet text. In addition, I am linking to the current post by passing ${post.frontmatter.path}. Finally I am passing my twitter handle.
There are other parameters you can pass to your share links. Things like hashtags, mentions, and more. Following the same pattern you can do the same for Facebook, LinkedIn and others.
A much cleaner approachYou may have noticed that I created the sharing snippet directly in the blog-post.js template. A much cleaner approach would be to create a new React component for all yoru sharing links and include the component in your blog-post.js.
Here's the full snippet for all the social channels I am using:
<Share> <ShareLabel>Share this post</ShareLabel> <ShareSocial> <ShareItem> <ShareLink href={`https://twitter.com/intent/tweet/?text=${ post.frontmatter.title }&url=https://mariohernandez.io${post.frontmatter.path}%2F&via=imariohernandez`} > <span> <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="icon icon-twitter" > <path d="M23 3a10.9 10.9 0 0 1-3.14 1.53 4.48 4.48 0 0 0-7.86 3v1A10.66 10.66 0 0 1 3 4s-4 9 5 13a11.64 11.64 0 0 1-7 2c9 5 20 0 20-11.5a4.5 4.5 0 0 0-.08-.83A7.72 7.72 0 0 0 23 3z" /> </svg> </span> <LinkLabel>Share on Twitter</LinkLabel> </ShareLink> </ShareItem> <ShareItem> <ShareLink href={`https://www.facebook.com/sharer/sharer.php?u=https://mariohernandez.io${ post.frontmatter.path }`} target="_blank" > <span> <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="icon icon-facebook" > <path d="M18 2h-3a5 5 0 0 0-5 5v3H7v4h3v8h4v-8h3l1-4h-4V7a1 1 0 0 1 1-1h3z" /> </svg> </span> <LinkLabel>Share on Facebook</LinkLabel> </ShareLink> </ShareItem> <ShareItem> <ShareLink href={`https://www.linkedin.com/shareArticle?mini=true&url=https://mariohernandez.io${ post.frontmatter.path }&title=${post.frontmatter.title}&source=${post.frontmatter.title}`} target="_blank" > <span> <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="icon icon-linkedin" > <path d="M16 8a6 6 0 0 1 6 6v7h-4v-7a2 2 0 0 0-2-2 2 2 0 0 0-2 2v7h-4v-7a6 6 0 0 1 6-6z" /> <rect x="2" y="9" width="4" height="12" /> <circle cx="4" cy="4" r="2" /> </svg> </span> <LinkLabel>Share on LinkedIn</LinkLabel> </ShareLink> </ShareItem> </ShareSocial> </Share> In ClosingIf you want to have a clean and light weight way to share your content with others, the Responsible Sharing Links may just be what you need.
Mario Hernandez: How to build a card component
One of the most popular components on any website I've worked on is the "Card" component. Depending on the website, the card component is used to group various pieces of content into a container which can be used throughout your website.
Read the full article on how to build a card comonent.
Mario Hernandez: Flexible Headings with Twig
Proper use of headings h1-h6 in your project presents many advantages incuding semantic markup, better SEO ranking and better accesibility.
Updated April 3, 2020
Building websites using the component based approach presents all kinds of advantages over the traditional page building approach. Today I’m going to show how to create what would normally be an Atom if we use the atomic design approach for building components. We are going to take this simple component to a whole new level by providing a way to dynamically controlling how it is rendered.
The heading componentHeadings are normally used for page or section titles and are a big part of making your website SEO friendly. As simple as this may sound, headings need to be carefully planned. A typical heading would look like this:
<h1>This is a Heading 1</h1>The idea of components is that they are reusable, but how can we possibly turn what already looks like a bare bones component into one that provides options and flexibility? What if we wanted to use a h2 or h3? or what if the title field is a link to another page? Then the heading component would probably not work because we have no way of changing the heading level from h1 to any other level or add a URL. Let's improve the heading component so we make it more dynamic.
Enter Twig and JSONTwig offers many advantages over plain HTML and today we will use some logic to transform the static heading component into a more dynamic one.
Let’s start by creating a simple JSON object which we will use as data for Twig to consume. We will build some logic around this data to make the heading component more dynamic. This is typically how I build components on projects I work on.
- In your project, typically within the components/patterns directory create a new folder called heading
- Inside the heading folder create a new file called heading.json
- Inside the new file paste the code snippet below
So we created a simple JSON object with 4 keys: heading_level, modifier, title, and url.
- The heading_level is something we can use to change the headings from say, h1 to h2 or h3 if we need to.
- The modifier key allows us to pass a modifier CSS class when we make use of this component. The modifier class will make it possible for us to style the heading differently than other headings, if needed.
- The title key is the title's string of text that will become the title of a page or a component.
- ... and finally, the url key, if present, will allow us to wrap the title in an <a> tag, to make it a link.
- Inside the heading folder create a new file called heading.twig
- Inside the new file paste the code snippet below
Wow! What's all this? 😮
Let's break things down to explain what's happening here since the twig code has changed significantly:
- First we make use of heading.heading_level to complete the number part of the heading. If a value is not provided for heading_level in the JSON file, we are setting a default of 2. This will ensue that by default we will have a <h2> as the title, much better than <h1> as we saw before. This value can be changed every time the heading isused. The same approach is taken to close the heading tag at the last line of code.
- Also, in addition to adding a class of heading, we check whether there is a value for the modifier key in JSON. If there is, we pass it to the heading as a CSS class. If no value is provided nothing will be added.
- In the next line line, we check whether a URL was provided in the JSON file, and if so, we wrap the Flexible Headings with Twig variable in a <a> tag to turn the title into a link. The href value for the link is ``. If no URL is provided in the JSON file, we simply print the value of Flexible Headings with Twig as plain text.
Well, our heading component is ready but unfortunately the component on its own does not do any good. The best way to take advantage of our super smart component is to start using it within other components.
Putting the heading component to useAs previously indicated, the idea of components is so they can be reusable which eliminates code duplication. Now that we have the heading component ready, we can reuse it in other templates by taking advantage of twig’s include statements. That will look like this:
<article class="card"> {% include '@components/heading/heading.twig' with { "heading": heading } only %} </article>The example above shows how we can reuse the heading component in the card component by using a Twig’s include statement.
NOTE: For this to work, the same data structure for the heading needs to exist in the card’s JSON file. Or, you could also alter the heading's values in twig, like this:
<article class="card"> {% include '@components/heading/heading.twig' with { "heading": { "heading_level": 3, "modifier": 'card__title', "title": "This is a super flexible and smart heading", "url": "https://mariohernandez.io" } } only %} </article>You noticed the part @components? this is only an example of a namespace. If you are not familiar with the component libraries Drupal module, it allows you to create namespaces for your theme which you can use to nest or include components as we see above.
End resultThe heading component we built above would look like this when it is rendered:
<h3 class="heading card__title"> <a href="https://mariohernandez.io" class="heading__link"> This is a super flexible and smart heading </a> </h3> In closingThe main goal of this post is to bring light on how important it is to build components that are not restricted and can be used throughout the site in a way that does not feel like you are repeating yourself.
Additional Resources:Mario Hernandez: Getting started with Gatsby
As many developers, when I hear the words "static website" I immediate think of creating flat HTML pages and editing them by hand. Times have changed. As you will see, Static Site Generators (SSG), offer some of the most advanced features and make use of latest technologies available on the web.
Static Site Generators are nothing new. If you search for SSG you will find many. One of the most popular ones is Jekyll, which I have personally worked with and it's a really good one. However, this post focusing on Gatsby. Probably one of the hottest system for creating static sites.
What is Gatsby?Gatsby's primarily objective is to build static sites, but as you will learn, that's just the tip of the iceberg.
Gatsby is a blazing-fast static site generator for React.
How does Gatsby work?While other SSGs use templating languages like Mustache, Handlebars, among others, Gatsby uses React. This not only allows for building modern component-driven websites, it also provides an incredible fast page rendering. Like mind-blowing fast.
Extending GatsbyOne of the most powerful features of Gatsby is its growing number of "Plugins". Plugins are the building blocks of Gatsby. They allow you to implement new features and functionality by running a couple of commands and making some configuration changes. Anything from adding Sass to your React project, creating a blog, configuring Google Anaylitics and many many more.
Plugins are contributed code kindly provided by the generous Open Source community which totally rocks. Anyone is able to write plugins and make them available to the world to consume and use.
Check out their Plugins page for a full list of ways you can take your static site to the next level.
Editorial ProcessSo we are building static sites and you may be wondering How do I create content for my site? There are several ways in which you can create a content editign workflow for your site. Probably the easiest way is to use static Markdown files. Markdown is a lightweight markup language with plain text formatting syntax. It is designed so that it can be converted to HTML and many other formats. Markdown is often used to format readme files, for writing messages in online discussion forums, and to create rich text using a plain text editor. This blog is using markdown. Since I am the only creating content I don't need a fancy administrative interface to crate content.
Markdown is only one of the ways you can create content for your static sites. Others include more advanced methods such as plugging in Gatsby with your Content Management System (CMS) of choice. This includes Wordpress, Drupal, Netlify, ContentaCMS, Contenful, and others. This means if you currently use any of those CMSs, you can continue to use them to retain a familiar workflow while moving your front-end workflow to a simpler and easier to manage process. This method is usually referred to as decoupled or headless, as your back-end is independent of your front-end.
Quering DataAs previously mentioned, Gatsby with the power of React create the perfect system for building robust, flexible and super fast static sites. However, there is a third component that takes that power to a whole new level, and that is GraphQL.
GraphQL is a query language for APIs and a runtime for fulfilling those queries with your existing data. GraphQL provides a complete and understandable description of the data in your API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.
Deploying GatsbyHosting for a Gatsby site can be done anywhere where React apps can run. Nowadays that's pretty much anywhere. However, before investing in an expensive and highly complicated hosting environment, take a look at some of the simpler and less expensive options on this page.
You will see that for a basic website, you can use several of the free options such as github pages, netlify and others which already include advanced continuous integration workflows. For more advanced sites where a CMS may be involved, you can also find options for deployment that will simplify your DevOps process.
My own blog is running out of a github repo that automatically get deployed when I push new updates. This is happening via Netlify which to me is probably the easiest way I have ever deployed a website.
In closingDon't worry if you are a little skeptical about static site generators. I was too. However, I gave Gatsby a try and I see myself building more gatsby sites in the future. Before Gatsby I worked with Jekyll which is also a great static site generator, but what sets Gatsby apart is its seamless integration with React and GraphQL. The combination of those 3 provides endless posibilities in your web building process. Check it out.
Mario Hernandez: Running a training workshop
Update 1-10-19
I wrote an extended version of this post at Mediacurrent's blog, check it out.
As long as I can remember I've enjoyed public speaking. This doesn't mean I am good at it, it simply means I enjoy it. School events, class president, my jobs, etc., they all taught me great lessons about public speaking. So when I started as a developer, sharing my knowledge with others at conferences or meetups came pretty natural.
I'd like to clarify, that after years of doing talks and other methods of public speaking, I am still terrified. I get nervous, my hands sweat, my legs shake, and my voice gets weird. Basically what I am trying to say is that I'm not an expert by any means, but I overcome the phobia of public speaking by doing it frequently.
For many years I have speaking at conferences, but in the past few years I started conducting longer workshops. I first started doing online workshops, which have their pros and cons. While they don't put you face to face with your audience, it also does not give you a good sense for how effective your training is because you can't see people reactions. For this reason I prefer to do face-to-face training.
As part of my job I conduct periodic Front-End training workshops for clients and recently I started conducting all-day training workshops at conferences. I really enjoy it and I'ld like to share some of the lessons learned.
Ideally you want to pick a topic you feel 100% confident about. I have learned that people attending your training or talks welcome any information you can share no matter how simple or elementary it may feel to you. Don't ever think what you know may not be of interest to others because you would be wrong.
Lately I have been challenging myself a little more when picking a topic to train about. While is good to know the topic well, it is also extremely rewarding to pick a topic you'd like to learn more about. This may feel contrary to what I said ealier but hear me out. When you decide to train on a topic, you will spend a lot of time preparing, training, testing and reharsing. This is exactly how you learn a new skill. I can't tell you how many times I come out of training I did knowing more about the topic than before and also learning from people who attended the training. If you want to learn a new skill, teaching others about it could be the best way for you to learn it.
Preparing for the trainingEveryone has their own style for teaching or doing a presentation. Some people like to use slides and screenshots, others show recordings of their project or code. My personal preference is to build a working prototype. This to me presents many advantages, but it also means you will spend more time getting ready.
My training workshops usually include very little slides because the majority of the training will be spent writing actual code and building the prototype during the training.
Here's my typical process for preparing for a training workshop:
-
Identify a prototype that serves the purpose of the training. If I am teaching a workshop about component based development I would normally pick something that involves the different aspects of component based development (attoms, nested components, reusable components, etc.)
-
Build the prototype upfront to ensure you have a working model to demo and go by.
-
Once prototype is built, create a public repo so you can share the working prototype
-
Write step-by-step instructions to building the prototype. Normally I would break the prototype down into small components, atoms.
-
Test, test and test. You want to make sure yoru audience will not run insto unexpected issues while following your instructions. For this reason you need to make sure you test your instructions. Ask a friend or colleague to go through each of your excercises to ensure thing work as expected.
-
Provide a pre-training evaluation. A quick set of questions that will give you an idea of people's skills level as well as environment (Linux, Widows, OSX). This will help you plan ahead of time.
-
Build a simple slide deck for introductions and agenda purposes. Mainly I move away from slides as soon as introduction and agenda is done. The rest of the training is all hands on.
As you will learn, one thing that can really kill a lot of the time during training is assisting people with their local environment setup. I have conducted training workshops where I've spent half the time helping people with their environment. For this reason, nowadays I communicate with the people ahead of time to ensure everyone's local environment is ready to go.
I normally make myself available once or twice in an evening through a google hangout to assit anyone who may need help. I also provide detailed instructions on how to get their local environment ready. This could save you a lot of time during training. In addition, for those who did spend the time on getting their environment ready, it's not fair that they have to be held back because someone did not make an effort to setup their environment.
I make myself available ahead of training but if someone is still having issues because of neglect, I don't hold the rest of the class back. I try to help them but at some point I move on.
If possible, get help from someone who is also well-versed with the topic so they can assist you help people who may get stuck. Nothing is more frustrating that havign to break the flow of the training to help people who get stuck. Having someone else help you with this allows you to continue with the training and not have everyone loose momentum.
FinallyEnjoy yourself. Make sure you and your audience have fun. If you show excitement in what you are doing people will get excited as well.
Dirk Eddelbuettel: nanotime 0.3.8 on CRAN: More Maintenance
Leonardo and I are happy to annunce that a new version 0.3.8 of our nanotime package arrived on CRAN today. It is the first release in over 1 1/2 years. nanotime relies on the RcppCCTZ package (as well as the RcppDate package for additional C++ operations) and offers efficient high(er) resolution time parsing and formatting up to nanosecond resolution, using the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from a rigorous refactoring by Leonardo who not only rejigged nanotime internals in S4 but also added new S4 types for periods, intervals and durations.
This release responds to a number of enhancements including a new paramter accurate for POSIXct to nanotime conversions, a vector date converter, a switch to double return value when durations objects are dividded – as well as a small battery of CRAN requests for changes and updates. This started with a move away from the now ‘non-API’ function SET_S4_OBJECT which has been replaced by use of Rf_asS4. We also no longer need a custom compiler flag on Windows (where for some reasons nobody understands or remembers, bitfields are not packed) to small enhancements of manual page formatting and last-but-not-least avoidance of some new UBSAN warnings. The NEWS snippet has the full details.
Changes in version 0.3.8 (2024-06-19)Time format documentation now has a reference to RcppCCTZ
The package no longer sets a default C++ compilation standard of C++11 (Dirk initially in #114, and later switched to C++17)
New accurate parameter for conversion from POSIXct to nanotime (Davor Josipovic and Leonardo in #116 closing #115)
The as.Date() function is now vectorized and can take a TZ argument (Leonardo and Dirk in #119 closing #118)
Use of internal function SET_S4_OBJECT has been replaced by API function Rf_asS4 (Leonardo in #121 closing #120)
An nanoduration / nanoduration expression now returns a double (Leonardo in #122 closing #117)
Bitfield calculations no longer require an Windows-only compiler switch (Leonardo in #124)
A simple manual page format nag involving has been addressed (Dirk in #126 fixing #125)
An set of tests tickling an UBSAN issue via Rcpp code no longer run unless CI is set (Dirk in #127 fixing #123)
Thanks to my CRANberries, there is a diffstat report for this release. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository – and all documentation is provided at the nanotime documentation site.
If you like this or other open-source work I do, you can sponsor me at GitHub.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
KDE PIM Sprint 2024 Report
In 2021 I decided to take a break from contributing to KDE, since I felt that I’ve been losing motivation and energy to contribute for a while… But I’ve been slowly getting back to hacking on KDE stuff for the past year, which ended in me going to Toulouse this year to attend the annual KDE PIM Sprint, my first in 5 years.
I’m very happy to say that we have /a lot/ going on in PIM, and even though not everything is in the best shape and the community is quite small (there were only four of us at the sprint), we have great plans for the future, and I’m happy to be part of it.
Day 0The sprint was officially supposed to start on Saturday, but everyone arrived already on Friday, so why wait? We wrote down the topics to discuss, put them on a whiteboard and got to it.
We’ve managed to discuss some pretty important topics - how we want to proceed with deprecation and removal of some components, how to improve our test coverage or how to improve indexing and much much more.
I arrived to the sprint with two big topics to discuss: milestones and testing:
MilestonesThe idea is to create milestones for all our bigger efforts that we work (or want to work) on. The milestones should be concrete goals that are achievable within a reasonable time frame and have clear definition of done. Each milestones should then be split to smaller tasks that can be tackled by individuals. We hope that this will help to make KDE PIM more attractive to new contributors, who can now clearly see what is being worked on and can find very concrete, bite-sized tasks to work on.
As a result, we took all the ongoing tasks and turned most of them into milestones in Gitlab. It’s still very much work in progress, we still need to break down many milestones to smaller tasks, but the general ideas are out there.
E2E Testing of ResourcesAkonadi Resources provide “bridge” between Akonadi Server and individual services, like IMAP servers, DAV servers, Google Calendar etc. But we have no tests to verify that our Resources can talk to the services and vice versa. The plan is to create a testing framework (in Python) so that we can have automated nightly tests to verify that e.g. IMAP resource interfaces properly with common IMAP server implementations, including major proprietary ones like Gmail or Office365. We want to achieve decent coverage for all our resources. This is a big project, but I think it’s a very exciting one as it includes not just programming, but also figuring out and building some infrastructure to run e.g. Dovecot, NextCloud and others in a Docker to test against.
Day 1On Saturday we started quite early, all the delicious french pastry is not going to eat itself, is it? After breakfast we continued with discussions, we dicussed tags support, how to improve our PR. But we also managed to produce some code. I implemented syncing of iCal categories with Akonadi tags, so the tags are becoming more useful. I also prepared Akonadi to be cleanly handle planned deprecation and retirement of KJots, KNotes and their acompanying resources, as well as planned removal of the Akonadi Kolab Resource (in favor of using IMAP+DAV).
One of the tasks I want to look into is improving how we do database transactions in the Akonadi Server. To get some data out of it, I shoved Prometheus exporter into Akonadi, hooked it up to a local Prometheus service, thrown together a Grafana dashboard, and here we are:
We decided to order some pizzas for dinner and stayed at the venue hacking until nearly 11 o’clock.
Day 2On the last day of the sprint we wrapped up on the discussions and focused on actually implementing some of the ideas. I spent most of the time extending the Migration agent to extract tags from all existing events and todos already stored in Akonadi and helped to create some of the milestones on the Gitlab board. We also came up with a plan for KDE PIM BoF on this years Akademy, where we want to present out progress on the respective milestones and to give a chance to contributors to learn what are the biggest hurdles they are facing when trying to contribute to KDE PIM and how we can help make it easier for them to get involved.
ConclusionI think it was a very productive sprint and I am really excited to be involved in PIM again. Can’t wait to meet up with everyone again on Akademy in September.
Go check out Kevin’s and Carl’s reports to see what else have they been up to during the sprint.
Did some of the milestones caught your eye, or do you have have any questions? Come talk to us in our matrix channel.
Finally, many thanks to Kevin for organizing the sprint, Étincelle Coworking for providing us with nice and spacious venue and KDE e.V. for supporting us on travel.
Finally, if you like such meetings to happen in the future so that we can push forward your favorite software, please consider making a tax-deductible donation to the KDE e.V. foundation.
Django Weblog: DjangoCon US: Call for Venue Proposal 2025
DEFNA is seeking proposals for a venue for DjangoCon US 2025 and ideally 2026. You can read the details on DEFNA’s site.
For 2025, we are looking at conference dates of October 5-10, 2025 or October 12-17, 2025.
The deadline for submissions is July 28, 2024. If you have any questions or concerns, please reach out to the DEFNA board at hello AT defna.org. We look forward to hearing from you!
Real Python: Build a Guitar Synthesizer: Play Musical Tablature in Python
Have you ever wanted to compose music without expensive gear or a professional studio? Maybe you’ve tried to play a musical instrument before but found the manual dexterity required too daunting or time-consuming. If so, you might be interested in harnessing the power of Python to create a guitar synthesizer. By following a few relatively simple steps, you’ll be able to turn your computer into a virtual guitar that can play any song.
In this tutorial, you’ll:
- Implement the Karplus-Strong plucked string synthesis algorithm
- Mimic different types of string instruments and their tunings
- Combine multiple vibrating strings into polyphonic chords
- Simulate realistic guitar picking and strumming finger techniques
- Use impulse responses of real instruments to replicate their unique timbre
- Read musical notes from scientific pitch notation and guitar tablature
At any point, you’re welcome to download the complete source code of the guitar synthesizer, as well as the sample tablature and other resources that you’ll use throughout this tutorial. They might prove useful in case you want to explore the code in more detail or get a head start. To download the bonus materials now, visit the following link:
Get Your Code: Click here to download the free sample code that you’ll use to build a guitar synthesizer in Python.
Take the Quiz: Test your knowledge with our interactive “Build a Guitar Synthesizer” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
Build a Guitar SynthesizerIn this quiz, you'll test your understanding of what it takes to build a guitar synthesizer in Python. By working through this quiz, you'll revisit a few key concepts from music theory and sound synthesis.
Demo: Guitar Synthesizer in PythonIn this step-by-step guide, you’ll build a plucked string instrument synthesizer based on the Karplus-Strong algorithm in Python. Along the way, you’ll create an ensemble of virtual instruments, including an acoustic, bass, and electric guitar, as well as a banjo and ukulele. Then, you’ll implement a custom guitar tab reader so that you can play your favorite songs.
By the end of this tutorial, you’ll be able to synthesize music from guitar tablature, or guitar tabs for short, which is a simplified form of musical notation that allows you to play music without having to learn how to read standard sheet music. Finally, you’ll store the result in an MP3 file for playback.
Below is a short demonstration of the synthesizer, re-creating the iconic soundtracks of classic video games like Doom and Diablo. Click the play button to listen to the sample output:
E1M1 - At Doom's Gate (Bobby Prince), Tristram (Matt Uelmen)Once you find a guitar tab that you like, you can plug it into your Python guitar synthesizer and bring the music to life. For example, the Songsterr website is a fantastic resource with a wide range of songs you can choose from.
Project OverviewFor your convenience, the project that you’re about to build, along with its third-party dependencies, will be managed by Poetry. The project will contain two Python packages with distinctly different areas of responsibility:
- digitar: For the synthesis of the digital guitar sound
- tablature: For reading and interpreting guitar tablature from a file
You’ll also design and implement a custom data format to store guitar tabs on disk or in memory. This will allow you to play music based on a fairly standard tablature notation, which you’ll find in various places on the Internet. Your project will also provide a Python script to tie everything together, which will let you interpret the tabs with a single command right from your terminal.
Now, you can dive into the details of what you’ll need to set up your development environment and start coding.
PrerequisitesAlthough you don’t need to be a musician to follow along with this tutorial, a basic understanding of musical concepts such as notes, semitones, octaves, and chords will help you grasp the information more quickly. It’d also be nice if you had a rough idea of how computers represent and process digital audio in terms of sampling rate, bit depth, and file formats like WAV.
But don’t worry if you’re new to these ideas! You’ll be guided through each step in small increments with clear explanations and examples. So, even if you’ve never done any music synthesis before, you’ll have a working digital guitar or digitar by the end of this tutorial.
Note: You can learn music theory in half an hour by watching an excellent and free video by Andrew Huang.
The project that you’ll build was tested against Python 3.12 but should work fine in earlier Python versions, too, down to Python 3.10. In case you need a quick refresher, here’s a list of helpful resources covering the most important language features that you’ll take advantage of in your digital guitar journey:
- Assignment expressions
- Data classes
- Enumerations
- Protocols (static duck typing)
- Structural pattern matching
- Type hints
Other than that, you’ll use the following third-party Python packages in your project:
- NumPy to simplify and speed up the underlying sound synthesis
- Pedalboard to apply special effects akin to electric guitar amplifiers
- Pydantic and PyYAML to parse musical tablature representing finger movements on a guitar neck
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Real Python: Quiz: Creating Great README Files for Your Python Projects
Test your understanding of how a great README file can make your Python project stand out and how to create your own README files.
Take this quiz after reading our Creating Great README Files for Your Python Projects tutorial.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
PyCharm: How to Move From pandas to Polars
This is a guest post from Cheuk Ting Ho, a data scientist who contributes to multiple open-source libraries, such as pandas and Polars.
You’ve probably heard about Polars – it is now firmly in the spotlight in the data science community.
Are you still using pandas and would like to try out Polars? Are you worried that it will take a lot of effort to migrate your projects from pandas to Polars? You might be concerned that Polars won’t be compatible with your existing pipeline or the other tools you are currently using.
Fear not! In this article, I will answer these questions so you can decide whether to migrate to using Polars or not. I will also provide some tips for those of you who have already decided to migrate.
How is Polars different from pandas?Polars is known for its speed and security, as it is written in Rust and based on Apache Arrow. For details about Polars vs. pandas, you can see our other blog post here. In short, while Polars’ backend architecture is different from pandas’, the creator and community around Polars have tried to maintain a Python API that is very similar to pandas’. At first glance, Polars code is very similar to pandas code. Fun fact – some contributors to pandas are also contributors to Polars. Due to this, the barrier for pandas users to start using Polars is relatively low. However, as it is still a different library, it is worth double-checking the differences between the two.
Advantages of using PolarsHave you struggled when using pandas for a relatively large data set? Do you think pandas is using too much RAM and slowing your computer down while working locally? Polars may solve this problem by using its lazy API. Intermediate steps won’t be executed unless needed, saving memory for the intermediate steps in some cases.
Another advantage Polars has is that, since it is written in Rust, it can make use of concurrency much better than pandas. Python is traditionally single-threaded, and although pandas uses the NumPy backend to speed up some operations, it is still mainly written in Python and has certain limitations in its multithreading capabilities.
Tools that make the switch easyAs Polars’ popularity grows, there is more and more support for Polars in popular tools for data scientists, including scikit-learn and HoloViz.
PyCharm, the most popular IDE used by data scientists, provides a similar experience when you work with pandas and Polars. This makes the process of migration smoother. For example, interactive tables allow you to easily see the information about your DataFrame, such as the number of rows and columns.
PyCharm has an excellent pagination feature – if you want to see more results per page, you can easily configure that via a drop-down menu:
You can see the statistical summary for the data when you hover the cursor over the column name:
You can also sort the data for inspection with a few clicks in the header. You can also use the multi-sorting functionality – after sorting the table once, press and hold ⌥ (macOS) or Alt (Windows) and click on the second column you want the table to be sorted by. For example, here, we can sort by island and bill_length_mm in the table.
To get more insights from the DataFrame, you can switch to chat view with the icon on the left:
You can also change how the data is shown in the settings, showing different columns and using different graph types:
It also helps you to auto-complete methods when using Polars, very handy when you are starting to use Polars and not familiar with all of the methods that it provides. To understand more about full line code completion in JetBrains IDEs, please check out this article.
You can also access the official documentation quickly by clicking the Polars icon in the top-right corner of the table, which is really handy.
How to migrate from pandas to PolarsIf you’re now convinced to migrate to Polars, your final questions might be about the extent of changes needed for your existing code and how easy it is to learn Polars, especially considering your years of experience and muscle memory with pandas.
Similarities between pandas and PolarsPolars provides APIs similar to pandas, most notably the read_csv(), head(), tail(), and describe() for a glance at what the data looks like. It also provides similar data manipulation functions like join() and groupby()/ group_by(), and aggregation functions like mean() and sum().
Before going into the migration, let’s look at these code examples in Polars and pandas.
Example 1 – Calculating the mean score for each class
pandas
import pandas as pd df_student = pd.read_csv("student_info.csv") print(df_student.dtypes) df_score = pd.read_csv("student_score.csv") print(df_score.head()) df_class = df_student.join(df_score.set_index("name"), on="name").drop("name", axis=1) df_mean_score = df_class.groupby("class").mean() print(df_mean_score)Polars
import polars as pl df_student = pl.read_csv("student_info.csv") print(df_student.dtypes) df_score = pl.read_csv("student_score.csv") print(df_score.head()) df_class = df_student.join(df_score, on="name").drop("name") df_mean_score = df_class.group_by("class").mean() print(df_mean_score)Polars provides similar io methods like read_csv. You can also inspect the dtypes, do data cleaning with drop, and do groupby with aggregation functions like mean.
Example 2 – Calculating the rolling mean of temperatures
pandas
import pandas as pd df_temp = pd.read_csv("temp_record.csv", index_col="date", parse_dates=True, dtype={"temp":int}) print(df_temp.dtypes) print(df_temp.head()) df_temp.rolling(2).mean()Polars
import polars as pl df_temp = pl.read_csv("temp_record.csv", try_parse_dates=True, dtypes={"temp":int}).set_sorted("date") print(df_temp.dtypes) print(df_temp.head()) df_temp.rolling("date", period="2d").agg(pl.mean("temp"))Reading with date as index in Polars can also be done with read_csv, with a slight difference in the function arguments. Rolling mean (or other types of aggregation) can also be done in Polars.
As you can see, these code examples are very similar, with only slight differences. If you are an experienced pandas user, I am sure your journey using Polars will be quite smooth.
Tips for migrating from pandas to PolarsAs for code that was previously written in pandas, how can you migrate it to Polars? What are the differences in syntax that may trip you up? Here are some tips that may be useful:
Selecting and filteringIn pandas, we use .loc / .iloc and [] to select part of the data in a data frame. However, in Polars, we use .select to do so. For example, in pandas df["age"] or df.loc[:,"age"] becomes df.select("age") in Polars.
In pandas, we can also create a mask to filter out data. However, in Polars, we will use .filter instead. For example, in pandas df["age" > 18] becomes df.filter(pl.col("a") > 18) in Polars.
All of the code that involves selecting and filtering data needs to be rewritten accordingly.
Use .with_columns instead of .assignA slight difference between pandas and Polars is that, in pandas we use .assign to create new columns by applying certain logic and operations to existing columns. In Polars, this is done with .with_columns. For example:
In pandas
df_rec.assign( diameter = lambda df: (df.x + df.y) * 2, area = lambda df: df.x * df.y )becomes
df_rec.with_columns( diameter = (pl.col("x") + pl.col("y")) * 2, area = lambda df: pl.col("x") * pl.col("y") )in Polars.
.with_columns can replace groupbyIn addition to assigning a new column with simple logic and operations, .with_columns offers more advanced capabilities. With a little trick, you can perform operations similar to groupby in pandas by using window functions:
In pandas
df = pd.DataFrame({ "class": ["a", "a", "a", "b", "b", "b", "b"], "score": ["80", "39", "67", "28", "77", "90", "44"], }) df["avg_score"] = df.groupby("class")["score"].transform("mean")becomes
df.with_columns( pl.col("score").mean().over("class").alias("avg_score") )in Polars.
Use scan_csv instead of read_csv if you canAlthough read_csv also works in Polars, by using scan_csv instead of read_csv it will turn to lazy evaluation mode and benefit from the lazy API mentioned above.
Building pipelines properly with lazy APIIn pandas, we usually use .pipe to build data pipelines. However, since Polars works a bit differently, especially when using the lazy API, we want the pipeline to be executed only once. So, we need to adjust the code accordingly. For example:
Instead of this pandas code snippet:
def discount(df): df["30_percent_off"] = df["price"] * 0.7 return df def vat(df): df["vat"] = df["price"] * 0.2 return df def total_cost(df): df["total"] = df["30_percent_off"] + df["vat"] return df (df .pipe(discount) .pipe(vat) .pipe(total_cost) )We will have the following one in Polars:
def discount(input_col)r: return pl.col(input_col).mul(0.7).alias("70_percent_off") def vat(input_col): return pl.col(input_col).mul(0.2).alias("vat") def total_cost(input_col1, input_col2): return pl.col(input_col1).add(pl.col(input_col2).alias("total") df.with_columns( discount("price"), val("price"), total_cost("30_percent_off", "vat"), ) Missing data: No more NaNDo you find NaN in pandas confusing? There is no NaN in Polars! Since NaN is an object in NumPy and Polars doesn’t use NumPy as the backend, all missing data will now be null instead. For details about null and NaN in Polars, check out the documentation.
Exploratory data analysis with PolarsPolars provides a similar API to pandas, and with hvPlot, you can easily create a simple plotting function with exploratory data analysis in Polars. Here I will show two examples, one creating simple statistical information from your data set, and the other plotting simple graphs to understand the data.
Summary statistics from datasetWhen using pandas, the most common way to get a summary statistic is to use describe. In Polars, we can also use describe in a similar manner. For example, we have a DataFrame with some numerical data and missing data:
We can use describe to get summary statistics:
Notice how object types are treated – in this example, the column name gives a different result compared to pandas. In pandas, a column with object type will result in categorical data like this:
In Polars, the result is similar to numeric data, which makes less sense:
Simple plotting with Polars DataFrame
To better visualize of the data, we might want to plot some graphs to help us evaluate the data more efficiently. Here is how to do so with the plot method in Polars.
First of all, since Polars uses hvPlot as backend, make sure that it is installed. You can find the hvPlot User Guide here. Next, since hvPlot will output the graph as an interactive Bokeh graph, we need to use output_notebook from bokeh.plotting to make sure it will show inline in the notebook. Add this code at the top of your notebook:
Also, make sure your notebook is trusted. This is done by simply checking the checkbox in the top-right of the display when using PyCharm.
Next, you can use the plot method in Polars. For example, to make a scatter plot, you have to specify the columns to be used as the x- and y-axis, and you can also specify the column to be used as color of the points:
df.plot.scatter(x="body_mass_g", y="bill_length_mm", color="species")This will give you a nice plot of the different data points of different penguin species for inspection:
Of course, scatter plots aren’t your only option. In Polars, you can use similar steps to create any type of plot that is supported by hvPlot. For example, hist can be done like this:
df.plot.hist("body_mass_g", by=["species","sex"])For a full list of plot types supported by hvPlot, you can have a look at the hvPlot reference gallery.
ConclusionI hope the information provided here will help you on your way with using Polars. Polars is an open-source project that is actively maintained and developed. If you have suggestions or questions, I recommend reaching out to the Polars community.
About the authorCheuk Ting Ho
Cheuk has been a Data Scientist at various companies – a job that demands high numerical and programming skills, especially in Python. Following her passion for the tech community, Cheuk has been a Developer Advocate for three years. She also contributes to multiple open-source libraries like Hypothesis, Pytest, pandas, Polars, PyO3, Jupyter Notebook, and Django. Cheuk is currently a consultant and trainer at CMD Limes.