FLOSS Project Planets

Jonathan Dowland: Shared notes and TODO lists

Planet Debian - Mon, 2019-08-19 15:55

When it comes to organising myself, I've long been anachronistic. I've relied upon paper notebooks for most of my life. In the last 15 years I've stuck to a particular type of diary/notebook hybrid, with a week-to-view on the left-hand side of pages and lined notebook pages on the right.

This worked well for me for my own personal stuff but obviously didn't work well for family things that need to be shared. Trying to find systems that work for both my wife and I has proven really challenging. The best we've come up with so far is a shared (IMAP) account and Apple's notes apps.

On iOS, Apple's low-frills note-taking app lets you synchronise your notes with a mail account (over IMAP). It stores them individually in HTML format, one email per note page, in a mailbox called "Notes". You can set up note syncing to the same account from multiple devices, and so we have a "family" mailbox set up on both my phone and my wife's. I can also get at the notes using any other mail client if I need to.

This works surprisingly well, but not perfectly. In particular synchronising changes to notes can go wrong if we both have the same note page open at the same time. The failure mode is not the worst: it duplicates the note into two; but it's still a problem.

Can anyone recommend a simple, more robust system for sharing notes — and task lists — between people? For task lists, it would be lovely (but not essential) if we could tick things off. At the moment we manage that just as free-form text.

Categories: FLOSS Project Planets

KMyMoney 5.0.6 released

Planet KDE - Mon, 2019-08-19 15:48

The KMyMoney development team today announces the immediate availability of version 5.0.6 of its open source Personal Finance Manager.

Another maintenance release is ready: KMyMoney 5.0.6 comes with some important bugfixes. As usual, problems have been reported by our users and the development team fixed some of them in the meantime. The result of this effort is the brand new KMyMoney 5.0.6 release.

Despite even more testing we understand that some bugs may have slipped past our best efforts. If you find one of them, please forgive us, and be sure to report it, either to the mailing list or on bugs.kde.org.

From here, we will continue to fix reported bugs, and working to add many requested additions and enhancements, as well as further improving performance.

Please feel free to visit our overview page of the CI builds at https://kmymoney.org/build.php and maybe try out the lastest and greatest by using a daily crafted AppImage version build from the stable branch.

The details

Here is the list of the bugs which have been fixed. A list of all changes between v5.0.5 and v5.0.6 can be found in the ChangeLog.

  • 408361 Hardly distinguishable line colors in reports
  • 410091 Open sqlite database under kmymoney versions >= 5.0
  • 410865 Access to german online banking requires product key
  • 411030 Attempt to move one split to new category moves all splits with same category

Here is the list of the enhancements which have been added:

  • The default price precision for the Indonesian Rupiah in new files has been raised to 10 decimals.
Categories: FLOSS Project Planets

PSF GSoC students blogs: Week 12: Blog Post (#6)

Planet Python - Mon, 2019-08-19 15:41

Last week, I worked on several steps to finalize my project.
Some work was put into correcting and improving my pull requests, so they can be merged soon. I created an example file to demonstrate the useability of this project, and opened up a new pull request for it (next to another PR, that introduces the plotting support I implemented the previous week).
As mentioned in my last blog post, I also thought about an alternative solution, i.e. to create a genuine plotting function for SourceTFR. But as I already suspected, this will not work without a lot of additional work. I originally intended to introduce a new type of plot derived from MNE's "TimeViewer", where you can not only skim along the time axis through the plotting GUI, but also along the frequency (and maybe the epochs) axis. After trying out some stuff, my best intuition to reaching this goal would be to simplify the "TimeViewer" class, and make it some kind of a "AxisViewer" class. This class would allow a developer to decide which axis (or even multiple axes) should be manipulable through the GUI. Yet, this would be only a part of the work, as this does not include the actual plotting function used by SourceTFR, and also would only work for the plots made on surface type source level data (since volume type source level data employs an entirely different plotting GUI). In my opinion, this functionality should rather be added later, so I can now concentrate on the finishing touches of the rest of the project.
One such finishing touch is for example time and memory saving computation of tfr_stockwell, which is for now only available for tfr_morlet and tfr_multitaper. I already made attempts to tackle this problem, and currently try to make it pass the equivalence tests. But this will definitely be one of the things to do this week, as it will have big implications on the practical useability of the function.

Another thing I worked on last week (and will continue to work on this week) was the project page where I'll upload my final results next week. I decided to create a GitHub gist for this purpose, since it is simple and good looking at the same time. Most of the gist is already finished, only the links to the PRs and some direct examples will need to be added there. So I hope, everything will work out this week, and I'll be able to show you a nice conclusion on my Google Summer of Code project until next week.

So, for the last time (this summer?!): Stay tuned!

Categories: FLOSS Project Planets

Kontact and Google Integration Issues

Planet KDE - Mon, 2019-08-19 14:53

Lately there were some issues with the Google integration in Kontact which caused that it is no longer possible to add new Google Calendar or Gmail account in Kontact because the log in process will fail. This is due to an oversight on our side which lead to Google blocking Kontact as it did not comply with Google’s policies. We are working on resolving the situation, but it will take a little bit.

Existing users should not be affected by this - if you already had Google Calendar or Gmail set up in Kontact, the sync should continue to work. It is only new accounts that cannot be created.

In case of Gmail the problem can mostly be worked around when setting up the IMAP account in KMail by selecting PLAIN authentication1 method in the Advanced tab and using your email and password. You may need to enable Less Secure Applications in your Google account settings in order to be able to log in with regular email address and password.

If you are interested in the technical background of this issue, the problem comes from Google’s OAuth App Verification process. When a developer wants to connect their app to a Google service they have to select which particular services their app needs access to, and sometimes even which data within each service they want to access. Google will then verify that the app is not trying to access any other data or that it is not misusing the data it has access to - this serves to protect Google users as they might sometimes approve apps that will access their calendars or emails with malicious intent without them realizing that.

When I registered Kontact I forgot to list some of the data points that Kontact needs access to. Google has noticed this after a while and asked us to clarify the missing bits. Unfortunately I wasn’t able to react within the given time limit and so Google has preemptively blocked login for all new users.

I’m working on clarifying the missing bits and having Google review the new information, so hopefuly the Google login should start working again soon.

  1. Despite its name, the PLAIN authentication method does not weaken the security. Your email and password are still sent safely encrypted over the internet. 

Categories: FLOSS Project Planets

Holger Levsen: 20190818-cccamp

Planet Debian - Mon, 2019-08-19 14:49
Home again

Two days ago I finally arrived home again and was greeted with this very nice view when entering the area:

(These images were taken yesterday from inside the venue.)

To give an idea of scale, the Pesthörnchen flag on top is 2m wide

Since today, there's also a rainbow flag next to the Pesthörnchen one. I'm very much looking forward to the next days, though buildup is big fun already.

Categories: FLOSS Project Planets

Antoine Beaupré: KNOB attack: Is my Bluetooth device insecure?

Planet Debian - Mon, 2019-08-19 13:58

A recent attack against Bluetooth, called KNOB, has been making waves last week. In essence, it allows an attacker to downgrade the security of a Bluetooth so much that it's possible for the attacker to break the encryption key and spy on all the traffic. The attack is so devastating that some have described it as the "stop using bluetooth" flaw.

This is my attempt at answering my own lingering questions about "can I still use Bluetooth now?" Disclaimer: I'm not an expert in Bluetooth at all, and just base this analysis on my own (limited) knowledge of the protocol, and some articles (including the paper) I read on the topic.

Is Bluetooth still safe?

It really depends what "safe" means, and what your threat model is. I liked how the Ars Technica article put it:

It's also important to note the hurdles—namely the cost of equipment and a surgical-precision MitM—that kept the researchers from actually carrying out their over-the-air attack in their own laboratory. Had the over-the-air technique been easy, they almost certainly would have done it.

In other words, the active attack is really hard to do, and the researchers didn't actually do one at all! It's a theoretical flaw, at this point, and while it's definitely possible, it's not what the researchers did:

The researchers didn't carry out the man-in-the-middle attack over the air. They did, however, root a Nexus 5 device to perform a firmware attack. Based on the response from the other device—a Motorola G3—the researchers said they believe that both attacks would work.

This led some researchers to (boldy) say they would still use a Bluetooth keyboard:

Dan Guido, a mobile security expert and the CEO of security firm Trail of Bits, said: "This is a bad bug, although it is hard to exploit in practice. It requires local proximity, perfect timing, and a clear signal. You need to fully MitM both peers to change the key size and exploit this bug. I'm going to apply the available patches and continue using my bluetooth keyboard."

So, what's safe and what's not, in my much humbler opinion?

Keyboards: bad

The attack is a real killer for Bluetooth keyboards. If an active attack is leveraged, it's game over: everything you type is visible to the attacker, and that includes, critically, passwords. In theory, one could even input keyboard events into the channel, which allows basically arbitrary code execution on the host.

Some, however, made the argument that it's probably easier to implant a keylogger in the device than actually do that attack, but I disagree: this requires physical access, while the KNOB attack can be done remotely.

How far this can be done, by the way, is still open to debate. The Telegraph claimed "a mile" in a click-bait title, but I think such an attacker would need to be much closer for this to work, more in the range of "meters" than "kilometers". But it still means "a black van sitting outside your house" instead of "a dude breaking into your house", which is a significant difference.

Other input devices: hum

I'm not sure mice and other input devices are such a big deal, however. Extracting useful information from those mice moving around the screen is difficult without seeing what's behind that screen.

So unless you use an on-screen keyboard or have special input devices, I don't think those are such a big deal when spied upon.

They could be leveraged with other attacks to make you "click through" some things an attacker would otherwise not be able to do.

Speakers: okay

I think I'll still keep using my Bluetooth speakers. But that's because I don't have much confidential audio I listen to. I listen to music, movies, and silly cat videos; not confidential interviews with victims of repression that should absolutely have their identities protected. And if I ever come across such material, I now know that I should not trust that speaker..

Otherwise, what's an attacker going to do here: listen to my (ever decreasing) voicemail (which is transmitted in cleartext by email anyways)? Listen to that latest hit? Meh.

Do keep in mind that some speakers have microphones in them as well, so that's not the entire story...

Headsets and microphones: hum

Headsets and microphones are another beast, as they can listen to other things in your environment. I do feel much less comfortable using those devices now. What makes the entire thing really iffy is some speakers do have microphones in them and all of a sudden everything around you can listen on your entire life.

(It seems like a given, with "smart home assistants" these days, but I still like to think my private conversations at home are private, in general. And I generally don't want to be near any of those "smart" devices, to be honest.)

One mitigating circumstance here is that the attack needs to happen during the connection (or pairing? still unclear) negociation, which doesn't happen that often if everything works correctly. Unfortunately, this happens more than often exactly with speakers and headsets. That's because many of those devices stupidly have low limits on the number of devices they can pair with. For example, the Bose Soundlink II can only pair with 8 other devices. If you count three device by person (laptop, workstation, phone), you quickly hit the limit when you move the device around. So I end up repairing that device quite often.

And that would be if the attack takes place during the pairing phase. As it turns out, the attack window is much wider: the attack happens during the connexion stage (see Figure 1, page 1049 in the paper), after devices have paired. This actually happens way more often than just during pairing. Any time your speaker or laptop will go to sleep, it will disconnect. Then to start using the device again, the BT layer will renegociate that keysize, and the attack can happen again.

(I have written the authors of the paper to clarify at which stage the attack happens and will update this post when/if they reply. Update: Daniele Antonioli has confirmed the attack takes place at connect phase.)

Fortunarely, the Bose Soundlink II has no microphone, which I'm thankful of. But my Bluetooth headset does have a microphone, which makes me less comfortable.

File and contact transfers: bad

Bluetooth, finally, is also used to transfer stuff other than audio of course. It's clunky, weird and barely working, but it's possible to send files over Bluetooth, and some headsets and car controllers will ask you permission to list your contacts so that "smart" features like "OK Google, call dad please" will work.

This attack makes it possible for an attacker to steal your contacts, when connecting devices. It can also intercept file transfers and so on.

That's pretty bad, to say the least.

Unfortunately, the "connection phase" mitigation described above is less relevant here. It's less likely you'll be continuously connecting two phones (or your phone and laptop) together for the purpose of file transfers. What's more likely is you'll connect the devices for explicit purpose of the file transfer, and therefore an attacker has a window for attack at every transfer.

I don't really use the "contacts" feature anyways (because it creeps me the hell out in the first place), so that's not a problem for me. But the file transfer problem will certainly give me pause the next time I ever need to feel the pain of transfering files over Bluetooth again, which I hope is "never".

It's interesting to note the parallel between this flaw, which will mostly affect Android file transfers, and the recent disclosure of flaws with Apple's Airdrop protocol which was similarly believed to be secure, even though it was opaque and proprietary. Now, think a bit about how Airdrop uses Bluetooth to negociate part of the protocol, and you can feel like I feel that everything in security just somewhat keeps crashes down and we don't seem to be able to make any progress at all.

Overall: meh

I've always been uncomfortable with Bluetooth devices: the pairing process has no sort of authentication whatsoever. The best you get is to enter a pin, and it's often "all zeros" or some trivially easy thing to bruteforce. So Bluetooth security has always felt like a scam, and I especially never trusted keyboards with passwords, in particular.

Like many branded attacks, I think this one might be somewhat overstated. Yes, it's a powerful attack, but Bluetooth implementations are already mostly proprietary junk that is undecipherable from the opensource world. There are no or very few open hardware implementations, so it's somewhat of expected we find things like this.

I have also found the response from the Bluetooth SIG is particularly alarming:

To remedy the vulnerability, the Bluetooth SIG has updated the Bluetooth Core Specification to recommend a minimum encryption key length of 7 octets for BR/EDR connections.

7 octets is 56 bits. That's the equivalent of DES, which was broken in 56 hours back, over 20 years ago. That's far from enough. But what's more disturbing is that this key size negociation protocol might be there "because 'some' governments didn't want other governments to have stronger encryption", ie. it would be a backdoor.

The 7-byte lower bound might also be there because of Apple lobbying. Their AirPods were implemented as not-standards-compliant and already have that lower 7-byte bound, so by fixing the standard to match one Apple implementation, they would reduce the cost of their recall/replacements/upgrades.

Overally, this behavior of the standards body is what should make us suspicious of any Bluetooth device going forward, and question the motivations of the entire Bluetooth standardization process. We can't use 56 bits keys anymore, and I can't believe I need to explicitely say so, but it seems it's where we're at with Bluetooth these days.

Categories: FLOSS Project Planets

Qt Visual Studio Tools 2.4 RC Released

Planet KDE - Mon, 2019-08-19 13:17

We have released Qt Visual Studio Tools 2.4 RC (version 2.4.0); the installation package is available in the Qt download page. This version features an improved integration of Qt tools with the Visual Studio project system, addressing some limitations of the current integration methods, most notably the inability to have different Qt settings for each project configuration, and lack of support for importing those settings from shared property sheets.

Using Qt with the Visual Studio Project System

The Visual Studio Project System is widely used as the build system of choice for C++ projects in VS. Under the hood, MSBuild provides the project file format and build framework. The Qt VS Tools make use of the extensibility of MSBuild to provide design-time and build-time integration of Qt in VS projects — toward the end of the post we have a closer look at how that integration works and what changed in the new release.

Up to this point, the Qt VS Tools extension managed its own project settings in an isolated manner. This approach prevented the integration of Qt in Visual Studio to fully benefit from the features of VS projects and MSBuild. Significantly, it was not possible to have Qt settings vary according to the build configuration (e.g. having a different list of selected Qt modules for different configurations), including Qt itself: only one version/build of Qt could be selected and would apply to all configurations, a significant drawback in the case of multi-platform projects.

Another important limitation that users of the Qt VS Tools have reported is the lack of support for importing Qt-related settings from shared property sheet files. This feature allows settings in VS projects to be shared within a team or organization, thus providing a single source for that information. Up to now, this was not possible to do with settings managed by the Qt VS Tools.

To overcome these and other related limitations, all Qt settings — such as the version of Qt, which modules are to be used or the path to the generated sources — will now be stored as fully fledged project properties. The current Qt Settings dialog will be removed and replaced by a Qt Settings property page. It will thus be possible to set the values of all Qt settings according to configuration, as well as import those values from property sheet files.

 

A closer look

An oversimplified primer might describe MSBuild as follows:

  • An MSBuild project consists of references to source files and descriptions of actions to take in order to process those source files — these descriptions are called targets.
  • The build process runs in the context of a project configuration (e.g. Debug, Release, etc.) A project may contain any number of configurations.
  • Data associated to source files and the project itself is accessible through properties. MSBuild properties are name-value definitions, specified per configuration (i.e. each configuration has its own set of property definitions).

Properties may apply to the project itself or to a specific file in the project, and can be defined globally or locally:

  • Project scope properties are always global (e.g. the project’s output directory or target file name).
  • Properties applying to source files can be defined globally, in which case the same value will apply to all files (e.g. default compiler warning level is defined globally at level 3).
  • Such a global, file-scope definition may be overridden for a specific file by a locally defined property with the same name (e.g. one of the source files needs to be compiled with warning level 4).
  • Global definitions are stored in the project file or imported from property sheet files.
  • Local property definitions are stored in the project file, within the associated source file references.

The Qt Visual Studio Tools extension integrates with the MSBuild project system by providing a set of Qt-specific targets that describe how to process files (e.g. a moc header) with the appropriate Qt tools.

The current integration has some limitations, with respect to the capabilities of the MSBuild project system:

  • User-managed Qt build settings are copied to project properties on change. Given this one-way synchronization, project properties may become out-of-sync with the corresponding Qt settings.
  • The value of the Qt build settings is the same for all configurations, e.g. the same Qt build and modules will be used, regardless of the selected configuration.
  • It is not possible to override properties in generated files like the meta-object source code output of moc.
  • Qt settings can only be stored in the project file. As such, it is not possible to import Qt definitions from shared property sheets, e.g. a common Qt build shared across several projects.

As discussed above, the solution for these limitations has been to make Qt settings fully fledged project properties. In this way, Qt settings will be guaranteed in-sync with all other properties in the project, and have the possibility of being defined differently for each build configuration. It will be possible to import Qt settings from property sheets, and the property page of Qt tools that generate C++ code, like moc, will now allow overriding compiler properties in generated files.

The post Qt Visual Studio Tools 2.4 RC Released appeared first on Qt Blog.

Categories: FLOSS Project Planets

Jacob Rockowitz: Requesting a medical appointment online begins a patient's digital journey

Planet Drupal - Mon, 2019-08-19 12:59

Experience

My experience with healthcare, Drupal, and webforms

For the past 20 years, I have worked in healthcare helping Memorial Sloan Kettering Cancer Center (MSKCC) evolve their digital platform and patient experience. About ten years ago, I persuaded MSKCC to switch to Drupal 6, which was followed by a migration to Drupal 8. More recently, I have become the maintainer of the Webform module for Drupal 8. Now, I want to leverage my experience and expertise in healthcare, webforms, and Drupal, to start exploring how we can improve patient and caregiver’s digital experience related to online appointment requests.

It’s important that we understand the problem/challenge of requesting an appointment online, examine how hospitals are currently solving this problem, and then offer some recommendations and ways to improve existing approaches. Instead of writing one very long blog post, I’m going to break up this discussion into a series of three blog posts. This initial post is going to address the patient journey and experience around an appointment request form.

These blog posts are not Drupal-specific, but my goal is to create and share an exemplary "Request an appointment" form template for the Webform module for Drupal 8.

Improving patient and caregiver’s digital experience

Improving the patient and caregiver digital experience is a very broad, massive, and challenging topic. Personally, my goal when working with doctors, researcher, and caregivers is…

Making things "easy" for patients and caregivers in healthcare is easier said...Read More

Categories: FLOSS Project Planets

Jonathan Dowland: NAS upgrade

Planet Debian - Mon, 2019-08-19 11:13

After 5 years of continuous service, the mainboard in my NAS recently failed (at the worst possible moment). I opted to replace the mainboard with a more modern version of the same idea: ASRock J4105-ITX featuring the Intel J4105, an integrated J-series Celeron CPU, designed to be passively cooled, and I've left the rest of the machine as it was.

In the process of researching which CPU/mainboard to buy, I was pointed at the Odroid-H2: a single-board computer (SBC) designed/marketed at a similar sector to things like the Raspberry PI (but featuring the exact same CPU as the mainboard I eventually settled on). I've always felt that the case I'm using for my NAS is too large, but didn't want to spend much money on a smaller one. The ODroid-H2 has a number of cheap, custom-made cases for different use-cases, including one for NAS-style work, which is in a very small footprint: the "Case 1". Unfortunately this case positions two disk drives flat, one vertically above the other, and both above the SBC. I was too concerned that one drive would be heating the other, and cumulatively both heating the SBC at that orientation. The case is designed with a fan but I want to avoid requiring one. I had too many bad memories of trying to control the heat in my first NAS, the Thecus n2100, which (by default) oriented the drives in the same way (and for some reason it never occurred to me to rotate that device into the "toaster" orientation).

I've mildly revised my NAS page to reflect the change. Interestingly most of the niggles I was experiencing were all about the old mainboard, so I've moved them on a separate page (J1900N-D3V) in case they are useful to someone.

At some point in the future I hope to spend a little bit of time on the software side of things, as some of the features of my set up are no longer working as they should: I can't remote-decrypt the main disk via SSH on boot, and the first run of any backup fails due to some kind of race condition in the systemd unit dependencies. (The first attempt does not correctly mount the backup partition; the second attempt always succeeds).

Categories: FLOSS Project Planets

Python Software Foundation: Humble Bundle by No Starch supports the Python Software Foundation!

Planet Python - Mon, 2019-08-19 10:07
We are super excited to announce that the Python Software Foundation is featured as a charity in a Humble Bundle by No Starch Press this month.



This bundle features books such as Python Playground, Mission Python, Invent your own Computer Games with Python, and so much more.

Click here to see all the books being featured and GET THE BUNDLE before it closes! The bundle will run from August 19th to September 2nd (11am Pacific). Proceeds received help charities such as the Python Software Foundation (PSF). Once you click to get your bundle, you can also choose where your money goes if you'd like to customize the split of proceeds.



Humble Bundle sells games, ebooks, software, and other digital content. Their mission is to support charity while providing awesome content to customers at great prices. Thanks to past Humble Bundles that the PSF has been a part of, this program has helped the PSF raise more than $300,000 since 2017! The PSF and the Python community thank Humble Bundle and all of the featured products that have selected the PSF as one of their charities. This funding has had a positive impact for Pythonistas all around the world.

No Starch is a long time community contributor supporting the PSF in various Bundles and supporting Young Coder classes that happen at PyCon US.  “As one of the leading publishers of Python books worldwide, No StarchPress is very excited to support the organization at the core of the Python programming language” said No Starch Press Founder Bill Pollock. “Python is at the core of so much technical work today and very much at the core of our publishing program.”

The PSF staff and board of directors send a big "Thank You!" to everyone involved.


Categories: FLOSS Project Planets

Real Python: How to Make a Discord Bot in Python

Planet Python - Mon, 2019-08-19 10:00

In a world where video games are so important to so many people, communication and community around games are vital. Discord offers both of those and more in one well-designed package. In this tutorial, you’ll learn how to make a Discord bot in Python so that you can make the most of this fantastic platform.

By the end of this article you’ll learn:

  • What Discord is and why it’s so valuable
  • How to make a Discord bot through the Developer Portal
  • How to create Discord connections
  • How to handle events
  • How to accept commands and validate assumptions
  • How to interact with various Discord APIs

You’ll begin by learning what Discord is and why it’s valuable.

Free Bonus: Click here to get access to a chapter from Python Tricks: The Book that shows you Python's best practices with simple examples you can apply instantly to write more beautiful + Pythonic code.

What Is Discord?

Discord is a voice and text communication platform for gamers.

Players, streamers, and developers use Discord to discuss games, answer questions, chat while they play, and much more. It even has a game store, complete with critical reviews and a subscription service. It is nearly a one-stop shop for gaming communities.

While there are many things you can build using Discord’s APIs, this tutorial will focus on a particular learning outcome: how to make a Discord bot in Python.

What Is a Bot?

Discord is growing in popularity. As such, automated processes, such as banning inappropriate users and reacting to user requests are vital for a community to thrive and grow.

Automated programs that look and act like users and automatically respond to events and commands on Discord are called bot users. Discord bot users (or just bots) have nearly unlimited applications.

For example, let’s say you’re managing a new Discord guild and a user joins for the very first time. Excited, you may personally reach out to that user and welcome them to your community. You might also tell them about your channels or ask them to introduce themselves.

The user feels welcomed and enjoys the discussions that happen in your guild and they, in turn, invite friends.

Over time, your community grows so big that it’s no longer feasible to personally reach out to each new member, but you still want to send them something to recognize them as a new member of the guild.

With a bot, it’s possible to automatically react to the new member joining your guild. You can even customize its behavior based on context and control how it interacts with each new user.

This is great, but it’s only one small example of how a bot can be useful. There are so many opportunities for you to be creative with bots, once you know how to make them.

Note: Although Discord allows you to create bots that deal with voice communication, this article will stick to the text side of the service.

There are two key steps when you’re creating a bot:

  1. Create the bot user on Discord and register it with a guild.
  2. Write code that uses Discord’s APIs and implements your bot’s behaviors.

In the next section, you’ll learn how to make a Discord bot in Discord’s Developer Portal.

How to Make a Discord Bot in the Developer Portal

Before you can dive into any Python code to handle events and create exciting automations, you need to first create a few Discord components:

  1. An account
  2. An application
  3. A bot
  4. A guild

You’ll learn more about each piece in the following sections.

Once you’ve created all of these components, you’ll tie them together by registering your bot with your guild.

You can get started by heading to Discord’s Developer Portal.

Creating a Discord Account

The first thing you’ll see is a landing page where you’ll need to either login, if you have an existing account, or create a new account:

If you need to create a new account, then click on the Register button below Login and enter your account information.

Important: You’ll need to verify your email before you’re able to move on.

Once you’re finished, you’ll be redirected to the Developer Portal home page, where you’ll create your application.

Creating an Application

An application allows you to interact with Discord’s APIs by providing authentication tokens, designating permissions, and so on.

To create a new application, select New Application:

Next, you’ll be prompted to name your application. Select a name and click Create:

Congratulations! You made a Discord application. On the resulting screen, you can see information about your application:

Keep in mind that any program that interacts with Discord APIs requires a Discord application, not just bots. Bot-related APIs are only a subset of Discord’s total interface.

However, since this tutorial is about how to make a Discord bot, navigate to the Bot tab on the left-hand navigation list.

Creating a Bot

As you learned in the previous sections, a bot user is one that listens to and automatically reacts to certain events and commands on Discord.

For your code to actually be manifested on Discord, you’ll need to create a bot user. To do so, select Add Bot:

Once you confirm that you want to add the bot to your application, you’ll see the new bot user in the portal:

Notice that, by default, your bot user will inherit the name of your application. Instead, update the username to something more bot-like, such as RealPythonTutorialBot, and Save Changes:

Now, the bot’s all set and ready to go, but to where?

A bot user is not useful if it’s not interacting with other users. Next, you’ll create a guild so that your bot can interact with other users.

Creating a Guild

A guild (or a server, as it is often called in Discord’s user interface) is a specific group of channels where users congregate to chat.

Note: While guild and server are interchangeable, this article will use the term guild primarily because the APIs stick to the same term. The term server will only be used when referring to a guild in the graphical UI.

For example, say you want to create a space where users can come together and talk about your latest game. You’d start by creating a guild. Then, in your guild, you could have multiple channels, such as:

  • General Discussion: A channel for users to talk about whatever they want
  • Spoilers, Beware: A channel for users who have finished your game to talk about all the end game reveals
  • Announcements: A channel for you to announce game updates and for users to discuss them

Once you’ve created your guild, you’d invite other users to populate it.

So, to create a guild, head to your Discord home page:

From this home page, you can view and add friends, direct messages, and guilds. From here, select the + icon on the left-hand side of the web page to Add a Server:

This will present two options, Create a server and Join a Server. In this case, select Create a server and enter a name for your guild:

Once you’ve finished creating your guild, you’ll be able to see the users on the right-hand side and the channels on the left:

The final step on Discord is to register your bot with your new guild.

Adding a Bot to a Guild

A bot can’t accept invites like a normal user can. Instead, you’ll add your bot using the OAuth2 protocol.

Technical Detail: OAuth2 is a protocol for dealing with authorization, where a service can grant a client application limited access based on the application’s credentials and allowed scopes.

To do so, head back to the Developer Portal and select the OAuth2 page from the left-hand navigation:

From this window, you’ll see the OAuth2 URL Generator.

This tool generates an authorization URL that hits Discord’s OAuth2 API and authorizes API access using your application’s credentials.

In this case, you’ll want to grant your application’s bot user access to Discord APIs using your application’s OAuth2 credentials.

To do this, scroll down and select bot from the SCOPES options and Administrator from BOT PERMISSIONS:

Now, Discord has generated your application’s authorization URL with the selected scope and permissions.

Disclaimer: While we’re using Administrator for the purposes of this tutorial, you should be as granular as possible when granting permissions in a real-world application.

Select Copy beside the URL that was generated for you, paste it into your browser, and select your guild from the dropdown options:

Click Authorize, and you’re done!

Note: You might get a reCAPTCHA before moving on. If so, you’ll need to prove you’re a human.

If you go back to your guild, then you’ll see that the bot has been added:

In summary, you’ve created:

  • An application that your bot will use to authenticate with Discord’s APIs
  • A bot user that you’ll use to interact with other users and events in your guild
  • A guild in which your user account and your bot user will be active
  • A Discord account with which you created everything else and that you’ll use to interact with your bot

Now, you know how to make a Discord bot using the Developer Portal. Next comes the fun stuff: implementing your bot in Python!

How to Make a Discord Bot in Python

Since you’re learning how to make a Discord bot with Python, you’ll be using discord.py.

discord.py is a Python library that exhaustively implements Discord’s APIs in an efficient and Pythonic way. This includes utilizing Python’s implementation of Async IO.

Begin by installing discord.py with pip:

$ pip install -U discord.py

Now that you’ve installed discord.py, you’ll use it to create your first connection to Discord!

Creating a Discord Connection

The first step in implementing your bot user is to create a connection to Discord. With discord.py, you do this by creating an instance of Client:

# bot.py import os import discord from dotenv import load_dotenv load_dotenv() token = os.getenv('DISCORD_TOKEN') client = discord.Client() @client.event async def on_ready(): print(f'{client.user} has connected to Discord!') client.run(token)

A Client is an object that represents a connection to Discord. A Client handles events, tracks state, and generally interacts with Discord APIs.

Here, you’ve created a Client and implemented its on_ready() event handler, which handles the event when the Client has established a connection to Discord and it has finished preparing the data that Discord has sent, such as login state, guild and channel data, and more.

In other words, on_ready() will be called (and your message will be printed) once client is ready for further action. You’ll learn more about event handlers later in this article.

When you’re working with secrets such as your Discord token, it’s good practice to read it into your program from an environment variable. Using environment variables helps you:

  • Avoid putting the secrets into source control
  • Use different variables for development and production environments without changing your code

While you could export DISCORD_TOKEN={your-bot-token}, an easier solution is to save a .env file on all machines that will be running this code. This is not only easier, since you won’t have to export your token every time you clear your shell, but it also protects you from storing your secrets in your shell’s history.

Create a file named .env in the same directory as bot.py:

# .env DISCORD_TOKEN={your-bot-token}

You’ll need to replace {your-bot-token} with your bot’s token, which you can get by going back to the Bot page on the Developer Portal and clicking Copy under the TOKEN section:

Looking back at the bot.py code, you’ll notice a library called dotenv. This library is handy for working with .env files. load_dotenv() loads environment variables from a .env file into your shell’s environment variables so that you can use them in your code.

Install dotenv with pip:

$ pip install -U python-dotenv

Finally, client.run() runs your Client using your bot’s token.

Now that you’ve set up both bot.py and .env, you can run your code:

$ python bot.py RealPythonTutorialBot#9643 has connected to Discord!

Great! Your Client has connected to Discord using your bot’s token. In the next section, you’ll build on this Client by interacting with more Discord APIs.

Interacting With Discord APIs

Using a Client, you have access to a wide range of Discord APIs.

For example, let’s say you wanted to write the name and identifier of the guild that you registered your bot user with to the console.

First, you’ll need to add a new environment variable:

# .env DISCORD_TOKEN={your-bot-token} DISCORD_GUILD={your-guild-name}

Don’t forget that you’ll need to replace the two placeholders with actual values:

  1. {your-bot-token}
  2. {your-guild-name}

Remember that Discord calls on_ready(), which you used before, once the Client has made the connection and prepared the data. So, you can rely on the guild data being available inside on_ready():

# bot.py import os import discord from dotenv import load_dotenv load_dotenv() TOKEN = os.getenv('DISCORD_TOKEN') GUILD = os.getenv('DISCORD_GUILD') client = discord.Client() @client.event async def on_ready(): for guild in client.guilds: if guild.name == GUILD: break print( f'{client.user} is connected to the following guild:\n' f'{guild.name}(id: {guild.id})' ) client.run(TOKEN)

Here, you looped through the guild data that Discord has sent client, namely client.guilds. Then, you found the guild with the matching name and printed a formatted string to stdout.

Note: Even though you can be pretty confident at this point in the tutorial that your bot is only connected to a single guild (so client.guilds[0] would be simpler), it’s important to realize that a bot user can be connected to many guilds.

Therefore, a more robust solution is to loop through client.guilds to find the one you’re looking for.

Run the program to see the results:

$ python bot.py RealPythonTutorialBot#9643 is connected to the following guild: RealPythonTutorialServer(id: 571759877328732195)

Great! You can see the name of your bot, the name of your server, and the server’s identification number.

Another interesting bit of data you can pull from a guild is the list of users who are members of the guild:

# bot.py import os import discord from dotenv import load_dotenv load_dotenv() TOKEN = os.getenv('DISCORD_TOKEN') GUILD = os.getenv('DISCORD_GUILD') client = discord.Client() @client.event async def on_ready(): for guild in client.guilds: if guild.name == GUILD: break print( f'{client.user} is connected to the following guild:\n' f'{guild.name}(id: {guild.id})\n' ) members = '\n - '.join([member.name for member in guild.members]) print(f'Guild Members:\n - {members}') client.run(TOKEN)

By looping through guild.members, you pulled the names of all of the members of the guild and printed them with a formatted string.

When you run the program, you should see at least the name of the account you created the guild with and the name of the bot user itself:

$ python bot.py RealPythonTutorialBot#9643 is connected to the following guild: RealPythonTutorialServer(id: 571759877328732195) Guild Members: - aronq2 - RealPythonTutorialBot

These examples barely scratch the surface of the APIs available on Discord, be sure to check out their documentation to see all that they have to offer.

Next, you’ll learn about some utility functions and how they can simplify these examples.

Using Utility Functions

Let’s take another look at the example from the last section where you printed the name and identifier of the bot’s guild:

# bot.py import os import discord from dotenv import load_dotenv load_dotenv() TOKEN = os.getenv('DISCORD_TOKEN') GUILD = os.getenv('DISCORD_GUILD') client = discord.Client() @client.event async def on_ready(): for guild in client.guilds: if guild.name == GUILD: break print( f'{client.user} is connected to the following guild:\n' f'{guild.name}(id: {guild.id})' ) client.run(TOKEN)

You could clean up this code by using some of the utility functions available in discord.py.

discord.utils.find() is one utility that can improve the simplicity and readability of this code by replacing the for loop with an intuitive, abstracted function:

# bot.py import os import discord from dotenv import load_dotenv load_dotenv() TOKEN = os.getenv('DISCORD_TOKEN') GUILD = os.getenv('DISCORD_GUILD') client = discord.Client() @client.event async def on_ready(): guild = discord.utils.find(lambda g: g.name == GUILD, client.guilds) print( f'{client.user} is connected to the following guild:\n' f'{guild.name}(id: {guild.id})' ) client.run(TOKEN)

find() takes a function, called a predicate, which identifies some characteristic of the element in the iterable that you’re looking for. Here, you used a particular type of anonymous function, called a lambda, as the predicate.

In this case, you’re trying to find the guild with the same name as the one you stored in the DISCORD_GUILD environment variable. Once find() locates an element in the iterable that satisfies the predicate, it will return the element. This is essentially equivalent to the break statement in the previous example, but cleaner.

discord.py has even abstracted this concept one step further with the get() utility:

# bot.py import os import discord from dotenv import load_dotenv load_dotenv() TOKEN = os.getenv('DISCORD_TOKEN') GUILD = os.getenv('DISCORD_GUILD') client = discord.Client() @client.event async def on_ready(): guild = discord.utils.get(client.guilds, name=GUILD) print( f'{client.user} is connected to the following guild:\n' f'{guild.name}(id: {guild.id})' ) client.run(TOKEN)

get() takes the iterable and some keyword arguments. The keyword arguments represent attributes of the elements in the iterable that must all be satisfied for get() to return the element.

In this example, you’ve identified name=GUILD as the attribute that must be satisfied.

Technical Detail: Under the hood, get() actually uses the attrs keyword arguments to build a predicate, which it then uses to call find().

Now that you’ve learned the basics of interacting with APIs, you’ll dive a little deeper into the function that you’ve been using to access them: on_ready().

Responding to Events

You already learned that on_ready() is an event. In fact, you might have noticed that it is identified as such in the code by the client.event decorator.

But what is an event?

An event is something that happens on Discord that you can use to trigger a reaction in your code. Your code will listen for and then respond to events.

Using the example you’ve seen already, the on_ready() event handler handles the event that the Client has made a connection to Discord and prepared its response data.

So, when Discord fires an event, discord.py will route the event data to the corresponding event handler on your connected Client.

There are two ways in discord.py to implement an event handler:

  1. Using the client.event decorator
  2. Creating a subclass of Client and overriding its handler methods

You already saw the implementation using the decorator. Next, take a look at how to subclass Client:

# bot.py import os import discord from dotenv import load_dotenv load_dotenv() token = os.getenv('DISCORD_TOKEN') class CustomClient(discord.Client): async def on_ready(self): print(f'{self.user} has connected to Discord!') client = CustomClient() client.run(token)

Here, just like before, you’ve created a client variable and called .run() with your Discord token. The actual Client is different, however. Instead of using the normal base class, client is an instance of CustomClient, which has an overridden on_ready() function.

There is no difference between the two implementation styles of events, but this tutorial will primarily use the decorator version because it looks similar to how you implement Bot commands, which is a topic you’ll cover in a bit.

Technical Detail: Regardless of how you implement your event handler, one thing must be consistent: all event handlers in discord.py must be coroutines.

Now that you’ve learned how to create an event handler, let’s walk through some different examples of handlers you can create.

Welcoming New Members

Previously, you saw the example of responding to the event where a member joins a guild. In that example, your bot user could send them a message, welcoming them to your Discord community.

Now, you’ll implement that behavior in your Client, using event handlers, and verify its behavior in Discord:

# bot.py import os import discord from dotenv import load_dotenv load_dotenv() token = os.getenv('DISCORD_TOKEN') client = discord.Client() @client.event async def on_ready(): print(f'{client.user.name} has connected to Discord!') @client.event async def on_member_join(member): await member.create_dm() await member.dm_channel.send( f'Hi {member.name}, welcome to my Discord server!' ) client.run(token)

Like before, you handled the on_ready() event by printing the bot user’s name in a formatted string. New, however, is the implementation of the on_member_join() event handler.

on_member_join(), as its name suggests, handles the event of a new member joining a guild.

In this example, you used member.create_dm() to create a direct message channel. Then, you used that channel to .send() a direct message to that new member.

Technical Detail: Notice the await keyword before member.create_dm() and member.dm_channel.send().

await suspends the execution of the surrounding coroutine until the execution of each coroutine has finished.

Now, let’s test out your bot’s new behavior.

First, run your new version of bot.py and wait for the on_ready() event to fire, logging your message to stdout:

$ python bot.py RealPythonTutorialBot has connected to Discord!

Now, head over to Discord, log in, and navigate to your guild by selecting it from the left-hand side of the screen:

Select Invite People just beside the guild list where you selected your guild. Check the box that says Set this link to never expire and copy the link:

Now, with the invite link copied, create a new account and join the guild using your invite link:

First, you’ll see that Discord introduced you to the guild by default with an automated message. More importantly though, notice the badge on the left-hand side of the screen that notifies you of a new message:

When you select it, you’ll see a private message from your bot user:

Perfect! Your bot user is now interacting with other users with minimal code.

Next, you’ll learn how to respond to specific user messages in the chat.

Responding to Messages

Let’s add on to the previous functionality of your bot by handling the on_message() event.

on_message() occurs when a message is posted in a channel that your bot has access to. In this example, you’ll respond to the message '99!' with a one-liner from the television show Brooklyn Nine-Nine:

@client.event async def on_message(message): if message.author == client.user: return brooklyn_99_quotes = [ 'I\'m the human form of the 💯 emoji.', 'Bingpot!', ( 'Cool. Cool cool cool cool cool cool cool, ' 'no doubt no doubt no doubt no doubt.' ), ] if message.content == '99!': response = random.choice(brooklyn_99_quotes) await message.channel.send(response)

The bulk of this event handler looks at the message.content, checks to see if it’s equal to '99!', and responds by sending a random quote to the message’s channel if it is.

The other piece is an important one:

if message.author == client.user: return

Because a Client can’t tell the difference between a bot user and a normal user account, your on_message() handler should protect against a potentially recursive case where the bot sends a message that it might, itself, handle.

To illustrate, let’s say you want your bot to listen for users telling each other 'Happy Birthday'. You could implement your on_message() handler like this:

@client.event async def on_message(message): if 'happy birthday' in message.content.lower(): await message.channel.send('Happy Birthday! 🎈🎉')

Aside from the potentially spammy nature of this event handler, it also has a devastating side effect. The message that the bot responds with contains the same message it’s going to handle!

So, if one person in the channel tells another “Happy Birthday,” then the bot will also chime in… again… and again… and again:

That’s why it’s important to compare the message.author to the client.user (your bot user), and ignore any of its own messages.

So, let’s fix bot.py:

# bot.py import os import random import discord from dotenv import load_dotenv load_dotenv() token = os.getenv('DISCORD_TOKEN') client = discord.Client() @client.event async def on_ready(): print(f'{client.user.name} has connected to Discord!') @client.event async def on_member_join(member): await member.create_dm() await member.dm_channel.send( f'Hi {member.name}, welcome to my Discord server!' ) @client.event async def on_message(message): if message.author == client.user: return brooklyn_99_quotes = [ 'I\'m the human form of the 💯 emoji.', 'Bingpot!', ( 'Cool. Cool cool cool cool cool cool cool, ' 'no doubt no doubt no doubt no doubt.' ), ] if message.content == '99!': response = random.choice(brooklyn_99_quotes) await message.channel.send(response) client.run(token)

Don’t forget to import random at the top of the module, since the on_message() handler utilizes random.choice().

Run the program:

$ python bot.py RealPythonTutorialBot has connected to Discord!

Finally, head over to Discord to test it out:

Great! Now that you’ve seen a few different ways to handle some common Discord events, you’ll learn how to deal with errors that event handlers may raise.

Handling Exceptions

As you’ve seen already, discord.py is an event-driven system. This focus on events extends even to exceptions. When one event handler raises an Exception, Discord calls on_error().

The default behavior of on_error() is to write the error message and stack trace to stderr. To test this, add a special message handler to on_message():

# bot.py import os import random import discord from dotenv import load_dotenv load_dotenv() token = os.getenv('DISCORD_TOKEN') client = discord.Client() @client.event async def on_ready(): print(f'{client.user.name} has connected to Discord!') @client.event async def on_member_join(member): await member.create_dm() await member.dm_channel.send( f'Hi {member.name}, welcome to my Discord server!' ) @client.event async def on_message(message): if message.author == client.user: return brooklyn_99_quotes = [ 'I\'m the human form of the 💯 emoji.', 'Bingpot!', ( 'Cool. Cool cool cool cool cool cool cool, ' 'no doubt no doubt no doubt no doubt.' ), ] if message.content == '99!': response = random.choice(brooklyn_99_quotes) await message.channel.send(response) elif message.content == 'raise-exception': raise discord.DiscordException client.run(token)

The new raise-exception message handler allows you to raise a DiscordException on command.

Run the program and type raise-exception into the Discord channel:

You should now see the Exception that was raised by your on_message() handler in the console:

$ python bot.py RealPythonTutorialBot has connected to Discord! Ignoring exception in on_message Traceback (most recent call last): File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/client.py", line 255, in _run_event await coro(*args, **kwargs) File "bot.py", line 42, in on_message raise discord.DiscordException discord.errors.DiscordException

The exception was caught by the default error handler, so the output contains the message Ignoring exception in on_message. Let’s fix that by handling that particular error. To do so, you’ll catch the DiscordException and write it to a file instead.

The on_error() event handler takes the event as the first argument. In this case, we expect the event to be 'on_message'. It also accepts *args and **kwargs as flexible, positional and keyword arguments passed to the original event handler.

So, since on_message() takes a single argument, message, we expect args[0] to be the message that the user sent in the Discord channel:

@client.event async def on_error(event, *args, **kwargs): with open('err.log', 'a') as f: if event == 'on_message': f.write(f'Unhandled message: {args[0]}\n') else: raise

If the Exception originated in the on_message() event handler, you .write() a formatted string to the file err.log. If another event raises an Exception, then we simply want our handler to re-raise the exception to invoke the default behavior.

Run bot.py and send the raise-exception message again to view the output in err.log:

$ cat err.log Unhandled message: <Message id=573845548923224084 pinned=False author=<Member id=543612676807327754 name='alexronquillo' discriminator='0933' bot=False nick=None guild=<Guild id=571759877328732195 name='RealPythonTutorialServer' chunked=True>>>

Instead of only a stack trace, you have a more informative error, showing the message that caused on_message() to raise the DiscordException, saved to a file for longer persistence.

Technical Detail: If you want to take the actual Exception into account when you’re writing your error messages to err.log, then you can use functions from sys, such as exc_info().

Now that you have some experience handling different events and interacting with Discord APIs, you’ll learn about a subclass of Client called Bot, which implements some handy, bot-specific functionality.

Connecting a Bot

A Bot is a subclass of Client that adds a little bit of extra functionality that is useful when you’re creating bot users. For example, a Bot can handle events and commands, invoke validation checks, and more.

Before you get into the features specific to Bot, convert bot.py to use a Bot instead of a Client:

# bot.py import os import random from dotenv import load_dotenv # 1 from discord.ext import commands load_dotenv() token = os.getenv('DISCORD_TOKEN') # 2 bot = commands.Bot(command_prefix='!') @bot.event async def on_ready(): print(f'{bot.user.name} has connected to Discord!') bot.run(token)

As you can see, Bot can handle events the same way that Client does. However, notice the differences between Client and Bot:

  1. Bot is imported from the discord.ext.commands module.
  2. The Bot initializer requires a command_prefix, which you’ll learn more about in the next section.

The extensions library, ext, offers several interesting components to help you create a Discord Bot. One such component is the Command.

Using Bot Commands

In general terms, a command is an order that a user gives to a bot so that it will do something. Commands are different from events because they are:

  • Arbitrarily defined
  • Directly called by the user
  • Flexible, in terms of their interface

In technical terms, a Command is an object that wraps a function that is invoked by a text command in Discord. The text command must start with the command_prefix, defined by the Bot object.

Let’s take a look at an old event to better understand what this looks like:

# bot.py import os import random import discord from dotenv import load_dotenv load_dotenv() TOKEN = os.getenv('DISCORD_TOKEN') client = discord.Client() @client.event async def on_message(message): if message.author == client.user: return brooklyn_99_quotes = [ 'I\'m the human form of the 💯 emoji.', 'Bingpot!', ( 'Cool. Cool cool cool cool cool cool cool, ' 'no doubt no doubt no doubt no doubt.' ), ] if message.content == '99!': response = random.choice(brooklyn_99_quotes) await message.channel.send(response) client.run(TOKEN)

Here, you created an on_message() event handler, which receives the message string and compares it to a pre-defined option: '99!'.

Using a Command, you can convert this example to be more specific:

# bot.py import os import random from discord.ext import commands from dotenv import load_dotenv load_dotenv() token = os.getenv('DISCORD_TOKEN') bot = commands.Bot(command_prefix='!') @bot.command(name='99') async def nine_nine(ctx): brooklyn_99_quotes = [ 'I\'m the human form of the 💯 emoji.', 'Bingpot!', ( 'Cool. Cool cool cool cool cool cool cool, ' 'no doubt no doubt no doubt no doubt.' ), ] response = random.choice(brooklyn_99_quotes) await ctx.send(response) bot.run(token)

There are several important characteristics to understand about using Command:

  1. Instead of using bot.event like before, you use bot.command(), passing the invocation command (name) as its argument.

  2. The function will now only be called when !99 is mentioned in chat. This is different than the on_message() event, which was executed any time a user sent a message, regardless of the content.

  3. The command must be prefixed with the exclamation point (! ) because that’s the command_prefix that you defined in the initializer for your Bot.

  4. Any Command function (technically called a callback) must accept at least one parameter, called ctx, which is the Context surrounding the invoked Command.

A Context holds data such as the channel and guild that the user called the Command from.

Run the program:

$ python bot.py

With your bot running, you can now head to Discord to try out your new command:

From the user’s point of view, the practical difference is that the prefix helps formalize the command, rather than simply reacting to a particular on_message() event.

This comes with other great benefits as well. For example, you can invoke the help command to see all the commands that your Bot handles:

If you want to add a description to your command so that the help message is more informative, simply pass a help description to the .command() decorator:

# bot.py import os import random from discord.ext import commands from dotenv import load_dotenv load_dotenv() token = os.getenv('DISCORD_TOKEN') bot = commands.Bot(command_prefix='!') @bot.command(name='99', help='Responds with a random quote from Brooklyn 99') async def nine_nine(ctx): brooklyn_99_quotes = [ 'I\'m the human form of the 💯 emoji.', 'Bingpot!', ( 'Cool. Cool cool cool cool cool cool cool, ' 'no doubt no doubt no doubt no doubt.' ), ] response = random.choice(brooklyn_99_quotes) await ctx.send(response) bot.run(token)

Now, when the user invokes the help command, your bot will present a description of your command:

Keep in mind that all of this functionality exists only for the Bot subclass, not the Client superclass.

Command has another useful functionality: the ability to use a Converter to change the types of its arguments.

Converting Parameters Automatically

Another benefit of using commands is the ability to convert parameters.

Sometimes, you require a parameter to be a certain type, but arguments to a Command function are, by default, strings. A Converter lets you convert those parameters to the type that you expect.

For example, if you want to build a Command for your bot user to simulate rolling some dice (knowing what you’ve learned so far), you might define it like this:

@bot.command(name='roll_dice', help='Simulates rolling dice.') async def roll(ctx, number_of_dice, number_of_sides): dice = [ str(random.choice(range(1, number_of_sides + 1))) for _ in range(number_of_dice) ] await ctx.send(', '.join(dice))

You defined roll to take two parameters:

  1. The number of dice to roll
  2. The number of sides per die

Then, you decorated it with .command() so that you can invoke it with the !roll_dice command. Finally, you .send() the results in a message back to the channel.

While this looks correct, it isn’t. Unfortunately, if you run bot.py, and invoke the !roll_dice command in your Discord channel, you’ll see the following error:

$ python bot.py Ignoring exception in command roll_dice: Traceback (most recent call last): File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 63, in wrapped ret = await coro(*args, **kwargs) File "bot.py", line 40, in roll for _ in range(number_of_dice) TypeError: 'str' object cannot be interpreted as an integer The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/bot.py", line 860, in invoke await ctx.command.invoke(ctx) File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 698, in invoke await injected(*ctx.args, **ctx.kwargs) File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 72, in wrapped raise CommandInvokeError(exc) from exc discord.ext.commands.errors.CommandInvokeError: Command raised an exception: TypeError: 'str' object cannot be interpreted as an integer

In other words, range() can’t accept a str as an argument. Instead, it must be an int. While you could cast each value to an int, there is a better way: you can use a Converter .

In discord.py, a Converter is defined using Python 3’s function annotations:

@bot.command(name='roll_dice', help='Simulates rolling dice.') async def roll(ctx, number_of_dice: int, number_of_sides: int): dice = [ str(random.choice(range(1, number_of_sides + 1))) for _ in range(number_of_dice) ] await ctx.send(', '.join(dice))

You added : int annotations to the two parameters that you expect to be of type int. Try the command again:

With that little change, your command works! The difference is that you’re now converting the command arguments to int, which makes them compatible with your function’s logic.

Note: A Converter can be any callable, not merely data types. The argument will be passed to the callable, and the return value will be passed into the Command.

Next, you’ll learn about the Check object and how it can improve your commands.

Checking Command Predicates

A Check is a predicate that is evaluated before a Command is executed to ensure that the Context surrounding the Command invocation is valid.

In an earlier example, you did something similar to verify that the user who sent a message that the bot handles was not the bot user, itself:

if message.author == client.user: return

The commands extension provides a cleaner and more usable mechanism for performing this kind of check, namely using Check objects.

To demonstrate how this works, assume you want to support a command !create_channel <channel_name> that creates a new channel. However, you only want to allow administrators the ability to create new channels with this command.

First, you’ll need to create a new member role in the admin. Go into the Discord guild and select the {Server Name} → Server Settings menu:

Then, select Roles from the left-hand navigation list:

Finally select the + sign next to ROLES and enter the name admin and select Save Changes:

Now, you’ve created an admin role that you can assign to particular users. Next, you’ll update bot.py to Check the user’s role before allowing them to initiate the command:

# bot.py import os import discord from discord.ext import commands from dotenv import load_dotenv load_dotenv() TOKEN = os.getenv('DISCORD_TOKEN') bot = commands.Bot(command_prefix='!') @bot.command(name='create-channel') @commands.has_role('admin') async def create_channel(ctx, channel_name='real-python'): guild = ctx.guild existing_channel = discord.utils.get(guild.channels, name=channel_name) if not existing_channel: print(f'Creating a new channel: {channel_name}') await guild.create_text_channel(channel_name) bot.run(TOKEN)

In bot.py, you have a new Command function, called create_channel() which takes an optional channel_name and creates that channel. create_channel() is also decorated with a Check called has_role().

You also use discord.utils.get() to ensure that you don’t create a channel with the same name as an existing channel.

If you run this program as it is and type !create-channel into your Discord channel, then you’ll see the following error message:

$ python bot.py Ignoring exception in command create-channel: Traceback (most recent call last): File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/bot.py", line 860, in invoke await ctx.command.invoke(ctx) File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 691, in invoke await self.prepare(ctx) File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 648, in prepare await self._verify_checks(ctx) File "/Users/alex.ronquillo/.pyenv/versions/discord-venv/lib/python3.7/site-packages/discord/ext/commands/core.py", line 598, in _verify_checks raise CheckFailure('The check functions for command {0.qualified_name} failed.'.format(self)) discord.ext.commands.errors.CheckFailure: The check functions for command create-channel failed.

This CheckFailure says that has_role('admin') failed. Unfortunately, this error only prints to stdout. It would be better to report this to the user in the channel. To do so, add the following event:

@bot.event async def on_command_error(ctx, error): if isinstance(error, commands.errors.CheckFailure): await ctx.send('You do not have the correct role for this command.')

This event handles an error event from the command and sends an informative error message back to the original Context of the invoked Command.

Try it all again, and you should see an error in the Discord channel:

Great! Now, to resolve the issue, you’ll need to give yourself the admin role:

With the admin role, your user will pass the Check and will be able to create channels using the command.

Note: Keep in mind that in order to assign a role, your user will have to have the correct permissions. The easiest way to ensure this is to sign in with the user that you created the guild with.

When you type !create-channel again, you’ll successfully create the channel real-python:

Also, note that you can pass the optional channel_name argument to name the channel to whatever you want!

With this last example, you combined a Command, an event, a Check, and even the get() utility to create a useful Discord bot!

Conclusion

Congratulations! Now, you’ve learned how to make a Discord bot in Python. You’re able to build bots for interacting with users in guilds that you create or even bots that other users can invite to interact with their communities. Your bots will be able to respond to messages and commands and numerous other events.

In this tutorial, you learned the basics of creating your own Discord bot. You now know:

  • What Discord is
  • Why discord.py is so valuable
  • How to make a Discord bot in the Developer Portal
  • How to create a Discord connection in Python
  • How to handle events
  • How to create a Bot connection
  • How to use bot commands, checks, and converters

To read more about the powerful discord.py library and take your bots to the next level, read through their extensive documentation. Also, now that you’re familiar with Discord APIs in general, you have a better foundation for building other types of Discord applications.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Stack Abuse: Debugging Python Applications with the PDB Module

Planet Python - Mon, 2019-08-19 08:41
Introduction

In this tutorial, we are going to learn how to use Python's PDB module for debugging Python applications. Debugging refers to the process of removing software and hardware errors from a software application. PDB stands for "Python Debugger", and is a built-in interactive source code debugger with a wide range of features, like pausing a program, viewing variable values at specific instances, changing those values, etc.

In this article, we will be covering the most commonly used functionalities of the PDB module.

Background

Debugging is one of the most disliked activities in software development, and at the same time, it is one of the most important tasks in the software development life cycle. At some stage, every programmer has to debug his/her code, unless he is developing a very basic software application.

There are many different ways to debug a software application. A very commonly used method is using the "print" statements at different instances of your code to see what is happening during execution. However, this method has many problems, such as the addition of extra code that is used to print the variables' values, etc. While this approach might work for a small program, tracking these code changes in a large application with many lines of code, spread over different files, can become a huge problem. The debugger solves that problem for us. It helps us find the error sources in an application using external commands, hence no changes to the code.

Note: As mentioned above, PDB is a built in Python module, so there is no need to install it from an external source.

Key Commands

To understand the main commands or tools that we have at our disposal in PDB, let's consider a basic Python program, and then try to debug it using PDB commands. This way, we will see with an example what exactly each command does.

# Filename: calc.py operators = ['+', '-', '*', '/'] numbers = [10, 20] def calculator(): print("Operators available: ") for op in operators: print(op) print("Numbers to be used: ") for num in numbers: print(num) def main(): calculator() main()

Here is the output of the script above:

Operators available: + - * / Numbers to be used: 10 20

I have not added any comments in the code above, as it is beginner friendly and involves no complex concepts or syntax at all. It's not important to try and understand the "task" that this code achieves, as its purpose was to include certain things so that all of PDB's commands could be tested on it. Alright then, let's start!

Using PDB requires use of the Command Line Interface (CLI), so you have to run your application from the terminal or the command prompt.

Run the command below in your CLI:

$ python -m pdb calc.py

In the command above, my file's name is "calc.py", so you'll need to insert your own file name here.

Note: The -m is a flag, and it notifies the Python executable that a module needs to be imported; this flag is followed by the name of the module, which in our case is pdb.

The output of the command looks like this:

> /Users/junaid/Desktop/calc.py(3)<module>() -> operators = [ '+', '-', '*', '/' ] (Pdb)

The output will always have the same structure. It will start with the directory path to our source code file. Then, in brackets, it will indicate the line number from that file that PDB is currently pointing at, which in our case is "(3)". The next line, starting with the "->" symbol, indicates the line currently being pointed to.

In order to close the PDB prompt, simply enter quit or exit in the PDB prompt.

A few other things to note, if your program accepts parameters as inputs, you can pass them through the command line as well. For instance, had our program above required three inputs from the user, then this is what our command would have looked like:

$ python -m pdb calc.py var1 var2 var3

Moving on, if you had earlier closed the PDB prompt through the quit or exit command, then rerun the code file through PDB. After that, run the following command in the PDB prompt:

(Pdb) list

The output looks like this:

1 # Filename: calc.py 2 3 -> operators = ['+', '-', '*', '/'] 4 numbers = [10, 20] 5 6 def calculator(): 7 print("Operators available: ") 8 for op in operators: 9 print(op) 10 11 print("Numbers to be used: ") (Pdb)

This will show the first 11 lines of your program to you, with the "->" pointing towards the current line being executed by the debugger. Next, try this command in the PDB prompt:

(Pdb) list 4,6

This command should display the selected lines only, which in this case are lines 4 to 6. Here is the output:

4 numbers = [10, 20] 5 6 def calculator(): (Pdb) Debugging with Break Points

The next important thing that we will learn about is the breakpoint. Breakpoints are usually used for larger programs, but to understand them better we will see how they function on our basic example. Break points are specific locations that we declare in our code. Our code runs up to that location and then pauses. These points are automatically assigned numbers by PDB.

We have the following different options to create break points:

  1. By line number
  2. By function declaration
  3. By a condition

To declare a break point by line number, run the following command in the PDB prompt:

(Pdb) break calc.py:8

This command inserts a breakpoint at the 8th line of code, which will pause the program once it hits that point. The output from this command is shown as:

Breakpoint 1 at /Users/junaid/Desktop/calc.py: 8 (Pdb)

To declare break points on a function, run the following command in the PDB prompt:

(Pdb) break calc.calculator

In order to insert a breakpoint in this way, you must declare it using the file name and then the function name. This outputs the following:

Breakpoint 2 at /Users/junaid/Desktop/calc.py:6

As you can see, this break point has been assigned number 2 automatically, and the line number i.e. 6 at which the function is declared, is also shown.

Break points can also be declared by a condition. In that case the program will run until the condition is false, and will pause when that condition becomes true. Run the following command in the PDB prompt:

(Pdb) break calc.py:8, op == "*"

This will track the value of the op variable throughout execution and only break when its value is "*" at line 8.

To see all the break points that we have declared in the form of a list, run the following command in the PDB prompt:

(Pdb) break

The output looks like this:

Num Type Disp Enb Where 1 breakpoint keep yes at /Users/junaid/Desktop/calc.py: 8 2 breakpoint keep yes at /Users/junaid/Desktop/calc.py: 6 breakpoint already hit 1 time 3 breakpoint keep yes at /Users/junaid/Desktop/calc.py: 8 stop only if op == "*" (Pdb)

Lastly, let's see how we can disable, enable, and clear a specific break point at any instance. Run the following command in the PDB prompt:

(Pdb) disable 2

This will disable breakpoint 2, but will not remove it from our debugger instance.

In the output you will see the number of the disabled break point.

Disabled breakpoint 2 at /Users/junaid/Desktop/calc.py:6 (Pdb)

Lets print out the list of all break points again to see the "Enb" value for breakpoint 2:

(Pdb) break

Output:

Num Type Disp Enb Where 1 breakpoint keep yes at /Users/junaid/Desktop/calc.py:8 2 breakpoint keep no at /Users/junaid/Desktop/calc.py:4 # you can see here that the "ENB" column for #2 shows "no" breakpoint already hit 1 time 3 breakpoint keep yes at /Users/junaid/Desktop/calc.py:8 stop only if op == "*" (Pdb)

To re-enable break point 2, run the following command:

(Pdb) enable 2

And again, here is the output:

Enabled breakpoint 2 at /Users/junaid/Desktop/calc.py:6

Now, if you print the list of all break points again, the "Enb" column's value for breakpoint 2 should show a "yes" again.

Let's now clear breakpoint 1, which will remove it all together.

(Pdb) clear 1

The output is as follows:

Deleted breakpoint 1 at /Users/junaid/Desktop/calc.py:8 (Pdb)

If we re-print the list of breakpoints, it should now only show two breakpoint rows. Let's see the "break" command's output:

Num Type Disp Enb Where 2 breakpoint keep yes at /Users/junaid/Desktop/calc.py:4 breakpoint already hit 1 time 3 breakpoint keep yes at /Users/junaid/Desktop/calc.py:8 stop only if op == "*"

Exactly what we expected.

Before we move ahead from this section, I want to show you all what is displayed when we actually run the code until the specified breakpoint. To do that, let's clear all the previous breakpoints and declare another breakpoint through the PDB prompt:

1. Clear all breakpoints

(Pdb) clear

After that, type "y" and hit "Enter". You should see an output like this appear:

Deleted breakpoint 2 at /Users/junaid/Desktop/calc.py:6 Deleted breakpoint 3 at /Users/junaid/Desktop/calc.py:8

2. Declare a new breakpoint

What we wish to achieve is that the code should run up until the point that the value of num variable is greater than 10. So basically, the program should pause before the number "20" gets printed.

(Pdb) break calc.py:13, num > 10

3. Run the code until this breakpoint

To run the code, use the "continue" command, which will execute the code until it hits a breakpoint or finishes:

(Pdb) continue

You should see the following output:

Operators available: + - * / Numbers to be used: 10 > /Users/junaid/Desktop/calc.py(13)calculator() -> print(num)

This is exactly what we expected, the program runs until that point and then pauses, now it's up to us if we wish to change anything, inspect variables, or if we want to run the script until completion. To instruct it to run to completion, run the "continue" command again. The output should be the following:

20 The program finished and will be restarted > /Users/junaid/Desktop/calc.py(3)<module>() -> operators = [ '+', '-', '*', '/' ]

In the above output, it can be seen that the program continues from exactly where it left off, runs the remaining part, and then restarts to allow us to debug it further if we wish. Let's move to the next section now.

Important Note: Before moving forward, clear all the breakpoints by running the "clear" command, followed by typing in "y" in the PDB prompt.

Next and Step Functions

Last, but not the least, let's study the next and step functions; these will be very frequently used when you start debugging your applications, so let's learn what they do and how they can be implemented.

The step and next functions are used to iterate throughout our code line by line; there's a very little difference between the two. While iterating, if the step function encounters a function call, it will move to the first line of that function's definition and show us exactly what is happening inside the function; whereas, if the next function encounters a function call, it will run all lines of that function in a single go and pause at the next function call.

Confused? Let's see that in an example.

Re-run the program through PDB prompt using the following command:

$ python -m pdb calc.py

Now type in continue in the PDB prompt, and keep doing that until the program reaches the end. I'm going to show a section of the whole input and output sequence below, which is sufficient to explain the point. The full sequence is quite long, and would only confuse you more, so it will be omitted.

> /Users/junaid/Desktop/calc.py(1)<module>() -> operators = [ '+', '-', '*', '/' ] (Pdb) step > /Users/junaid/Desktop/calc.py(2)<module>() -> numbers = [ 10, 20 ] . . . . > /Users/junaid/Desktop/calc.py(6)calculator() -> print("Operators available: " ) (Pdb) step Operators available: > /Users/junaid/Desktop/calc.py(8)calculator() -> for op in operators: (Pdb) step > /Users/junaid/Desktop/calc.py(10)calculator() -> print(op) (Pdb) step + > /Users/junaid/Desktop/calc.py(8)calculator() -> for op in operators: (Pdb) step > /Users/junaid/Desktop/calc.py(10)calculator() -> print(op) . . . .

Now, re-run the whole program, but this time, use the "next" command instead of "step". I have shown the input and output trace for that as well.

> /Users/junaid/Desktop/calc.py(3)<module>() -> operators = ['+', '-', '*', '/'] (Pdb) next > /Users/junaid/Desktop/calc.py(4)<module>() -> numbers = [10, 20] (Pdb) next > /Users/junaid/Desktop/calc.py(6)<module>() -> def calculator(): (Pdb) next > /Users/junaid/Desktop/calc.py(15)<module>() -> def main(): (Pdb) next > /Users/junaid/Desktop/calc.py(18)<module>() -> main() (Pdb) next Operators available: + - * / Numbers to be used: 10 20 --Return--

Alright, now that we have output trace for both of these functions, let's see how they are different. For the step function, you can see that when the calculator function is called, it moves inside that function, and iterates through it in "steps", showing us exactly what is happening in each step.

However, if you see the output trace for the next function, when "main" function is called, it doesn't show us what happens inside that function (i.e. a subsequent call to the calculator function), and then directly prints out the end result in a single go/step.

These commands are useful if you are iterating through a program and want to step through certain functions, but not others, in which case you can utilize each command for its purposes.

Conclusion

In this tutorial, we learned about a sophisticated technique for debugging python applications using an built-in module named PDB. We dove into the different troubleshooting commands that PDB provides us with, including the next and step statements, breakpoints, etc. We also applied them to a basic program to see them in action.

Categories: FLOSS Project Planets

Codementor: How to Find and Hire a Python/Django Development Company

Planet Python - Mon, 2019-08-19 06:52
Learn how to find and hire a Python development company.
Categories: FLOSS Project Planets

Codementor: Top 7 Compelling Reasons to Hire Ukrainian Developers

Planet Python - Mon, 2019-08-19 06:51
Find out why it’s worth hiring Ukrainian developers.
Categories: FLOSS Project Planets

PSF GSoC students blogs: Weekly blog #6 (week 12): 12/08 to 18/08

Planet Python - Mon, 2019-08-19 06:34

Hello there! We are at the end of GSoC! Week 12 was the last week where we’d do any major coding. We still have the next week where we do a submission (and a final check-in?), so let me keep it short and tell you what I worked on this week and what I got stuck on.

 

This week I focused on my two open PR’s - the text PR and the PyTorch PR.

 

For the text PR, I did a couple of things:

  • I updated the image tutorial and image tests with new features introduced in this PR, such as explanation “modifiers”.

  • I fleshed out (padded with comments) the text tutorial so that it looks nice.

  • I opened issues on GitHub for some TODO’s and FIXME’s.

  • One of the mentors reviewed the PR so I made any appropriate changes, including:

    • API changes - instead of letting an argument take multiple types, have separate arguments for each type.

    • An interesting change - when trying to choose a hidden layer for Grad-CAM, we will now default to one of the latter layers (closer to output). This is closer to how Grad-CAM is defined (since picking earlier layers would be like using some other gradient method for explanations).

 

One interesting issue that arose as a result of the above change (defaulting to “higher level” layers) was that my integration tests broke and the tutorial explanations didn’t look as good. With the guidance of my mentor I resolved the failing tests by explicitly picking a layer that works. For the tutorials I made a comment that using earlier layers gave better results. Interesting issues with text!

 

Next for the PyTorch PR I did the following:

  • Added code that can extract an image from an input tensor (a simple transform in torchvision), like what we do for Keras.

  • Added a very short image tutorial.

 

I got stuck on the text part of PyTorch because my test model was quite large (large word embeddings). It looks like I will have to train my own embedding layer with a small vocabulary.

 

What’s next

  • Final submission coming soon.

  • Slowly work on getting text and pytorch PR’s merged as a regular open source contributor!

    • Text PR still has some TODO’s and reviews to resolve.

    • PyTorch PR needs a text tutorial, integration tests, and unit tests. We’ll scale down so hopefully this doesn’t take too much time.

 

That’s all the technical details! I think we have one more blog next week, so I can talk about GSoC in broader terms then?

 

Thanks for reading once again!

Tomas Baltrunas

Categories: FLOSS Project Planets

Agiledrop.com Blog: Top 10 Drupal Accessibility Modules

Planet Drupal - Mon, 2019-08-19 06:09

In this post, we'll take a look at some of the most useful modules that will help make your Drupal site more accessible to developers, content editors and users alike.

READ MORE
Categories: FLOSS Project Planets

PSF GSoC students blogs: Twelveth week of GSoC: Getting ready for the final week of GSoC

Planet Python - Mon, 2019-08-19 06:03

My GSoC is soon coming to an end so I took some time to write down what still needs to be done:

Making a release of MNE-BIDS

In the past months, there were substantial additions, fixes, and cosmetic changes made to the codebase and documentation of MNE-BIDS. The last release has happened in April (about 4 months ago) and we were quite happy to observe some issues and pull requests raised and submitted by new users. With the next release we can provide some new functionality for this growing user base.

Handling coordinates for EEG and iEEG in MNE-BIDS

In MNE-BIDS, the part of the code that handles the writing of sensor positions in 3D space (=coordinates) is so far restricted to MEG data. Extending this functionality to EEG and iEEG data has been on the to do list for a long time now. Fortunately, I have been learning a bit more about this topic during my GSoC, and Mainak has provided some starting points in an unrelated PR that I can use to finish this issue. (After the release of MNE-BIDS though, to avoid cramming in too much last-minute content before the release)

Writing a data fetcher for OpenNeuro to be used in MNE-Python

While working with BIDS and M/EEG data, the need for good testing data has come up time and time again. For the mne-study-template we solved this issue with a combination of DataLad and OpenNeuro. Meanwhile, MNE-BIDS has its own dataset.py module ... however, we all feel like this module is duplicating the datasets module of MNE-Python and not advancing MNE-BIDS. Rather, it is confusing the purpose of MNE-BIDS.

As a solution, we want to write a generalized data fetching function for MNE-Python that works with OpenNeuro ... without adding the DataLad (and hence Git-Annex) dependency). Once this fetching function is implemented, we can import it in MNE-BIDS and finally deprecate MNE-BIDS' dataset.py module.

Make a PR in MNE-Python that will support making Epochs for duplicate events (will fix ds001971 PR)

In MNE-Python, making data epochs is not possible, if two events share the same time. This became apparent with the dataset ds001971 that we wanted to add to the mne-study-template pipeline: https://github.com/mne-tools/mne-study-template/pull/41. There was a suggestion on how to solve this issue by merging the event codes that occurred at the same time. Once this fix is implemented in MNE-Python, we can use this to finish the PR in the mne-study-template.

Salvage / close the PR on more "read_raw_bids" additions

Earlier in this GSoC, I made a PR intended to improve the reading functionality of MNE-BIDS (https://github.com/mne-tools/mne-bids/pull/244). However, the PR was controversially discussed, because it was not leveraging BIDS and instead relying on introducing a dictionary as a container for keyword arguments.

After lots of discussion, we agreed to solve the situation in a different way (by leveraging BIDS) and Mainak made some initial commits into that direction. However in the further progress, the PR was dropped because other issues had higher priority.

Before finishing my GSoC, I want to salvage what's possible from this PR and then close it ... and improving the original issue report so that the next attempt at this PR can rely on a more detailed objective.

Categories: FLOSS Project Planets

Dries Buytaert: Low-code and no-code tools continue to drive the web forward

Planet Drupal - Mon, 2019-08-19 04:35

A version of this article was originally published on Devops.com.

Twelve years ago, I wrote a post called Drupal and Eliminating Middlemen. For years, it was one of the most-read pieces on my blog. Later, I followed that up with a blog post called The Assembled Web, which remains one of the most read posts to date.

The point of both blog posts was the same: I believed that the web would move toward a model where non-technical users could assemble their own sites with little to no coding experience of their own.

This idea isn't new; no-code and low-code tools on the web have been on a 25-year long rise, starting with the first web content management systems in the early 1990s. Since then no-code and low-code solutions have had an increasing impact on the web. Examples include:

While this has been a long-run trend, I believe we're only at the beginning.

Trends driving the low-code and no-code movements

According to Forrester Wave: Low-Code Development Platforms for AD&D Professionals, Q1 2019, In our survey of global developers, 23% reported using low-code platforms in 2018, and another 22% planned to do so within a year..

Major market forces driving this trend include a talent shortage among developers, with an estimated one million computer programming jobs expected to remain unfilled by 2020 in the United States alone.

What is more, the developers who are employed are often overloaded with work and struggle with how to prioritize it all. Some of this burden could be removed by low-code and no-code tools.

In addition, the fact that technology has permeated every aspect of our lives — from our smartphones to our smart homes — has driven a desire for more people to become creators. As the founder of Product Hunt Ryan Hoover said in a blog post: As creating things on the internet becomes more accessible, more people will become makers..

But this does not only apply to individuals. Consider this: the typical large organization has to build and maintain hundreds of websites. They need to build, launch and customize these sites in days or weeks, not months. Today and in the future, marketers can embrace no-code and low-code tools to rapidly develop websites.

Abstraction drives innovation

As discussed in my middleman blog post, developers won't go away. Just as the role of the original webmaster has evolved with the advent of web content management systems, the role of web developers is changing with the rise of low-code and no-code tools.

Successful no-code approaches abstract away complexity for web development. This enables less technical people to do things that previously could only by done by developers. And when those abstractions happen, developers often move on to the next area of innovation.

When everyone is a builder, more good things will happen on the web. I was excited about this trend more than 12 years ago, and remain excited today. I'm eager to see the progress no-code and low-code solutions will bring to the web in the next decade.

Categories: FLOSS Project Planets

Erik Marsja: The Easiest Data Cleaning Method using Python &amp; Pandas

Planet Python - Mon, 2019-08-19 04:27

The post The Easiest Data Cleaning Method using Python & Pandas appeared first on Erik Marsja.

In this post we are going to learn how to do simplify our data preprocessing work using the Python package Pyjanitor. More specifically, we are going to learn how to:

  • Add a column to a Pandas dataframe
  • Remove missing values
  • Remove an empty column
  • Cleaning up column names

That is, we are going to learn how clean Pandas dataframes using Pyjanitor. In all Python data manipulation examples, here we are also going to see how to carry out them using only Pandas functionality.

What is Pyjanitor?

What is Pyjanitor? Before we continue learning on how to use Pandas and Pyjanitor to clean our datasets, we will learn about this package. The python package Pyjanitor extends Pandas with a verb-based API. This easy to use API is providing us with convenient data cleaning techniques. Apparently, it started out as a port of the R package janitor. Furthermore, it is inspired by the ease-of-use and expressiveness of the r-package dplyr. Note, there are some different ways how to work with the methods and this post will not cover all of them (see the documentation).

How to install Pyjanitor

There are two easy methods to install Pyjanitor:

1. Installing Pyjanitor using Pip pip install pyjanitor 2. Installing Pyjanitor using Conda: conda -c install conda-forge pyjanitor

Now that we know what Pyjanitor is and how to install the package we soon can continue the Python data cleaning tutorial by learning how to remove missing values from Pandas. Note, that this Pandas tutorial will walk through each step on how to do it using Pandas and Pyjanitor. In the end, we will have a complete data cleaning example using only Pyjanitor and a link to a Jupyter Notebook with all code.

Fake Data

In the first Python data manipulation examples, we are going to work with a fake dataset. More specifically, we are going to create a dataframe, with an empty column, and missing values. In this part of the post we are, further, going to use the Python packages SciPy, and NumPy. That is, these packages also need to be installed.

In this example we are going create three columns; Subject, RT (response time), and Deg. To create the response time column, we will use SciPy norm to create data that is normally distributed.

import numpy as np import pandas as pd from scipy.stats import norm from random import shuffle import janitor subject = ['n0' + str(i) for i in range(1, 201)] Python Normal Distribution using Scipy

In the next code chunk we create a variable, for response time, using a normal distribution.

a = 457 rt = norm.rvs(a, size=200) Shuffling the List and Adding Missing Values

Furthermore, we are adding some missing values and shuffling the list of normally distirbuted data:

# Shuffle the response times shuffle(rt) rt[4], rt[9], rt[100] = np.nan, np.nan, np.nan Dataframe from Dictionary

Finally, we are creating a dictionary of our two variables and use the dictionary to create a Pandas dataframe.

data = { 'Subject': subject, 'RT': rt, } df = pd.DataFrame(data) df.head() Dataframe created from dictionary Data Cleaning in Python with Pandas and Pyjanitor How to Add a Column to Pandas Dataframe

Now that we have created our dataframe from a dictionary we are ready to add a column to it. In the examples, below, we are going to use Pandas and Pyjanitors method.

1. Append a Column to Pandas Dataframe

It’s quite easy to add a column to a dataframe using Pandas. In the example below we will append an empty column to the Pandas dataframe:

df['NewColumnName'] = np.nan df.head() Column added to dataframe 2. Adding a Column to Pandas Dataframe using Pyjanitor

Now, we are going to use the method add_column to append a column to the dataframe. Adding an empty column is not as easy as using the method above. However, as you will see towards the end of this post, we can use all of the methods when creating our dataframe:

newcolvals = [np.nan]*len(df['Subject']) df = df.add_column('NewColumnName2', newcolvals) df.head() Append column to Pandas dataframe How to Remove Missing Values in Pandas Dataframe

It is quite common that our dataset is far from complete. This may be due to error in the measurement instruments, people forgetting, or refusing, to answer certain questions, amongst many other things. Despite the reason behind missing information these rows are called missing values. In the framework of Pandas the missing values are coded by the symbol NA, much like in R statistical environment. Pandas have the function isna() to help us identify missings in our dataset. If we want to drop missing values, Pandas have the function dropna().

1 Dropping Missing Values using Pandas dropna method

In the code example below we are dropping all rows with missing values. Note, if we want to modify the dataframe we should add the inplace parameter and set it to true.

df.dropna(subset=['RT']).head()

Dropping Missing Values from Pandas Dataframe using PyJanitor

The method to drop missing values from a Pandas Dataframe using Pyjanitor is the same the one above. That is, we are going to use the the dropna method. However, when using Pyjanitor we also use the parameter subset to select which column(s) we are going to use when removing missing data from the dataframe:

df.dropna(subset=['RT']) How to Remove an Empty Column from Pandas Dataframe

In the next Pandas data manipulation example, we are going to remove the empty column from the dataframe. First, we are going to use Pandas to remove the empty column and, then, we are going to use Pyjanitor. Remember, towards the end of the post we will have a complete example in which we carry out all data cleaning while actually creating the Pandas Dataframe.

1. Removing an Empty Column from Pandas Dataframe

When we want to remove an empty column (e.g., with missing values) we use the Pandas method dropna again. However, we use the axis method and set it to 1 (for column). Furthermore, we also have to use the parameter how and set it to ‘all’. If we don’t it will remove any column with missing values

df.dropna(axis=1, how='all').head() Removed empty columns 2. Deleting an Empty Column from Pandas Dataframe using Pyjanitor

It’s a bit easier to remove an empty column using Pyjanitor:

df.remove_empty() How to Rename Columns in Pandas Dataframe

Now that we know how to remove missing values, add a column to a Pandas dataframe, and how to remove a column, we are going to continue this data cleaning tutorial learning how to rename columns.

For instance, in the post where we learned how to load data from a JSON file to a Pandas dataframe, we renamed columns to make it easier to work with the dataframe later. In the example below, we will read a JSON file, and rename columns using both Pandas dataframe method rename and Pyjanitor

import requests from pandas.io.json import json_normalize url = "https://datahub.io/core/s-and-p-500-companies-financials/r/constituents-financials.json" resp = requests.get(url=url) df = json_normalize(resp.json()) df.iloc[:,0:6].head()

More about loading data to dataframes:

1 Renaming Columns in Pandas Dataframe

As can be seen in the image above, there are some whitespaces and special characters that we want to remove. In the first renaming columns example we are going to use Pandas rename method together with regular expressions to rename the columns (i.e., we are going to replace whitespaces and \ with underscores).

import re df.rename(columns=lambda x: re.sub('(\s|/)','_',x), inplace=True) df.keys()

2. How to Rename Columns using Pyjanitor and clean_names

The task to rename a column (or many columns) is way easier using Pyjanitor. In fact, when we have imported this Python package, we can just use the clean_names method and it will give us the same result as using Pandas rename method. In fact, using clean_names we also get all letters in the column names to lowercase:

df = df.clean_names().head() df.keys()

How to Clean Data when Loading the Data from Disk

The cool thing with using Pyjanitor to clean our data is that we can do use all of the above methods when loading our data. For instance, in the final data cleaning example we are going to add a column to the dataframe, remove empty columns, drop missing data, and clean the column names. This is what makes working with Pyjanitor our lives easier.

data_id = [1]*200
url = 'https://raw.githubusercontent.com/marsja/jupyter/master/SimData/DF_NA_Janitor.csv' df = ( pd.read_csv(url, index_col=0) .add_column('data_id', data_id) .remove_empty() .dropna() .clean_names() ) df.head()

Aggregating Data using Pyjanitor

In the last example we are going to use Pandas methods agg, groupby, and reset_index together with the Pyjanitor method collapse_levels to calculate the mean and standard for each sector:

df.groupby('sector').agg(['mean', 'std']).collapse_levels().reset_index()

More about grouping and aggregating data using Python and Pandas:

Conclusion:

In this post we have learned how to do some data cleaning methods. Specifically, we have learned how to append a column to a Pandas dataframe, remove empty columns, handling missing values, and renaming the columns (i.e., getting better column names). There are, of course, many more data cleaning methods available, both when it comes to Pandas and Pyjanitor.

In conclusion, the methods added by the Python package are both s imilar to the one of the R-package janitor and dplyr. These methods will make our lives easier when preprocessing our data.

What is your favorite data cleaning method and/or Package? It can be either using R, Python, or any other programming language. Leave a comment below!

Add a column to a Pandas dataframe > Remove missing values > Remove an empty column > Cleaning up column names And all of this using only a few lines of code (end of the post)." data-app-id="27344152" data-app-id-name="category_below_content">

The post The Easiest Data Cleaning Method using Python & Pandas appeared first on Erik Marsja.

Categories: FLOSS Project Planets

Reuven Lerner: Weekly Python Exercise A3 (beginner objects) is open

Planet Python - Mon, 2019-08-19 04:10

If you’ve been programming in Python for any length of time, then you’ve undoubtedly heard that “everything is an object.”

But what does that mean? And who cares?  And what effect does that have on you as a developer — or on Python, as a language?

Indeed, how can (and should) you take advantage of Python’s object-oriented facilities to make your code more readable, maintainable, standard, and (dare I say it) Pythonic?

If you’re relatively new to Python, and have been struggling with some of these same questions, or if you’re just wondering about the differences between instances, classes, methods, and attributes, then I have good news for you: The upcoming cohort of Weekly Python Exercise is all about object-oriented programming.

In this 15-week course, you’ll learn in the best way I know, by solving problems and discussing them with others. As you work through the exercises, you’ll get a better understanding of:

  • Instances and classes
  • Attributes, at both the attribute and class level
  • Methods
  • Composition of objects
  • Inheritance
  • Magic methods

Weekly Python Exercise, of course, is a family of 15-week classes designed to help improve your Python fluency.  Each course works the same:

  • On Tuesday, you receive a problem (along with “pytest” tests)
  • On the following Monday, you receive my detailed solution. 
  • In between, we can discuss it in the forum.
  • I also have live office hours, when people can ask any questions that they might have.

WPE A-level courses are for beginners, while B-level courses are for more advanced Python developers. But you can take any or all of them, in any order — and there’s no overlap between the exercises in these classes and any of the previous books/courses I’ve given.

This new cohort (A3) will be starting on Tuesday, September 17th.  To join, you must sign up before September 10th.  But if you sign up by September 3rd, you’ll get the early-bird discount, bringing the price down to $80 — more than $20 off the full price.

I won’t be offering these exercises for at least one more year. So if you want to sharpen your OO skills before the autumn of 2020, then you should act now.

As always, you can get an even better price if you’re a student, pensioner/retiree/senior citizen, or living permanently outside of the world’s 30 richest countries. Just reply to this e-mail, and I”ll send you the appropriate coupon code.

And if several people (at least five) from your company want to join together?  Let me know, and I’ll give you an additional discount, too.

There’s lots more to say about Weekly Python Exercise, now in its third year of helping Python developers from around the world to write better code — doing more in less time, and getting better jobs than before.  You can read more, and try to some sample exercises, at https://WeeklyPythonExercise.com/ .

But if you’ve always wanted to improve your fluency with Python objects, then you can just sign up at https://WeeklyPythonExercise.com/ .

Don’t wait, though! The early-bird discount ends on September 3rd.

The post Weekly Python Exercise A3 (beginner objects) is open appeared first on Reuven Lerner.

Categories: FLOSS Project Planets

Pages