Feeds
FSF Blogs: The Licensing & Compliance Team, running at full steam for your freedom
The Licensing & Compliance Team, running at full steam for your freedom
PyCoder’s Weekly: Issue #636 (July 2, 2024)
#636 – JULY 2, 2024
View in Browser »
In this video course, you’ll learn how to create graphical user interface (GUI) applications with Python and PyQt. Once you’ve covered the basics, you’ll build a fully functional desktop calculator that can respond to user events with concrete actions.
REAL PYTHON course
Umbra Space has released a data set consisting of satellite based radar images of shipping. This article from Mark shows you how to grab the data, visualize, and annotate it.
MARK LITWINTSCHIK
Logfire, by the makers of Pydantic, is an observability platform that will help you understand your app’s behavior with less code and time. Built on OpenTelemetry, it features user-friendly dashboards, SQL querying, and Python-specific integrations. Get started today →
PYDANTIC sponsor
This is a very detailed list of best practices for developing in Python. It includes tools, language features, application design, which libraries to use an more.
STUART ELLIS
In this tutorial, you’ll build a guitar synthesizer using the Karplus-Strong algorithm in Python. You’ll model vibrating strings, simulate strumming techniques, read musical notation and tablature, and apply audio effects. By the end, you’ll have created a digital guitar that can play any song. This tutorial was also discussed on Real Python Podcast Episode #210.
REAL PYTHON
The backdoor introduced to the xz-utils compression project through social engineering was one of the topics at the Python Language Summit. Participants discussed what can be done to prevent similar social engineering attacks on the Python source.
PYTHON SOFTWARE FOUNDATION
“With PropelAuth, I think I’ve spent about a day – total – on auth over the past year.” PropelAuth is easy to integrate and provides all the tools your team needs to manage your users - dashboards, user insights, impersonation, SSO and more →
PROPELAUTH sponsor
“Prettier” is a JavaScript based linting tool for templates. For folks not familiar with the world of npm, it can be a bit daunting to get it going. Simon fiddled with it so you don’t have to and posted how he got it working on his system.
SIMON WILLISON
An often overlooked aspect of software development is architecture at the module & function level. It is important to write code that is simple and easy to move from one place to another.
DAVIDVUJIC.BLOGSPOT.COM • Shared by David Vujic
This is a simple, quick guide to getting started running LLMs on your local computer. It covers the basics of the powerful libraries Ollama and LangChain for controlling these AI models.
JOSHUA COOK • Shared by Joshua Cook
This tutorial covers hash tables, collision handling, performance optimization and how it relates to the implementation of the set data structure in Python.
ABHINAV UPADHYAY
Keeping a counter across objects in Django means having to be careful about race conditions. This article outlines several approaches to the problem.
GONÇALO VALÉRIO
Brett talks about the different ways you can support the many maintainers of open source projects, and often times just saying “thanks” means a lot.
BRETT CANNON
Learn all about weak references in Python: reference counting, garbage collection, and practical uses of the weakref module
MARTIN HEINZ • Shared by Martin Heinz
In this tutorial, you’ll learn how to create, organize, and format high-quality README files for your Python projects.
REAL PYTHON
This quick TIL post from Rodrigo shows you how to get information about the terminal size from the shutil module.
RODRIGO GIRÃO SERRÃO
Learn how to use pytest fixtures for writing maintainable and isolated tests.
STANLEY ULILI
GITHUB.COM/POMPONCHIK • Shared by pomponchik
oxo: Security Scanning Orchestrator jax: Composable Transformations of Python+NumPy Programs dbt-utils: Utility Functions for DBT Projects Events Weekly Real Python Office Hours Q&A (Virtual) July 3, 2024
REALPYTHON.COM
July 4, 2024
MEETUP.COM
July 4, 2024
SYPY.ORG
July 8 to July 15, 2024
EUROPYTHON.EU
July 8 to July 14, 2024
SCIPY.ORG
July 10 to July 14, 2024
PYCON.ORG
Happy Pythoning!
This was PyCoder’s Weekly Issue #636.
View in Browser »
[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]
Dima Kogan: vnlog.slurp() with non-numerical data
For a while now I'd see an annoying problem when trying to analyze data. I would be trying to import into numpy an innocuous-looking data file like this:
# image x y z temperature image1.png 1 2 5 34 image2.png 3 4 1 35As usual, I would be using vnlog.slurp() (a thin wrapper around numpy.loadtxt()) to read this in, but that doesn't work: the image filenames aren't parseable as numerical values. Up until now I would work around this by using the suprocess module to fork off a vnl-filter -p !image and then slurp that, but it's a pain and slow and has other issues. I just solved this conclusively using the numpy structured dtypes. I can now do this:
dtype = np.dtype([ ('image', 'U16'), ('x y z', int, (3,)), ('temperature', float), ]) arr = vnlog.slurp("data.vnl", dtype=dtype)This will read the image filename, the xyz points and the temperature into different sub-arrays, with different types each. Accessing the result looks like this:
print(arr['image']) ---> array(['image1.png', 'image2.png'], dtype='<U16') print(arr['x y z']) ---> array([[1, 2, 5], [3, 4, 1]]) print(arr['temperature']) ---> array([34., 35.])Notes:
- The given structured dtype defines both how to organize the data, and which data to extract. So it can be used to read in only a subset of the available columns. Here I could have omitted the temperature column, for instance
- Sub-arrays are allowed. In the example I could say either
dtype = np.dtype([ ('image', 'U16'),
('x y z', int, (3,)),
('temperature', float), ])
or
dtype = np.dtype([ ('image', 'U16'), ('x', int), ('y', int), ('z', int), ('temperature', float), ])The latter would read x, y, z into separate, individual arrays. Sometime we want this, sometimes not.
- Nested structured dtypes are not allowed. Fields inside other fields are not supported, since it's not clear how to map that to a flat vnlog legend
- If a structured dtype is given, slurp() returns the array only, since the field names are already available in the dtype
We still do not support records with any null values (-). This could probably be handled with the converters kwarg of numpy.loadtxt(), but that sounds slow. I'll look at that later.
This is available today in vnlog 1.38.
Bits from Debian: Bits from the DPL
Dear Debian community,
Statement on Daniel PocockThe Debian project has successfully taken action to secure its trademarks and interests worldwide, as detailed in our press statement. I would like to personally thank everyone in the community who was involved in this process. I would have loved for you all to have spent your volunteer time on more fruitful things.
Debian Boot team might need helpI think I've identified the issue that finally motivated me to contact our teams: for a long time, I have had the impression that Debian is driven by several "one-person teams" (to varying extents of individual influence and susceptibility to burnout). As DPL, I see it as my task to find ways to address this issue and provide support.
I received private responses from Debian Boot team members, which motivated me to kindly invite volunteers to some prominent and highly visible fields of work that you might find personally challenging. I recommend subscribing to the Debian Boot mailing list to see where you might be able to provide assistance.
/usrmergeHelmut Grohne confirmed that the last remaining packages shipping aliased files inside the package set relevant to debootstrap were uploaded. Thanks a lot for Helmut and all contributors that helped to implement DEP17.
Contacting more teamsI'd like to repeat that I've registered a BoF for DebConf24 in Busan with the following description:
This BoF is an attempt to gather as much as possible teams inside Debian to exchange experiences, discuss workflows inside teams, share their ways to attract newcomers etc.
Each participant team should prepare a short description of their work and what team roles (“openings”) they have for new contributors. Even for delegated teams (membership is less fluid), it would be good to present the team, explain what it takes to be a team member, and what steps people usually go to end up being invited to participate. Some other teams can easily absorb contributions from salsa MRs, and at some point people get commit access. Anyway, the point is that we work on the idea that the pathway to become a team member becomes more clear from an outsider point-of-view.
I'm lagging a bit behind my team contacting schedule and will not manage to contact every team before DebConf. As a (short) summary, I can draw some positive conclusions about my efforts to reach out to teams. I was able to identify some issues that were new to me and which I am now working on. Examples include limitations in Salsa and Salsa CI. I consider both essential parts of our infrastructure and will support both teams in enhancing their services.
Some teams confirmed that they are basically using some common infrastructure (Salsa team space, mailing lists, IRC channels) but that the individual members of the team work on their own problems without sharing any common work. I have also not read about convincing strategies to attract newcomers to the team, as we have established, for instance, in the Debian Med team.
DebConf attendanceThe amount of money needed to fly people to South Korea was higher than usual, so the DebConf bursary team had to make some difficult decisions about who could be reimbursed for travel expenses. I extended the budget for diversity and newcomers, which enabled us to invite some additional contributors. We hope that those who were not able to come this year can make it next year to Brest or to MiniDebConf Cambridge or Toulouse
tag2uploadOn June 12, Sean Whitton requested comments on the debian-vote list regarding a General Resolution (GR) about tag2upload. The discussion began with technical details but unfortunately, as often happens in long threads, it drifted into abrasive language, prompting the community team to address the behavior of an opponent of the GR supporters. After 560 emails covering technical details, including a detailed security review by Russ Allbery, Sean finally proposed the GR on June 27, 2024 (two weeks after requesting comments).
Firstly, I would like to thank the drivers of this GR and acknowledge the technical work behind it, including the security review. I am positively convinced that Debian can benefit from modernizing its infrastructure, particularly through stronger integration of Git into packaging workflows.
Sam Hartman provided some historical context [1], [2], [3], [4], noting that this discussion originally took place five years ago with no results from several similarly lengthy threads. My favorite summary of the entire thread was given by Gregor Herrmann, which reflects the same gut feeling I have and highlights a structural problem within Debian that hinders technical changes. Addressing this issue is definitely a matter for the Debian Project Leader, and I will try to address it during my term.
At the time of writing these bits, a proposal from ftpmaster, which is being continuously discussed, might lead to a solution. I was also asked to extend the GR discussion periods which I will do in separate mail.
Talk: Debian GNU/Linux for Scientific ResearchI was invited to have a talk in the Systems-Facing Track of University of British Columbia (who is sponsoring rack space for several Debian servers). I admit it felt a bit strange to me after working more than 20 years for establishing Debian in scientific environments to be invited to such a talk "because I'm DPL". 😉
Kind regards Andreas.
direvent @ Savannah: GNU Direvent Version 5.4
GNU direvent version 5.4 is available for download.
New in this version:
It is possible to limit number of command instances that are allowed to run simultaneously for a particular watcher. This is done using
the max-instances statement in watcher section.
In previous version, watchers waited for the handler to terminate, unless given the nowait option explicitly. It is now fixed and nowait is the default, as described in the documentation.
In some cases setting the sentinel effectively removed the original watcher. That happened if the full file name of the original watcher
and its directory part produced the same hash code.
gdbm @ Savannah: GNU dbm version 1.24
GNU dbm version 1.24 is available for download. New in this version:
The --update (-U) option instructs gdbm_load to update an existing database.
The --replace (-r) is valid only when used together with --update.
In contrast to gdbm_load and gdbm_load_from_file, which derive the value of the flag parameter for gdbm_open from the value of their replace argument, this function allows the caller to specify it explicitly.
- Fix binary dump format for key and/or data of zero size (see bug 656)
- Fix location tracking and recover command in gdbtool (see bug 566)
- Fix possible buffer underflow in gdbmload.
- Ensure any padding bytes in avail_elem structure are filled with 0. This fixes debian bug 1031276.
- Improve the documentation.
Mike Gabriel: Polis - a FLOSS Tool for Civic Participation -- Introduction (episode 1/5)
This is the first article of a 5-episode blog post series written by Guido Berhörster, member of staff at my company Fre(i)e Software GmbH. Thanks, Guido for being on the Polis project.
Enjoy the read on the work Guido has been doing over the past months,
Mike
A team lead by Raoul Kramer/BetaBreak is currently adapting Polis for evaluation and testing by several Dutch provincial governments and central government ministries. Guido Berhörster (author of this article) who is an employee at Fre(ie) Software GmbH has been involved in this project as the main software developer. This series of blog posts describes how and why Polis was initially modified and adapted, what issues the team ran into and how this ultimately lead them to start a new Open Source project called Particiapp for accelerating the development of alternative Polis frontends compatible to but independent from the upstream project.
Table of Contents of the Blog Post Series- Introduction (this article)
- Initial evaluation and adaptation
- Issues extending Polis and adjusting our goals
- Creating (a) new frontend(s) for Polis
- Current status and roadmap
Polis is a platform for participation which helps to gather, analyze and understand viewpoints of large groups of participants on complex issues. In practical terms participants take part in “conversations” on a predefined topic by voting on statements or submitting their own statements (referred to as “comments” in Polis) for others to vote on1.
Through statistical analysis including machine learning participants are sorted into groups based on similarities in voting behavior. In addition, group-informed and overall consensus statements are identified and presented to participants in real-time. This allows for participants to react to and refine statements and either individually or through a predefined process to come to an overall consensus.
Furthermore, the order in which statements are presented to participants is influenced by a complex weighting system based on a number of factors such as variance, recency, and frequency of skipping. This so called “comment routing” is intended to facilitate a meaningful contribution of participants without requiring them to vote on each of a potentially huge number of statements 2.
Polis open-ended nature sets it apart from online surveys using pre-defined questions and allows its users to gather a more accurate picture of the public opinion. In contrast to a discussion forum or comment section where participants directly reply to each other, it discourages unproductive behavior such as provocations or personal attacks by not presenting statements in chronological order in combination with voting. Finally, its “comment routing” is intended to provide scalability towards a large number of participants which generate a potentially large number of statements.
The project was developed and is maintained by The Computational Democracy Project, a USA-based non-profit organization which provides a hosted version and offers related services. It is also released as Open Source software under the AGPL 3.0 license.
Polis has been used in a variety of different contexts as part of broader political processes facilitating broader political participation and opinion-forming, and gathering feedback and creative input.
Use of Polis in TaiwanOne prominent use case of Polis is its adoption as part of the vTaiwan participatory governance project. Established by the g0v civic tech community in the wake of the 2014 mass protests by the Sunflower movement, the vTaiwan project enables consultations on proposed legislation among a broad range of stakeholders including government ministries, lawmakers, experts, interest groups, civil society as well as the broader public. Although the resulting recommendations are non-binding, they exert pressure on the government to take action and recommendations have been adopted into legislation.345
vTaiwan uses Polis for large-scale online deliberations as part of a structured participation process. These deliberations take place after identifying and involving stakeholders and experts and providing through information about the topic at hand to the public. Citizens are then given the opportunity to vote on statements or provide alternative proposals which allows for the refinement of ideas and ideally leads to a consensus at the end. The results of these online deliberations are then curated, discussed in publicly broadcast face-to-face meetings which ultimately produce concrete policy recommendations. vTaiwan has in numerous cases given impulses resulting in government action and provided significant input e.g. on legislation regulating Uber or technological experiments by Fintech startups.35
See also- Polis Case Studies
- BBC Click: Can Taiwan Reboot Democracy?
- Polis: Scaling Deliberation by Mapping High Dimensional Opinion Spaces
-
https://compdemocracy.org/Polis/ ↩
-
https://compdemocracy.org/comment-routing/ ↩
-
https://www.theguardian.com/world/2020/sep/27/taiwan-civic-hackers-polis-consensus-social-media-platform ↩
-
https://www.technologyreview.com/2018/08/21/240284/the-simple-but-ingenious-system-taiwan-uses-to-crowdsource-its-laws/ ↩ ↩
Real Python: Defining Python Constants for Code Maintainability
In programming, the term constant refers to names representing values that don’t change during a program’s execution. Constants are a fundamental concept in programming, and Python developers use them in many cases. However, Python doesn’t have a dedicated syntax for defining constants. In practice, Python constants are just variables that never change.
To prevent programmers from reassigning a name that’s supposed to hold a constant, the Python community has adopted a naming convention: use uppercase letters. For every Pythonista, it’s essential to know what constants are, as well as why and when to use them.
In this video course, you’ll learn how to:
- Properly define constants in Python
- Identify some built-in constants
- Use constants to improve your code’s readability, reusability, and maintainability
- Apply different approaches to organize and manage constants in a project
- Use several techniques to make constants strictly constant in Python
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
The Drop Times: CMS Usage Patterns in USA Charity and Non-Profit Organizations: FOSS Takes the Lead
Matt Glaman: Running Drupal on the Edge with WebAssembly
At DrupalCon Portland, Dries announced Starshot during his State of Drupal presentation. Part of Starshot is the idea that we have Drupal CMS and Core. The big difference is that the Drupal CMS offering comes with standard contributed modules used by almost every existing Drupal build. Then, Dries showed a proposed wireframe for the Drupal.org download page. One of the first things I noticed was the "Launch" in the Drupal CMS section. I wasn't sure about the end goal:
LN Webworks: How Important Is Low-code development to your company?
Low-code solutions are a quick and adaptable substitute for the conventional development cycle as digitization picks up speed.
Low-code developments are quick, flexible, and affordable but perform better at certain tasks than others. Businesses usually use them mainly in these particular areas- Workflow Automation, Customer Relationship Management (CRM), Enterprise Resource Planning (ERP), Data Integration, Mobile and Web Applications, Business Process Management (BPM), Internal Tools and Dashboards, and E-commerce Platforms.
Before this, let’s delve into the concept of low-code development to gain insight into how it benefits businesses.
Colin Watson: Free software activity in June 2024
My Debian contributions this month were all sponsored by Freexian.
- I switched man-db and putty to Rules-Requires-Root: no, thanks to a suggestion from Niels Thykier.
- I moved some files in pcmciautils as part of the /usr move.
- I upgraded libfido2 to 1.15.0.
- I made an upstream release of multipart 0.2.5.
- I reviewed some security-update patches to putty.
- I packaged yubihsm-connector, yubihsm-shell, and python-yubihsm.
- openssh:
- I did a bit more planning for the GSS-API package split, though decided not to land it quite yet to avoid blocking other changes on NEW queue review.
- I removed the user_readenv option from PAM configuration (#1018260), and prepared a release note.
- Python team:
- I packaged zope.deferredimport, needed for a new upstream version of python-persistent.
- I fixed some incompatibilities with pytest 8: ipykernel and ipywidgets.
- I fixed a couple of RC or soon-to-be-RC bugs in khard (#1065887 and #1069838), since I use it for my address book and wanted to get it back into testing.
- I fixed an RC bug in python-repoze.sphinx.autointerface (#1057599).
- I sponsored uploads of python-channels-redis (Dale Richards) and twisted (Florent ‘Skia’ Jacquet).
- I upgraded babelfish, django-favicon-plus-reloaded, dnsdiag, flake8-builtins, flufl.lock, ipywidgets, jsonpickle, langtable, nbconvert, requests, responses, partd, pytest-mock, python-aiohttp (fixing CVE-2024-23829, CVE-2024-23334, CVE-2024-30251, and CVE-2024-27306), python-amply, python-argcomplete, python-btrees, python-cups, python-django-health-check, python-fluent-logger, python-persistent, python-plumbum, python-rpaths, python-rt, python-sniffio, python-tenacity, python-tokenize-rt, python-typing-extensions, pyupgrade, sphinx-copybutton, sphinxcontrib-autoprogram, uncertainties, zodbpickle, zope.configuration, zope.proxy, and zope.security to new upstream versions.
You can support my work directly via Liberapay.
Python Software Foundation: The 2024 PSF Board Election is Open!
It’s time to cast your vote! Voting is open starting today Tuesday, July 2nd, through Friday, July 16th, 2024 2:00 pm UTC. Check the Elections page to see how much time you have left to vote.
If you are a voting member of the PSF that affirmed your intention to participate in this year’s election, you will receive an email from “OpaVote Voting Link <noreply@opavote.com>” with a link to your ballot. The subject line will read “Python Software Foundation Board of Directors Election 2024”. If you haven’t seen your ballot by Wednesday, please check your spam folder for a message from “noreply@opavote.com”. If you don’t see anything get in touch by emailing psf-elections@python.org so we can look into your account and make sure we have the most up-to-date email for you.
Three seats on the board are open, but you can approve as many of the 19 candidates as you like. We’re delighted by how many of you are willing to contribute to the Python community by serving on the PSF Board! Make sure you take some time to look at all the nominee statements and choose your candidates carefully. ATTN: Choose carefully before you press the big green vote button. Once your vote is cast, it cannot be changed.
You need to be a Contributing, Managing, Supporting, or Fellow member and have affirmed your voting intention by June 25th, 2024, to vote in this election. If you’d like to learn more or sign up as a PSF Member, check out our membership types. You can check your membership status on your User Information page on psfmember.org (you will need to be logged in). If you have questions about your membership or the election please email psf-elections@python.org
Python Bytes: #390 Coding in a Castle
Junichi Uekawa: July.
roose.digital: Tips for creating calculators in Drupal using Webforms and the Computed Twig element
Tryton News: Newsletter June 2024
In the last month we focused on speeding-up performance issues, fixing bugs and improving the behaviour of things - building on the changes from our last release. We also added some new features which we would like to introduce to you in this newsletter.
For an in depth overview of the Tryton issues please take a look at our issue tracker or see the issues and merge requests filtered by label.
Changes for the User Sales, Purchases and ProjectsWe use a dedicated Web Shop-page on the product form which contains web shop related fields.
We’ve added relates from sale and purchase lines to their stock moves and invoice lines.
Purchase and sale amendments now allow to update the secondary unit of its lines.
Now Tryton deletes a purchase request when its related product is deleted. Previously such a purchase request was kept in the system, but we decided that is better to remove it.
Accounting, Invoicing and PaymentsPayments with zero amount are allowed again in the system. This allows to correctly handle full refunds for some payment gateways that use zero amounts on them.
Stock, Production and ShipmentsWhen counting inventories with lots we now also show the lot in addition to the product, as a product may have many lots.
User InterfaceSao now uses a grid to display trytond.model.fields.Dict items to add more flexibility.
To make Tryton more accessible we now make the contents of the message-dialog selectable and copiable.
Data and ConfigurationWe improved the user experience when importing CSV data. This eases the adoption of Tryton by lowering the barrier to load initial data in to the system. Here is a list of the relevant changes:
- The import wizard does no longer close the import window on error.
- Error messages include row and column numbers to ease the manual error correction.
- We raise an user error when the import data has a different number of columns as defined in the import wizard.
The CSV export also got new features. It now supports different languages per column in one export. This is specially useful when working with translatable master data like for example product names.
We now replace the “Accounting Party” user access group by the “Accounting” user access group. There is no need to limit accounting fields from party to a specific group by default.
New DocumentationThe ldap_authentication module is now documented.
Did you know, that a Model._rec_name must point to a trytond.model.fields.Char field?
New ReleasesWe released bug fixes for the currently maintained long term support series
7.0 and 6.0, and for the penultimate series 7.2.
We added a new configuration section [report] with option convert_command to be able to use a different document converter.
Now the trytond-admin command validates the email-value. The interactive email input loops until a valid email address is entered.
Changes for Implementers and DevelopersWe added the option --export-translations to the trytond-admin command. It exports the translation of any activated module to their respective locale folder.
1 post - 1 participant
Specbee: Why Stylus is more flexible than SCSS and how to implement it in Drupal
Zato Blog: Understanding API rate-limiting techniques
Enabling rate-limiting in Zato means that access to Zato APIs can be throttled per endpoint, user or service - including options to make limits apply to specific IP addresses only - and if limits are exceeded within a selected period of time, the invocation will fail. Let's check how to use it all.
API rate limiting works on several levels and the configuration is always checked in the order below, which follows from the narrowest, most specific parts of the system (endpoints), through users which may apply to multiple endpoints, up to services which in turn may be used by both multiple endpoints and users.
- First, per-endpoint limits
- Then, per-user limits
- Finally, per-service limits
When a request arrives through an endpoint, that endpoint's rate limiting configuration is checked. If the limit is already reached for the IP address or network of the calling application, the request is rejected.
Next, if there is any user associated with the endpoint, that account's rate limits are checked in the same manner and, similarly, if they are reached, the request is rejected.
Finally, if the endpoint's underlying service is configured to do so, it also checks if its invocation limits are not exceeded, rejecting the message accordingly if they are.
Note that the three levels are distinct yet they overlap in what they allow one to achieve.
For instance, it is possible to have the same user credentials be used in multiple endpoints and express ideas such as "Allow this and that user to invoke my APIs 1,000 requests/day but limit each endpoint to at most 5 requests/minute no matter which user".
Moreover, because limits can be set on services, it is possible to make it even more flexible, e.g. "Let this service be invoked at most 10,000 requests/hour, no matter which user it is, with particular users being able to invoke at most 500 requests/minute, no matter which service, topping it off with per separate limits for REST vs. SOAP vs. JSON-RPC endpoint, depending on what application is invoke the endpoints". That lets one conveniently express advanced scenarios that often occur in practical situations.
Also, observe that API rate limiting applies to REST, SOAP and JSON-RPC endpoints only, it is not used with other API endpoints, such as AMQP, IBM MQ, SAP, task scheduler or any other technologies. However, per-service limits work no matter which endpoint the service is invoked with and they will work with endpoints such as WebSockets, ZeroMQ or any other.
Lastly, limits pertain to with incoming requests only - any outgoing ones, from Zato to external resources - are not covered by it.
Per-IP restrictionsThe architecture is made even more versatile thanks to the fact that for each object - endpoint, user or service - different limits can be configured depending on the caller's IP address.
This adds yet another dimension and allows to express ideas commonly witnessed in API-based projects, such as:
- External applications, depending on their IP addresses, can have their own limits
- Internal users, e.g. employees of the company using VPN, may have hire limits if their addresses are in the 172.x.x.x range
- For performance testing purposes, access to Zato from a few selected hosts may have no limits at all
IP-based limits work hand in hand are an integral part of the mechanism - they do not rule out per-endpoit, user or service limits. In fact, for each such object, multiple IP-using limits can be set independently, thus allowing for highest degree of flexibility.
Exact or approximateRate limits come in two types:
- Exact
- Approximate
Exact rate limits are just that, exact - they en that a limit is not exceeded at all, not even by a single request.
Approximate limits may let a very small number of requests to exceed the limit with the benefit being that approximate limits are faster to check than exact ones.
When to use which type depends on a particular project:
-
In some projects, it does not really matter if callers have a limit of 1,000 requests/minute or 1,005 requests/minute because the difference is too tiny to make a business impact. Approximate limits work best in this case.
-
In other projects, there may be requirements that the limit never be exceeded no matter the circumstances. Use exact limits here.
Alright, let's check how to define the limits in the Zato Dashboard. We will use the sample service below:
# -*- coding: utf-8 -*- # Zato from zato.server.service import Service class Sample(Service): name = 'api.sample' def handle(self): # Return a simple string on response self.response.payload = 'Hello there!\n'Now, in web-admin, we will configure limits - separately for the service, a new and a new REST API channel (endpoint).
Points of interest:
- Configuration for each type of object is independent - within the same invocation some limits may be exact, some may be approximate
- There can be multiple configuration entries for each object
- A unit of time is "m", "h" or "d", depending on whether the limit is per minute, hour or day, respectively
- All limits within the same configuration are checked in the order of their definition which is why the most generic ones should be listed first
Now, all is left is to invoke the service from curl.
As long as limits are not reached, a business response is returned:
$ curl http://my.user:password@localhost:11223/api/sample Hello there! $But if a limit is reached, the caller receives an error message with the 429 HTTP status.
$ curl -v http://my.user:password@localhost:11223/api/sample * Trying 127.0.0.1... ... < HTTP/1.1 429 Too Many Requests < Server: Zato < X-Zato-CID: b8053d68612d626d338b02 ... {"zato_env":{"result":"ZATO_ERROR","cid":"b8053d68612d626d338b02eb", "details":"Error 429 Too Many Requests"}} $Note that the caller never knows what the limit was - that information is saved in Zato server logs along with other details so that API authors can correlate what callers get with the very rate limiting definition that prevented them from accessing the service.
zato.common.rate_limiting.common.RateLimitReached: Max. rate limit of 100/m reached; from:`10.74.199.53`, network:`*`; last_from:`127.0.0.1; last_request_time_utc:`2020-11-22T15:30:41.943794; last_cid:`5f4f1ef65490a23e5c37eda1`; (cid:b8053d68612d626d338b02)And this is it - we have created a new API rate limiting definition in Zato and tested it out successfully!
More resources➤ Python API integration tutorial
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?