Feeds
GNU Taler news: Video interview with Nic Eigel, co-author of the GNU Taler real-time auditor
Bits from Debian: Wind River Platinum Sponsor of DebConf24
We are pleased to announce that Wind River has committed to sponsor DebConf24 as a Platinum Sponsor.
For nearly 20 years, Wind River has led in commercial open source Linux solutions for mission-critical enterprise edge computing. With expertise across aerospace, automotive, industrial, telecom, more, the company is committed to open source through initiatives like eLxr, Yocto, Zephyr, and StarlingX.
With this commitment as Platinum Sponsor, Wind River is contributing to make possible our annual conference, and directly supporting the progress of Debian and Free Software, helping to strengthen the community that continues to collaborate on Debian projects throughout the rest of the year.
Wind River plans to announce an exiting new project based on Debian at this year's DebConf!
Thank you very much, Wind River, for your support of DebConf24!
Become a sponsor too!DebConf24 will take place from 28th July to 4th August 2024 in Busan, South Korea, and will be preceded by DebCamp, from 21st to 27th July 2024.
DebConf24 is accepting sponsors! Interested companies and organizations should contact the DebConf team through sponsors@debconf.org, or visit the DebConf24 website at https://debconf24.debconf.org/sponsors/become-a-sponsor/.
Kushal Das: Friends, the most important part of any conference
At the beginning one goes to the conferences to listen to the talks and make new contacts. You meet a lot of new faces every time. Over time a few of them will become great friends and then all conferences will become about friends.
We wait for the conferences so that we can meet our friends. I went back to PyCon US this year after 5 years, means I met many friends after 5 years. It was so happy feeling to see them again.
Last week I went to my first ever Euro Python in Prague, finally the visa was good in the right days of the year. This means I managed to meet more friends, a few of them just after a month (as they were present in PyCon US) and some after many many years. Really enjoyed the social event place selections by the organizers.
Personally the social events allowed me to go full scale nerd out on technical and social issues with friends. I was really missing these discussions. Heard more stories and discussed about fun ideas. One is below :)
$ python Python 3.12.4 (main, Jun 7 2024, 00:00:00) [GCC 14.1.1 20240607 (Red Hat 14.1.1-5)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> hello 🤌🤌🤌 Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name 'hello' is not defined. Did you mean: 'help'? >>> [].set("different exception") 🤌🤌🤌 Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'list' object has no attribute 'set' >>>KDE Plasma 6.1.3, Bugfix Release for July
Tuesday, 16 July 2024. Today KDE releases a bugfix update to KDE Plasma 6, versioned 6.1.3.
Plasma 6.1 was released in June 2024 with many feature refinements and new modules to complete the desktop experience.
This release adds two weeks' worth of new translations and fixes from KDE's contributors. The bugfixes are typically small but important and include:
- KWin Plugins nightlight: Relax custom times constraints. Commit. Fixes bug #489366
- Fix KCM Clock save on non-systemd distros. Commit.
- KWin Tiling: Don't put maximized windows in tile. Commit. Fixes bug #489463
How to: Profile your display in the Plasma Wayland session
Profiling displays is already not a super simple thing on its own, but things get more complicated when you try to profile your display in Wayland - profiling applications don’t support Wayland yet, some APIs on the compositor side to make it work well are still missing, and there’s a general lack of information on the topic. So today I’ll show you how to profile your display in the Plasma Wayland session.
I did this in Fedora 40, but you can follow these steps in other distributions as well.
Step 1: Install DisplayCal and start itThis sounds easy, but
- it’s not packaged for Fedora. That’s being worked on, but right now it’s not an option | edit: turns out there is a COPR for it
- installing it with pip just gave me a bunch of compilation errors, and I haven’t figured out how to fix them
- the package on Flathub is really old and broken
To work around that, I used distrobox to install the Arch Linux package for DisplayCAL:
sudo dnf install distrobox distrobox create --name archinabox --image archlinux:latest distrobox enter archinabox sudo pacman -S displaycal distrobox-export --app displaycal exitAfter running these commands, DisplayCAL can be started from any app launcher, like Kickoff or KRunner.
Step 2: SetupTo get correct measurement results, the compositor needs to pass the pixel data from the profiling app directly to the display, and not do any color management itself. This will be automated at some point, but for now you need to manually ensure that
- HDR is disabled
- the color profile of the display is set to “None” in the display settings
- night light is off, or at least suspended in the system tray
- all KWin effects that modify colors, like the color blindness correction effect, are disabled
- if you’re on a new-ish AMD laptop and want to profile the internal display, that you’re either plugged in to a power source, or have the power profile set to performance, to disable a power saving feature that changes the colors
Now start DisplayCal and head to the Calibration tab. Here it’s important to set the tone curve to “as measured”, and untick interactive display adjustment, as those don’t work correctly right now and will mess up the profile.
You’ve done everything correctly if the button on the bottom of the application shows “Profile only”.
Last but not least, you also need to adjust the display settings to what you want to use with the profile later, as the profile is only correct for one specific set of display settings. This includes the brightness of the display!
Step 3: ProfileIn the profiling tab of DisplayCAL, select your desired settings - in most cases the default will be sufficient - and click “Profile only”. When it asks if you want to continue with the current calibration curves, select “use linear calibration instead” and de-select “embed calibration curves in profile”. Then put the colorimeter in the center of the screen, and let it do its thing.
Once it’s done, it’ll ask you to install the profile. Installing it will not automatically enable that profile to be used, but it’ll save the profile in ~/.config/color/icc/devices/display/ and you can select that file in the display settings.
Step 4: Verification (optional)If you’d like to make sure the profile is correct or accurate enough, you can use DisplayCAL to verify the result. Make sure you’ve set the profile in the display settings, switch to the verification tab in DisplayCAL and select your newly created profile in the “settings”
Here again, because DisplayCAL doesn’t support Wayland yet, you need to adjust a few settings for everything to work correctly. You need to select the simulation profile “Rec.709 ITU-R BT.709”, select “Use simulation profile as display profile” and set the tone curve to “Gamma 2.2”. Afterwards, click on “Measurement report”, choose a location to save it in, put the colorimeter in the center of the screen again and wait for it to complete.
Don’t be alarmed if the result says the whitepoint is wrong, this is simply caused by DisplayCAL assuming we want to target the whitepoint of the simulation profile, which doesn’t necessarily match the whitepoint of your display.
What about calibration though?To calibrate the display, that is, to adjust brightness, tone curves for non color managed applications and the whitepoint of the display, DisplayCAL uses an X11 API to set the gamma lookup tables of the GPU. That API doesn’t work in the Wayland session and the profiling process doesn’t handle that situation properly, which is why all calibration needs to be disabled for the created profile to be correct.
DisplayCAL (or ArgyllCMS, which does the actual profiling) could add support for applying a lookup table in the application instead of having the compositor do it, but we can also handle calibration entirely on the compositor side instead, which offers a bit more flexibility.
Changing the tone curves for non color managed applications doesn’t make sense in the Plasma Wayland session, as all windows are always color managed, so that part is already dealt with. Adjusting the brightness on screens that don’t have any native means of brightness control is already implemented for Plasma 6.2, and I have a working proof of concept for changing the whitepoint of the display without needing a new ICC profile too, so we should be at feature parity soon. I’ll talk more about these adjustments in a future post.
The Open Source Initiative joins CMU in launching Open Forum for AI: A human-centered approach to AI development
The Open Source Initiative (OSI) is pleased to share that we are joining the founding team of Open Forum for AI (OFAI), an initiative designed by Carnegie Mellon University (CMU) to foster a human-centered approach to artificial intelligence. OFAI aims to enhance our understanding of AI and its potential to augment human capabilities while promoting responsible development practices.
The missions of OSI and OFAI are well-aligned; at the heart of OFAI is a commitment to ensuring that AI development serves the public interest. With the support of renowned partners like Omidyar Network, NobleReach Foundation, and internal CMU funding, OFAI is positioned to serve as a pivotal platform for shaping AI strategies and policies that prioritize safety, privacy, and equity.
The OSI is proud to be part of this project. Stefano Mafulli and Deb Bryant from the OSI will participate in OFAI, integrating their efforts toward a standard Open Source AI Definition through a collaborative process involving stakeholders from the Open Source community, industry, and academia as well as their contributions to public policy.
A collective effortThe success of OFAI hinges on the diverse expertise it convenes. Leading this initiative is Sayeed Choudhury, Associate Dean for Digital Infrastructure at CMU and a member of the OSI Board. Alongside him, a team of CMU faculty members and external advisors will contribute knowledge in ethics, computational technologies, and inclusive AI research.
Notable participants like Michele Jawando from Omidyar Network and Arun Gupta from NobleReach Foundation have emphasized the importance of Open Source AI in driving innovation and inclusivity as well as the need for a human-centered, trust-based approach to AI development.
OFAI’s ambitious goalsOFAI aims to influence AI policy by coordinating research and policy objectives and advocating for transparent and inclusive AI development. The initiative will focus on five key areas:
- Research
- Technical prototypes
- Policy recommendations
- Community engagement
- Talent for service
Deb Bryant will lead Community Engagement, building in part upon the broad community of interest gathered through the public process of OSI’s Defining Open Source AI.
One of OFAI’s foundational projects is the creation of an “Openness in AI” framework, which seeks to make AI development more transparent and inclusive. This framework will serve as a vital resource for policymakers, researchers, and the broader community.
Looking aheadWith the OSI set to deliver a stable version of the Open Source AI Definition at All Things Open in October, the launch of OFAI magnifies the importance of this work to bring together diverse stakeholders to ensure AI technologies align with societal values and public interests.
Drupal Association blog: Drupal Business Survey 2024
The Drupal Business Survey investigates the trends in the digital market, in particular from service providers involved with the open source DXP Drupal. Digital agencies from all over the world participate in the yearly survey. Business insights on market share and growth opportunities are shared in a comprehensive report. The results are presented at DrupalCon Europe, the international Drupal conference. Digital business owners from all over the world are invited to participate.
The Drupal Business Survey shares valuable business insights from Drupal service providers worldwide. Drupal’s open source ecosystem is supported by a strong community of tens of thousands professionals worldwide, working together on the popular digital experience platform. Because Drupal is open source, anyone can work with Drupal or make changes to it. The Drupal Business Survey gives meaningful data for business owners and decision makers to build their next year’s business strategy on.
The Drupal Business Survey results in a comprehensive report on business outlook and customer engagement. It has been a valuable guide for digital service providers, even to those working with other technologies but Drupal. Participants are from all continents, with most of the companies being in business for 10 years or more.
Results in 2023In 2023, the Business Survey underscored Drupal’s powerful selling points including security, flexibility and reliability. Business decision makers choosing Drupal for the digital experiences, found these values to be significantly more important. Drupal’s market share increased notably in 2023 in the finance industry as well as industry and automotive. Education and government remain consistent Drupal users.
Open Source remains one of the most powerful reasons for choosing Drupal. Being vendor independent and using the power is the basis for Drupal’s success. Companies have continued their contribution to the open source ecosystem, in spite of the harder market outlook in 2023.
Also the need for Drupal talent continues to be strong. More than 70% of the companies stated the business growth or need of expertise as their main reason for hiring new staff.
Imre Gmelig Meijling: “We continue to see a strong position of Drupal as valuable solution. Agencies and service providers remain committed to Drupal and pitch it as one of the best content management solutions on the market. The strong commitment to Drupal by so many organizations worldwide proves that Drupal is a solid choice for many brands and organizations across the globe.”
CEO DinnerThe Drupal market is looking forward to how digital service providers are using Drupal and how the Drupal market has evolved this year. The results of the 2024 Drupal Business Survey will be presented at the CEO Dinner at DrupalCon Europe. DrupalCon Europe is the yearly Drupal summit where over 1500 Drupal users and professionals meet to exchange ideas and further evolve Drupal. The conference is held from 24-27 September in Barcelona, Spain. The CEO Dinner will be on Wednesday 25 September, extending the conversation to a BOF (Birds of a Feather) session on 26 September.
About the Business SurveyThe Drupal Business Survey support Drupal businesses worldwide and is organized by a team of industry experts Imre Gmelig Meijling (React Online), Janne Kalliola (Exove) and Michel van Velde (Craftmore) in collaboration with the Drupal Association.
Drupal is the open source Digital Experience Platform used by many organizations worldwide including Nestlé, Lufthansa and WWF.
Participate and share your insightsDrupal experts are invited to share their Drupal business insights through the Business Survey anonymously and come to DrupalCon Europe to review the results together.
You can participate in the Drupal Business Survey anonymously here: The Drupal Business Survey 2024. The survey closes on 4 September.
Open Source AI Definition – Weekly update July 15
It has been quiet over the 4th of July weekend on the forums and OSI has been speaking at different events:
- @stefano spoke in a panel at the UN event OSPOs for Good. Access the recording here.
- @mer is speaking at Open Source Community Africa
- OSI was present at the Linux Foundation hosted AI_dev: Open Source GenAI & ML Summit Europe 2024. Read about the takeaways here.
- @jberkus expresses concern about the extensive resources required to certify AI systems, estimating that it would take weeks of work per system. This scale makes it impractical for a volunteer committee like License Review.
- @shujisado reflects on past controversies over license conformity, noting that Open Source AI has the potential for a greater economic impact than early Open Source” He acknowledges the need for a more robust certification process given this increased significance. He suggests that cooperation from the machine learning community or consortia might be necessary to address technical issues and monitor the certification process neutrally. He offers to help spread the word about OSAID within the Japanese ML/LLM development community.
@jberkus clarifies that the OSI would need full-time paid staff to handle the certifications, as the work cannot be managed by volunteers alone.
GNU Taler news: Videos from main talks of Privacy, Identity and Payment in the Next Generation Internet event at BFH
Talking Drupal: Talking Drupal #459 - Off The Cuff 8
Today we are talking about Config Actions, The Panels Favorite Drupal Modules, and Drupal Contribution. We’ll also cover Transform API as our module of the week.
For show notes visit: www.talkingDrupal.com/459
Topics- New Config Action: Place Block
- Favorite Contrib modules
- Slack channels
- Preparing for Drupal 11
- Drupal events
- Config Action Place Block
- Front End Editing Drupal Module
- Gin Admin Theme
- Migrate Boost
- Keysave
- Navigation
- Matt Glaman Smart Date Code Blog Post
Nic Laflin - nLighteneddevelopment.com nicxvan John Picozzi - epam.com johnpicozzi Martin Anderson-Clutz - mandclu.com mandclu Baddý Sonja Breidert - 1xINTERNET baddysonja
MOTW CorrespondentMartin Anderson-Clutz - mandclu.com mandclu
- Brief description:
- Have you ever wanted to expose your Drupal site’s data as JSON using view modes, formatters, blocks, and more? There’s a module for that.
- Module name/project name:
- https://www.drupal.org/project/transform_api
- Transform API
- Brief history
- How old: created in Sep 2023 by LupusGr3y, aka Martin Giessing of Denmark
- Versions available: 1.1.0-beta4 and 1.0.2 versions available, both of which work with Drupal 9 and 10
- Maintainership
- Actively maintained, in fact the latest commit was earlier today
- Security coverage
- Documentation: in-depth README and a full user guide
- Number of open issues: 14 open issues, 3 of which are bugs, but none against the current branch
- Usage stats:
- 2 sites
- Module features and usage
- After installing Transform API, you should be able to get the JSON for any entities on your site by adding “format=json” as a parameter to the URL
- To get more fields exposed as JSON, you can configure a Transform mode, using a Field UI configuration very similar to view modes
- You can also add transform blocks to globally include specific data in all transformed URLs, in the same way you would use normal blocks to show information on your entity pages. The output of transform blocks is segmented into regions,
- Where Drupal’s standard engine produces render arrays that ultimately become HTML, Transform API replaces it with an engine that produces Transform Arrays that will ultimately become JSON
- Where Drupal’s standard JSON:API supports more or less exposes all information as raw data for the front end to format, Transform API allows for more of the formatting to be managed on the back end, where it will use Drupal’s standard caching mechanisms, permission-based access, and more
- Transform API also supports lazy transformers, which are callbacks that will be called after caching but before the JSON response is sent
- You can also use alter hooks to manipulate the transformed data
Mike Driscoll: Creating Images in Your Terminal with Python and Rich Pixels
A newer Python package called Rich Pixels allows you to create images in your terminal and display them. Darren Burns, one of the team members from the Textual project, created this package.
Anyway, let’s find out how to use Rich Pixels!
InstallationYou can install Rich Pixels using Python’s pip utility. Here’s how:
python -m pip install rich-pixelsOnce you have Rich Pixels installed, you can try it out!
Displaying Images in the TerminalRich Pixels lets you take a pre-existing image and show it in your terminal. The higher the image’s resolution, the better the output will be. However, if your image has too many pixels, it probably won’t fit in your terminal, and much of it will be drawn off-screen.
For this example, you will use the Python Show Podcast logo and attempt to draw it in your terminal.
Open up your favorite Python editor and add the following code to it:
from rich_pixels import Pixels from rich.console import Console console = Console() pixels = Pixels.from_image_path("python_show200.jpg") console.print(pixels)For this example, you will use a square image that is 200×200 pixels. You can run the code like this in your terminal:
python pixels.py
When you execute the command above, you will see something like this in your terminal:
As you can see, the image is a little pixelated and gets cut off at the bottom. Of course, this all depends on your monitor’s resolution.
Here’s what happens when you use an image that is 80×80 pixels:
You can also. use the Pillow package to create an image object and pass that the Rich Pixels too. Here’s how that might look:
with Image.open("path/to/image.png") as image: pixels = Pixels.from_image(image)You can create or draw your images using Pillow. There is some coverage of this topic in my article, Drawing Shapes on Images with Python and Pillow which you could then pass to Rich Pixels to display it.
Wrapping UpRich Pixels is a fun way to add extra pizzazz to your terminal applications. Rich Pixels can also be used in a Textual application. While there probably aren’t a lot of use cases for this package, it’s a lot of fun to play around with.
Give it a try, and let me know what you create!
The post Creating Images in Your Terminal with Python and Rich Pixels appeared first on Mouse Vs Python.
The Drop Times: Open is the New Default
Dear Readers,
Governments worldwide are increasingly embracing open-source software (OSS) to enhance their operations' transparency, security, and efficiency. Open source allows anyone to inspect, modify, and improve software, fostering a culture of collaboration and innovation. This shift is transforming the way public sector bodies function, ensuring digital solutions are cost-effective and tailored to the unique needs of each community.
Switzerland's recent mandate for OSS in its public sector is a prime example of this trend. The "Federal Law on the Use of Electronic Means for the Fulfilment of Governmental Tasks" (EMBAG) requires government-developed software to be open source, marking a significant step towards greater transparency and efficiency. However, Switzerland is not alone in this initiative; many countries are adopting similar policies to reduce dependency on costly proprietary software and foster innovation.
One of the notable benefits of open source in government is its potential for increased security and reliability. Continuous scrutiny and improvement by a global community of developers result in robust and adaptable solutions that can quickly respond to changing needs and security threats. Additionally, OSS helps prevent vendor lock-in, allowing governments to choose the best solutions without being tied to specific providers.
A recent study by Veniz Guzman on The DropTimes highlights the growing adoption of the open-source CMS Drupal among government entities of all sizes. Larger governments with complex digital needs are increasingly turning to Drupal for its flexibility and robust features. Interestingly, smaller to mid-sized entities are also recognizing Drupal's benefits, with rising adoption rates among smaller cities and counties.
With that, let's move on to the important stories of last week.
Drupal Starshot has announced the formation of its Advisory Council, a significant step in its efforts to expand the reach and impact of Drupal. This council will provide strategic input and feedback to ensure that the initiative meets the needs of key stakeholders and end users. To learn more about the composition of the Starshot Advisory Council, click here.
In "Using Drupal Migrations to Modify Content Within a Site," Jaymie Strecker demonstrates how to use the Drupal Migrate API for restructuring content within a site. The article provides practical examples and assumes basic knowledge of the Migrate API.
In other news, the Drupal Association has announced HeroDevs as the inaugural partner for the new Drupal 7 Extended Security Support Provider Program. This initiative aims to assist organizations unable to migrate from Drupal 7 by the end-of-life date of January 5, 2025, ensuring their systems remain secure and compliant.
Additionally, Drupal.org has announced that it will end support for Composer 1 on Packages.Drupal.org. This move is part of efforts to prepare its infrastructure for providing automatic updates for Drupal and upgrading Drupal.org itself. New packages and releases will no longer be available for Composer 1 starting August 12, 2024, and full support will be dropped on October 1, 2024.
Doubling the spirit of community, DrupalCon Europe has announced Tomasz Rogalski as the winner of the Merce Drawing Contest. His drawing of Mercè, the celebrated figure of Barcelona's annual festival, will be featured on the official event stickers at DrupalCon Barcelona.
The Women in Drupal award will be presented shortly at DrupalCon Barcelona, an event dedicated to the Drupal community. This award aims to recognize and celebrate the significant contributions made by women within the Drupal ecosystem. To nominate a deserving Drupal artist, attendees can fill out the nomination form provided.
Drupal GovCon 2024 is only a month away, scheduled to happen in August 13 - 15, 2024. It will feature a session titled "What the Heck is ARIA? A Beginner's Guide to ARIA for Accessibility," presented by Kat Shaw, a Lead Engineer and Accessibility Advocate (CPACC) at Lullabot. DrupalCamp Colorado has announced its second Keynote Speaker for this year's event, Matthew Saunders.
The DropTimes has compiled notable Drupal events happening throughout the week of July 15 to July 21. This curated list offers a glimpse into the varied activities taking place within the Drupal community, catering to enthusiasts of all skill levels. Read here.
amazee.io announces the launch of a new cloud region on AWS in Tokyo, Japan, enhancing the company's services in the Asia Pacific region. This expansion provides Japanese customers with reduced latency, improved fault tolerance, and compliance with local data privacy laws.
Roderik Muit and Alexandru Ieremia of drunomics presented the new Custom Elements module at Drupal Developer Days in Burgas, in June 2024. This session, part of an informal "Birds of a Feather" discussion, introduced the latest version of the Custom Elements module, a vital part of drunomics' headless Drupal solutions.
We acknowledge that there are more stories to share. However, due to selection constraints, we must pause further exploration for now.
To get timely updates, follow us on LinkedIn, Twitter and Facebook. Also, join us on Drupal Slack at #thedroptimes.
Thank you,
Sincerely
Alka Elizabeth
Sub-editor, The DropTimes.
The Drop Times: Using Drupal Migrations to Deploy New Content
Python Bindings for KDE Frameworks: GSoC Midterm Review
Week 7 of my 14-week GSoC project has finished, which means that it’s time for a midterm update! Many things have happened since the last update.
I added support for KI18n, KGuiAddons, KNotifications, KUnitConversion and KXMLGui. That was faster than expected. I also created a small unit conversion demo using KUnitConversion (it’s written in Python):
We have decided to move on to upstream the bindings to their corresponding repositories. For that, I set up a development environment and I’m now adding the code to each library.
Real Python: Split Your Dataset With scikit-learn's train_test_split()
One of the key aspects of supervised machine learning is model evaluation and validation. When you evaluate the predictive performance of your model, it’s essential that the process be unbiased. Using train_test_split() from the data science library scikit-learn, you can split your dataset into subsets that minimize the potential for bias in your evaluation and validation process.
In this tutorial, you’ll learn:
- Why you need to split your dataset in supervised machine learning
- Which subsets of the dataset you need for an unbiased evaluation of your model
- How to use train_test_split() to split your data
- How to combine train_test_split() with prediction methods
In addition, you’ll get information on related tools from sklearn.model_selection.
Get Your Code: Click here to download the free sample code that you’ll use to learn about splitting your dataset with scikit-learn’s train_test_split().
Take the Quiz: Test your knowledge with our interactive “Split Your Dataset With scikit-learn's train_test_split()” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
Split Your Dataset With scikit-learn's train_test_split()In this quiz, you'll test your understanding of how to use the train_test_split() function from the scikit-learn library to split your dataset into subsets for unbiased evaluation in machine learning.
The Importance of Data SplittingSupervised machine learning is about creating models that precisely map the given inputs to the given outputs. Inputs are also called independent variables or predictors, while outputs may be referred to as dependent variables or responses.
How you measure the precision of your model depends on the type of a problem you’re trying to solve. In regression analysis, you typically use the coefficient of determination, root mean square error, mean absolute error, or similar quantities. For classification problems, you often apply accuracy, precision, recall, F1 score, and related indicators.
The acceptable numeric values that measure precision vary from field to field. You can find detailed explanations from Statistics By Jim, Quora, and many other resources.
What’s most important to understand is that you usually need unbiased evaluation to properly use these measures, assess the predictive performance of your model, and validate the model.
This means that you can’t evaluate the predictive performance of a model with the same data you used for training. You need evaluate the model with fresh data that hasn’t been seen by the model before. You can accomplish that by splitting your dataset before you use it.
Training, Validation, and Test SetsSplitting your dataset is essential for an unbiased evaluation of prediction performance. In most cases, it’s enough to split your dataset randomly into three subsets:
-
The training set is applied to train or fit your model. For example, you use the training set to find the optimal weights, or coefficients, for linear regression, logistic regression, or neural networks.
-
The validation set is used for unbiased model evaluation during hyperparameter tuning. For example, when you want to find the optimal number of neurons in a neural network or the best kernel for a support vector machine, you experiment with different values. For each considered setting of hyperparameters, you fit the model with the training set and assess its performance with the validation set.
-
The test set is needed for an unbiased evaluation of the final model. You shouldn’t use it for fitting or validation.
In less complex cases, when you don’t have to tune hyperparameters, it’s okay to work with only the training and test sets.
Underfitting and OverfittingSplitting a dataset might also be important for detecting if your model suffers from one of two very common problems, called underfitting and overfitting:
-
Underfitting is usually the consequence of a model being unable to encapsulate the relations among data. For example, this can happen when trying to represent nonlinear relations with a linear model. Underfitted models will likely have poor performance with both training and test sets.
-
Overfitting usually takes place when a model has an excessively complex structure and learns both the existing relations among data and noise. Such models often have bad generalization capabilities. Although they work well with training data, they usually yield poor performance with unseen test data.
You can find a more detailed explanation of underfitting and overfitting in Linear Regression in Python.
Prerequisites for Using train_test_split()Now that you understand the need to split a dataset in order to perform unbiased model evaluation and identify underfitting or overfitting, you’re ready to learn how to split your own datasets.
You’ll use version 1.5.0 of scikit-learn, or sklearn. It has many packages for data science and machine learning, but for this tutorial, you’ll focus on the model_selection package, specifically on the function train_test_split().
Note: While this tutorial is tested with this specific version of scikit-learn, the features that you’ll use are core to the library and should work equivalently in other versions of scikit-learn as well.
You can install sklearn with pip:
Read the full article at https://realpython.com/train-test-split-python-data/ »[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Week 7 recap - vector approach
Real Python: Quiz: How to Use Generators and yield in Python
In this quiz, you’ll test your understanding of Python generators.
Generators and the Python yield statement can help you when you’re working with large datasets that might overwhelm your machine’s memory. Another use case is when you have a complex function that needs to maintain an internal state every time it’s called.
When you understand Python generators, then you’ll be able to work with large datasets in a more Pythonic fashion, create generator functions and expressions, and apply your knowledge towards building efficient data pipelines.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Real Python: Quiz: How to Write Beautiful Python Code With PEP 8
In this quiz, you’ll test your understanding of how to write beautiful Python code with PEP 8.
By working through this quiz, you’ll revisit the key guidelines laid out in PEP 8 and how to set up your development environment to write PEP 8 compliant Python code.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Steinar H. Gunderson: Pull requests via git push
This project inspired me to investigate whether git.sesse.net could start accepting patches in a format that was less friction than email, and didn't depend on custom SSH-facing code written by others. And it seems it really can! The thought was to simply allow git push from anyone, but that git push doesn't actually push anything; it just creates a pull request (by email). It was much simpler than I'd thought. First make an empty hooks directory with this pre-receive hook (make sure it is readable by your web server, and marked as executable):
#! /bin/bash set -e read oldsha newsha refname git send-email --to=steinar+git@gunderson.no --suppress-cc=all --subject-prefix="git-anon-push PATCH" --quiet $oldsha..$newsha echo '' echo 'Thank you for your contribution! The patch has been sent by email and will be examined for inclusion.' echo 'The push will now exit with an error. No commits have actually been pushed.' exit 1Now we can activate this hook and anonymous push in each project (I already run git-http-backend on the server for pulling, and it supports just fine if you tell it to), and give www-data write permissions to store the pushed objects temporarily:
git config core.hooksPath /srv/git.sesse.net/hooks git config http.receivepack true sudo chgrp -R www-data . chmod -R g+w .And now any attempts to git push will send me patch emails that I can review and optionally include!
It's not perfect. For instance, it doesn't support multipush, and if you try to push to a branch that doesn't exist already, will error out since $oldsha is all-zeros. And the From: header is always www-data (but I didn't want to expose myself to all sorts of weird injection attacks by trying to parse the committer email). And of course, there's no spam control, but if you want to spam me with email, then you could just like… send email?
(I have backups, in case someone discovers some sort of evil security hole.)
Thomas Lange: FAIme adds Korean language support
In two weeks DebConf24, the Debian conference starts in Busan, South Korea. Therefore I've added support for the Korean language into the web service of FAI:
https://fai-project.org/FAIme/
Another new feature of the FAIme service will be announced at DebConf24 in August.