Planet Python

Subscribe to Planet Python feed
Planet Python - http://planetpython.org/
Updated: 6 hours 16 min ago

ListenData: NumPy argmax() Function : Learn with Examples

Sun, 2023-11-05 21:02

In this tutorial, we will see how to use the NumPy argmax() function in Python along with examples.

The numpy.argmax() function in Python is used to find the indices of the maximum element in an array.

Syntax of NumPy argmax() Function

Below is the syntax of the NumPy argmax() function:

import numpy as np np.argmax(array, axis, out) To read this article in full, please click hereThis post appeared first on ListenData
Categories: FLOSS Project Planets

Read the Docs: Read the Docs newsletter - November 2023

Sun, 2023-11-05 19:00
News and updates
  • Work continues on hardening Addons, our new in-documentation JavaScript client that supports all documentation tools. We’re looking for people in the community to test out this new functionality, and will be expanding access in the near future.

  • Python 3.12 is now supported on builds, and is the default version used when you specify build.tools.python: 3 in your configuration file.

  • Language codes are now normalized in URLs: https://blog.readthedocs.com/language-codes-are-now-normalized/.

  • Our search infrastructure was upgraded to Elastic Search 8, which should provide faster and better search results.

  • We submitted a fix to the API Key library we use to make API calls to our servers much faster, which should result in faster build times for all users.

Upcoming changes
  • We are working to expand the functionality of our redirects feature to support more use cases. More will be announced here in the coming month.

  • We are planning an upgrade to our dashboard notification system, so that users have more control and better context for on-site notifications.

  • Our beta dashboard continues to be tested in public beta, and new functionality for Addons configuration will only be available in that new interface.

Want to follow along with our development progress? View our full roadmap 📍️

Possible issues

We don’t have any possible issues to report this month.

Questions? Comments? Ideas for the next newsletter? Contact us!

Categories: FLOSS Project Planets

Marcos Dione: automating-blender-based-hillshading-with-python

Sun, 2023-11-05 11:19

Remember my Blend based hillshading? I promised to try to automate it, right? Well, it seems I have the interest and stamina now, so that's what I'm doing. But boys and girls and anything in between and beyond, the stamina is waning and the culprit is Blender's internals being exposed into a non-Pythonic API[3]. I swear if I worked in anything remotely close to this, I would be writing a wrapper for all this. But in the meantime, it's all a discovery path to something that does not resemble a hack. Just read some of Blender's Python Quickstart:

When you are familiar with other Python APIs you may be surprised that new data-blocks in the bpy API cannot be created by calling the class:

bpy.types.Mesh() Traceback (most recent call last): File "<blender_console>", line 1, in <module> <span class="createlink">TypeError</span>: bpy_struct.__new__(type): expected a single argument

This is an intentional part of the API design. The Blender Python API can’t create Blender data that exists outside the main Blender database (accessed through bpy.data), because this data is managed by Blender (save, load, undo, append, etc).

Data is added and removed via methods on the collections in bpy.data, e.g:

mesh = bpy.data.meshes.new(name="MyMesh")

That is, instead of making the constructor call this internal API, they make it fail miserably and force you to use the internal API! Today I was mentioning that Asterisk's programming language was definitely designed by a Telecommunications Engineer, so I guess this one was designed by a 3D artist? But I digress...

One of the first thing about Blender's internals is that one way to work is based on Contexts. This makes sense when developing plugins, where you mostly need to apply things to the selected object, but for someone really building everything from scratch like I need to, it feels weird.

One of the advantages is that you can open a Python console and let Blender show you the calls it makes for every step you make on the UI, but it's so context based that the results is useless as a script. Or for instance, linking the output of a thing into he the input of another is registered as a drag-and-drop call that includes the distance the mouse moved during the drag, so it's relative of the output dot where you started and what it links to also depends on the physical and not logical position of the things you're linking,

bpy.ops.node.link(detach=False, drag_start=(583.898, 257.74))

It takes quite a lot of digging around in a not very friendly REPL[1] with limited scrollback and not much documentation to find more reproducible, less context dependent alternatives. This is what's eating up my stamina, it's not so fun anymore. Paraphrasing someone on Mastodon: What use is a nice piece of Open Software if it's documentation is not enough to be useful[2]?

Another very important thing is that all objects have two views: one that has generic properties like position and rotation, which can be reacheched by bpy.data.objects; and one that has specific properties like a light's power or a camera's lens angle, which can be reached by f.i. bpy.data.cameras. This was utterly confusing, specially since all bpy.data's documentation is 4 lines long. Later I found out you can get specific data from the generic one in the .data attribute, so the take out is: always get your objects from bpy.data.objects.

Once we get over that issue, things are quite straightforward, but not necessarily easy. The script as it is can already be used with blender --background --python <script_file>, but have in account that when you do that, you start with the default generic 3D setup, with a light, a camera and a cube. You have to delete the cube, but you can get a reference to the other two to reuse them.

Then comes the administrative stuff around just rendering the scene. To industrialize it and be able to quickly test stuff, you can try to get command line options. You can use Python's argparser module for this, but have in account that those --background --python blender.py options are going to be passed to the script, so you either ignore unknown options or you declare those too:

mdione@ioniq:~/src/projects/elevation$ blender --background --python blender.py Blender 3.6.2 Read prefs: "/home/mdione/.config/blender/3.6/config/userpref.blend" usage: blender [-h] [--render-samples RENDER_SAMPLES] [--render-scale RENDER_SCALE] [--height-scale HEIGHT_SCALE] FILE blender: error: unrecognized arguments: --background --python

Also, those options are going to be passed to Blender! So at the end of your run, Blender is going to complain that it doesn't understand your options:

unknown argument, loading as file: --render-samples Error: Cannot read file "/home/mdione/src/projects/elevation/--render-samples": No such file or directory Blender quit

The other step you should do is to copy the Geo part of GeoTIFF to the output file. I used rasterio, mostly because at first I tried gdal (I was already using gdal_edit.py to do this in my previous manual procedure), but it's API was quite confusing and rasterio's is more plain. But, rasterio can't actually open a file just to write the metadata like gdal does, so I had to open the output file, read all data, open it again for writing (this truncates the file) and write metadata and data.

Now, some caveats. First, as I advanced in my last post, the method as it is right now has issues at the seams. Blender can't read GDAL VRT files, so either I build 9 planes instead of 1 (all the neighbors are needed to properly calculate the shadows because Blender is also taking in account light reflected back from other features, meaning mountains) or for each 1x1 tile I generate another with some buffer. I will try the first one and see if it fixes this issue without much runtime impact.

Second, the script is not 100% parametrized. Sun size and power are fixed based on my tests. Maybe in the future. Third, I will try to add a scattering sky, so we get a bluish tint to the shadows, and set the Sun's color to something yellowish. These should probably be options too.

Fourth, and probably most important. I discovered that this hillshading method is really sensible to missing or bad data, because they look like dark, deep holes. This is probably a deal breaker for many, so you either fix your data, or you search for better data, or you live with it. I'm not sure what I'm going to do.

So, what did I do with this? Well, first, find good parameters, one for render samples and another for height scale. Render time grows mostly linearly with render samples, so I just searched for the one before detail stopped appearing; the value I found was 120 samples. When we left off I was using 10 instead of 5 for height scale, but it looks too exaggerated on hills (but it looks AWESOME in mountains like the Mount Blanc/Monte Bianco! See below), so I tried to pinpoint a good balance. For me it's 8, maybe 7.

Why get these values right? Because like I mentioned before, a single 1x1°, 3601x5137px tile takes some 40m in my laptop at 100 samples, so the more tuned the better. One nice way to quickly test is to lower the samples or use the --render-scale option of the script to reduce the size of the output. Note that because you reduce both dimensions at the same time, the final render (and the time that takes) is actually the square of this factor: 50% is actually 25% (because 0.50 * 0.50 = 0.25).

So, without further addo, here's my script. If you find it useful but want more power, open issues or PRs, everything is welcome.

https://github.com/StyXman/blender_hilllshading [5]

Try to use the main branch; develop is considered unstable and can be broken.

A couple of images of the shadows applied to my style as teaser, both using only 20 samples and x10 height scale:

Dhaulagiri:

Mont Blanc/Monte Bianco:

Man, I love the fact that the tail of the Giacchiaio del Miage is in shadows, but the rest is not; or how Monte Bianco/Mont Blanc's shadow reaches across the valley to the base of la Tête d'Arp. But also notice the bad data close to la Mer de Glace.

blender python openstreemap gdal elevation hillshading rasterio gis dem

[1] Ok, TBH here, I'm very much used to ipython's console, it's really closer to the plain python one. No tab completion, so lots of calls to dir() and a few help()s.

[2] I couldn't find it again. Mastodon posts are not searchable by default, which I understand is good for privacy, but on the other hand the current clients don't store anything locally, so you can't even search what you already saw. I have several semi-ranting posts about this and I would show them to you, but they got lost on Mastodon. See what I mean?

[3] So you have an idea, this took me a whole week of free time to finish, including but not in the text, my old nemesis, terracing effect. This thing is brittle.

[4] Yeah, maybe the API is mostly designed for this.

[5] My site generator keeps breaking. This is the second time I have to publicly admit this. Maybe next weekend I'll gather steam and replace it with nikola.

Categories: FLOSS Project Planets

Mirek Długosz: 10 years in testing

Sat, 2023-11-04 08:39

Exactly 10 years ago I started my first job as a software tester.

That doesn’t mean I started testing 10 years ago. Back when I was in high school and at university, I did spend some time doing testing and quality-related stuff for various open source projects - Debian, KDE, Kadu, LibreOffice, Cantata, and some more. I don’t remember any longer which was the first and when exactly that happened. I imagine my first contribution was pretty uneventful - perhaps a message on users forum, a response confirming this is a bug, and an encouragement to report it on bug tracking system or devs mailing list.

Nonetheless, “first job as software tester” is a good place to start counting. First, it’s easy - I have papers to prove the exact date. Second, from that day I have spent about eight hours a day, five days a week, every week, on testing-related things. That adds up to a lot of time, but it’s the consistency that sets it apart from any open source work I have done. Last but not least, the decision to start this specific job set me on a path to treat testing much more seriously, and which eventually led me to where I am today.

I’m not much of a job hopper. In these 10 years, I have only had two employers. But I did change teams and projects quite a lot - I’ve been on 4 projects in first company, and now I’m on my 5th project in second company. The longest time I’ve ever been in a single project is 2 years and 7 months. Details are on LinkedIn.

I came into testing after getting a degree in sociology. In my time at university, I had an opportunity to get my feet wet in empirical social research. I approached testing the same way I approached empirical sociology, even if only because I didn’t really know anything else - I assumed there’s a number of things the team would like to know and my job is to learn about them and report my findings. The hard part is that we don’t have direct access to some of the things we would like to know more about, so we need to depend on a number of proxies of uncertain reliability. X can be caused by Y, and we observed X, but is this because of Y, or some other factor Z? How can we rule out Z? Today, I can confidently say this is not the worst way to approach testing.

When I started my first job, I have been using Linux as my main operating system for about 7 years. During that time I learned how to use shell, I got familiar with the idea that things change and move around, I faced various breakages after updates. Often trying to fix them was frustrating, but I did learn how to search for information, I picked up few tricks and I learned how various components can interact in complex system. That was another major source of experiences that influenced my approach to testing.

I guess I also have certain character traits that helped me to become a decent tester. I tend to be stubborn, I don’t give up easily, I self-identify as perfectionist and I strive to actually understand the thing I am dealing with.

After a year and a half I decided that I want to know more about testing, especially established testing techniques and solutions. My work was praised, but it was all based on intuition and past experiences from other fields. I felt I was missing fundamentals and I feared I might be missing some obvious and elementary testing techniques or skills. I tried to fill these gaps by attending an ISTQB preparation course, but it did not deliver what I was looking for.

My manager knew about my disappointment and at one point presented me with the opportunity to attend a testing conference in another city. One of the talks given there was called “Context-Driven Testing: A New Hope”. This is a funny title, as Context-Driven Testing was already 15 years old at that time and “schools of testing” debate has long left community conciousness. I don’t remember many details of the talk itself, but I did left the conference with a feeling that I should learn more about CDT, as they might have at least some of the answers I was looking for.

I think I started by reading “Lessons Learned in Software Testing”, and what a book it was! It not only revolutionized the way I think about testing to this day, but also gave me much-needed confidence. I found I was already doing some of the things that book recommended, but now I knew why they were worth doing. This is the book that everyone who is serious about testing should read, and probably re-read thorough their career. I think I read it at very good moment, too - I had about three years of experience at the time. I feel I wouldn’t get that much from it if I read it earlier.

Later I have read “Perfect Software” by late Jerry Weinberg. I think this is a great book for people who just start in testing. It surely helped to establish some of my knowledge, but I don’t think it was as influential for me as “Lessons Learned”. It would have been if I read it earlier.

Finally, I have read the complete archives of James Bach and Michael Bolton blogs. This is not something I can recommend to anyone, as both are very prolific writers - each authored few hundreds articles. I think it took me well over a year to get through them all. Nonetheless, this allowed me to fully immerse myself in their thinking and I can confidently say I understand where they are coming from and where they are going to. This also allowed me to stumble upon few very valuable articles and resources that I still refer to.

There’s a lot that I learned from all these resources, but I would like to point out two overarching principles that I often come back to. One, my role as a tester is to show possibilities and broaden the view of the team. My job is to go beyond simple and quick answers. Two, every single day I need to ask myself: what is the most important, most impactful thing I can do right now? And then do this exact thing, even if it means putting aside earlier plans and ideas. Change is something to embrace, not to be afraid of.

About five years into my career, I began to slowly move into more software development-heavy role. To some extent, that was out of necessity - I saw many tasks that could be rectified with a tiny bit of programming. At the same time, I was in the environment where development was considered higher on organizational totem pole than “manual testing”, and showing programming skills was a clear way for more respectable assignments and higher salary. Similar to my testing journey, that was not the moment I started to learn programming - I have written my first shell scripts and perl programs back in high school. While I did struggle, I felt confident enough in my programming prowess to do some simple things.

The event that really helped me to take off to the next level happened about a year after I joined Red Hat. We had a UI test automation framework, which was recently rewritten by a couple of contractors. They worked in a silo and as a result most of the team was not familiar with that code. My job was to learn it, contribute to it and become one of the maintainers.

I think contractors felt threatened by my presence and thought their job security depended on them being the only people capable of working with the framework. As a result, they made code review a nightmare. They threw it all - passive-aggressive comments, unhelpful comments, misleading comments, requests to change code that was already approved in earlier review cycle, demands to explain almost every single line of code, replying anytime between a day and a week. That was all on top of working with unfamiliar, complex and barely documented libraries.

I don’t look back at that time with fondness, but I have to admit it was an effective learning exercise. I was forced to understand things above my capabilities, and eventually I did understand them. This was very much the moment programming finally clicked for me. Also, I learned precisely what to avoid during code reviews and when teaching others.

Since then, my interests started to move more in direction of software design and architecture. I know I can write good enough code that works. But I also want to write code that is maintainable in the long term and allows for adjustments in response to changing environment or requirements.

In these 10 years, I have primarily been an individual contributor. This is the role I feel comfortable in and which I think suits me well. However, I did act as a kind of team lead in two separate occasions. Both times I was not formally a manager for other people and I didn’t feel I have all the tools necessary to make them do the required work. The first time I was completely unprepared for a challenge in front of me. The second time went a little bit better, as I knew more about ways to informally influence people.

These would be the rough summary and most important highlights of my 10 years in testing. There’s no narrative closure, as I am still here and intend to stay for a while longer. I’m happy to talk about testing, open source, software engineering and related topics, so feel free to get in touch with me if this is something you find interesting, or if you would like to draw from my experience.

Categories: FLOSS Project Planets

Pages