It’s not outdated, and maybe you didn’t get what I was talking about, but probably because I didn’t word it well. So let me quote someone else instead:
While pure Python code should work unchanged, code written in other languages or using the CPython C API may not. The GIL was implicitly protecting a lot of thread-unsafe C, C++, Cython, Fortran, etc. code - and now it no longer does. Which may lead to all sorts of fun outcomes (crashes, intermittent incorrect behavior, etc.)
This covers the layers below python, but still doesn’t fully cover the process wide restrictions (it’s not just about thread safety, global initialization and resources are very relevant too). It also doesn’t cover the exact same problem which also exists for any layer calling into python from above, which also relies/relied on the GIL for synchronization. This means that you can’t effectively, confidently, and safely scale python without scaling the number of python runtime instances, each in its own full OS process.
The fact that multiprocessing became popular, instead of just scaling/parallelizing with threads, is itself an extension of the GIL/no-GIL circus. But its not the primary point here.
So not only the proposition (which someone referenced in another sub-thread) to move everything into python is in and of itself a self-defeating argument, since it proves a language being a problem (through forcing language lock-in). But even within a language (python), not all problems are solvable still.
Note that this is not all theoretical for me. In one of my previous projects, I had to call into a python library (which didn’t have an adequate alternative) from rust code (through pyo3). And I was forced to scale with multiple python runtimes, each getting one request/message at a time.
And we still didn’t get to any low-level details yet 😉
Ah okay, I see what you’re saying. GIL removal can break FFI code through the C API.
I have a lot of thoughts on this, and most of it is too incoherent to put into a comment someone else can understand, but for the most part, I think the issues are still pretty overblown. I just can’t see a world where Python needs true multithreading support when multiprocessing works just fine for most applications. You find a hot loop, you parallelize it in multiprocessing and just like that, your process runs much faster.
I do think that GIL removal is a terrible idea in general, for the reasons in the quote you provided, but at least it’s optional. I would personally not enable it in anything I maintain for several years if I didn’t have to.
I also think that polyglot code running in the same process is also a bad idea. I can understand calling C or Fortran from Python for numerical processing for example, but doing the reverse (like calling Python from Rust) is something I try to avoid at all costs. In any project I’ve worked on, if significant portions of the codebase were written in different languages, you would just use something like protobuf to specify the I/O protocol between the different languages and call them across a socket. It’s easier to maintain in my opinion since it defines a concrete boundary between the two different portions of code, and also makes it much easier to test each component separately too.
If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases (which is what the intellectually deficient OP tried to argue against), then I’m not sure you’re qualified to opine on the matter (hint: spawning a thread per task is already too much overhead for many workloads). And I don’t mean any disrespect to you here. Your replies were far from the worst in this thread.
Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.
And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.
If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases
That’s way too broad of a statement to be true in any absolute sense. It really depends on what you’re doing, and sure, there are plenty of ways to do it stupidly, but a lot of people that are smarter than you and me have thought about ways that the overhead of spawning processes can be mitigated. All I can say is that I’ve had perfectly good success using multiprocessing.Pool and getting near perfect speedups. Is it as fast as threading? No. But it’s good enough. If it’s not good enough for you, then maybe we just have different use-cases, but without talking about concrete examples, we can’t make any specific claims.
And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.
What do you want to talk about? Are you writing kernel drivers in Python or something? Why would we talk about writing low level code in Python, that sounds absurdly stupid. Most of the high performance libraries people use are written in Fortran or C. The entire field of data science uses Python, if performance were not a heavily optimized problem, then a lot of people out there must really love wasting their money on compute. Python is a glue language for high performance low level code, everybody knows this already.
Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.
It’s not outdated, and maybe you didn’t get what I was talking about, but probably because I didn’t word it well. So let me quote someone else instead:
This covers the layers below python, but still doesn’t fully cover the process wide restrictions (it’s not just about thread safety, global initialization and resources are very relevant too). It also doesn’t cover the exact same problem which also exists for any layer calling into python from above, which also relies/relied on the GIL for synchronization. This means that you can’t effectively, confidently, and safely scale python without scaling the number of python runtime instances, each in its own full OS process.
The fact that
multiprocessingbecame popular, instead of just scaling/parallelizing with threads, is itself an extension of the GIL/no-GIL circus. But its not the primary point here.So not only the proposition (which someone referenced in another sub-thread) to move everything into python is in and of itself a self-defeating argument, since it proves a language being a problem (through forcing language lock-in). But even within a language (python), not all problems are solvable still.
Note that this is not all theoretical for me. In one of my previous projects, I had to call into a python library (which didn’t have an adequate alternative) from rust code (through pyo3). And I was forced to scale with multiple python runtimes, each getting one request/message at a time.
And we still didn’t get to any low-level details yet 😉
Ah okay, I see what you’re saying. GIL removal can break FFI code through the C API.
I have a lot of thoughts on this, and most of it is too incoherent to put into a comment someone else can understand, but for the most part, I think the issues are still pretty overblown. I just can’t see a world where Python needs true multithreading support when multiprocessing works just fine for most applications. You find a hot loop, you parallelize it in multiprocessing and just like that, your process runs much faster.
I do think that GIL removal is a terrible idea in general, for the reasons in the quote you provided, but at least it’s optional. I would personally not enable it in anything I maintain for several years if I didn’t have to.
I also think that polyglot code running in the same process is also a bad idea. I can understand calling C or Fortran from Python for numerical processing for example, but doing the reverse (like calling Python from Rust) is something I try to avoid at all costs. In any project I’ve worked on, if significant portions of the codebase were written in different languages, you would just use something like protobuf to specify the I/O protocol between the different languages and call them across a socket. It’s easier to maintain in my opinion since it defines a concrete boundary between the two different portions of code, and also makes it much easier to test each component separately too.
If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases (which is what the intellectually deficient OP tried to argue against), then I’m not sure you’re qualified to opine on the matter (hint: spawning a thread per task is already too much overhead for many workloads). And I don’t mean any disrespect to you here. Your replies were far from the worst in this thread.
Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.
And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.
That’s way too broad of a statement to be true in any absolute sense. It really depends on what you’re doing, and sure, there are plenty of ways to do it stupidly, but a lot of people that are smarter than you and me have thought about ways that the overhead of spawning processes can be mitigated. All I can say is that I’ve had perfectly good success using
multiprocessing.Pooland getting near perfect speedups. Is it as fast as threading? No. But it’s good enough. If it’s not good enough for you, then maybe we just have different use-cases, but without talking about concrete examples, we can’t make any specific claims.What do you want to talk about? Are you writing kernel drivers in Python or something? Why would we talk about writing low level code in Python, that sounds absurdly stupid. Most of the high performance libraries people use are written in Fortran or C. The entire field of data science uses Python, if performance were not a heavily optimized problem, then a lot of people out there must really love wasting their money on compute. Python is a glue language for high performance low level code, everybody knows this already.
At least we agree on something!