Ah okay, I see what you’re saying. GIL removal can break FFI code through the C API.
I have a lot of thoughts on this, and most of it is too incoherent to put into a comment someone else can understand, but for the most part, I think the issues are still pretty overblown. I just can’t see a world where Python needs true multithreading support when multiprocessing works just fine for most applications. You find a hot loop, you parallelize it in multiprocessing and just like that, your process runs much faster.
I do think that GIL removal is a terrible idea in general, for the reasons in the quote you provided, but at least it’s optional. I would personally not enable it in anything I maintain for several years if I didn’t have to.
I also think that polyglot code running in the same process is also a bad idea. I can understand calling C or Fortran from Python for numerical processing for example, but doing the reverse (like calling Python from Rust) is something I try to avoid at all costs. In any project I’ve worked on, if significant portions of the codebase were written in different languages, you would just use something like protobuf to specify the I/O protocol between the different languages and call them across a socket. It’s easier to maintain in my opinion since it defines a concrete boundary between the two different portions of code, and also makes it much easier to test each component separately too.
If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases (which is what the intellectually deficient OP tried to argue against), then I’m not sure you’re qualified to opine on the matter (hint: spawning a thread per task is already too much overhead for many workloads). And I don’t mean any disrespect to you here. Your replies were far from the worst in this thread.
Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.
And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.
If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases
That’s way too broad of a statement to be true in any absolute sense. It really depends on what you’re doing, and sure, there are plenty of ways to do it stupidly, but a lot of people that are smarter than you and me have thought about ways that the overhead of spawning processes can be mitigated. All I can say is that I’ve had perfectly good success using multiprocessing.Pool and getting near perfect speedups. Is it as fast as threading? No. But it’s good enough. If it’s not good enough for you, then maybe we just have different use-cases, but without talking about concrete examples, we can’t make any specific claims.
And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.
What do you want to talk about? Are you writing kernel drivers in Python or something? Why would we talk about writing low level code in Python, that sounds absurdly stupid. Most of the high performance libraries people use are written in Fortran or C. The entire field of data science uses Python, if performance were not a heavily optimized problem, then a lot of people out there must really love wasting their money on compute. Python is a glue language for high performance low level code, everybody knows this already.
Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.
Ah okay, I see what you’re saying. GIL removal can break FFI code through the C API.
I have a lot of thoughts on this, and most of it is too incoherent to put into a comment someone else can understand, but for the most part, I think the issues are still pretty overblown. I just can’t see a world where Python needs true multithreading support when multiprocessing works just fine for most applications. You find a hot loop, you parallelize it in multiprocessing and just like that, your process runs much faster.
I do think that GIL removal is a terrible idea in general, for the reasons in the quote you provided, but at least it’s optional. I would personally not enable it in anything I maintain for several years if I didn’t have to.
I also think that polyglot code running in the same process is also a bad idea. I can understand calling C or Fortran from Python for numerical processing for example, but doing the reverse (like calling Python from Rust) is something I try to avoid at all costs. In any project I’ve worked on, if significant portions of the codebase were written in different languages, you would just use something like protobuf to specify the I/O protocol between the different languages and call them across a socket. It’s easier to maintain in my opinion since it defines a concrete boundary between the two different portions of code, and also makes it much easier to test each component separately too.
If you didn’t automatically register that requiring spawning full OS processes to scale performance immediately disqualifies a language from a whole plethora of use-cases (which is what the intellectually deficient OP tried to argue against), then I’m not sure you’re qualified to opine on the matter (hint: spawning a thread per task is already too much overhead for many workloads). And I don’t mean any disrespect to you here. Your replies were far from the worst in this thread.
Frankly this thread has been very illuminating. My perception of the average p.d user and their concrete knowledge was very inflated.
And we didn’t even get to low level stuff yet. People are already failing at the macro level. Sad.
That’s way too broad of a statement to be true in any absolute sense. It really depends on what you’re doing, and sure, there are plenty of ways to do it stupidly, but a lot of people that are smarter than you and me have thought about ways that the overhead of spawning processes can be mitigated. All I can say is that I’ve had perfectly good success using
multiprocessing.Pooland getting near perfect speedups. Is it as fast as threading? No. But it’s good enough. If it’s not good enough for you, then maybe we just have different use-cases, but without talking about concrete examples, we can’t make any specific claims.What do you want to talk about? Are you writing kernel drivers in Python or something? Why would we talk about writing low level code in Python, that sounds absurdly stupid. Most of the high performance libraries people use are written in Fortran or C. The entire field of data science uses Python, if performance were not a heavily optimized problem, then a lot of people out there must really love wasting their money on compute. Python is a glue language for high performance low level code, everybody knows this already.
At least we agree on something!