> Static vs dynamic typing is a spicy topic in software engineering and almost everyone has an opinion on it. I will let the reader decide when they should write types, but I think you should at least know that Python 3 supports type hints.
I think this gives readers the impression that type hints have anything to do with dynamic vs. static type systems, which isn't true. These are merely annotations that are attached to values. In fact, it's legal to use full strings as type annotations — they have no inherent semantic value.
You can gain semantic value by using a tool like MyPy or an IDE like PyCharm, but the type annotations do not in themselves do anything. I think it could be worth clarifying this for readers who are unaware.
The most underrated part of type annotations are so-called stub files [0], which allow you to keep your annotations separate from the actual code, thus reducing its size while still benefiting from the static analysis with mypy.
Unfortunately the IDE support for those files (.pyi extension) is quite limited -- I have even suggested to the PyCharm team to implement an option to automatically save all annotations to stub files, while still displaying them alongside the code in the editor.
Does PyCharm support these? Like if I write the stub files manually, will type hints and autocompletion work correctly in the main file? This could be a great way to reduce the amount of syntactic overhead, at least in function definitions and top-level code.
At the moment, you can click on a (say) class definition, and, if it has the corresponding entry in the pyi file, it will open it. But I would like it to be a bit more integrated.
Another advantage is that you can use pyi files for static checking, but remove them when deploying the application.
Sounds sort of like the .mli files in Ocaml. As I understand it, the compiler uses (and enforces) the types found therein, though can infer a lot from the code in .ml files as well.
Yeah, I'm pretty sure Python itself explicitly uses the term "annotations" everywhere instead of "type hints". The point is that they can be used for anything, typing being one such use.
As for libraries, other than MyPy, there's also Pyre by Facebook and Pytype by Google. I'm definitely excited to see more advanced static analysis being implemented beyond simple type checking. Some of these libraries are starting to explore that.
Permitting arbitrary annotations is a rather nice feature that, besides static analysis, opens the door for expressive runtime pre/postcondition checking functionality like the Clojure spec library[1]. Does Python have something similar?
Unfortunately PEP 563 [1] means that any non-type-hint usage of annotations is considered deprecated. You could still do things with decorators (like mentioned in a sibling comment), although I suppose that doesn't offer such an elegant syntax.
Oh my goodness. I've committed some naughty deeds with the ast module before, but this is truly, wonderfully, heinous. You push this idea so far that I wouldn't even recognize some of these examples as syntactically valid, let alone predict their effects. I'm awestruck and a little frightened of you.
As mentioned in the other comment, this is just a library used to give type hints as annotations. The feature itself is called annotations, this is one specific use-case.
The reason this library exists in stdlib is so that all the different type hint libraries can interoperate. Otherwise if they each brought their own typing, you couldn't switch between different static analysis libraries.
They are available in the runtime, so you can do runtime reflection on the annotations and perform dependency injection if that is your thing. In that sense they do something: they provide information to the runtime which you can use if you want.
I think this is just a semantic argument at this point, but to me what you mention isn't evidence of the annotations doing anything. They make it possible for things to be done, just as I said.
The distinction I'm drawing is to statically-typed languages where simply adding annotations will actually change semantics of the program (e.g., by forcing a downcast or something).
But you're absolutely right that annotations can be useful for many things!
I think this falls under the realm of "gain[ing] semantic value by using a tool like MyPy". The implementation of static typing by way of processing type annotations is an implementation choice, and not a part of the Python language specification.
The implementation of static typing by way of processing type annotations is an implementation choice, and not a part of the Python language specification.
That comes close to being the worst idea in the history of programming languages. Types aren't checked, you can't trust the annotations, and you give up a big optimization opportunity. But Guido's naive interpreter will still work.
If you say something is a float, the compiler needs to both enforce that and use that. Then maybe the Python crowd wouldn't need to call out to C code whenever they needed to do some number crunching.
> Then maybe the Python crowd wouldn't need to call out to C code whenever they needed to do some number crunching.
Numpy and Scipy use a lot of Fortran at the backend, and unfortunately, beating Fortran when it comes to number crunching is insanely difficult, not just because of the language, but because those libraries like LAPACK have spent decades being refined into incredibly fast and accurate systems.
Python's ability to talk to those libraries is a bonus. But Python itself could never compete directly with them. Pure-python code isn't capable of some of the insane speed those libraries can pull off.
>That comes close to being the worst idea in the history of programming languages. Types aren't checked, you can't trust the annotations
As for types not being checked, that's not a problem. JITs do extra specialization on assumed types even without annotations all the time, and just drop them at runtime when they are not valid anymore (e.g. a variable that only pointed to integers now stores a string). Still, this runtime overhead of dropping specialized code aside, those optimizations make e.g. v8 hella fast.
(And of course you could also just enable optimizations after you've statically checked the whole program types with something like mypy -- this is even easier and less dynamic than what v8 does).
>and you give up a big optimization opportunity
You don't give anything up, since an implementation can take advantage of this "big optimization opportunity" if it wants and has the manpower to add it.
>But Guido's naive interpreter will still work.
It's also less work, which unless we have a volunteer here, was and remains the intention.
Why is calling out to C code via something like numpy a bad thing?
Python is never going to be as fast as C. Ever. Even if you somehow managed to decouple numeric types from PyObject in a backwards compatible way it would not be as fast.
Type Hints can’t be used as an optimization in the interpreter due to the way they are resolved. And in any case, it would not be safe to do so and there are very few cases where optimizations make sense.
Though nothing prevents it from becoming "part of the Python language specification" -- except the backend implementation (besides Python doesn't have a specification, it's what the CPython does).
Well I think you could argue that Python is specified in part by CPython, but also in part by the PEPs. As static type checking is inherently opposed to the Python philosophy, it would take a PEP to force CPython to change its implementation accordingly.
This makes sense when you see that the variable production in the Python grammar [0] has been updated to essentially read "either a lone variable or a variable with a type annotation". (I highlighted the relevant line of the grammar for you in the link.)
F strings have been amazing in my everyday programming. I have been programming in Python for about 8 years now. I'd say some additions to this list are
Conda for managing environments and Deep Learning builds such as Cuda on Ubuntu
Jupyter notebooks for quick prototyping
IPython if you have never seen it before.
Requests Library
This is maybe not directly visible in the parent comment, and particularly not on the website, that is compatible with old versions of python also, but to me there's a huge difference in usability also in the ability to use variable names in the current scope without additional verbose crap by default.
This seems secondary to a lot of people, but to me that changes everything:
>>> x = 1
>>> f"{x}"
'1'
This is much more readable than:
>>> "{x}".format(x=x)
'1'
>>> "{}".format(x)
'1'
It makes the code immensely more readable than having to count parameters, especially for long strings with a lot of data in them.
Counting is for computers, not programmers; I shouldn't have to count anything for such a trivial task.
So, thanks to f"", no need for what I find to be an ugly additional call to .format(...), that is just visual clutter for most cases.
The only reasons I could see for a call with a dedicated dict is data exfiltration from a user provided format string executed on python code they do not control, or renaming/subsetting of keywords for cleanliness. Both are special cases, and that's the effect the current implementation has, thankfully.
I see your point, but the syntax inside of fstrings forms an entirely new sub-language (DSL). I can look at old python and know exactly how the .format() calls parse with positional and named keyword arguments, but looking at the fstrings I need to be familiar with all of the new magic.
I've heard similar complaints about the loop macro in Common Lisp, and this feels a bit like shell or Tcl strings which could mean anything (arguments to sed/awk or paths to widgets/urls). However, I guess people are familiar with regular expressions as a completely foreign nested language (DSL), so this isn't much different than that.
How is it implicit? Every insertion is expressed in exactly the place where it will print. It's way more explicit than having a placeholder and a value on completely different parts of the expression.
I did mention named parameters, you are using positional parameters to make your point. You don't even list the variables in your example. That seems pretty implicit to me.
I think this is more explicit:
"My name is {name}, I'm {feet} tall and live in {location}.".format(name="Bob", location="USA", feet=7)
I love F-strings, my only gripe is the chosen syntax for invoking them.
For starters, F may or may not be capitalized, so these two lines are identical.
F"My string with {interpolated_value}"
f'My string with {interpolated_value}'
Like using apostrophes or quotation marks to enclose strings (or F-strings), I find myself wasting time questioning the trivial notion of whether to capitalize or not.
I think Groovy's syntax is great, where quotation marks denote an interpolated string, and apostrophes denote a plain old string.
"This is my ${interpolated_value} string"
'This is my plain old string'
But these are nitpicks, and it's really too late to implement something like that.
Apache Groovy's string syntax is deficient, not great. It doesn't have the docstrings syntax that Python, Ruby, and Perl all use.
An early beta version of Groovy 1.0 had it (thanks to a Sam Pullara), but it was later yanked out. Groovy's self-styled Project Manager at the time said he only wanted syntax in Groovy that would cause the Java syntax highlight rules in Eclipse and Netbeans to highlight Groovy code similar to Java, so if a manager was walking around the programming area, the screens would look like the programmers were using Java.
And that's how Groovy got its deficient string syntax.
This just feels like another example of Python slowly creeping more esoteric syntax into the language, making it harder for beginners to approach the language.
Beginners don't have to know or touch any of this though. They can keep using normal strings until at some point they learn about f strings and see a need for them. You don't need to stop adding features to a language just because a beginner won't be able to learn everything. They don't need to.
I think the point is, a beginner coming across some code using such esoteric features is going to be a bit lost as to what's going on. With most of Python you can see what it's doing even if you didn't know the syntax before. That's true of basic f-strings, but not so much with some of this stuff.
Yeah languages are meant to be read even by beginners. I think a language like python shines by being small (in design) yet powerful (by capability). Having myriads of "nice for one use case" feature in the language is not the way forward. I long for the time where % format and .format are removed from the language spec. Even if they still work on the implementation for compatibility. Please remove things.
We're not done with Python 2 yet. This would be a Python 4 for sure. That's not according the philosophy of backward compatibility of Python (even if I agree with you).
Yes backward compatibility is not an option (lets avoid redoing python3 trauma ) and simply delegating coexistence of old and new features to implementation is a bit lazy. However documenting one way as the current one and other features as old ways, for backward compatibility can simplify things.
>>> 'User {user} has logged in and did an action {action}.'.format(**locals())
'User Jane Doe has logged in and did an action buy.'
I thought it was clever at first, but noticed it was frustrating to refactor because it was hard to see the scope of variables (especially if the template string is defined elsewhere). I don't lean heavily on IDEs, but at the time it would flag many required variables as unused because it couldn't grok this.
Do you find the same problems when using f strings?
>>> 'User {user} has logged in and did an action {action}.'.format(**locals())
D:
To me, this is pretty gross. This implicitly requires all the correct variables to be bound in the local scope, and there's no easy way to check that this is the case without reading all of the code leading up to this point (which is bad, in my opinion). Because what this really does is create string-level variables `user` and `action` and pass the values of similarly-named (i.e., shadowed) variables in the outer (but still local) scope in their place. This is why IDEs struggle with this: they can't simply know the contents of the dictionary returned by the function `locals()` without actually executing the code (unless they special-case calls to functions named `locals`, but this causes problems if the user ever shadows this name).
F-strings directly use the bound variables in the outer (but still local) scope:
>>> f'User {user} has logged in and did an action {action}.'
'User Jane Done has logged in and did an action buy.'
This avoids the shadowing problem, thus making it easy to use with IDEs and (IMO) easier to grok while reading directly, although maybe most people don't think about shadowing to the same extent I do and wouldn't be as bothered by this in general. Still, the IDE support is incredibly useful.
> Do you find the same problems when using f strings?
No, you can't define f strings elsewhere, and in theory your IDE can statically analyze them just as the Python interpreter does. In practice, your IDE might suck, I don't know.
I just want to thank the author for not using Medium.
Far too many tech blog posts use that platform now and I don't like it very at all as it feels really bloated and I can never be sure if what I'm about to click is a 'premium' Medium post or not.
Alternatively if you have uBlock and are scared of one-off extensions, you can add this generic modal filter[0] list from the webannoyances repo.
I found this by searching "modal" on filterlists.com and clicking the subscribe button. IMO there should be a Medium-specific one like they have for stackoverflow or youtube.
I cannot personally vouch if the modal filters are overzealous. I'd rather avoid Medium than putting up with this sort of thing.
I only found the "one-click" subscribe for the masterlist in the repo readme. Additionally I couldn't link a specific filterlists.com result to the modal filter "one-click" subscribe that I use.
I'd have preferred a ublock subscribe link but I'm not sure how to make it clickable without an href.
Python 2 is free software. Nobody and no proper subset of people can unilaterally end it. It ends when people stop using it, writing patches for it, and sharing those patches. There might be a lacuna of coordination when the current maintainers step down in 8 months, but the leftover stubborn Python 2 users will come up with some way to share their patches for, probably, decades.
foo = json.dumps({'a': 1, 'b': {'c': 2}}) # Pretend you've downloaded some JSON...
bar = json.loads(foo, object_hook=lambda x: types.SimpleNamespace(**x))
print(bar.a)
print(bar.b.c)
print(bar.b.d) # Error
bar.b.d = 3
print(bar.b.d) # Works now
I mean, you could pretty much do the same sort of thing with namedtuple, except this is mutable and doesn't require a pre-declaration of the keys. It's easier to write off the cuff.
Whoa, I've never seen the `types` module before! This will actually be very handy for me for what I've been working on lately. Thanks for pointing it out!
It's definitely worthwhile going over the standard library[0] every now and then. I always find something interesting to read up on. I recently learned about cmd[1] (which has been around for a long time) and have been using it to write a text adventure game.
Originally small dynamic languages like Python and Javascript are constantly growing new language/stdlib features but rarely taking anything away. I wonder how this affects learnability, how long will it take for a programming newbie to be able to jump into an existing project or read other people's code. I guess this could be measured and studied, I wonder if someone has done it.
I think Python was probably at a pretty good local sweet spot in the complexity:power ratio around the 2.0 version.
Python has taken many things away, hence the 2 to 3 transition. And people didn't like it. Not one bit. We even had to bring features back (u'', %, etc) after taking them away.
Now I agree, if I could snap my gantlet I would remove a lot more. We don't need Template(), the wave module or @static and so many other things. And we could have cleaned up Python more deeply in the transition.
But reality is very messy.
Plus, you gotta realize that the Python community has nowhere near the resources of languages such as Javascript. JS is developed by giants that poured hundreds of millions in it and set dozens of geniuses to work solely on improving the implementations.
Python barely (and this is recent, we had less before!) has a few millions for operating the PSF related activities, which includes organizing PyCons all over the world and hosting python.org or pypi.org. Only a handful of people are actually paid to work on Python.
So you have a giant, diverse community, with billions of lines of code for so many different activities, and not enough resources to work on cleaning up everything.
Welcome to life, this stuff where if you want to get things done, you have to compromise.
In practice, at a workplace, you can probably remove quite a lot through linting or some other form on enforcement. In general, as long as the majority of people out there use the same good style and don't use stuff like Template(), I don't think it'll be an issue for newbies. The real issue is when you have many methods of doing the same thing that are all popular.
Yes but I do understand the problem. E.g learning packaging in python is a terrible experience, not because it's hard, it's really not, but because of the noise we accumulated.
> not because it's hard, it's really not, but because of the noise we accumulated.
In an effort to avoid the noise, would you be so kind as to point me in the direction of the current way to handle packaging in python? I don't work in Python consistently enough to be up to date on this, but every time I do it's hard for me to suss out the best way to package up and distribute the work for internal end users.
I don't know of any resource that make a good summary of everything, that is up to date and pragmatic.
The least worst is https://packaging.python.org/ but it skip major issues I know beginners have (multiple installed Python, path problems, etc), it's not clear what works on what OS. It also promotes pyproject.toml over setup.cfg, and ignore pex and nuikta.
That's one of the numerous short pocket books I should write.
Thanks for the link! I've worked with Python long enough to be familiar with those beginner issues, just infrequent enough (and even more so for stuff that needs packaged) that I've never quite been able to wade through the noise.
Also,
> not because it's hard, it's really not
and then
> Give me a book deal, I'll do it :)
gave me chuckle. Because they're both true: it's not necessarily hard, and yet despite that it still needs a short book to really tackle the subject. That's such an out of place state for something related to a core component of Python.
2 new syntaxes a years is not a not. I find it quite balanced for a 25 years old language.
It's funny, because I always hear people complaining Python move too slowly to stay modern on one side, and then right after some other people saying it goes too fast.
My take on this is that people just like to complain.
I don't think it's the case that a language needs all of its syntactic sugar to be intuitive or obvious to the initiate. These are things you will acquire over time as you either search the internet for "how do I do X" or you read other people's code that exemplifies them.
Though I'd argue that anybody who adopts the "pythonista" label is likely to be somebody who looks through version release notes for these kinds of things. I certainly do!
I'm training people in Python regularly and my classroom experience is: ALWAYS USE __INIT__.PY EVERYWHERE. Don't use this namespace feature. You think you understand how it works, but you don't. It will bite you.
> Much as I wish python importing was as powerful as js
In what way is python importing less powerful than JS? My experience is very much the opposite (to the extent that JS even has an import system).
> I don't get why it is, but requests isn't.
Because pathlib was proposed (PEP 428), considered useful and self-contained.
The requests project does not want it to be included (this was discussed at length back in 2015), and it would require merging chardet and urllib3 into the stdlib first.
My post was unclear, it's fair that base JS doesn't have an import system; I was talking about ES6 imports.
And yes I'm aware why requests isn't in the stdlib, but that doesn't make it "correct". And chardet should definitely be in the stdlib. I mean, ffs, `mimetypes` is in the stdlib. urllib3 is more contentious but honestly? It probably should be as well.
Namespaces are useful for library providers and os packagers to split sub features of one package and distribute the chunks separately.
It will virtually merge dirs with the same name under some circonstances, and won't be importable in some others. You also lose the benefit of having an init file in which you explicitly setup the object you expose for import.
It also won't let you cache on mutable values, so you can't pass in lists or dicts. I've used it on occasion, a lot of the time I ended up rolling my own instead. Usually without expiration since I didn't need it.
The "Implicit namespace packages" item is not correct. Those are for a a specific packaging scenario which doesn't come up in normal usage. Packages should have an __init__.py file in general.
If the file is going to be empty I fail to justify the effort. It is implicitly a package. Is there a case where an empty init file makes sense in Python 3?
The example of f-string in the article is quite unfortunate, it suggests using it for logging messages.
F-strings shouldn't be used in loggers, because they aren't lazy evaluated.
TBF for the vast majority of logging it doesn't really matter. Not to mention the laziness is only the (usually cheap) formatting itself. If you're logging the result of non-trivial expressions (which would otherwise not be computed at all) you have to handroll it.
f-strings are mostly unusable for translatable contexts. And format-strings also don't work then (they're a security issue).
I already experienced performance issue, just because log messages (millions/s) were evaluated, for logger in debug mode, so it really depends on the system.
Wow, today I learned about f strings. That will make my code so much more readable! And no more annoying bugs because I skipped one of the seven references in the string!
Glad to hear you like it. As a fun fact (and self-back patting), it was I who braved the gauntlet at the python-ideas mailing list to get the idea accepted. Eric Smith did everything else so deserves the credit.
Background: turns out that the string prefix was the only ASCII syntax left in Python3, that's why it was chosen. The name "eff string" makes me cringe, but once it stuck there was no stopping it.
> it was I who braved the gauntlet at the python-ideas mailing list to get the idea accepted
That explains why you're all over this post ;)
Fantastic feature, though. Really. Thanks!
Out of curiosity, what name would you have preferred over "f-strings"? "Format strings"? "Interpolation strings"? "I-will-evaluate-expressions-in-curly-braces strings"?
Well, I usually participate in Python threads. This post had only one part on f-strings.
What to call it? Anything I guess. String interpolation is what it is called elsewhere. But the f prefix is what everyone sees, though as mentioned, it is only an implementation detail because other forms of syntax were already spoken for. I personally chose f for "format."
Early on Guido changed the scope to include expressions as well. People were already starting to say "eff string" so I changed my proposal to "e-string" for "expression." I also like the sound of it better, sounds like email, etc. But Guido (and Eric) decided to stick with f. Maybe because Guido learned English later he doesn't realize how unfortunate it sounds. But, years later we use the "iPad" and forget it sounds like a feminine napkin. So, no big deal in the end.
Don't forget @singledispatch! Every once and a while this tool is glorious for a polymorphic handler. Ie, `process_entity` that will fan-out to `process_movie` or `process_tv_show` etc...
I miss it in a Python 2.7 project I am on right now.
It took place three times actually; Ping proposed string interpolation in 2000 in PEP 215 https://www.python.org/dev/peps/pep-0215/ and it was rejected, or rather simplified by Barry Warsaw in 2002 in PEP 292, which was actually implemented: https://www.python.org/dev/peps/pep-0292/ and remains in modern Python. Barry's optimism that this would satisfy the demand for template strings was of course foolish, but it met with Guido's misguided approval at the time. Fifteen years later, it was proposed a third time as PEP 498 and finally got in in a sane form: https://www.python.org/dev/peps/pep-0498/
Exactly, the f-string syntax looks so close to PHP’s way of handling strings that I’m really amazed that it managed to pass and be implemented. I mean, I don’t see that much of a difference between
PHP has many problems, but I always considered string handling one of PHP's strengths. Considering how PHP's string interpolation has gone mainstream I don't seem to be the only one:
That's an interesting question. I couldn't find any summary of the history of string interpolation. The first version of PHP in '94 had it. Bash predates that by 5 years and Perl by another 2, but I'm not sure about the initial feature set of either of those. The first POSIX standard for command interpreters is from 1992. I suspect the feature comes from early versions of csh in the early 80s.
However I think it's fair to credit PHP with introducing generations of programers to the idea.
String interpolation was there in the first versions of the Bourne shell, and I suspect the pre-Bourne shell in PDP-7 Unix in 1970 or so, but of course programmers have been generating strings by pasting together bits of constant and variable strings for a lot longer than that; Church defined the λ-calculus in terms of string interpolation in the 1930s, before there were even any computers. Computer-generated printed letters were already a commonplace occurrence in the US in the 1950s (at least if references in the fiction of the era are to be believed), and those are of course built with string interpolation. Around that time, assembly languages became the first languages to claim to make computer programming obsolete (because at the time "programming" meant programming in binary), and rather quickly some of them developed macro capabilities — before 1960, I think. Macro assemblers typically do their macro magic with string interpolation, even today. High-level languages of the time commonly used template strings, like those in (1964?) BASIC's PRINT USING statement, though FORTRAN's "edit descriptors" used a non-interpolation-based syntax. COBOL's "edited picture" feature is a kind of string interpolation, but I have no idea when it was added to COBOL.
In the 1970s string interpolation really got going with languages like sh, cpp, and m4, which was popularized on many platforms by the "Software Tools" book (which in a sense is a slow buildup to the presentation of m4); m4 is Turing-complete entirely through string interpolation (more so even than Tcl decades later).
I suspect Perl had string interpolation from the beginning (1988), given its shell roots, but I don't have a copy of Perl 1 to test with.
Bourne shells as early as Version 7 Unix (1979, [1]) had rather elaborate interpolation of variables into strings (i.e., not just ${foo} but ${foo?bar}, etc.).
I'm reasonably sure that earlier versions of sh had some facility for this as well.
So, it's not really a PHP innovation, although some people might have seen it first there.
Apparently Unix Version 6 (1975) used Thompson Shell which (provided the manuals are correct [1][2]) didn't support arbitrary variables (and even environment variables were not a thing back then). However for use in scripts $0, $1, etc were replaces with the script name and the arguments.
About 15 years ago when I was starting my life as a professional programmer PHP's way of handling strings was seen as inferior to stuff like:
> name = 'Jane'
> print 'Hello, %s' % (name)
but I can't exactly reproduce all the discussions from back then. As for me personally I can see that I dislike the new
> print(f'Hello {name}')
format because my editor cannot instantly show me that {name} is in a fact a variable, and now that I wrote this down I remember that this was one of the complaints made against PHP's use of "Hello, $name" . But maybe newer editors do in fact recognize {name} as a Python variable, in any case it looks and feels "impure", for a lack of a better word.
VScode and PyCharm recognise variables in f-strings correctly. I'm not sure how "%s" is more pure than "{name}". They are both variables, only in the case of "%s" you add a level of indirection.
f-strings are not the same way to do things but a different way. You can't explode a dictionary into an f-string to fill in the fields, for example, but you can in .format().
Another useful feature of 3.7+ (also available as a backport by the same name) I’ve not seen mentioned is: ContextVars
The gist is you get a single interface for any type of threadlocal storage that also works with coroutines (asyncio). I use it pretty consistently for context logging of long running processes in our systems. However it has implications on other concurrency paradigms usages as well.
> Data classes sounds good but there should be only one way to do things
Dataclasses greatly simplify generation of simple classes which are unnecessarily filled with boilerplate. I'd argue that they are the one way to do this, and that choosing to manually implement the same naive `__init__` function for every simple class you write is the "wrong" way.
(Here I take "simple class" to mean a class which takes in some values during initialization and stores them without anything terribly interesting going on during that initialization. A simple class can have methods defined on it, though.)
> type-hints are ok for very very big projects.
D:
> No comment on implicit-namespace-packages yet as still trying to understand solid use-case.
I think it's just... trying to keep your directories "clean"? I also don't really understand the use of this haha.
> Dataclasses greatly simplify generation of simple classes which are unnecessarily filled with boilerplate
I would love to solve these boilerplate kinds of problems using plugging like emmets. Magic like this feels good when writing but troubles during debugging and customizing.
Additionally naive programmers always use such features in wrong situation if they don't get use-case. I really feel sad after seeing so much bad-django-code in production.
> 4. type-hints are ok for very very big projects.
Great for smaller ones too! I type annotate every function, even
def parse_args() -> argparse.Namespace:
pass
at the top of simple scripts. With a good IDE (VS Code fits here too), the extra language lookups/insight you get are awesome. Command+mouse_hover gives me a great bit of information about args passed into the function if I annotate the inputs.
I tried to use type hints but I do not see enough value in them to justify making code more complex. It looks to me that they are similar to hungarian notation as they make refactoring harder but they are not reliable and does provide little value in checking program correctness. They are just documentation embedded into variable declaration used only for linting so I tend to not trust them.
Because it is not used by runtime. Mypy is just linter which adds additional restrictions on code which are not enforced by interpreter. You can have false positives so correct python code must be fixed to pass linter checks. In my opinion if you need static type checks you should use statically typed language and not hurt your coding speed by such partial solution. Also majority of python libraries do not use type annotations so you are limited to your code.
Well, yes, this is exactly how the type checkers of
statically-typed languages work: as linters, during compilation.
The largest difference is that typing in Python is gradual, so not
everything needs to be typed as you say.
I agree that statically typed languages are preferable. I don't consider static
typing optional so I'm glad Python is growing a solution that supports it,
though.
In my experience, mypy works surprisingly well. It doesn't hurt my
coding speed at all but enhances it to levels that were not
previously possible in Python, due to a lack of typing. False
positives are rather rare.
importlib.resources is an excellent replacement for pkg_resources.
with importlib.resources.open_binary('my.package', 'foo.png') as f:
...
Avoids the overhead of importing pkg_resources, and hassle of having to call cleanup_resources before the interpreter shuts down. Backports are available for Python <= 3.6 too!
They're basic but also things you'd never look up, especially if you've been doing python a long time.
For example, I just learned about f strings, because why would I need to ever look up if there is a replacement for .format()?
And the LRU cache. I've been hand rolling that for years, but I never thought to see if they had added it to the standard library, because why would I look?
On a related note, every couple of years I make an effort to man page every cli tool I use, and re-read the man page, just in case. Most times I discover there is some argument that I've missed before in a tool (or has been added in the intervening time)
EDIT: nope, nope, this is all wrong. My mistake. Ignore everything else here.
> classes don't need (object) anymore
My friend, old-style classes became unnecessary with the release of Python 2.2 — almost 20 years ago now [0]. I'm afraid you've been living in the past. :)
Extending `object` was necessary until 2.7 when it became the default and old style classes were removed entirely in 3. So I think while the GP comment recently learned they don’t need to extend `object` anymore, they are/were familiar with new style vs old style classes.
A funky thing about the Enum class is you may also want the `@enum.unique` decorator in most cases, which enforces the enum/value pair being unique. I found the documentation a little misleading until I found that [0].
I'm not sure I follow the purpose of this, and the documentation is not terribly elucidating haha.
What's a case where I'll have a problem if I don't use this decorator? (I've used the Enum class a bit since it became available, but not in any particularly interesting implementations so maybe that's why I've never heard of this.)
``@enum.unique`` will raise a ValueError if you accidentally include the same value twice in an enum definition:
In [21]: @enum.unique
...: class Animal(enum.Enum):
...: DOG = 1
...: CAT = 2
...: PIG = 2
...:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-21-cbc1625bb41c> in <module>
1 @enum.unique
----> 2 class Animal(enum.Enum):
3 DOG = 1
4 CAT = 2
5 PIG = 2
~/.virtualenvs/py3/lib/python3.6/enum.py in unique(enumeration)
834 ["%s -> %s" % (alias, name) for (alias, name) in duplicates])
835 raise ValueError('duplicate values found in %r: %s' %
--> 836 (enumeration, alias_details))
837 return enumeration
838
ValueError: duplicate values found in <enum 'Animal'>: PIG -> CAT
It's intended to catch programmer errors - not really useful for small enums like this one, but in a large enum, having duplicates could be difficult to notice, and might cause some insidious bugs.
So this is mitigated by using `enum.auto` for setting the values. But I can definitely see how `enum.unique` would be useful if your enums had more specific values.
> What's a case where I'll have a problem if I don't use this decorator?
If you are explicitly setting enum values, it causes accidental aliases (duplicate values) to be an error early rather than a source of (potentially subtle) problems later.
It also specifically communicates intent to other readers of the code, which can be just as important if your code has a lot of enums,some of which do use aliasing intentionally.
I was curious to see how asyncio has matured into such a list.. but it was missing the list. Is it common enough for people to use in scripts to speed up concurrent work yet?
From where I sit (I've been soloing python projects for a few months now) For IO-blocking tasks, asyncio is the way to go and threading is very rarely needed if any. It is very simple to place a bunch of tasks into a list and do asyncio.gather on them, or add timeouts to async methods.
For CPU intensive tasks, asyncio actually provides it's own syntax for process pools which are easily interchangeable with thread pools (just change an import alias)
We actually tried switching to asyncio in production, but got worse cpu utilisation and worse latency (vs greenlet). After some benchmarking and tuning, we got a bit better latency, but CPU utilisation was still worse, so we actually ended up swithcing back to just using greenlet :(
Is type hinting being adopted by Pythonistas today? I've been aware of the feature but haven't used it. To be a modern Python programmer, should I start using it?
I use it everywhere because it vastly improves code readability/usability in my IDE (PyCharm). But I don't know if I'm representative of the general Python population.
Not sure anyone will see this but this article is partly trash. When your post is titled "things you are not using in python3 but probably should" and the first section is on f-strings... I was tempted to close the tab then , but i followed thru and did find some interesting things.
But seriously f-strings? people have been talking about these for years now, it's not new is it?
I really hate that __init__.py thing about modules it confuses the F out of teachers and students. Using files without a "program" on the command line is a massive leap as it is.
Are you saying that you exited the C++ scene because dealing with types is tedious? For me, C++ career opportunities were extremely rare to come by and I never had the luxury to change jobs because I didn't like the language in some way.
There's no reason why Python 2 can't have the vast majority of these features. It's a shame that the Python continues to ignore the reality of the mess they made with Python 3, and failing to question what was an originally bad idea.
> There's literally no point in discussing it any more. CPython 2.7 support ends in 7 months. It's time to move on.
Ah, the classic "nothing to see here, keep moving on" argument. The lack of admission of a bad idea is itself a problem. What happens in a few years when Python 4 renders all Python 3 code incompatible?
> What happens in a few years when Python 4 renders all Python 3 code incompatible?
Ah, the classic "they did it once so they'll definitely do it every single time from now on" argument.
Nick Coghlan, one of the Python core developers, said "Python 4.0 will merely be "the release that comes after Python 3.9"" [0], so I think your concerns are ill-based.
> Not to mention some of us are glad lots of things got fixed and not sore in the slightest.
The people who are not "sore in the slightest" probably don't have to deal with large Python 2 codebases that rely on libraries that have no plan to move to Python 3. They are typically people who are writing code from scratch.
Rendering a huge codebase obsolete in order to upgrade print statements, import statements, and internationalization (i.e., unicode) was not a fair trade for many existing projects.
My stuff was not overly concerned with encodings, that part was luck. I moved to the logging module early, avoiding print problems, that was smart. Other changes were trivial mechanical fixes, that was easy.
For the unlucky, obtuse, or resource challenged, you've had an extra ten years of support. That's sufficient IMHO. It is time to suck it up and port, or retire the app. Constant bitching just gets old.
Meanwhile, I'm using a great language that will be relevant for years.
> one of the Python core developers, said "Python 4.0 will merely be "the release that comes after Python 3.9"" [0], so I think your concerns are ill-based.
Why would they say that? They know breaking compatibility was a huge blunder, but they ignore the folks that it affected. Python 3 promoters are typically folks that don't have to deal with mess it created. If the Python foundation actually admitted to their mistake, it would engender far more confidence and credibility than pretending it does not exist.
No, there's no reason Python 2 can't have these features, but there's no particularly compelling reason to invest the engineering effort to develop it.
The move to Unicode is not "incremental" by, like, any measure. And other things, like moving `print` to be a function instead of a statement, were absolutely good design choices and worth breaking for.
People have had plenty of time to work on porting their code from 2.x to 3.x. Honestly, I think it's remarkable the devs kept maintaining 2.x as long as they did. I mean, look at Swift, where they had breaking changes year after year, and yet their user base has fairly exploded since the language's release not so long ago.
Just... get on with it, you know? Move to Python 3.x and be done with it. There's no particularly good reason to stick to 2.x forever.
I'm not particularly excited about a lot of other new features though.
I don't see a real reason for type annotations from my perspective (if I wanted typed I'd use a typed language, if I wanted types indicated I'd indicate in the doc strings).
Also f-strings which I haven't used it. They probably are better, but I'm barely getting used to `format` as opposed to `%s` (which admittedly `%s` does suck as I've said many time while counting items on the screen with a pencil).
> I don't see a real reason for type annotations from my perspective (if I wanted typed I'd use a typed language, if I wanted types indicated I'd indicate in the doc strings).
I definitely get this perspective, but I actually use type annotations a lot. I do the vast majority of my Python development in PyCharm, and using type annotations greatly facilitates auto-complete features. It also helps me do some very simple debugging while writing, as the IDE will tell me "Hey! I don't think these types match up!", which is often enough to save me the headache of debugging.
Realistically, I just want Python but with a static type system. Which is why I'm implementing that as my own project haha.
> Also f-strings which I haven't used it.
Oh, dude, you gotta get on board with f-strings. They're soooo much simpler than the alternatives. Just skip learning `.format` and go straight to f-strings. I moved to f-strings when they first became available a few years ago and haven't looked back since.
Dataclasses are also great, IMO. I prefer algebraic data types in functional languages like Haskell or OCaml, but I think Python's dataclasses are a "good enough" solution for me to use in the meantime. I often define very simple data types and hate the default `__repr__` implementations and writing naive `__init__` functions. Dataclasses make all this much faster and, I think, more readable.
There's no question that Python 3 is an improvement, but the question is whether it's an improvement worth breaking every existing Python app. Especially when many of Python 3's features could easily have been backported to Python 2. Python 3 is effectively a new language. People will be running Python 2 for many more years to come.
Ah, yes, I just meant they shouldn't focus on using `.format` and should instead just use f-strings directly. Didn't mean to imply that they were totally separate things, though that's definitely how my phrasing came across.
Yep, the other way is easier (if, and only if it's short), it has simpler rules, less typing, and that's how I learned it and it's in muscle memory without having to stop and think or look at examples.
> The move to Unicode is not "incremental" by, like, any measure.
The unicode improvements in Python 3 are definitely a step in the right direction. But there were certainly good ways of improving unicode issues without breaking compatibility. Things like print are a joke, no one should care about its semantics.
> Just... get on with it, you know? Move to Python 3.x and be done with it.
Pretty easy to say. But try convincing product managers to take 5 devs off their current projects for 6 months to port a working production app to a newer programming language, with little concrete evidence that it will have a meaningful impact on the project (other than introducing bugs).
Guido had a good retrospective on the shift, while the change was hard and painful, it needed to happen for the language to be easier to work with longer term.
> Guido had a good retrospective on the shift, while the change was hard and painful, it needed to happen for the language to be easier to work with longer term.
He was wrong. It is far harder than he anticipated, and it did not "need" to happen to grow the language.
Can you expand on why/where Guido was wrong beyond why he cites where he (and the core team) made mistakes in the linked talk? I'd disagree about the "need", as the fundamental changes (meaning non-package renaming changes) were pretty important for the increasingly international economy we are in.
There is a reason to get rid of bloat. It makes for a more orthogonal language. You don't have randomly sprinkled syntax for printing characters, raising exceptions, catching exceptions, handling strings, implicit type conversions when comparing values, etc which were bad ideas and good riddance.
> Static vs dynamic typing is a spicy topic in software engineering and almost everyone has an opinion on it. I will let the reader decide when they should write types, but I think you should at least know that Python 3 supports type hints.
I think this gives readers the impression that type hints have anything to do with dynamic vs. static type systems, which isn't true. These are merely annotations that are attached to values. In fact, it's legal to use full strings as type annotations — they have no inherent semantic value.
You can gain semantic value by using a tool like MyPy or an IDE like PyCharm, but the type annotations do not in themselves do anything. I think it could be worth clarifying this for readers who are unaware.