A nice comment in a forum thread (extracted below, but also see the shorter more facetious version below that) about references and their lifetimes in structs. Here is a link to the full thread over on users.rust-lang.org
I feel like I needed to hear or read this, and I feel like this sort of clarification is not put front and centre enough in rust learning material (as others in the thread say too). “The Book” certainly doesn’t seem interested in clarifying perspectives like this.
The Comment
Other languages use term “reference” for storing things “by reference” or just referencing any object anywhere in general. It’s not like that in Rust.
What Rust calls “reference” is a much more specific thing, that is no so general-purpose. It has a narrower, restrictive usage. Rust references are more like read-only or write-exclusive locks. They make their target unmovable and immutable for entire duration of their existence. They can’t exist on their own, only as a counterpart of an owned value.
References in structs also make the whole struct itself temporary, and everything that touches that struct becomes temporary and tied to the scope of the borrowed value that started it.
If these restrictions (that cause you fight with the borrow checker) aren’t what you want to achieve, then you don’t want temporary references.
99% of the time when you need to store something “by reference”, Box (or Arc or String or PathBuf or Vec or some other owned type) is the right answer.
Note that &T and Box<T> have identical representation in memory — a pointer. They differ by ownership.
In Short
From here
You’re not allowed to use references in structs until you think Rust is easy. They’re the evil-hardmode of Rust that will ruin your day.
😉
Use Box or Arc to store things in structs “by reference”. Temporary borrows don’t do what you think they do.
Hmmm. On the one hand I heavily agree with the analogy to read only and write exclusive locks (I think I gave it myself in the ownership chapter recap/discussion thread). On the other, I feel like this is singling out “references” for something that is much broader in cause.
I could just as easily say that references in rust are “just” references, and that it’s the rules around ownership (like the compiler automatically dropping things when changing scope) that restrict what you can do with them, in practice.
I dunno, given rust’s history and explicit purpose of preventing memory errors, notably in multi threaded code, if you still assume you can just take a reference to “any object in memory” and do whatever you want with it then you either haven’t paid enough attention to how the book introduces references or maybe rust isn’t cut out for your way of thinking.
There are ways to do raw pointers in rust if you need to. But then you need to more or less do all of the compiler’s job yourself, notably figuring out when you can safely free that memory.
In other words, references don’t exist on their own, and so how the author describes them “in other languages” feels to me like they never had to implement a language themselves and safely deal with shared references, nor use them that thoroughly in a language with “general” references like C. Otherwise I don’t think they would have developed this mental model that expects references to be considered their own thing, almost separate from the underlying data.
I can’t tell if the author is inexperienced with languages that “do a lot” and have a more cohesive design, or is so much more knowledgeable and experienced than I that they’ve already taken this into consideration and are arriving at this point regardless.
I got the feeling from the thread that the author’s perspective and framing is coming from having to help many people with their problems and likely misuses and misunderstandings of rust references, where your point of not having “paid enough attention to how the book introduces references” may be applicable to many of the people they’ve helped.
For me, while there isn’t anything conceptually new in their framing, I found the emphasis and perspective helpful and affirming, in part because I hadn’t got to the point of verbalising the “locks” metaphor as you say you probably did but instead was still thinking of it all as a set of constraints to work within and problems to avoid. Framing the situation in a more “active”/“positive” (as in posit) way where the emphasis is on what references “do” rather than just on what they “prevent” seems helpful to me, and that’s the part that I feel like is missing from “The Book” or anything else I’ve read. It also feels like a better way of explaining why lifetime annotations become necessary (generally wondering which being how I ended up reading those threads)
Ahh, that makes a ton more sense to me! Thanks.
I like your verbiage regarding “things you do” vs “an [ambient] set of constraints”, and I would go one further (or in any case my mental model is): there are a set of valid states or configurations for references that is smaller than (and contained within) the set of possible or imaginable configurations.
By configuration, here, I kinda literally mean “the state of the program/machine executing the program, as that execution plays out”. So, “3 read-only borrows of some piece of data” is a valid configuration, while “2 simultaneous mutable borrows” is an invalid state for the program to be in. It’s sort of the same conceptual approach as types: you express the valid/allowed configurations that are (almost always) a subset of the overall possible states. Except in this case, it’s the language that has defined it once, ahead of time, rather than us programmers who define things as we go. Maybe that just gets us back to the same point that wasn’t originally clicking for you, though.
Sidenote: I wonder if a language could be developed that allowed the programmer to define or setup their own set of constraints around lifetimes and references. Sort of like what Zig seems to allow for memory allocator/allocation strategies, but specifically for ownership semantics. As I type this out, I can’t shake the feeling that this is basically what Rust has already done with their
&vs&mutvsBoxvsRCvsRefCelletc. It’s not like there is a way to safely have 2 mutable references to the same memory/variable that is both general and simpler than existing mechanisms.Re: lifetime notations, it’s good to know that this helped you. For me it was the combination of encountering the particular part in chap 4 of The Book where they examine the case of a function that accepts a tuple and returns a ref to the first element (or something along those lines), and trying to define (and then use) a struct that would store a ref to some other data in a hobby project.
Overall, this makes me realize how much of Rust I have already internalized!
Sidenote: I wonder if a language could be developed that allowed the programmer to define or setup their own set of constraints around lifetimes and references. Sort of like what Zig seems to allow for memory allocator/allocation strategies, but specifically for ownership semantics. As I type this out, I can’t shake the feeling that this is basically what Rust has already done with their & vs &mut vs Box vs RC vs RefCell etc.
Yep to all of your post … and this is my impression too (without having gotten on top of the smart pointers yet).
Only thing I can imagine adding though would be an optional garbage collector (that is shipped in the binary like with Go). I’m not sure how helpful one would be beyond RC/RefCell (I’d imagine a good GC could handle a complex network of references better, which wouldn’t be very “rust”-like, but expanding the scope of the language toward “just getting stuff done” territory would be the main point of a GC). A quick search turned up this project which seems to be very small but still active (interestingly, the blog post linked there points out that rust used to have a built in GC but removed it to be a 3rd party library instead).
Re: lifetime notations, it’s good to know that this helped you. For me it was the combination of encountering the particular part in chap 4 of The Book where they examine the case of a function that accepts a tuple and returns a ref to the first element (or something along those lines), and trying to define (and then use) a struct that would store a ref to some other data in a hobby project.
Yea for me the structure and framing of The Book when lifetimes were first brought up (in Ch 4) didn’t work for me as it came off to me as another series of problems to solve or constraints to navigate.
What I find interesting and compelling about the framing in my top post is that it conceptualises references and borrowing by starting with lifetimes. I’m not as experienced as you and haven’t internalised rust like you have (awesome to read though!), but I think I would have found it better to go from the basic idea of pointers as a way of not taking ownership and then going straight into “everything has a lifetime with changing permissions/locks over that time depending on what other references exist and while rust infers these lifetime durations sometimes, sufficient complexity requires you to specify them yourself” and then building out from that basis. For instance, I’m not even sure The Book makes it clear that Rust is inferring variable lifetimes automatically (??).
I was reading this last evening and ultimately decide against pinging you with it at the time, lol.
It directly riffs off of, and complements, the article you posted. It also speaks to your remarks on an optional garbage collector, as well as “how to think about the lifetime/borrow checker/permissions interplay”.
but I think I would have found it better to go from the basic idea of pointers as a way of not taking ownership
small quibble:
String,Box, andVecare all technically pointers that do take ownership (rather, they have ownership of what they’re pointing to). It’s really only “references” in Rust that don’t take ownership. Which, IIRC, is more or less how The Book introduces references in chap 4. So I’m not really sure how what you’re describing would differ from the current state of the book. Nonetheless, I understand the confusion that comes from “learning about” lifetimes so “late”.I suspect Rust is so complex, and like a living organism its parts have co-evolved together so much, that there is no linear explanation or introduction that can “truly” do it justice. You sort of have to throw yourself in, and repeat that process from a handful of directions/trajectories, to start to get a cohesive idea of the whole. I say “co-evolved” notably because so much of “modern” Rust seems to me to be syntactic sugar over “core/original/2015” rust.
I haven’t gotten to the chapters on explicit lifetime annotation this time around, but I expect The Book to clearly state, then, that everything has a lifetime and the compiler actually infers most of them for you as part of this “syntactic sugar”.
Thanks for the link! Will try to read it.
small quibble: String, Box, and Vec are all technically pointers that do take ownership (rather, they have ownership of what they’re pointing to). It’s really only “references” in Rust that don’t take ownership. Which, IIRC, is more or less how The Book introduces references in chap 4. So I’m not really sure how what you’re describing would differ from the current state of the book. Nonetheless, I understand the confusion that comes from “learning about” lifetimes so “late”.
Yea, I was going to specify (lol) that my first time through I really missed or didn’t focus on what
BoxandVecwere as possibly alternative tools to references or at least interesting objects in terms of combining ownership and pointers to data on the heap … and how I’m not sure that’s really on me given the order of ideas in The Book and the emphasis on references. Again, for me right now, it seems that “lifetimes” is the connecting concept in that it clarifies what’s going on and what problems are being solved/created. For me, instead of the section in Ch 4, it was an early section in the Rustonomicon (that I was just casually browsing through … I know nothing about unsafe rust) that leaned in hard on the centrality of lifetimes.Nonetheless, I’m a little keen now to get a grip on how a
Box(or other owning + pointing type) can be used as an easier/cleaner substitute for references. I don’t have a clear image in my mind and I think I just need some application to work with or example to read through (maybe in the blog post you linked).Thanks again for the link!
I finally clicked over to the thread, and you’re totally correct; the author is clearly talking about their collective encounters with newcomers rather than their own rust “journey”.
This makes me want to share a growing hunch of mine: rust is, currently, made by and for someone like me, who’s had the privilege of developing a (rudimentary) working knowledge of the fundamentals involved in how today’s computers work, from the physics involved all the way up to something like OCaml or python, before ever hearing about Rust itself.
More specifically, there’s so much that The Book leaves assumed-yet-unsaid, even when it really tries to provide background and rationale for concepts.
I don’t think it’s reasonable to expect [The Book’s writing team] to properly teach everything that it later relies on, and it already does say that it expects a certain level of programming experience from the reader near its very beginning. I think it could use a more explicit and concrete list of “requirements”, though, like a dependency tree of prior concepts to acquire. Maybe that could help it strike a better balance between over- and under-explaining things.
Generally agreed. Awesome to hear that you’re gelling with the language! If I may ask, what particular experience or background knowledge do you think makes you and rust such a good fit? Knowing OCaml (I’ve certainly heard of OCaml as an adjacent language to rust in terms of concepts and interests)?
I think it could use a more explicit and concrete list of “requirements”, though, like a dependency tree of prior concepts to acquire. Maybe that could help it strike a better balance between over- and under-explaining things.
Yea, agreed. In general my policy is that a text book or monographs has the task of mapping as many prior backgrounds to new understandings as possible. This often requires constructing non-linear content and guides to navigating it but can often be worth while IMO.
In relation to rust, it seems to me that it’s the sort of thing that benefits from openly engaging in directionless “horizontal” learning in order to build up a necessary foundation for then building “vertically” once enough pieces are in place. At least more so than more basic languages.
A “steep” learning curve, IMO, is often confused for this sort of thing, where in reality you’re not learning difficult or “steep” things sequentially, but meandering confusedly around a space non-linearly until things finally “click” (ie, until the foundation is sufficiently formed). The thing is that openly embracing a structure that isn’t always building “upward” can make the “journey” feel much less steep than it is.
I used to run a workshop on git for non-CS academic researchers and I opened with a warning that it was going to be structured like climbing a spiral, starting with the problems it solves, then basic ideas, then how those ideas can solve the problems, then a demonstration of each followed by painting a complete picture of how git would fit into one’s workflows. And I found it worked well. People often found it boring at first, but really appreciated it once things started clicking and had fewer confusions I think than if things were done differently.
f I may ask, what particular experience or background knowledge do you think makes you and rust such a good fit? Knowing OCaml (I’ve certainly heard of OCaml as an adjacent language to rust in terms of concepts and interests)?
Overall it’s from having gone through [most of] a Computer Science and Applied Math engineering course. Some of the courses:
- functional programming with OCaml
- proving correctness in simple programs
- Computer Architecture 101 that walked us from building simple logic gates out of transistors to simulating a 32-bit processor on an FPGA
- implementing some common collections, like Rust’s
Vecor a linked list, in C - parsing and regular grammars
- intro to compilers
This gave me an appreciation for many of the things that Rust forces you to confront head-on like stack vs heap and why stack frames are created & dropped, copy vs clone vs move, types sometimes needing to have sizes known at compile-time, and of course all the use-after-free, double-free, etc shenanigans that come from pointers.
Edit: and getting pretty familiar with Python as my primary language (which had nothing to do with attending engineering school). Rust’s traits, and the way that many of the “higher-level” language features are directly implemented using canonical traits, is very similar in practice with how Python uses “dunder” methods (=“double-underscore”, of the form
__methodname__()) to duck-type its language features like iterators, collection length, collection indexing, dot notation access/manipulation, context managers, and even inheritance. Context managers in particular are almost equivalent to how Rust drops things that exit scope (they allow you to attach behavior to exiting scope in an otherwise runtime garbage-collected + interpreted language).I would describe OCaml as if Rust had been invented from a pure maths point of view of computation, as opposed to how in reality Rust was invented from a “when can [heap] memory be predicted by the compiler to be free-able” (my own words, obviously). So you can do things like specify a “true” generic function in OCaml that infers the restrictions that apply to the valid types it can receive, and the compiler takes care of mono-morphizing that code for the various types that your program actually ends up manipulating/passing to that function at runtime. All functions are partial and composable, and to mutate things you have to explicitly delineate a block of imperative code (everything is immutable + functional by default). You end up doing a lot of pattern matching. Many problems can be compactly solved by recursively pattern matching over arrays and tuples while shuffling around the data that you’re working on (aka dynamic programming).
In relation to rust, it seems to me that it’s the sort of thing that benefits from openly engaging in directionless “horizontal” learning in order to build up a necessary foundation for then building “vertically” once enough pieces are in place. At least more so than more basic languages.
Yes. I made the analogy to biological organisms in my other reply, but you could also make one with human societies; you can’t understand a society by starting from any single member and slowly zooming out. You’ll need to at least repeat that process from multiple different starting points before you begin to form any true/deep understanding of that society as a whole. I.E., you need to examine the garbage collectors and the town priests and the brothel workers, not just the landowners or the factory workers.
I really like that “climbing a spiral” pitch! I wonder how adaptable it would be to learning Rust, however. Or rather, how one could construct said spiral differently; it already feels like The Book spirals upward and outward from a core of general/abstracted programming.
Thanks for the reply! Also nice curriculum there. I haven’t done most of that (or not enough) but I’ve basically written up that as a list of shit I should have under my belt … so nice for me to see personally too.
I really like that “climbing a spiral” pitch! I wonder how adaptable it would be to learning Rust, however. Or rather, how one could construct said spiral differently; it already feels like The Book spirals upward and outward from a core of general/abstracted programming.
Yea maybe. For me, and I’d imagine many who’ve read The Book, a more Spiral-ish or biological/horizontal learning approach on references/pointers etc would go far I think. I haven’t searched hard for it, but from about mid-way through Ch4 I’ve thought that a good deep dive on working with the borrow checker would go far. Given the blog posts and forum threads we’ve linked to here, it almost feels like there’s a hole in the available material. It could work a bit like a reference too so long as it has a good amount of examples well organised along conceptual grounds, which I think it should. But if it addressed all the required concepts and then dug into good examples, both trivial and realistic/applied and mapped the relevant problems and solutions back to all the concepts and their treatment elsewhere in the book, while also providing reading guides for people of differing backgrounds … I think it could go quite far. Maybe you’d be just the person? 😉
You’re welcome!
I strongly agree with the feeling that there is a hole in the current material. At the same time, given the glimpses I’ve had of the other chapters and their sections, I wonder if we “just” need to get through the rest of the book.
The current state of The Book and our progress through it as a community specifically reminds me of being at school: the lessons build on each other, and it always takes a certain amount of material covered before you start to really make sense of things and “refactor”, almost, your prior understanding.
The author of these comments also runs the account @[email protected] over on mastodon if you’re interested. It seems to be a dedicated rust tips account.
