I'm reminded of the time I was on the InfoSec team at a company you've all heard of where all software development was done on a handful of Unix servers with a shared network file system.
I reached out to the Systems team to suggest they look into mounting the home directories with the no-setuid option, and one of them invited me over to pair at their desk. They explained that there was a trick, where you could leave the existing mount point as is, with its existing mount options, and then mount the same device somewhere else with different options. So they got a root shell, went into some scratch directory, mounted the homedirs there, and we tested things out.
I forget how that went, or what happened next, but it seems that we neglected to clean up after this experiment, and at some point, another Systems engineer was looking around the scratch directory and perhaps ran something like `du -ak` (forgetting to specify the option that disables crossing onto new devices), finding what appeared to be a gigantic copy of everyone's home directories.
So they started `rm -rf`'ing it, and made a pretty huge dent before the phones started ringing and they realized that what they were erasing was not a copy on a local scratch space, but rather the real thing over on the NFS drive.
I was one ROLLBACK away from the Users table in production. Mid 90's, we fixed things in production all the time, and I left the where clause out. There were no soft deletes back then, we just deleted the row, and that would trigger the deletion of user resources elsewhere...
I'm sure we had backups, but it would have been bad and resulted in a lot more fixes in production. I guess the upside of the "seat of the pants" days is that there was poor referential integrity, so it wouldn't have taken out anything else like billing history.
edit: I never typed anything so carefully as I typed that rollback. I almost want to say there was an autocommit when closing the session, but my memory on that is fuzzy. MS SQL Server, if anyone knows. From then on, when called on for sketchy but necessary actions, I write the where clause first, run it as a select, see the number of results, wrap it in a transaction, etc...
I was manually monitoring a slow query of mine with "SHOW FULL PROCESSLIST", and at one point I saw "delete from readings" pop up. Readings being a massive, massive table that drove most of our functionality. I quickly killed the query and went about trying to see who else was connected to the DB, but then I saw the query come up again. So I killed it again, and second later it came back.
I had to shout one of the sales guys over, and tell him to go office to office and demand anybody who was deleting rows to stop. Meanwhile I stayed at my desk and kept killing the query.
Shortly after, I heard our CTO yell "shit!", and the query went away. He was trying to clear out his local testing DB, and didn't realize he was in a production shell. I don't know if I saved the company, but I certainly saved us from a couple of days of downtime and maybe an equal or larger amount of lost data. But the CTO couldn't praise me publicly or he'd look like an idiot, so that was the end of that story!
I remember learning that when Pixar did a pan-and-scan version of one of their movies, they wouldn't just use a floating 4:3 frame over the 16:9 original -- they'd re-render the movie, moving items around as necessary, to get a better framing. The perfectionism was palpable when we'd meet with them.
This seems like a huge advantage with mostly-digital movies.
I wonder how easily a Marvel or Starwars movie could be re-rendered like this. Technically there are actors of course, but they are usually the main focus of the scene anyway...
For green screen shots this would be fine. For VFX LED wall shots, this wouldn’t work. However, movies and shows today are shot in a high resolution that you can crop/edit into most formats.
Green screen shots lend itself to resizing better due to the ability to fill in more of the scene while pulling the subject matter away a bit to fit the format.
Many LED wall shots are completely rotoscoped anyway just to dial it in that much further. But you still get the advantages of environmental lighting and the efficiency of real time decision making.
VFX shots are not pure CGI, it's a whole lot of layers composited on top of each other, think Photoshop but video. So making tweaks to any given shot would not be as easy.
There's wider adopted 16:9 aka 1.78 aspect target and then there's everything from 1.85 to 2.39 and ridiculousness like 2.44. Most common three today being 1.78 (aka 16:9), 1.85 and 2.39 (2.35 instead sometimes). 1.85 to 1.78 is usually a fit frame type of crop and call it a day, but 2.39 to 1.78 for home/TV release is same story all over again - do you do the letterbox or are we gonna play re-frame this shot game?
Disney did this with "Lady and the Tramp". Cinemascope and an academy ratio versions were animated individually in order to make the best use of the frame shape.
Early in my career I was working for a well-known anti-virus firm. They had a crash of their Perforce servers and it turns out the backups were not good and the most recent good one was several months old. They painstakingly restored it from the good backup and forensic recovery of the drives, but there were several files that were still missing to fully restore the archive. They did know the hashes of the files, so what they did was push out a bunch of custom virus signatures to the dogfooding (internal testing) environment which flagged any files with those hashes as viruses, and then asked anyone who had a hit to send the file into central IT to get it resolved. It was fairly clever I thought!
The ending is sad -- that intrepid recovery mostly didn't end up used:
And then, some months later, Pixar rewrote the film from almost the ground up, and we made Toy Story 2 again. That rewritten film was the one you saw in theatres and that you can watch now on Blu-Ray.
I think we made How to Train Your Dragon three times at DreamWorks. So many rewrites, director changes, character redesign, etc. If you ever wonder why a feature length animated film can cost over $300 million dollars, this is one reason.
Oh you worked on How to Train Your Dragon? That's my favourite movie in the world!
Really random follow up question - there's a How to Train Your Dragon advert that I assume played before the first film in cinemas, advertising the cinema's popcorn and other concessions, but the only two copies on YouTube are in really terrible quality. (one in English https://www.youtube.com/watch?v=f37fI14Ne0I, the other in Finnish I think https://www.youtube.com/watch?v=W0Hr4mD3G1Q). I really really want a high quality version of this, and I'm wondering if you know of anyone at DreamWorks who I can contact about this? I've tried tweeting at a couple of people but unsurprisingly heard nothing back, so I'm maybe looking for someone in marketing?
Thanks for the advice, it's appreciated! Any ideas how to get in touch? If not I'll try my luck at public twitter (which has had a 50/50 success rate for me in the past...)
It can be real pressure cooker. Developers who come in not from the game development world are often shocked by the brutal hours. The artists have it the worst however. They tend to get laid off in droves as productions ramps down. Directors and writers also get fired. They seem to be less immune to being punished than other management types in more traditional software companies.
> Developers who come in _not_ from the game development
I've been working on hardware, firmware, backend, frontend, crypto, hft, mobile apps and games. I think that on average you need to rewrite your game 3 times so it feels 'right'.
In college I remember hearing from one of my animation professors that DreamWorks switched between NURBS and polygons for How To Train Your Dragon, I'm unsure which direction it was but I don't envy the artists tasked with that if true.
I don’t know about that. I would need more info. The PDI tool chain used for Dragon used the same clamped p-blend curves that had been used for several generations of movies. Technology was almost never the issue for delays. It was usually story, direction, editorial, etc. If you get more context, maybe I can find out more.
Yes, the recordings get done over many times. Sometimes we lose someone due to story changes or some other circumstance. This is a bummer, as there is more than the voice that is lost. While it is easier for some stars to drive over to Glendale from wherever they live in LA, they still "perform" the part and that performance is recorded. The video of the voice over work is referred to by the animators and influences a lot about the how the animated character behaves. I suppose in less expensive productions, you could just try and plop a voice over in, but all the top studios will use the video reference and ask the talent back for voice fixes.
In general, I found the talent to be really fun to be around and no-one just phoned it in. They really liked to work on the films. I ended up sitting next to Gerard Butler at the employee screening of Dragon at the now non-existent Universal Amphitheater. He stayed for the whole film. Unlike me, he hadn't seen it in thousands of tiny chunks during dailies, and had a blast.
I teach in an animation program. We tell our students over and over that animation is not like the movies: re-=shoots, pick-up shots, multiple take etc are simply not possible. That is why the bloopers reel in Toy Story was so ironic.
Cool. I hope your students are finding places at some good studios.
Not only did DWA have bloopers, there are entire sequences rendered and ready to go that didn't make it in the final movie. There is also some shorts and an entire film that made it to pre-viz that never were released.
The "original" Toothless was not Toothless. There were several other Night Fury designs. One amazing part of working in a studio (please try to tour one if you have any friend or reason to get on campus at Disney/Pixar/DWA) is seeing all of the storyboards and pre-viz art lining the walls. It is quite magical.
I don't know about sad necessarily - if the earlier iteration was necessary to realize they needed to make the changes that brought us the final version, it's all part of a larger journey. I can see how this could be frustrating for people involved, but reshoots / rewrites are part of the process.
And I suspect that the recovery of character models and other assets was still instrumental in making the final film, if not the actual shots/sequences themselves.
The sad part is that the thing they saved didn't actually get used and was just thrown out. So it wasn't exactly the big save it's made up to be. But if it wasn't framed as a big save maybe it wouldn't be posted every week or so.
If the thing they saved hadn’t been saved, it makes me wonder if they would have remade it as written, and then either have to throw that out or release what was by all accounts not a good first pass.
I suspect some character models were reused, and recovering what they did enabled the rewrite that came after.
I think it’s still fair to characterize this as a big save even if it wasn’t used as-is. They arguable needed the first version to decide to move away from it.
This seems to be par for the course for Pixar. I remember many of the deleted scenes from the Inside Out DVD were prefaced with explanation from Pixar that they were from a much earlier draft. They realized after working through the first draft that they now had a much better understanding of the characters, and then rewrote the narrative to match.
>
This decision came fairly late in the process, as animators had been working on the film for almost a year with only 17 months until it needed to be completed.
Nobody comes into a digital movie with all of their tools already written and running at top speed, right? Isn't this typically a process where frames per hour is pretty low at first and ramps up as you go?
My favorite part of this account isn't the "test your backups!" lesson. It's the "Have an amazing team" lesson. In one weekend, a team of actual humans sorted through thirty thousand files by hand.
Surround yourself with amazing people, and you will find amazing solutions.
Everyone talking about backups, and the need to avoid blame during disaster recovery. And that's all great.
But MY big takeaway is the final section. After all that, the studio threw away the movie anyway. Had their people make a different movie, working 100 hours per week for 9 months straight. That sounds like an absolutely dystopian hellscape of a culture, and the tone of the article seems to celebrate it. Marveling that those people had the "opportunity" to experience that level of "camaraderie".
My first couple of jobs out of college were pretty "death march"-ish. And yeah, maybe there's something to having an experience or two like that under your belt when you're young and single. But this Pixar example here is just straight up abusive and exploitive, and I'm not sure that a celebratory lens is how we should best view it.
I recommend restoring your backup as a test. Restoring a blank tape often doesn't do anything. And if it does do something, it's probably not what you want.
In the olden days when I was the admin for a small shop we would immediately restore the backup to a second system to be used as a playground. Win/win: verified restore and a place to test/train with production data.
I came to make this snide comment as well! Nobody tests their backups though, I've only ever seen it done inadvertently when the backups were used as part of the development process.
BTW how do you test a personal workstation backup? I don’t have backup workstations to rotate between by backing one up, restoring on the other, working and then cycling between them in this manner. How do you test the restore procedure without risking destruction of the original?
Most backup systems allow you to explore backups and extract individual files. Testing a bunch of files for restoring is at least better than completely ignoring it.. however it's difficult if it turns out part of your backup is simply missing, maybe an erroneous exclusion entry or some path miss (happened to me, I was without backup for a home folder for a year before I noticed it, the /home/user/ was a symlink to another volume that mostly had tmp data and so wasn't backed up). You might not even notice this even if you restore the whole backup, because you can't really be expected to go through everything every time either..
You can buy an extra storage device and swap them out (either physically or swapping the boot order), as long as your workstation doesn't have soldered in storage.
> In order for her to work on the movie while out, they had plugged the machine up to the local network and copied the whole file tree over.
Now image the same situation in 2022:
- there is no acceptable backup for the whole movie (or let's say, app)
- only you know there is one acceptable copy... but it's on your personal laptop. You did copy it back in the day because, well, let's say you wanted to run the app in our local machine
There is no way you would tell your team/boss: "Hey guys, no worries! I have a perfect copy of it!", because why on earth did you copy work material into your personal laptop?! That's the rule number 1 of every modern tech company: don't mix work stuff with personal laptop (and the other way around: don't use work laptop for personal stuff).
- If you can trust an employee with a $25K ($50K today) computer, trusting them with a complete source tree isn't a stretch.
- I'd be more pissed off that somebody didn't have a complete backup somewhere, even if it was a personal laptop they used for work.
- Back then, most people didn't have home computers. Hell nowadays a home pc isn't a guarantee, as most of my (non-engineering) coworkers only have cellphones and ipads at home. And while I agree don't mix work and play, most of these people do.
The computer in question was a Silicon Graphics workstation (pictured in the article). What the image does not convey is the sheer gargantuan size of SG machines. Desktop computing it may have been, but in some cases people actually used their SG boxes as physical desktops.
The SG Octane that features in this story was actually one of the more petit versions, but still it would take two people to move comfortably.
> The story likely takes place in 1998, though Jacob admits he’s foggy on the exact date.
In 1998 we were living in Internet Time. The amount of stuff that changed from '97 to '98 and from '98 to '99 made people's heads spin. Google incorporated in September of 1998. 56K modems were brand new that year. At the end of 1999 when Toy Story 2 came out, Napster had been out for 5 months, and Java 1.2 was just about to exit from beta into general availability.
People were still introducing each other to WinAmp when they started storyboarding this movie.
True. Most of the people I knew with computers back then had them provided by their work, or bought them for university.
But then I lived in a small city in Texas not near a major metro. Most people there didn't buy a home pc until the DotCom boom brought those cheap emachines.
And by the next year that had jumped to 56% and 50% respectively. That's 25 million Americans who got the Internet in a 12 month period. And that 2000 number was up from 36.6% and 18% in 1997.
You can't project backward two years from a 2000 number during that era. You're missing data points during the steepest part of the curve.
Susman was the Supervising Technical Director and the machine was a Silicon Graphics workstation. Certainly sounds like she was given a work machine to do her job.
I kind of wonder if deduping posts would be good. I find it really annoying to see reposts and if definitely makes it feel like there really isn't all that much content out there, at least much content that has enough viral potential. I suppose it's possible for a company to fingerprint content easily enough to stop dupes. Maybe it would block the post but allow a repost of the original content? But is that really good for the company? Plenty of people haven't seen that content anyway, so maybe it's working for those people, and sometimes you do want to see the repost if it's been long enough. I personally think it might just seize the entire ecosystem if you prevented it because the volume of content would go down, and the vitality of things would go down too. Not to mention the people reposting are kind of keeping the whole thing alive for their virtual points, and if they can't do it at scale by posting the same video in 20 different places anymore things might just stop. I feel like a lot of social networks are relying on a an orbiting cloud of garbage that just recirculates constantly. I'm surprised no one has started uploading, upscaling and posting America's Funniest Home Videos clips.
> I feel like a lot of social networks are relying on a an orbiting cloud of garbage that just recirculates constantly
This is definitely true of mainstream social media, but I don't think it's that bad on HN. Based on the list of duplicates, this has been posted 13 times in 12 years, that's really not that bad. Things don't live on the front-page that long, someone reasonably could have missed it every time. If people are engaging with the post, what's the problem?
They must, because dupes are less obvious than those other things. You need to either have seen the story on HN before and remember it; or do a search.
this is insane to me because individual frames for a Pixar film are around 1GB in size.
recall how large laptop hard drives were before Toy Story 2 (2002)...
I have an extremely small amount of experience writing shaders for RenderMan (BMRT, actually), and those aren't large. animation data typically isn't HUGE, but models sure as heck are, and a frame you send to RenderMan is concrete (it used to be, maybe it isn't anymore?) meaning that all geometry was present in the file you gave to RenderMan, only the compiled shaders were referenced from it, because they were procedural and could not be statically included. so I guess the frames the renderer sees are sort of like the binaries of what I would now view as a software compilation process.
well that's my point. I always thought that what was lost were frame definition files, and not the models. surely every animator had copies of the models; you can't animate without them, and the movie was in production when the data was lost, so they had those models, or certainly most of them.
I think what was lost was the animations, and not the models or the frame/scene files.
I remember the stories from Ed Catmull's book Creativity Inc. Although not that they tossed the movie out after almost deleting it anyway, like it's said in the article. The story about starting from scratch was that they left the team that did it alone, but it was their first movie and it fell flat. Reading about it then felt like a dick move and kind of questionable, but even more so later when I'd read that Lassiter, who was the most critical one, is a certified creep / jerk. So I don't know, but I guess if you're successful in the end your opinion is never questioned.
I reached out to the Systems team to suggest they look into mounting the home directories with the no-setuid option, and one of them invited me over to pair at their desk. They explained that there was a trick, where you could leave the existing mount point as is, with its existing mount options, and then mount the same device somewhere else with different options. So they got a root shell, went into some scratch directory, mounted the homedirs there, and we tested things out.
I forget how that went, or what happened next, but it seems that we neglected to clean up after this experiment, and at some point, another Systems engineer was looking around the scratch directory and perhaps ran something like `du -ak` (forgetting to specify the option that disables crossing onto new devices), finding what appeared to be a gigantic copy of everyone's home directories.
So they started `rm -rf`'ing it, and made a pretty huge dent before the phones started ringing and they realized that what they were erasing was not a copy on a local scratch space, but rather the real thing over on the NFS drive.
Fortunately, this company did have great backups.