Hacker Newsnew | past | comments | ask | show | jobs | submit | miquong's commentslogin


Redirects my man, Learn em and love them


If you're following along and can't/don't want to remember the SQL syntax, use the examples from the post for LLM text-to-SQL context:

  Q: Which photo has the highest number of faces?
  A: SELECT SourceFile
  FROM photos
  WHERE RegionType IS NOT ''
  ORDER BY length(RegionType) DESC
  LIMIT 1;
  
  Q: ...
You can also fetch and use the table schema with `sqlite3 exif.db .schema`


The author has a highly amusing, off-beat Instagram feed: https://www.instagram.com/crime_pays_but_botany_doesnt


He also has a YouTube channel. I first saw him speak at SF Nerd Nite which is a speaker series in San Francisco. He’s very entertaining and basically a self taught botanist.


His youtube channel is great! I love his Bay Area hikes and plant explanations.

There's a nice article about him, he is a full-time train engineer who drives train shipments all around the west coast and while he was traveling he got curious about all the plants he would see from the train so he started going to the libraries on his breaks from work to learn about plants.

https://www.outsideonline.com/culture/books-media/botany-joe...

Interesting chilean high-elevation rare carrot: https://www.youtube.com/watch?v=UdodZcrFIPM&t=2s

old growth redwoods: https://www.youtube.com/watch?v=Xbma869jMQY&t=4s


I second the Youtube channel suggestion. Very entertaining and informative. Lots of trash talking and banter to keep things interesting. The SF tours are especially great. Explanations of why things should/should not be planted in areas really opened my eyes to how bad the urban designs can be.

Sidenote: I like that he tattooed his finger with measurement lines so he can use his fingers in the pictures as reference for size.


Any other folks you saw at nerd night that I need to read and follow?


It's been a while since I attended one. I remember some of the most interesting talks were the ones I was not particularly interested in from the descriptions. Mostly talks about the history of San Francisco:

* Ghost signs: https://sf.nerdnite.com/2014/06/04/nerd-nite-sf-49/

* Neon signs: https://sf.nerdnite.com/2017/10/11/nerd-nite-sf-89-brain-sci...

* Dairy farming: https://sf.nerdnite.com/2019/08/14/nerd-nite-sf-111-butterfl...


For those in the Bay Area he has a pretty hilarious video of him shopping at Berkeley Bowl too, shredding junk like the homeopathic bs but praising their produce selection. Bonus video of him analyzing the sad trees in the Emeryville Target parking lot.


I requested, but gave up and requested deletion after. I still haven’t received the requested data.

Still, I figure I have a few trillion backup copies… albeit in very raw format


Microsoft has an elaborate VS Code demo app that uses this API. Click "Open Folder" to see the API in action): https://vscode.dev


Tabs seem like they'd be great training material for LLMs and generative music. Does anyone know of projects attempting this?

Update: looks like some papers got into it last year:

Rock Guitar Tablature Generation via Natural Language Processing: https://arxiv.org/pdf/2301.05295

GTR-CTRL: Instrument and Genre Conditioning for Guitar-Focused Music Generation with Transformers: https://arxiv.org/abs/2302.05393


You may find this site interesting: https://folkrnn.org/tune/119105 Thing is tabs and other kinds of notation only capture a small part of the composition, often just the melody. So to generate "real" polyphonic music you need other kinds of data.


I don’t see how they’d be easier than midi.


Just want to send my thanks. I’ve been a regular user since the early testflights and it’s been great seeing the iterations and improvements, particularly for me the scheduled-but-no-show ghost buses.

Congrats to the team!


For image and layer manipulation, crane is awesome - as is the underlying go-containerregistry library.

It lets you add new layers, or edit any metadata (env vars, labels, entrypoint, etc) in existing images. You can also "flatten" an image with multiple layers into a single layer. Additionally you can "rebase" an image (re-apply your changes onto a new/updated base image). It does all this directly in the registry, so no docker needed (though it's still useful for creating the original image).

https://github.com/google/go-containerregistry/blob/main/cmd...

(updated: better link)


This is a great recommendation. It is worth noting that unlike Docker, crane is root- and daemonless which makes it work great in Nix (it's called 'crane' in the Nix repository). This allows for Nix to be used to manage dependencies for both building (e.g. Go) as well as packaging and deploying (e.g. gnu tar, crane).


Is there any performance benefit to having fewer layers? My understanding is that there's no gain by merging layers as the size of the image remains constant.


There are some useful cases — for example, if you're taking a rather bloated image as a base and trimming it down with `rm` commands, those will be saved as differential layers, which will not reduce the size of the final image in the slightest. Only merging will actually "register" these deletions.


Less performance and more security. Lots of ameteur images use a secret file or inadvertently store a secret to a layer without realizing an rm or other process in another layer doesn't actually eliminate it. If the final step of your build squashes the filesystem flat again you can remove a lot of potentially exposed metadata and secrets stored in intermediate layers


Eventually, once zstd support gets fully supported, and tiny gzip compression windows are not a limitation, then compressing a full layer would almost certainly have a better ratio over several smaller layers

https://github.com/opencontainers/image-spec/issues/803


Is it coming? That ticket doesn't fill me with hope, given its age and the disagreements over backwards compatibility.


If you've got a 50 layer image then each time you open a file, I believe the kernel has to look for that file in all 50 layers before it can fail with ENOENT.


It depends on your OCI engine; but this isn’t the case with containers. Each layer is successively “unpacked” upon a “snapshot”, from which containers are created.


Interesting. Certainly with podman I observe that the layers are combined by overlayfs.

I build this Dockerfile into a container with 'podman build -t test .':

    FROM registry.access.redhat.com/ubi9/ubi-minimal
    RUN touch /file1
    RUN touch /file2
    RUN touch /file3
Then run it with 'podman run --rm -it test':

Then I can inspect the container's storage configuration:

    $ podman inspect -l | jq '.[].GraphDriver'
    {
      "Name": "overlay",
      "Data": {
        "LowerDir": "/home/sam/.local/share/containers/storage/overlay/e0055211bf4b6a839f91759d9bbbb5b93c61f4e2969b25ec9d732428557c230f/diff:/home/sam/.local/share/containers/storage/overlay/efd666ad5dc6b7a92933043ed42a41341b4e25c80e981f6d01ab41927d8f8a89/diff:/home/sam/.local/share/containers/storage/overlay/1c108fb13c6708a93dcf237bea2637344cf65d3c9272693817aa0741b158fd7b/diff:/home/sam/.local/share/containers/storage/overlay/80c0d7946d02825018d47fbf34df054bec5dc433ae20f71ffabd8d3725448837/diff",
        "MergedDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/merged",
        "UpperDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/diff",
        "WorkDir": "/home/sam/.local/share/containers/storage/overlay/f14a1506ed692e3c40b06d5eb1ba9c011bad159c82df6ddee632b49ba02a5bfc/work"
      }
    }
Note that there are four directories in the LowerDir, and examining their contents reveals that there's one per layer in my 'test' image:

    $ podman inspect -l | jq '.[].GraphDriver.Data.LowerDir' -r | tr : '\n' | xargs -rt -n1 ls
    ls /home/sam/.local/share/containers/storage/overlay/e0055211bf4b6a839f91759d9bbbb5b93c61f4e2969b25ec9d732428557c230f/diff
    file3  run/
    ls /home/sam/.local/share/containers/storage/overlay/efd666ad5dc6b7a92933043ed42a41341b4e25c80e981f6d01ab41927d8f8a89/diff
    file2  run/
    ls /home/sam/.local/share/containers/storage/overlay/1c108fb13c6708a93dcf237bea2637344cf65d3c9272693817aa0741b158fd7b/diff
    file1  run/
    ls /home/sam/.local/share/containers/storage/overlay/80c0d7946d02825018d47fbf34df054bec5dc433ae20f71ffabd8d3725448837/diff
    afs/  bin@  boot/ dev/  etc/  home/ lib@  lib64@  lost+found/ media/  mnt/  opt/  proc/  root/  run/  sbin/ srv/  sys/  tmp/  usr/  var/
Same for a system (non-rootless) container:

    # podman inspect systemd-oxidized | jq '.[].GraphDriver.Data.LowerDir' -r | tr ':' '\n' | xargs -rt -n 1 ls -F
    ls -F /var/lib/containers/storage/overlay/062693a4d8fe24055645ee1c11bbc542507ba3fe101ed4e5b76b6457752d3ac5/diff 
    home/
    ls -F /var/lib/containers/storage/overlay/9d9c6a906aff40160f924c4fec96692a6fc59f27316b8df5ee2616091008a2fe/diff 
    home/
    ls -F /var/lib/containers/storage/overlay/e35e4cf6ce1057979baee991e609f82afb85360560dc8bf36ab069d8e8288f5f/diff 
    home/
    ls -F /var/lib/containers/storage/overlay/da5189fd0e740204d7a30e4b59b012251e8d36a1db488bf47ede61247a948fff/diff 
    home/
    ls -F /var/lib/containers/storage/overlay/702d48c0e8afce7cf5ce049939d8fdb596755e709a3735bbbf4900ec65c45529/diff 
    etc/  home/  var/
    ls -F /var/lib/containers/storage/overlay/877031909d6cf6038e0e0295c34f14932b71903835b3eb13ae6bf23396d65daf/diff 
    etc/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/b90fc2ebe1d7c19e5ee43033b8e8be95210b4e3a0012bb54db327f6d52e89d12/diff 
    tmp/
    ls -F /var/lib/containers/storage/overlay/aa2e131948d26d04f48f7c01c57aa98e2a308e4fc0d707c9532d30fa4ab89f5d/diff 
    root/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/6af6a203c83f697aa35b5007b5678b8c2b052ab482d9d7f7e5b322cd74682441/diff 
    root/  tmp/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/b237f20a22db5d282838c5a7c9d78f7d6e648d9b223a328fa8b8078e9c6b5bbe/diff 
    ls -F /var/lib/containers/storage/overlay/7d1bc5d46561795a4f7058e4098faa345e119617292761933e2ad7f185dbd45b/diff 
    ls -F /var/lib/containers/storage/overlay/88d52756a706920fbcb38cbbb6eaf43ea09be0045361ffe6920185ca7179a1d9/diff 
    tmp/
    ls -F /var/lib/containers/storage/overlay/0d0dedc3e4f8d7f39b9a0c78d1cf380ad1161023a49bd74ab8b51626f6d21f19/diff 
    root/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/d74968407a57d753c2ae232ea26e0e2d7fed30683a67a993b814b71684817eaa/diff 
    root/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/885b215dc2a8304a58a6d35a5668c74d3d80f27dacf56b97a3b8b13c1a423b33/diff 
    root/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/a4edd3819e3655d0ca90ac898a4a2b91ec50c83cbf61862186e10ae1b44cbcc0/diff 
    etc/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/954dcca00292635597e4c5ee9ebc35bf5fcc15a50dfd8f176df7b4c15e695e10/diff 
    bd_build/  etc/  root/ run/  usr/  var/
    ls -F /var/lib/containers/storage/overlay/fb670aa62e0f4f71382ef61a742e8ab1ad5d345b95267b0adc18c10e6c87f13b/diff 
    bd_build/
    ls -F /var/lib/containers/storage/overlay/7f5cbd8cc787c8d628630756bcc7240e6c96b876c2882e6fc980a8b60cdfa274/diff 
    bin@  boot/  dev/  etc/  home/ lib@  lib32@  lib64@  libx32@  media/  mnt/  opt/  proc/  root/  run/  sbin@  srv/  sys/  tmp/ usr/  var/
('podman system info' confirms that .store.graphDriverName == "overlay")

I'd be interested to see what this looks like with docker?


That seems ripe for optimization, if true, especially since those layers are all immutable.


A container runtime could optimize for speed by unpacking all those layers one by one into a single lower directory for the container to use; but at the cost of using lots of disk space, since those layers would no longer be shared between different containers.


One simple case where the resulting image is bigger than necessary:

``` COPY ./package.deb /tmp/package.deb

RUN dpkg -i /tmp/*.deb && rm -rf /tmp/*.deb ```

This results in two layers, with one layer containing a huge file, thus being part of the final image if you don't do multi-stage builds.


I'm working on a tool that does the opposite: to split layers into smaller, deterministic deltas.


If files are overwritten or removed in a lower layer, there can be size savings from that.


some startup performance savings in fewer http requests to fetch the image. small for sure but it's something?


In practice I've found the performance savings often goes the other way--for large (multi-GB) images it's faster to split it up into more layers that it can download in parallel from the registry. It won't parallelize the download of a single layer and in EC2+ECR you won't get particularly good throughput with a single layer.


Depends. If you would have to fetch a big layer often because of updates, that's not good. But if what is changing frequently is in a smaller layer, it will be more favorable


The dimmest phones are larger than any others


The included explanation is mind boggling to me:

> Things that are far away look smaller, but things that are REALLY far away look bigger, because when their light was emitted, the universe was small and they were close to us.


Lindows was my first Linux distro c. 1999. It was my first year at college, majoring in Business Administration, living on campus with internet speeds above 56kbps for the first time. Trawling through the internet at that time, I stumbled on a (likely bootleg) Lindows installer somewhere and fell down the rabbit hole.

Looking back, I realize now I’d been a self-denying computer geek before then but for whatever reason Lindows, it’s wacky installer, dual boot support, and fortunate hardware compatibility gave me the right nudge at the right time to send me on a lifetime of hacking.

Almost 25 years later, most as a professional software engineer, I have a completely biased affection for this strange OS.

Thanks for sharing!


Even though it became Lindpire, the spiritual successor to lightweight and windows like was LXDE and Lubuntu.

It was Slackware that pioneered the loadlin boot loader installed on a dos partition that I think li does/linspire picked-up. This was all gradually killed off by ntfs and windows 2k

It was a magical time for Linux with konqueror as a viable desktop browser and galeon as the best browser available until IBM started sponsoring the project to make it like epiphany. Eventually konqueror was absorbed into WebKit for safari, proprietary flash made browsing on Linux unnecessary difficult (gnash came later), and Microsoft lawsuits were customary towards all friendly UX not on a Mac.


linspire who was using ocaml and then haskell, a rare thing at the time http://lambda-the-ultimate.org/node/1506


Magical expect for compiling MPlayer by hand and doing voodoo to get the TV Tuner working. But XawTV and AleVT were cool.


Like 20 years ago I was obsessed with getting MythTV running with a TV tuner, but kept running into driver issues and low spec'd hardware. It was still magical to me, though.


TVtime worked faster but if didn't have LIRC support, I can't remember. If it did, it was to setup, much more lightweight but without recording support.

Nowadays Kodi/XBMC it's the spiritual sucessor.


Very fond of lubuntu, which I managed to get installed on NTFS in a directory on my C:drive allowing switching back-and-forth between it and my Windows install, and really wish that that was still a thing.


The focus turned to virtualization


Yeah, that doesn't work for me, since my machines aren't that fast, and I really, really want stylus support which wants hardware access.


What are some good applications which require stylus support on Linux?


Annotating and marking up PDFs.

Drawing in apps such as Inkscape and Arita.

Working up 3D models using OpenSCAD Graph Editor.

Taking notes w/ a pen in class using Write by Stylus Labs or Journal.


Xournal, not Journal.


Mine was DamnSmallLinux and Slax. I wasn't allowed to install Linux on my family's computer of course, so live CD's were my go-to. Though once I scrounged up enough spare parts from family friends to build a PC, I then got to learn how to force Slax to "install" to a hard drive, which was quite a challenge for me, as it really didn't want to be run off non-live-install media haha


Oh shit, DSL - that was my jam too, felt so special, even though had no clue what I was doing :D

Edit: The site is still alive! http://www.damnsmalllinux.org/


My first distro was Ubuntu 8.04 Server. Kind of funny because I was in high school back then and decided to install Ubuntu on my machine, coming from the Windows world, I thought that it would be cool to run the server version, thinking it would be like Windows Server 2008 (with a desktop). After installing I was shocked to find out that I had no desktop to work with, spent the next 3 days figuring out how to connect my machine to the internet and then eventually downloaded Gnome. Now almost 15 years later I still use Ubuntu as my daily driver, typing this on 22.04


I bought those business card shaped CD-R disks just so I could keep a copy of DSL with me before flash drives were popular.


Many here share that journey. For me, it was printing out the Gentoo "hard mode" installation guide where you compiled _everything_ from scratch, including the kernel.

Didn't make it but it taught me the terminal and some foundational concepts in an OS. Set me down the path of linux (ubuntu - bit easier to install) and hacking.


With my limited exposure to linux when I was a kid, I didn't understand why I needed to or what those `configure; make; make install` commands did. It was until very later when I knew what compiling was or that make is actually a tool to run arbitrary commands.


oh gods, the many many hours I "wasted" recompiling my gentoo installation trying to squeeze more performance out of my gaming machine and prove to all my gaming naysaying friends that linux is no good for gaming. This was back in the VoodooFX days...

fond memories prior to having a wife.


I never got there. My poor amdv4 couldn't even finish compiling the kernel without overheating. But I learned a lot about reducing kernel size.


My first mentor in Linux was a sysadmin at the hometown ISP, and my initiation was compiling Gentoo from Stage 1 (the "hard mode" install) on a dual socket P3 700mHz system. I've never done it ever again, but that foundational experience helped me immensely.


Oh that was the holy grail. I also never got there… had to settle for the lower levels of Gentoo purity.


I did it. Worst part was compiling KDE in an Athlon XP. Took days.


For me at least, the longest part was compiling OpenOffice.


Went on a weekend trip. Started the OOO compilation, allowed it to also install some dependencies like Blackdown (?) Java.

Decided, hey, might was well time it, for fun.

I came back 2 days later, it had just finished. The compilation took almost 2 full days :-)


I did it with a 300 MHz Celeron. You heard me.


I did multiple times on a 900-something mhz Athlon. Back when compile everything was the only option.


> For me, it was printing out the Gentoo "hard mode" installation guide where you compiled _everything_ from scratch, including the kernel.

There was another way to install Gentoo?


Yes, there were 3 stages, I think. Stage 3 was with some binaries throw in there. Stage 1 required recompiling the compiler at some point.


Oh shit, I remember the stages now, not much about them though. Did you really install Gentoo if you didn't compile the compiler you used to compile the rest of the system from the ground up though?


I'm not sure if this is possible anymore, but for a while, there was a way to copy over the compiled base system from the disk instead of compiling it all from scratch. Running your first emerge -vaDu world might end up recompiling everything anyway, depending on the age of the ISO, but you didn't have to do any compiling to get a base system deployed.


i was going to comment. I think even genkernel is compiled, not that i touch the stuff.


Also quirky was their Lindows Rock: https://m.youtube.com/watch?v=QSdRTOh2jeA


Do you remember the one they released when they changed name to linspire: https://www.youtube.com/watch?v=IIYtKHnU4mQ memorable


Never knew so many levels of parody and irony could be encapsulated in one flash animation!


Oops… it was WinLinux 2000[1] that came first for me and Lindows shortly after. Apparently the naming permutations worked for at least one person

[1] https://archive.org/details/win-linux-2000


Lindows was released on august 2001 (in case you like to have your memories properly ordered)


Yeah it must have been later than I remember. Even if it was a bootleg beta I found, it would have been 2000 at the earliest.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: