My First Regular Expressions

I’ve been reading Mastering Regular Expressions by Jeffrey E.F. Friedl, and since nobody in my life (aside from my wife) cares, I thought I’d share something I’m pretty proud of. My first set of regular expressions, that I wrote myself to manipulate the text I’m working with.

What’s I’m so happy about is that I wrote these expressions. I understand exactly what they do and the purpose of each character in each expression.

I’ve used regex in the past. Stuff cobbled together from stack overflow, but I never really understood how they worked or what the expressions meant, just that they did what I needed them to do at the time.

I’m only about 10% of the way through the book, but already I understand so much more than I ever did about regex (I also recognize I have a lot to learn).

I wrote the expressions to be used with egrep and sed to generate and clean up a list of filenames pulled out of tarballs. (movies I’ve ripped from my DVD collection and tarballed to archive them).

The first expression I wrote was this one used with tar and egrep to list the files in the tarball and get just the name of the video file:

tar -tzvf file.tar.gz | egrep -o ‘/[^/]*.m(kv|p4)’ > movielist

Which gives me a list of movies of which this is an example:

/The.Hunger.Games.(2012).[tmdbid-70160].mp4

Then I used sed with the expression groups to remove:

  • the leading forward slash
  • Everything from .[ to the end
  • All of the periods in between words

And the last expression checks for one or more spaces and replaces them with a single space.

This is the full sed command:

sed -Eie ‘s/^///; s/.[[a-z]±[0-9]+].m(p4|kv)//; s/[^a-zA-Z0-9()&-]/ /g; s/ +/ /g’ movielist

Which leaves me with a pretty list of movies that looks like this:

The Hunger Games (2012)

I’m sure this could be done more elegantly, and I’m happy for any feedback on how to do that! For now, I’m just excited that I’m beginning to understand regex and how to use it!

Edit: fixed title so it didn’t say “regex expressions”

figjam,

Congrats on your learning! I did a similar thing with music and converting all random songs to mp3

bizdelnick, (edited )

It is a great book, although a bit outdated. In particular, nowadays egrep is not recommended to use. grep -E is a more portable synonim.

Some notes on you script:

  1. You don’t need to escape slashes in grep regex. In the sed s/// command better use another character like s### so you also can leave slashes unescaped.
  2. You usually don’t need to pipe grep and sed, sed -n with regex address and explicit printing command gives the same result as grep.
  3. You could omit leading slash in your egrep regex, so you won’t need to remove it later.

So I would do the same with


<span style="color:#323232;">tar -tzvf file.tar.gz | sed -En '/.(mp4|mkv)$/{s#^.*/##; s#.[.*##; s#[^a-zA-Z0-9()&-]# #g; s/ +/ /g; p}'
</span>
SpaceCadet,
@SpaceCadet@feddit.nl avatar

nowadays egrep is not recommended to use. grep -E is a more portable synonim

Not directed at you personally, but this is the kind of pointless pedantry from upstream developers that grinds my gears.

Like, I’ve used egrep for 25 years. I don’t know of a still relevant Unix variant in existence that doesn’t have the egrep command. But suddenly now, when any other Unix variant but Linux is all but extinct, and all your shell scripts are probably full of bashisms and Linuxisms anyway, now there is somehow a portability problem, and they deem it necessary to print out a warning whenever I dare to run egrep instead of grep -E? C’mon now … If anything, they have just made it less portable by spitting out spurious warnings where there weren’t any before.

bizdelnick,

GNU grep, the most widespread implementation, does not include egrep, fgrep and rgrep for years. Distributions (not all, but many) provide shell scripts that simply run grep with corresponding option for backward compatibility. You can learn this from official documentation.

Also, my scripts are not full of bashisms, gnuisms, linuxisms and other -isms, I try to keep them portable unless it is really necessary to use some unportable command or syntax.

SpaceCadet, (edited )
@SpaceCadet@feddit.nl avatar

GNU grep, the most widespread implementation, does not include egrep, fgrep and rgrep for years. Distributions (not all, but many) provide shell scripts that simply run grep with corresponding option for backward compatibility. You can learn this from official documentation.

It seems you need to read the official documentation yourself. While it’s new information to me that egrep is no longer a symlink, as it used to be a couple of years ago, but a shell script wrapper to grep -E instead, the egrep command is to this day still provided by upstream GNU grep and is installed by default if you run ./configure; make; make install from source. So it is not a backward compatibility hack provided by the distribution.

You can check for yourself. Download the source from ftp.gnu.org/gnu/grep/grep-3.11.tar.gz, unpack and look for src/egrep.sh or line 1756 of src/Makefile. Apparently the change from symlink to shell script was done in 2014, and the deprecation warning was added only last year.

In any case, my larger point is that the depreciation of egrep was a pointless and arbitrary decision that does not benefit users, especially not veterans like myself who have become accustomed to its presence. I don’t mind change, but let’s be honest, most people are not in the habit of checking the minutiae of every little command line utility they use, so a change like this violates the principle of least surprise. It’s one thing if things are changed with a good reason and the users do not only suffer the inconvenience of the change but get to reap the benefits of it as well, but so far I haven’t found any justification for it yet, nor can I think of any.

So if there is a portability problem with using egrep now, it’s a self-inflicted portability problem that they caused by deprecating egrep in the first place.

Also, my scripts are not full of bashisms, gnuisms, linuxisms and other -isms, I try to keep them portable unless it is really necessary to use some unportable command or syntax.

Good for you. Do you want a cookie or something?

bizdelnick,

It seems you need to read the official documentation yourself.

I did. Debian man page, GNU grep manual.

I’m sorry for your loss, however the egrep deprecation is a fact. Of course you can continue using it as a veteran, but it is not correct to recommend this to beginners.

SpaceCadet, (edited )
@SpaceCadet@feddit.nl avatar

You are strawmanning, and your links are not countering any point I made. I never disputed the depreciation as fact, and I never recommended that beginners should use egrep over grep -E

I disputed your claims that the egrep command has just been a distro hack all these years, when in fact GNU to this day still distributes egrep through its source tarballs and only very recently started to warn about it through the wrapper script. And again, the only “portability problem” here is the fact that they deprecated it in the first place, i.e. a self-inflicted one.

Then as a Linux and Unix veteran I gave my subjective opinion by lamenting and criticizing the fact that this depreciation happened, and how changes like this always feel like unnecessary pedantry to me. Yes it’s an expression of frustration, but I am allowed to feel frustrated about it. I don’t need people like you invalidating how I feel about breaking changes in software that I use daily.

ReluctantMuskrat,

Good for you. Do you want a cookie or something?

I don’t know about that guy but you need a chill-pill dude.

SpaceCadet, (edited )
@SpaceCadet@feddit.nl avatar

Well he wrote it like he wanted to be applauded for it or something.

I also find the irony of your comment extremely funny … although that’s probably lost on you.

Later, dude.

mindlessLump,

I’ll have to check out this book. Just remember HTML cannot be parsed with regex

bizdelnick, (edited )

Well, technically it is possible with regex dialect that has lookarounds, but it is overcomplicated. There’s really no reason to do it.

ultra,

Thanks for that link.

danc4498,

I relearn regex from scratch every time I need to use it.

nis,

This is the way.

rustyricotta,

I stumbled upon this regex crossword puzzle a while back. I was never good enough to get it, but it seems like it could be fun.

FaceDeer,
@FaceDeer@kbin.social avatar

Just to chip in because I haven't seen it mentioned yet, but I fing LLMs like ChatGPT or Microsoft Copilot are really good at making regexes and also at explaining regexes. So if you're learning them or just want to get the darned thing to work so you can go to bed those are a good resource.

harsh3466,

You know, I haven’t yet used ChatGPT for anything, I might check it out for this reason.

spittingimage,
@spittingimage@lemmy.world avatar

I use it to tell me which page of the Pathfinder 1e manual I should look on for the rules I need.

Trent,

Just adding my congrats. Good job, OP. Regex is super useful stuff.

harsh3466,

Thank you!!!

Thorry84, (edited )

I can also recommend the book the TS mentioned, it is very good and after reading it you will understand regular expressions. It’s fine to use a cheat sheet if you want, cause if you don’t do it regularly the knowledge can sag, but the understanding is what matters. Also depending on the context, different implementations can have slightly different syntax or modifiers to be aware of.

I lent out the book to my brother once and he somehow lost it, so I never got it back. Don’t lend out book guys.

And remember not everything can be solved using a regular expression: xkcd.com/1171/

prowess2956,

I think the most impressive part of this is that your wife cares.

...does she have a sister?

sab,
@sab@kbin.social avatar

I'm currently seeing a girl I started dating after she had problems with her regex and I helped her out.

So far so good.

harsh3466,

She does but, I’d stay away from the sister. 🤣

mcepl, (edited )
@mcepl@lemmy.world avatar

Give a man a regular expression and he’ll match a string… teach him to make his own regular expressions and you’ve got a man with problems. – yakugo in regex.info/blog/2006-09-15/247#comment-3022 (and yes, it is http:// never https:// for this domain)

harsh3466,

Guess I’ve got problems!

OpenStars,
@OpenStars@kbin.social avatar

Regexps are awesome! And also not at the same time:-P. 🎉 Congrats👏!:-)

harsh3466,

Thank you!

davel,
@davel@lemmy.ml avatar

“regex” means “regular expression”, so “regex expression” means “regular expression expression”.

harsh3466,

Dang! I read through my post three times to make sure I didn’t do that and completely missed that I did it right in the title. (Now fixed).

adespoton,

I highly recommend alf.nu/RegexGolf?world=regex&level=r00

harsh3466,

That looks like a great way to practice

adespoton,

It’s definitely a way to get your regex-fu to the next level, especially if you have people to compete against.

harsh3466,

Oh gosh. There are regex competitions out there, aren’t there.

adespoton,

Yup, including for the largest “in production” regular expression….

stepanzak,

That’s really cool! I know some regex and I tried to learn vim regex, only to find out it’s a rabbithole so deep I’m afraid to look into. The feeling when you press enter and your carefully crafted regex does exactly what it’s supposed to do is awesome though. Good luck!

harsh3466,

Vim is on my list of things to learn. I didn’t even know vim had its own regex, but I suppose that makes sense. I’ve messed with vim a bit, but have stuck to nano so far.

stepanzak,

You might also like micro

leo,

Knowledge and understanding. Feels good, man.

Obligatory Xkcd.

https://sh.itjust.works/pictrs/image/d6d2db0a-cf5a-4270-b4bb-a898d9b88695.png

harsh3466,

It does feel good! And thanks for that xkcd! That one’s new to me.

CosmicTurtle,

Ah…the days when perl was the shit and python was still a glimmer in the eye of some frustrated programmer.

bionicjoey,

Relevant xkcd (specifically the alt text)

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • linux@lemmy.ml
  • localhost
  • All magazines
    •
    Loading…
    Loading the web debug toolbar…
    Attempt #