Category Archives: Technology

Interns, and College, and Certs. Oh My.

Posted on by
Reply

I do not have a college degree.

I think it’s important to lead with that, because while I’ve built a fairly successful career, I’ve done it without actually attaining a degree of any sort. I did attend college — a major university for 2 years, and a community college for a year. But in all that time (and all that debt), I never managed to piece together a degree.

Part of the problem is that like most college students, I changed my major multiple times. I started as an Electrical Engineer major. They seemed to make a lot of money, and, if I’m honest, that’s about all the thought I put into it. Then, in Calculus 3, I decided Electrical Engineers did things with numbers that just didn’t need to be done. I was also a Technical Writing major, and English major, and once I shifted to the community college, a “Liberal Arts” major. (I still don’t know what that actually means)

Don’t get me wrong, even though I didn’t get a degree, my college experience did actually help me significantly. I found myself skipping engineering classes at Michigan Tech, and hanging out in the computer labs all day (and night). There was a brand new NeXT computer lab, and it make Unix sexy. In fact, it was probably partially that time in a terminal window when I was supposed to be in engineering classes that made me fall in love with Unix/Linux.

Should I go to College?

This is question I get a lot. A lot a lot. It’s also a question I’m very hesitant to answer. Because the answer is a resounding maybe, and that’s not what anyone wants to hear. Another problem with the question is that the answer keeps changing. For example, back in 2012, I was asked this question at CBT Nuggets, and colleges were just starting to offer more than C++ programming as their only Computer Science class. Here was my response then:

This is still solid advice (well, as solid as advice from me gets anyway), but if anything, the college angle has gotten more attractive. Yes, education lags behind the cutting edge, but if you go into a university computer science program today, you’ll actually get a well rounded education on networking, operating systems, and actual useful programming languages. That still doesn’t mean it’s the right answer for everyone though, because college is very expensive, and you might be served better with a combination of certification programs, internships, and just plain old experience. When I was college-aged, there simply were no computer networking classes. Now there are, but there are also plenty of vocational programs that teach networking as well.

When I was college-aged, there simply were no computer networking classes. Now there are…

Let’s focus on my area of expertise; sourdough bread. No, just kidding, my technical specialty is Linux. And it’s an area that continues to attract more and more employers. Linux Insider posted an article during the pandemic pointing out the need for Linux-savvy workers, even as the industry moves away from traditional servers and hosts everything in the cloud. (Because guess what makes the cloud run? Yep. Linux.) And while colleges certainly offer Linux classes, they’re still lagging way behind current needs when it comes to employable skills. If you get your college degree, you’re still going to need to get certifications to not only prove your worth — but also to fill in the gaps dated college curriculum offers.

So College is a Waste?

Again, maybe. Here’s the thing, college does a couple things really, really well:

  • Teaches foundational knowledge that makes for better equipped professionals
  • Is structured in a way to teach a well-rounded educational base, wider than the specific topic of study
  • Gives students an opportunity to see if they like a variety of subjects (remember my Electrical Engineer “career”?)
  • Looks really good on a resume

And that last one is a real kicker. The current hiring process is largely automated at the early stages. Many employers use a college degree as a litmus test to determine whether or not to even interview a candidate. A college degree shows that a person has the stamina and hard work to achieve a difficult goal. Even if it doesn’t prove they’ll be a good employee, it’s often the first hurdle to even getting an interview. I personally think that’s sad, but I’m sure it’s a statistically viable way to sort the wheat from the chaff. Unfortunately, companies miss out on some really good folks who chose a different path.

I won’t lie, having a college degree does open doors, especially when applying for a job. It’s not the only way to get hired, but it’s important to judge if the cost and years spend getting educated at a university is worth it. It might be, especially if someone else is helping pay the bill. But going into $100k of debt will take a lot of years to pay off, even if you land a great job.

So What Else Is There?

This is where it’s much better to do what I say, and not what I did. When I left college, I started a small business. It failed miserably. (Like, really bad. It was ugly.) From there I got a tech support job at the local community college answering phones giving support to dialup Internet users. My experience in the computer labs at Michigan Tech, and my experience as a “small business owner” gave me enough resume fodder to get an interview. From there, after a series of very unfortunate events, I applied at a K12 school district for the technology director position. I shouldn’t have gotten that job, but I wrote a very compelling cover letter, and interviewed well. I also got very lucky. Getting lucky isn’t something you can prepare for, but all is not lost if you don’t go to college.

If you decide not to go to college, or at least not head off to university for a 4 year degree, there are a couple viable alternatives that will make you employable (even if it makes it tough to get past the automated resume filters):

  • Get an Associate’s degree at a community college.
  • Study on your own and get tech certifications from places like CompTIA, LPI, Cisco, Microsoft, etc.
  • Apply for an internship. If you can afford it, unpaid internships are an easy way to get experience.
  • Apply for an internship. Lots of interns get paid, if crappy. More on this later.
  • Get involved with some Open Source projects, especially if development is what you want to do.
  • Create an online presence. Certainly GitHub for developers, but also YouTube, Twitter, and LinkedIn.

I know this was the “no college” list, but a couple years at a community college is probably affordable, especially if you can stay home and commute. Having a degree, even an Associate’s degree, will open some doors. Even if your major is Underwater Basket-Weaving — just having the paper will potentially get you an interview.

Internships are incredible. They can be at a college, or a business in the field you’re interested in. They come in two flavors: Regular and crappy. Seriously though, some internships are unpaid. It’s the pre-employment equivalent of doing work for the exposure. It sucks. But, if there aren’t any other options, it’s a viable way to get your foot in the door and get some serious experience on your resume. There are many companies who pay their interns though, so don’t assume you’ll have to work for free. You’ll probably make a pittance, but it might be enough if you can still live at home, or have lots of roommates.

Lastly, the old adage is that nothing beats experience. That’s still true, especially if you can get to the actual interview process. An internship is incredible to put on a resume, but if you contribute to projects on GitHub (even documentation! EVERY project needs documentation!), or have projects of your own, that counts. And if a YouTube channel seems like a silly thing to start in order to get a job — I assure you, if you get to the interview stage, interviewers will check out your YouTube channel before they interview you. Same with blogging, tweeting, facebooking, linked-inning, etc. Put yourself in the hiring committee’s shoes, if they can “see” you before they actually see you, they’ll likely do so. Just be genuine, and it will be like an extended interview that you don’t have to sweat through!

You Avoided the Question Entirely

Yeah, I know. That’s what I meant at the start — it’s all maybe. Rather than a one-size-fits-all answer, hopefully this has given you a bit of insight to help you think through what will work best for you. Some of the ideas are valid regardless of your decision on college. All those bullet points above will make you more employable, and a better tech nerd in general. Getting involved in the community you’re interested in will only help make connections that might get you a job. Most of my “best” jobs have been because I knew someone who knew someone who saw my stuff. Good luck, and whatever your future looks like, I encourage you to seek after something you enjoy. If you don’t like calculus, don’t be an electrical engineer. Trust me.

Unlimited Local Storage for $12 per Month. Really.

Posted on by
4

I have a 48TB NAS in my basement. Granted, thanks to RAID6 I only (only!) have 36TB of usable space, but still, I assumed that would last me forever. Thanks to ripping DVDs and Blurays, I was so very, very wrong. Rather than spend a few thousand dollars on a new NAS, however, I decided to host my files in the cloud. The storage is unlimited for $12/month, and after 6 months or so, I can tell you it’s a viable alternative to local storage. Plus, it mounts exactly like a local NAS!

The Service

There are plenty of cloud-based storage solutions available, but they are all either very limited in storage space, or very expensive per GB. There is one solution, however, that provides unlimited storage for a set monthly price. Google Drive.

Officially, in order to get unlimited storage, you must get a Business Gsuite with 5 users. Each user is $12/month, and so you’d have to pay $60/month to get unlimited storage. Honestly, $60 a month for that much space is still insanely affordable — but if you open a Business Gsuite account with a single user (so only $12/mo), you still get unlimited storage. It might seem like an error Google would quickly fix, but it’s been that way for years. I’m currently using more than 40TB of space on my Google Drive, and only have a single user on my Gsuite for Business.

The Problem

Google Drive is nice, but let’s be honest, no one wants to use their web interface as bulk storage. It’s clumsy, slow, and as much as I love Google, the organization is confusing at best. Google does provide “Google Drive Stream”, but due to local caching, you still need local storage or you get “not enough space” errors.

Thankfully, Rclone makes direct access to Google Drive seamless. It allows you to create access via keypair (no annoying logging in all the time), and even lets you mount your remote share on your local filesystem. And in true Steve Jobs “one more thing” fashion, it also allows you to encrypt files and directories in real time, so your privacy is protected even if your data is stored on someone else’s hard drive. It’s seriously amazing. And Rclone? Open Source and totally free!

The Process

Rclone is in most Linux distributions, and even has Windows and OSX versions available that all work similarly. In this video, I show you how to quickly set up a share and mount it on the local filesystem. If it seems too good to be true, yeah, I get that. But I’ve been using it for months and I’ve been more than impressed. It’s been reliable, and robust enough to support a handful of users reading and writing at the same time.

You can do a bit of extra work to create your own application API, which will make the performance more robust. It doesn’t cost any extra money, but it’s admittedly a bit of confusing clicking.

The Training?

You probably know I create training for a living. I have more in-depth training on rclone over at CBT Nuggets. If you’re already a subscriber, you can go to https://snar.co/cbt-rclone to get to the skill directly.

If you’re not yet a subscriber at CBT Nuggets, you can see my Everything Linux course overview, which includes my rclone skill and many others. Feel free to sign up for a free trial if you want to view my training. https://snar.co/cbt-everythinglinux

(This isn’t a creepy bait and switch — the free video above really does walk you through the process. There’s just more capability if you’re interested in diving in deeper, and want to check out my professional DayJob training!)

Cryptocurrency and the IRS

Posted on by
Reply

[NOTE: This is a piece I wrote for Linux Journal a few years back. It’s still very relevant, and still important information for anyone dabbling in crypto. This seems like a good time of year to repost it.]

One for you, one for me, and 0.15366BTC for Uncle Sam.

When people ask me about bitcoin, it’s usually because someone told them about my days as an early miner. I had thousands of bitcoin, and I sold them for around a dollar each. At the time, it was awesome, but looking back—well you can do the math. I’ve been mining and trading with cryptocurrency ever since it was invented, but it’s only over the past few years that I’ve been concerned about taxes.

In the beginning, no one knew how to handle the tax implications of bitcoin. In fact, that was one of the favorite aspects of the idea for most folks. It wasn’t “money”, so it couldn’t be taxed. We could start an entire societal revolution without government oversight! Those times have changed, and now the government (at least here in the US) very much does expect to get taxes on cryptocurrency gains. And you know what? It’s very, very complicated, and few tax professionals know how to handle it.

What Is Taxable?

Cryptocurrencies (bitcoin, litecoin, ethereum and any of the 10,000 other altcoins) are taxed based on the “gains” you make with them. (Often in this article I mention bitcoin specifically, but the rules are the same for all cryptocurrency.) Gains are considered income, and income is taxed. What sorts of things are considered gains? Tons. Here are a few examples:

  • Mining.
  • Selling bitcoin for cash.
  • Trading one crypto coin for another on an exchange.
  • Buying something directly with bitcoin.

The frustrating part about taxes and cryptocurrency is that every transaction must be calculated. See, with cash transactions, a dollar is always worth a dollar (according to the government, let’s not get into a discussion about fiat currency). But with cryptocurrency, at any given moment, the coin is worth a certain amount of dollars. Since we’re taxed on dollars, that variance must be tracked so we are sure to report how much “money” we had to spend.

It gets even more complicated, because we’re taxed on the same bitcoin over and over. It’s not “double dipping”, because the taxes are only on the gains and losses that occurred between transactions. It’s not unfair, but it’s insanely complex. Let’s look at the life of a bitcoin from the moment it’s mined. For simplicity’s sake, let’s say it took exactly one day to mine one bitcoin:

1) After 24 hours of mining, I receive 1BTC. The market price for bitcoin that day was $1,000 per BTC. It took me $100 worth of electricity that day to mine (yes, I need to track the electrical usage if I want to deduct it as a loss).

Taxable income for day 1: $900.

2) The next day, I trade the bitcoin for ethereum on an exchange. The cost of bitcoin on this day is $1,500. The cost of ethereum on this day is $150. Since the value of my 1 bitcoin has increased since I mined it, when I make the trade on the exchange, I have to claim the increase in price as income. I now own 10 ethereum, but because of the bitcoin value increase, I now have more income. There are no deductions for electricity, because I already had the bitcoin; I’m just paying the capital gains on the price increase.

Taxable income for day 2: $500.

3) The next day, the price of ethereum skyrockets to $300, and the price of bitcoin plummets to $1,000. I decide to trade my 10 ethereum for 3BTC. When I got my ethereum, they were worth $1,500, but when I just traded them for BTC, they were worth $3,000. So I made $1,500 worth of profit.

Taxable income for day 3: $1,500.

4) Finally, on the 4th day, even though the price is only $1,200, I decide to sell my bitcoin for cash. I have 3BTC, so I get $3,600 in cash. Looking back, when I got those 3BTC, they were worth $1,000 each, so that means I’ve made another $600 profit.

Taxable income for day 4: $600.

It might seem unfair to be taxed over and over on the same initial investment, but if you break down what’s happening, it’s clear we’re only getting taxed on price increases. If the price drops and then we sell, our taxable income is negative for that, and it’s a deduction. If you have to pay a lot in taxes on bitcoin, it means you’ve made a lot of money with bitcoin!

Exceptions?

There are a few exceptions to the rules—well, they’re not really exceptions, but more clarifications. Since we’re taxed only on gains, it’s important to think through the life of your bitcoin. For example:

  1. Employer paying in bitcoin: I work for a company that will pay me in bitcoin if I desire. Rather than a check going into my bank account, every two weeks a bitcoin deposit goes into my wallet. I need to track the initial cost of the bitcoin as I receive it, but usually employers will send you the “after taxes” amount. That means the bitcoin you receive already has been taxed. You still need to track what it’s worth on the day you receive it in order to determine gain/loss when you eventually spend it, but the initial total has most likely already been taxed. (Check with your employer to be sure though.)
  2. Moving bitcoin from one wallet to another: this is actually a tougher question and is something worth talking about with your tax professional. Let’s say you move your bitcoin from a BitPay wallet to your fancy new Trezor hardware wallet. Do you need to count the gains/losses since the time it was initially put into your BitPay wallet? Regardless of what you and your tax professional decide, you’re not going to “lose” either way. If you decide to report the gain/loss, your cost basis for that bitcoin changes to the current date and price. If you don’t count a gain/loss, you stick to the initial cost basis from the deposit into the BitPay wallet.

The moral of the story here is to find a tax professional comfortable with cryptocurrency.

Accounting Complications

If you’re a finance person, terms like FIFO and LIFO make perfect sense to you. (FIFO = First In First Out, and LIFO = Last In First Out.) Although it’s certainly easy to understand, it wasn’t something I’d considered before the world of bitcoin. Here’s an example of how they differ:

  • Day 1: buy 1BTC for $100.
  • Day 2: buy 1BTC for $500.
  • Day 3: buy 1BTC for $1,000.
  • Day 4: buy 1BTC for $10,000.
  • Day 5: sell 1BTC for $12,000.

If I use FIFO to determine my gains and losses, when I sell the 1BTC on day 5, I have to claim a capital gain of $11,900. That’s considered taxable income. However, if I use LIFO to determine the gains and losses, when I sell the 1BTC on day 5, I have to claim only $2,000 worth of capital gains. The question is basically “which BTC am I selling?”

There are other accounting methods too, but FIFO and LIFO are the most common, and they should be okay to use with the IRS. Please note, however, that you can’t mix and match FIFO/LIFO. You need to pick one and stick with it. In fact, if you change the method from year to year, you need to change the method officially with the IRS, which is another task for your tax professional.

The Long and Short of It

Another complication when it comes to calculating taxes doesn’t have to do with gains or losses, but rather the types of gains and losses. Specifically, if you have an asset (such as bitcoin) for longer than a year before you sell it, it’s considered a long-term gain. That income is taxed at a lower rate than if you sell it within the first year of ownership. With bitcoin, it can be complicated if you move the currency from wallet to wallet. But if you can show you’ve had the bitcoin for more than a year, it’s very much worth the effort, because the long-term gain tax is significantly lower.

This was a big factor in my decision on whether to cash in ethereum or bitcoin for a large purchase I made this year. I had the bitcoin in a wallet, but it didn’t “age” as bitcoin for a full year. The ethereum had just been sitting in my Coinbase account for 13 months. I ended up saving significant money by selling the ethereum instead of a comparable amount of bitcoin, even though the capital gain amount might have been similar. The difference in long-term and short-term tax rates are significant enough that it’s worth waiting to sell if you can.

Overwhelmed?

If you’ve made only a couple transactions during the past year, it almost can be fun to figure out your gains/losses. If you’re like me, however, and you try to purchase things with bitcoin at every possible opportunity, it can become overwhelming fast. The first thing I want to stress is that it’s important to talk to someone who is familiar with cryptocurrency and taxes. This article wasn’t intended to prepare you for handling the tax forms yourself, but rather to show why you might need professional help!

Unfortunately, if you live in a remote rural area like I do, finding a tax professional who is familiar with bitcoin can be tough—or potentially impossible. The good news is that the IRS is handling cryptocurrency like any other capital gain/loss, so with the proper help, any good tax person should be able to get through it. FIFO, LIFO, cost basis and terms like those aren’t specific to bitcoin. The parts that are specific to bitcoin can be complicated, but there is an incredible resource online that will help.

If you head over to BitcoinTaxes (Figure 1), you’ll find an incredible website designed for bitcoin and crypto enthusiasts. I think there is a free offering for folks with just a handful of transactions, but for $29, I was able to use the site to track every single cryptocurrency transaction I made throughout the year. BitcoinTaxes has some incredible features:

  • Automatically calculates rates based on historical market prices.
  • Tracks gains/losses including long-term/short-term ramifications.
  • Handles purchases made with bitcoin individually and determines gains/losses per transaction (Figure 2).
  • Supports multiple accounting methods (FIFO/LIFO).
  • Integrates with online exchanges/wallets to pull data.
  • Creates tax forms.

The last bullet point is really awesome. The intricacies of bitcoin and taxes are complicated, but the BitcoinTaxes site can fill out the forms for you. Once you’re entered all your information, you can print the tax forms so you can deliver them to your tax professional. The process for determining what goes on the forms might be unfamiliar to many tax preparers, but the forms you get from BitcoinTaxes are standard IRS tax forms, which the tax pro will fully understand.

Figure 1. The BitcoinTaxes site makes calculating tax burdens far less burdensome.

Figure 2. If you do the math, you can see the price of bitcoin was drastically different for each transaction.

Do you need to pay $29 in order to calculate all your cryptocurrency tax information properly? Certainly not. But for me, the site saved me so many hours of labor that it was well worth it. Plus, while I’m a pretty smart guy, the BitcoinTaxes site was designed with the sole purpose of calculating tax information. It’s nice to have that expertise on hand.

My parting advice is please take taxes seriously—especially this year. The IRS has been working hard to get information from companies like Coinbase regarding taxpayer’s gains/losses. In fact, Coinbase was required to give the IRS financial records on 14,355 of its users. Granted, those accounts are only people who have more than $20,000 worth of transactions, but it’s just the first step. Reporting things properly now will make life far less stressful down the road. And remember, if you have a ton of taxes to pay for your cryptocurrency, that means you made even more money in profit. It doesn’t make paying the IRS any more fun, but it helps make the sore spot in your wallet hurt a little less.

PoE, PoE+, and Passive PoE

Posted on by
1

If you’re befuddled by every Poe other than Edgar Allen, after this short blog post, you’ll be confused… nevermore.

I’ve been installing a lot of POE devices recently, and the different methods for providing power over Ethernet cables can be very confusing. There are a few standards in place, and then there’s a method that isn’t a standard, but is widely used.

802.3af or Active PoE:

This is the oldest standard for providing power over Ethernet cables. It allows a maximum of 15.4 watts of power to be transmitted, and the devices (switch and peripheral) negotiate the amount of power and the wires on which the power is transmitted. If a device says it is PoE-compliant, that compliance is usually referring to 802.3af.

802.3at or PoE+:

The main difference between PoE and PoE+ is the amount of power that can be transmitted. There is still negotiation to determine the amount of power and what wires it’s transmitted on, but PoE+ supports up to 25.5 watts of power. Often, access points with multiple radios or higher-powered antennas require more power than 802.3af can supply.

Passive PoE:

This provides power over the Ethernet lines, but it doesn’t negotiate the amount of power or the wires on which the power is sent. Many devices use Passive PoE (notably, the Ubiquiti line of network hardware often uses 24v Passive PoE) to provide power to remote devices. With Passive PoE, the proprietary nature of the power specifics means that it’s often wise to use only power injectors or switches specifically designed for the devices that require Passive PoE. The power is “always on”, so it’s possible to burn out devices if they’re not prepared for electrified Ethernet wires, or if the CAT5 cabling is wired incorrectly.

Figure 1. This AP requires a Passive PoE 24v supply. It can be confusing, because even though it says it’s PoE, it won’t power on using a standard 802.3af switch.

The best practice for using power over Ethernet is either to use equipment that adheres to the 802.3af/at standards or to use the power injectors or switches specifically designed for the hardware. Usually, the standard-based PoE devices are more expensive, but the ability to use any brand PoE switch and device often makes the extra expense worthwhile. That said, there’s nothing wrong with Passive PoE, as long as the correct power is given to the correct devices.

Grepping is Awesome. Just Don’t Glob it Up!

Posted on by
Reply
Greps and pipes and greps and pipes and greps and pipes…

This article covers some grep and regex basics.

There are generally two types of coffee drinkers. The first type buys a can of pre-ground beans and uses the included scoop to make their automatic drip coffee in the morning. The second type picks single-origin beans from various parts of the world, accepts only beans that have been roasted within the past week and grinds those beans with a conical burr grinder moments before brewing in any number of complicated methods. Text searching is a bit like that.

For most things on the command line, people think of *.* or *.txt and are happy to use file globbing to select the files they want. When it comes to grepping a log file, however, you need to get a little fancier. The confusing part is when the syntax of globbing and regex overlap. Thankfully, it’s not hard to figure out when to use which construct.

Globbing

The command shell uses globbing for filename completion. If you type something like ls *.txt, you’ll get a list of all the files that end in .txt in the current directory. If you do ls R*.txt, you’ll get all the files that start with capital R and have the .txt extension. The asterisk is a wild card that lets you quickly filter which files you mean.

You also can use a question mark in globbing if you want to specify a single character. So, typing ls read??.txt will list readme.txt, but not read.txt. That’s different from ls read*.txt, which will match both readme.txt and read.txt, because the asterisk means “zero or more characters” in the file glob.

Here’s the easy way to remember if you’re using globbing (which is very simple) vs. regular expressions: globbing is done to filenames by the shell, and regex is used for searching text. The only frustrating exception to this is that sometimes the shell is too smart and conveniently does globbing when you don’t want it to—for example:


grep file* README.TXT

In most cases, this will search the file README.TXT looking for the regular expression file*, which is what you normally want. But if there happens to be a file in the current folder that matches the file* glob (let’s say filename.txt), the shell will assume you meant to pass that to grep, and so grep actually will see:


grep filename.txt README.TXT

Gee, thank you so much Mr. Shell, but that’s not what I wanted to do. For that reason, I recommend always using quotation marks when using grep. 99% of the time you won’t get an accidental glob match, but that 1% can be infuriating. So when using grep, this is much safer:


grep "file*" README.TXT

Because even if there is a filename.txt, the shell won’t substitute it automatically.

So, globs are for filenames, and regex is for searching text. That’s the first thing to understand. The next thing is to realize that similar syntax means different things.

Glob and Regex Conflicts

I don’t want this article to become a super in-depth piece on regex; rather, I want you to understand simple regex, especially as it conflicts with blobbing. Table 1 shows a few of the most commonly confused symbols and what they mean in each case.

Table 1. Commonly Used Symbols

Special CharacterMeaning in GlobsMeaning in Regex
*zero or more characterszero or more of the character it follows
?single occurrence of any characterzero or one of the character it follows but not more than 1
.literal “.” characterany single character

To add insult to injury, you might be thinking about globs when you use grep, but just because you get the expected results doesn’t mean you got the results for the correct reason. Let me try to explain. Here is a text file called filename.doc:


The fast dog is fast.
The faster dogs are faster.
A sick dog should see a dogdoc.
This file is filename.doc

If you type:


grep "fast*" filename.doc

The first two lines will match. Whether you’re thinking globs or regex, that makes sense. But if you type:


grep "dogs*" filename.doc

The first three lines will match, but if you’re thinking in globs, that doesn’t make sense. Since grep uses regular expressions (regex) when searching files, the asterisk means “zero or more occurrences of the previous character”, so in the second example, it matches dog and dogs, because having zero “s” characters matches the regex.

And let’s say you typed this:


grep "*.doc" filename.doc

This will match the last two lines. The asterisk doesn’t actually do anything in this command, because it’s not following any character. The dot in regex means “any character”, so it will match the “.doc”, but it also will match “gdoc” in “dogdoc”, so both lines match.

The moral of the story is that grep never uses globbing. The only exception is when the shell does globbing before passing the command on to grep, which is why it’s always a good idea to use quotation marks around the regular expression you are trying to grep for.

Use fgrep to Avoid Regex

If you don’t want the power of regex, it can be very frustrating. This is especially true if you’re actually looking for some of the special characters in a bunch of text. You can use the fgrep command (or grep -F, which is the same thing) in order to skip any regex substitutions. Using fgrep, you’ll search for exactly what you type, even if they are special characters. Here is a text file called file.txt:


I really hate regex.
All those stupid $, {}, and \ stuff ticks me off.
Why can't text be text?

If you try to use regular grep like this:


grep "$," file.txt

you’ll get no results. That’s because the “$” is a special character (more on that in a bit). If you’d like to grep for special characters without escaping them, or knowing the regex code to get what you want, this will work fine:


grep -F "$," file.txt

And, grep will return the second line of the text file because it matches the literal characters. It’s possible to build a regex query to search for special characters, but it can become complicated quickly. Plus, fgrep is much, much faster on a large text file.

Some Simple, Useful Regex

Okay, now that you know when to use globbing and when to use regular expressions, let’s look at a bit of regex that can make grepping much more useful. I find myself using the caret and dollar sign symbols in regex fairly often. Caret means “at the beginning of the line”, and dollar sign means “at the end of the line”. I used to mix them up, so my silly method to remember is that a farmer has to plant carrots at the beginning of the season in order to sell them for dollars at the end of the season. It’s silly, but it works for me!

Here’s a sample text file named file.txt:


chickens eat corn
corn rarely eats chickens
people eat chickens and corn
chickens rarely eat people

If you were to type:


grep "chickens" file.txt

you will get all four lines returned, because “chickens” is in each line. But if you add some regex to the mix:


grep "^chickens" file.txt

you’ll get both the first and fourth line returned, because the word “chickens” is at the beginning of those lines. If you type:


grep "corn$" file.txt

you will see the first and third lines, because they both end with “corn”. However, if you type:


grep "^chickens.*corn$" file.txt

you will get only the first line, because it is the only one that begins with chickens and ends with corn. This example might look confusing, but there are three regular expressions that build the search. Let’s look at each of them.

First, ^chickens means the line must start with chickens.

Second, .* means zero or more of any character, because remember, the dot means any character, and the asterisk means zero or more of the previous character.

Third, corn$ means the line must end with corn.

When you’re building regular expressions, you just mush them all together like that in a long string. It can become confusing, but if you break down each piece, it makes sense. In order for the entire regular expression to match, all of the pieces must match. That’s why only the first line matches the example regex statement.

A handful of other common regex characters are useful when grepping text files. Remember just to mush them together to form the entire regular expression:

  • \ — the backslash negates the “special-ness” of special characters, which means you actually can search for them with regex. For example, \$ will search for the $ character, instead of looking for the end of a line.
  • \s — this construct means “whitespace”, which can be a space or spaces, tabs or newline characters. To find the word pickle surrounded by whitespace, you could search for \spickle\s, and that will find “pickle” but not “pickles”.
  • .* — this is really just a specific use of the asterisk, but it’s a very common combination, so I mention it here. It basically means “zero or more of any characters”, which is what was used in the corn/chicken example above.
  • | — this means “or” in regex. So hi|hello will match either “hi” or “hello”. It’s often used in parentheses to separate it from other parts of the regular expression. For example, (F|f)rankfurter will search for the word frankfurter, whether or not it’s capitalized.
  • [] — brackets are another way to specify “or” options, but they support ranges. So the regex [Ff]rankfurter is the same as the above example. Brackets support ranges though, so ^[A-Z] will match any line that starts with a capital letter. It also supports numbers, so [0-9]$ will match any line that ends in a digit.

Your Mission

You can do far more complicated things with regular expressions. These basic building blocks are usually enough to get the sort of text you need out of a log file. If you want to learn more, by all means, either do some googling on regex, or get a book explaining all the nitty-gritty goodness. If you want me to write more about it, drop a comment and let me know.

I really, really encourage you to practice using regex. The best way to learn is to do, so make a few text files and see if the regex statements you create give you the results you expect. Thankfully, grep highlights the “match” it finds in the line it returns. That means if you’re getting more results than you expect, you’ll see why the regex matched more than you expected, because grep will show you.

The most important thing to remember is that grep doesn’t do globbing—that wild-card stuff is for filenames on the shell only. Even if globbing with grep seems to work, it’s probably just coincidence (look back at the dog/dogs example here if you don’t know what I’m talking about). Have fun grepping!