Linux Fu: Curling C

Sometimes, it pays to read the man pages of commands you use often. There might be a gem hidden in there that you don’t know about. Case in point: I’ve used curl (technically, cURL, but I’m going to stick with curl) many times to grab data from some website or otherwise make a web request. But what happens if you want to do the same thing from a C program? Well, you could be lazy and just spawn a copy of curl. But it turns out curl has a trick up its sleeve that can help you. If only I’d read the man page sooner!

First Things

The simplest use of curl is to just name a URL on the command line. For example, consider this session:

$ curl 
<head><title>301 Moved Permanently</title></head>
<center><h1>301 Moved Permanently</h1></center>

This isn’t so useful because it is a 301 response (to send you to the https server, in this case). The -L option will make curl go get the page instead of the redirect. Try:

$ curl -L

You probably want to pipe it through less or use the -o option to send the output to a file. If you want to see the details of the redirect, try:

$ curl -i

Jack of All

That’s just the very simplest thing you can do with curl. It can do a bewildering array of protocols, including FTP, SMB, POP3, IMAP, SFTP, and many more. It can form different requests and manipulate cookies, certificates, and many other things.

However, it turns out curl doesn’t really do any of those things. It just reads your input and manipulates libcurl which is where all the smarts are. If you have done much with Linux, then you realize that means you could use libcurl, too. But how?

Less Work, More Code

You could look up the details of libcurl. It isn’t a secret. In fact, there are two ways to use the library. The “easy” interface is quick to use and great for most things. If you need heavy-duty multiple transfers and other exotica, you might have to use the “multi” API, which is a bit more complex.

However, you don’t even have to start there. The curl program itself will help you. Here’s how it works. First, build a command line to get the results you want (just like we did earlier to read the front page of Hackaday). Then, add the –libcurl option to the command line along with the name of a C source file (that probably doesn’t already exist). The program will then write a skeletal piece of code to do the exact transfer you specified. You can compile in the usual way, just add -lcurl to the compile command.

You might have to tweak it a bit, and depending on your application, you might want to make some changes like modifying the URL. But you’ll get a great start.

For example, try:

$ curl -L -o output.txt --libcurl hackcurl.c

The skeleton will have a main function (you might want to change that if you are adding it to another program). In that main will be code reflecting most of the options you set on the command line.

There will also be a comment showing you some things you might want to set that it can’t figure out for you. Finally, there’s a little driver that just performs the operation, does some cleanup, and then exits with a status code. Here are the options the boilerplate code suggests:

/* Here is a list of options the curl code used that cannot get generated
as source easily. You may choose to either not use them or implement
them yourself.
CURLOPT_WRITEDATA set to a objectpointer
CURLOPT_INTERLEAVEDATA set to a objectpointer
CURLOPT_WRITEFUNCTION set to a functionpointer
CURLOPT_READDATA set to a objectpointer
CURLOPT_READFUNCTION set to a functionpointer
CURLOPT_SEEKDATA set to a objectpointer
CURLOPT_SEEKFUNCTION set to a functionpointer
CURLOPT_ERRORBUFFER set to a objectpointer
CURLOPT_STDERR set to a objectpointer
CURLOPT_HEADERFUNCTION set to a functionpointer
CURLOPT_HEADERDATA set to a objectpointer

An Example

The thing you’ll most often need to change is what happens to the data you read. The -o option isn’t what you wanted, probably, and so curl doesn’t build that into your C code. Of course, if you just wanted to send a request, that might be all you need. In my case, I will eventually want to know what the Hackaday server said back, so I have some work to do. Luckily, it isn’t much.

At the start of the code (available online). I added a section near the top that includes a few headers and defines a BUFFER structure. This is just a string with a length. I also made a simple function to create a buffer and another that libcurl can call to send me data (writefn).

That last function is the most complex of the custom code. It looks at how much data curl received, reallocates the buffer, and saves it. I didn’t intend this to be general purpose, so there’s no provision for fancy editing to the buffer. Things come in. The buffer grows. Stuff gets put on the end of the buffer. That’s all.

The next part of the custom code appears near the end. It tells libcurl about the writefn function and asks it to pass the address of buff (the buffer) to that function. Obviously, libcurl doesn’t care what that argument is. It just takes whatever you tell it and adds it to the call.

Once the boilerplate curl_easy_perform function returns successfully, the buff structure has the web page in it. For this example, I just print it out, which is boring, but you could, obviously, do whatever you wanted here.

That’s it, and you’ll find that for many tasks, this is sufficient. If it isn’t, the comments and the documentation suggest other ways to configure the library similar to how this example sets the write function.

More Curl Tricks

Need your current IP address (thanks, Amazon):


Want to check if a site is up?

curl -L -Is | head -n 1 | cut -d ' ' -f2

Need a QR code?


Can’t remember the definitions for the word inductor?


Of course, you don’t need C code to call these. If you do a quick search, you’ll find there are tons of services that curl can easily access from the command line or, using libcurl, from your programs, too.

Want to test your curl chops? We’ve looked at grabbing Hackaday automatically before.

11 thoughts on “Linux Fu: Curling C

Leave a Reply

Please be kind and respectful to help make the comments section excellent. (Comment Policy)

This site uses Akismet to reduce spam. Learn how your comment data is processed.