Recently, I wanted to build a small program to search things through the Brave API. Why? Well, I wanted to begin to understand how APIs work and, two, how search engines work. So, I set out to set up my brave account and make an API key. This is written here if you wish to follow along. https://tinyurl.com/mrxz873w
Saying Hello to the Brave API
I first tried the curl command to see how the API worked. (Just a side note for the sake of this blog: the command is on two lines to make the curl command work: move the -H "Accept-Encoding: gzip" -H "X-Subscription-Token: API_KEY_VALUE" up a level to make the command all one line.)
curl -s --compressed "https://api.search.brave.com/res/v1/web/search?q=brave+search" -H "Accept: application/json"
-H "Accept-Encoding: gzip" -H "X-Subscription-Token: API_KEY_VALUE"
As you can see, the Headers are not the normal restful headers when using an API. For example, most restful headers accept the following Bearer format when working with a token. -H "Authorization: Bearer <ACCESS_TOKEN>"
Using this Curl command will only get you authenticated to the API and will return the following: (This is just a snippet of what is returned; the whole JSON is very long and would make this a very long article.)
{"query":{"original":"brave search","show_strict_warning":false,"is_navigational":true,"local_decision":"drop","local_locations_idx":0,"is_news_breaking":false,"spellcheck_off":true,"country":"us","bad_results":false,"should_fallback":false,"postal_code":"","city":"","header_country":"","more_results_available":true,"state":""},"mixed":{"type":"mixed","main":[{"type":"web","index":0,"all":false},{"type":"web","index":1,"all":false},{"type":"videos","all":true},{"type":"web","index":2,"all":false},{"type":"web","index":3,"all":false},{"type":"web","index":4,"all":false},{"type":"web","index":5,"all":false},{"type":"web","index":6,"all":false},{"type":"web","index":7,"all":false},{"type":"web","index":8,"all":false},{"type":"web","index":9,"all":false},{"type":"web","index":10,"all":false},{"type":"web","index":11,"all":false},{"type":"web","index":12,"all":false},{"type":"web","index":13,"all":false},{"type":"web","index":14,"all":false},{"type":"web","index":15,"all":false},{"type":"web","index":16,"all":false},{"type":"web","index":17,"all":false},{"type":"web","index":18,"all":false},{"type":"web","index":19,"all":false}],"top":[],"side":[]},"type":"search","videos":{"type":"videos","results":[{"type":"video_result","url":"https://www.youtube.com/watch?v=gwho1EuwkRQ","title":"How To Change The Default Search Engine In The Brave ...","description":"How To Change The Default Search Engine In The Brave Web Browser | PC | *2022*This is a video tutorial on how to change the default search engine in the Brav...","age":"September 20, 2022","video":{},"meta_url":{"scheme":"https","netloc":"youtube.com","hostname":"www.youtube.com","favicon":"https://imgs.search.brave.com/Ux4Hee4evZhvjuTKwtapBycOGjGDci2Gvn2pbSzvbC0/rs:fit:32:32:1/g:ce/aHR0cDovL2Zhdmlj/b25zLnNlYXJjaC5i/cmF2ZS5jb20vaWNv/bnMvOTkyZTZiMWU3/YzU3Nzc5YjExYzUy/N2VhZTIxOWNlYjM5/ZGVjN2MyZDY4Nzdh/ZDYzMTYxNmI5N2Rk/Y2Q3N2FkNy93d3cu/eW91dHViZS5jb20v","path":"› watch"},"thumbnail":{"src":"https://imgs.search.brave.com/6g23ttPyzgvCVreUZ2pcnuTnGAhNHt8yQLyVNL3mlsc/rs:fit:200:200:1/g:ce/aHR0cHM6Ly9pLnl0/aW1nLmNvbS92aS9n/d2hvMUV1d2tSUS9t/YXhyZXNkZWZhdWx0/LmpwZz9zcXA9LW9h/eW13RW1DSUFLRU5B/RjhxdUtxUU1hOEFF/Qi1BSC1DWUFDMEFX/S0Fnd0lBQkFCR0Vn/Z0V5aF9NQTg9JmFt/cDtycz1BT240Q0xB/dzdIUkZsVTd1OUZT/YUR2WERZSE1TNnNj/U0p3"}},{"type":"video_result","url":"https://www.youtube.com/watch?v=ux4VacF3hxc","title":"Brave Browser And Search Engine Keep Getting Better - YouTube",
Lets Get Searching With Go
As you can see, we have yet to get searching. For that, we turn to Go.
In Go, you can use specific packages to interact with APIs. They are: "encoding/json", "net/http", and "net/url." There are a few others we are going to use, but they are just there to make this all work. I won't go into detail about them here; maybe in a future blog post.
Essentially, what we are dealing with here is JSON being passed to us, and we have to do something with that JSON. Now you have structured JSON and Unstructured JSON. To learn more about the difference and more about JSON, there is a great article on the Go Blog here: https://go.dev/blog/json
As we know the fields we are getting from the CURL command, we know to use a structured JSON approach. To do that, we set structs to match the fields of the Returned JSON. We do that like this:
type Searchrequest struct {
base *url.URL
token string
searchterm string
}
type Main struct {
Type string `json:"type"`
Index int `json:"index"`
All bool `json:"all"`
}
type Query struct {
Original string `json:"original"`
ShowStrictWarning bool `json:"show_strict_warning"`
IsNavigational bool `json:"is_navigational"`
IsNewsBreaking bool `json:"is_news_breaking"`
SpellcheckOff bool `json:"spellcheck_off"`
Country string `json:"country"`
BadResults bool `json:"bad_results"`
ShouldFallback bool `json:"should_fallback"`
PostalCode string `json:"postal_code"`
City string `json:"city"`
HeaderCountry string `json:"header_country"`
MoreResultsAvailable bool `json:"more_results_available"`
State string `json:"state"`
}
type Mixed struct {
Type string `json:"type"`
Main []Main `json:"main"`
Top []interface{} `json:"top"`
Side []interface{} `json:"side"`
}
type MetaURL struct {
Scheme string `json:"scheme"`
Netloc string `json:"netloc"`
Hostname string `json:"hostname"`
Favicon string `json:"favicon"`
Path string `json:"path"`
}
type Results struct {
Title string `json:"title"`
URL string `json:"url"`
IsSourceLocal bool `json:"is_source_local"`
IsSourceBoth bool `json:"is_source_both"`
Description string `json:"description"`
Language string `json:"language"`
FamilyFriendly bool `json:"family_friendly"`
Type string `json:"type"`
Subtype string `json:"subtype"`
MetaURL MetaURL `json:"meta_url"`
Age string `json:"age,omitempty"`
}
type Web struct {
Type string `json:"type"`
Results []Results `json:"results"`
FamilyFriendly bool `json:"family_friendly"`
}
type Response struct {
Query Query `json:"query"`
Mixed Mixed `json:"mixed"`
Type string `json:"type"`
Web Web `json:"web"`
}
This gets us ready to receive and send data and do something with it. We will touch on that in a minute first to tell Go to go (no pun intended) and interact with the API. We have to do what I like to call building the request.
Let's Build our Request
To build the request, we use the "net/http" package. We essentially tell the package this is going to be a GET request with a token, and in the "net/http" package, there is a function called http.NewRequest this basically controls our request and allows us to pass certain parameters to the function. We use http.MethodGet to be able to pass in a token and our search query.
The following code builds our request:
req, err := http.NewRequest(http.MethodGet, encodedurl, nil)
if err != nil {
log.Fatal(err)
}
req.Header = http.Header{
"Accept": {"application/json"},
"X-Subscription-Token": {searchrequest.token},
}
As you can see, our req variable receives the encoded URL (we will touch on this after and nil for error handling). We then pass the variable into a struct and add the following two headers "Accept:" which is how we wish to receive our data in this instance json.
The other header "X-Subscription-Token" is the token we are passing in. We are getting this token from a flag when we run the program as it is not good practice to put API keys into your code; we pass the value from this flag into a struct field called searchrequest.token
Passing in Flags
This piece of code is at the beginning of our program and looks like this:
// Ask For Token and Search Term
searchrequest := Searchrequest{
token: "",
searchterm: "",
}
flag.StringVar(&searchrequest.token, "token", "", "token")
flag.StringVar(&searchrequest.searchterm, "searchterm", "", "searchterm")
flag.Parse()
This asks our user for the token and the search term. Input is as follows:
go run getwebpage.go -token "VALUE" -searchterm "VALUE"
Working with Search Engines Search Terms
encodedsearchterm := url.PathEscape(searchrequest.searchterm)
encodedurl := fmt.Sprintf("https://api.search.brave.com/res/v1/web/search")
queryparam := url.Values{}
queryparam.Set("q", encodedsearchterm)
if len(searchrequest.searchterm) > 0 {
encodedurl += "?" + queryparam.Encode()
}
The reason why you see in the code the following variable encodedurl is not because of security but because of a simple problem when working with search engines. When you enter a search term into the search engine, you put your term like this ** Show Me Some Cats**. As you can see, you put your text in with spaces.
Now, the search engine automatically encodes your search term to something like this: show%20me%20some%20cats.This tells the rest of the engine this is a whole string with spaces.
Using the "net/url" package, you can encode the term first by passing it to the url struct called url.Values{} (this is in the net/url package). Then you get the search query and pass it to url.Values{} but not before you have encoded it with the Encode function. Now, for this to work, you also have to attach it to the "q" header, as this is what the Brave API only accepts to be able to use your term to search. The code that attaches our encoded query to the header of "q" is queryparam.Set("q", encodedsearchterm)
Finishing our Request
Now we have the API token and the query (our search term), we build a client to handle our request and then pass the req variable to that client.
// Send Request
client := &http.Client{}
repsonse, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
defer repsonse.Body.Close()
Receiving The Request
Essentially, by this point, all I wanted to do was receive my data and write to a file in the format of JSON.
To do that, I used the "json/unmarshall" package, and in this piece of code, I received the JSON and unmarshal it: json.Unmarshal([]byte(body), &res)
Then, this code will indent my JSON so that It is semi-readable, and it will write it to a file in the same directory as my program and call it reply.json.
content := fmt.Sprintf("Query: %+v\n, Results: %+v\n", res.Query, res.Web.Results)
bytes, _ := json.MarshalIndent(content, "", "")
contentjsonerr := os.WriteFile("reply.json", bytes, 0644)
if contentjsonerr != nil {
log.Fatal(contentjsonerr)
}
You can do a ton with that data, but this Blog post is already long enough, so I will stop here. I think I might write another post soon about what we can do with our cat's data.
If you want to write this program or run it yourself, you can find the code here: https://github.com/jasric89/headfirstgo/tree/master/Chapter13/HTTPProg
I wish to thank the Slack Gopher community (Join here: gophers.slack.com), particularly a fellow Gopher called Peter Hellberg, who helped me understand how APIs work. Without him, this blog post would not exist.
Happy Coding and The Cloud Dude will see you on the next Post.
